From patchwork Mon Sep 4 12:32:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 53438 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id AEEFC27BBEA; Mon, 4 Sep 2023 13:33:17 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 19DE327BBE2 for ; Mon, 4 Sep 2023 13:33:17 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qd8lL-0003y7-O0; Mon, 04 Sep 2023 08:33:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qd8lK-0003xq-Cz for guix-patches@gnu.org; Mon, 04 Sep 2023 08:33:02 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qd8lK-0002Wp-4y for guix-patches@gnu.org; Mon, 04 Sep 2023 08:33:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qd8lJ-00038o-VD for guix-patches@gnu.org; Mon, 04 Sep 2023 08:33:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#65732] [PATCH 1/2] gnu: Add bffe. References: <874jkadsto.fsf@cbaines.net> In-Reply-To: <874jkadsto.fsf@cbaines.net> Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 04 Sep 2023 12:33:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65732 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 65732@debbugs.gnu.org Received: via spool by 65732-submit@debbugs.gnu.org id=B65732.169383077112051 (code B ref 65732); Mon, 04 Sep 2023 12:33:01 +0000 Received: (at 65732) by debbugs.gnu.org; 4 Sep 2023 12:32:51 +0000 Received: from localhost ([127.0.0.1]:49009 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qd8l9-00038I-6k for submit@debbugs.gnu.org; Mon, 04 Sep 2023 08:32:51 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:51059) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qd8l6-000384-P0 for 65732@debbugs.gnu.org; Mon, 04 Sep 2023 08:32:49 -0400 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 4159A27BBE2 for <65732@debbugs.gnu.org>; Mon, 4 Sep 2023 13:32:48 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id f5f16138 for <65732@debbugs.gnu.org>; Mon, 4 Sep 2023 12:32:47 +0000 (UTC) From: Christopher Baines Date: Mon, 4 Sep 2023 13:32:46 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * gnu/packages/package-management.scm (bffe): New variable. --- gnu/packages/package-management.scm | 51 +++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) base-commit: 8d70ff3a49f7cdf0eab93b3a1c54ec2f016afc4a diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index aa24798071..a49f302bb7 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -961,6 +961,57 @@ (define-public rpm ;; The whole is GPLv2+; librpm itself is dual-licensed LGPLv2+ | GPLv2+. (license license:gpl2+))) +(define-public bffe + (let ((commit "3ce4613908bb4a42494323ef0597f6c3ae2dee24") + (revision "1")) + (package + (name "bffe") + (version (git-version "0" revision commit)) + (source (origin + (method git-fetch) + (uri (git-reference + (url "https://git.cbaines.net/guix/bffe") + (commit commit))) + (sha256 + (base32 + "0g7arfl8rwybqpmw8vc16ilvrva1bb435lqgbqypx0nphbcv2pxn")) + (file-name (string-append name "-" version "-checkout")))) + (build-system gnu-build-system) + (native-inputs + (list pkg-config + autoconf + automake + + ;; Guile libraries are needed here for cross-compilation. + (car (assoc-ref (package-native-inputs guix) "guile")) + guile-gnutls + guile-json-4 + guix + guix-data-service + guix-build-coordinator + guile-fibers-1.3 + guile-prometheus + guile-lib)) + (propagated-inputs + (list guile-gnutls + guile-json-4 + guix + guix-data-service + guix-build-coordinator + guile-fibers-1.3 + guile-prometheus + guile-lib)) + (home-page "https://git.cbaines.net/guix/bffe") + (synopsis "Build Farm Front-end for Guix") + (description + "The BFFE of Build Farm Front-end is an experimental frontend for Guix +build farms. It works together with the Guix Data Service and Guix Build +Coordinator to submit builds and monitor the activity. + +It functions as a Guile library, with the @code{run-bffe-service} procedure in +the @code{(bffe)} module as the entry point.") + (license license:gpl3+)))) + (define-public python-anaconda-client (package (name "python-anaconda-client") From patchwork Mon Sep 4 12:32:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 53439 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id F19D627BBEA; Mon, 4 Sep 2023 13:33:31 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, NUMERIC_HTTP_ADDR,SPF_HELO_PASS,URIBL_BLOCKED,WEIRD_PORT autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 8A39427BBE2 for ; Mon, 4 Sep 2023 13:33:30 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qd8lR-0003yV-0G; Mon, 04 Sep 2023 08:33:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qd8lK-0003xy-Pz for guix-patches@gnu.org; Mon, 04 Sep 2023 08:33:02 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qd8lK-0002Wv-Hn for guix-patches@gnu.org; Mon, 04 Sep 2023 08:33:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qd8lK-00038w-CD for guix-patches@gnu.org; Mon, 04 Sep 2023 08:33:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#65732] [PATCH 2/2] services: guix: Add bffe-service-type. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 04 Sep 2023 12:33:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65732 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 65732@debbugs.gnu.org Received: via spool by 65732-submit@debbugs.gnu.org id=B65732.169383077212058 (code B ref 65732); Mon, 04 Sep 2023 12:33:02 +0000 Received: (at 65732) by debbugs.gnu.org; 4 Sep 2023 12:32:52 +0000 Received: from localhost ([127.0.0.1]:49011 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qd8l9-00038K-JR for submit@debbugs.gnu.org; Mon, 04 Sep 2023 08:32:52 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42878) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qd8l6-000383-Ul for 65732@debbugs.gnu.org; Mon, 04 Sep 2023 08:32:49 -0400 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 5F67627BBE9 for <65732@debbugs.gnu.org>; Mon, 4 Sep 2023 13:32:48 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 38d3f82c for <65732@debbugs.gnu.org>; Mon, 4 Sep 2023 12:32:47 +0000 (UTC) From: Christopher Baines Date: Mon, 4 Sep 2023 13:32:47 +0100 Message-ID: <516c325562dcee78ef795b98ae0b5d739b790b89.1693830767.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches This is intended to replace the functionality of the Guix Build Coordinator queue builds script, and also provide a web interface for build farms. * gnu/services/guix.scm (): New record type. (bffe-configuration, bffe-configuration?, bffe-configuration-package, bffe-configuration-user, bffe-configuration-group, bffe-configuration-arguments bffe-configuration-extra-environment-variables): New procedures. (bffe-service-type): New variable. * gnu/tests/guix.scm (%test-bffe): New variable. * doc/guix.texi (Guix Services): Document the new service. --- doc/guix.texi | 59 +++++++++++++++++++ gnu/services/guix.scm | 131 +++++++++++++++++++++++++++++++++++++++++- gnu/tests/guix.scm | 81 +++++++++++++++++++++++++- 3 files changed, 269 insertions(+), 2 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index a6b74ce9c7..bf12f8945d 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -38083,6 +38083,65 @@ PAM Mount Service @node Guix Services @subsection Guix Services +@subsubheading Build Farm Front-End (BFFE) +The @uref{https://git.cbaines.net/guix/bffe/,Build Farm Front-End} +assists with building Guix packages in bulk. It's responsible for +submitting builds and displaying the status of the build farm. + +@defvar bffe-service-type +Service type for the Build Farm Front-End. Its value must be a +@code{bffe-configuration} object. +@end defvar + +@deftp {Data Type} bffe-configuration +Data type representing the configuration of the Build Farm Front-End. + +@table @asis +@item @code{package} (default: @code{bffe}) +The Build Farm Front-End package to use. + +@item @code{user} (default: @code{"bffe"}) +The system user to run the service as. + +@item @code{group} (default: @code{"bffe"}) +The system group to run the service as. + +@item @code{arguments} +A list of arguments to the Build Farm Front-End. These are passed to +the @code{run-bffe-service} procedure when starting the service. + +For example, the following value directs the Build Farm Front-End to +submit builds for derivations available from @code{data.guix.gnu.org} to +the Build Coordinator instance assumed to be running on the same +machine. + +@example +(list + #:build + (list + (build-from-guix-data-service + (data-service-url "https://data.guix.gnu.org") + (build-coordinator-url "http://127.0.0.1:8746") + (branches '("master")) + (systems '("x86_64-linux" "i686-linux")) + (systems-and-targets + (map (lambda (target) + (cons "x86_64-linux" target)) + '("aarch64-linux-gnu" + "i586-pc-gnu"))) + (build-priority (const 0)))) + #:web-server-args + '(#:event-source "https://example.com" + #:controller-args + (#:title "example.com build farm"))) +@end example + +@item @code{extra-environment-variables} (default: @var{'()}) +Extra environment variables to set via the shepherd service. + +@end table +@end deftp + @subsubheading Guix Build Coordinator The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build Coordinator} aids in distributing derivation builds among machines diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index 99b21f52d8..e9db2a231d 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -140,7 +140,17 @@ (define-module (gnu services guix) nar-herder-cached-compression-configuration-type nar-herder-cached-compression-configuration-level nar-herder-cached-compression-configuration-directory - nar-herder-cached-compression-configuration-directory-max-size)) + nar-herder-cached-compression-configuration-directory-max-size + + bffe-configuration + bffe-configuration? + bffe-configuration-package + bffe-configuration-user + bffe-configuration-group + bffe-configuration-arguments + bffe-configuration-extra-environment-variables + + bffe-service-type)) ;;;; Commentary: ;;; @@ -1030,3 +1040,122 @@ (define nar-herder-service-type nar-herder-account))) (description "Run a Nar Herder server."))) + + +;;; +;;; Build Farm Front-end (BFFE) +;;; + +(define-record-type* + bffe-configuration make-bffe-configuration + bffe-configuration? + (package bffe-configuration-package + (default bffe)) + (user bffe-configuration-user + (default "bffe")) + (group bffe-configuration-group + (default "bffe")) + (arguments bffe-configuration-arguments) + (extra-environment-variables + bffe-configuration-extra-environment-variables + (default '()))) + +(define (bffe-shepherd-services config) + (define bffe-package + (bffe-configuration-package config)) + + (define start-script + (program-file + "run-bffe" + (with-extensions (cons + bffe-package + ;; This is a poorly constructed Guile load path, + ;; since it contains things that aren't Guile + ;; libraries, but it means that the Guile + ;; libraries needed for BFFE don't need to be + ;; individually specified here. + (map second (package-transitive-propagated-inputs + bffe-package))) + #~(begin + (use-modules (bffe) + (bffe manage-builds)) + + (setvbuf (current-output-port) 'line) + (setvbuf (current-error-port) 'line) + + (simple-format #t "starting the bffe:\n ~A\n" + (current-filename)) + + (apply run-bffe-service + (append + (list #:pid-file "/var/run/bffe/pid") + #$(bffe-configuration-arguments config))))) + #:guile guile-3.0)) + + (match-record config + (package user group arguments extra-environment-variables) + + (list + (shepherd-service + (documentation "Build Farm Front-end") + (provision '(bffe)) + (requirement '(networking)) + (start #~(make-forkexec-constructor + (list #$start-script) + #:user #$user + #:group #$group + #:pid-file "/var/run/bffe/pid" + #:directory "/var/lib/bffe" + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8" + #$@extra-environment-variables) + #:log-file "/var/log/bffe/server.log")) + (stop #~(make-kill-destructor)))))) + +(define (bffe-activation config) + #~(begin + (use-modules (guix build utils)) + + (define %user + (getpw #$(bffe-configuration-user + config))) + + (chmod "/var/lib/bffe" #o755) + + (mkdir-p "/var/log/bffe") + + ;; Allow writing the PID file + (mkdir-p "/var/run/bffe") + (chown "/var/run/bffe" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (bffe-account config) + (match-record config + (user group) + (list (user-group + (name group) + (system? #t)) + (user-account + (name user) + (group group) + (system? #t) + (comment "BFFE user") + (home-directory "/var/lib/bffe") + (shell (file-append shadow "/sbin/nologin")))))) + +(define bffe-service-type + (service-type + (name 'bffe) + (extensions + (list + (service-extension shepherd-root-service-type + bffe-shepherd-services) + (service-extension activation-service-type + bffe-activation) + (service-extension account-service-type + bffe-account))) + (description + "Run the Build Farm Front-end."))) diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm index ad0980a10c..240ded4825 100644 --- a/gnu/tests/guix.scm +++ b/gnu/tests/guix.scm @@ -37,7 +37,8 @@ (define-module (gnu tests guix) #:use-module (ice-9 match) #:export (%test-guix-build-coordinator %test-guix-data-service - %test-nar-herder)) + %test-nar-herder + %test-bffe)) ;;; ;;; Guix Build Coordinator @@ -325,3 +326,81 @@ (define %test-nar-herder (name "nar-herder") (description "Connect to a running Nar Herder server.") (value (run-nar-herder-test)))) + + +;;; +;;; Build Farm Front-end +;;; + +(define %bffe-os + (simple-operating-system + (service dhcp-client-service-type) + (service guix-build-coordinator-service-type) + (service bffe-service-type + (bffe-configuration + (arguments + #~(list + #:web-server-args + '(#:port 8767 + #:controller-args + (#:title "Test title")))))))) + +(define (run-bffe-test) + (define os + (marionette-operating-system + %bffe-os + #:imported-modules '((gnu services herd) + (guix combinators)))) + + (define forwarded-port 8767) + + (define vm + (virtual-machine + (operating-system os) + (memory-size 1024) + (port-forwardings `((,forwarded-port . 8767))))) + + (define test + (with-imported-modules '((gnu build marionette)) + #~(begin + (use-modules (srfi srfi-11) (srfi srfi-64) + (gnu build marionette) + (web uri) + (web client) + (web response)) + + (define marionette + (make-marionette (list #$vm))) + + (test-runner-current (system-test-runner #$output)) + (test-begin "bffe") + + (test-assert "service running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (match (start-service 'bffe) + (#f #f) + (('service response-parts ...) + (match (assq-ref response-parts 'running) + ((pid) (number? pid)))))) + marionette)) + + (test-equal "http-get" + 200 + (let-values + (((response text) + (http-get #$(simple-format + #f "http://localhost:~A/" forwarded-port) + #:decode-body? #t))) + (response-code response))) + + (test-end)))) + + (gexp->derivation "bffe-test" test)) + +(define %test-bffe + (system-test + (name "bffe") + (description "Connect to a running Build Farm Front-end.") + (value (run-bffe-test))))