From patchwork Sat Sep 19 10:10:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 24217 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id EA2F127BBE8; Sat, 19 Sep 2020 11:12:23 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,UNPARSEABLE_RELAY,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 417D327BBE7 for ; Sat, 19 Sep 2020 11:12:23 +0100 (BST) Received: from localhost ([::1]:53962 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kJZqw-0002tU-Ce for patchwork@mira.cbaines.net; Sat, 19 Sep 2020 06:12:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49712) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJZqc-0002sx-A1 for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34160) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kJZqc-0008TW-0s for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kJZqb-0007Cn-TC for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43494] [PATCH v2 1/4] gnu: Add guix-build-coordinator. References: <87o8m39bnc.fsf@cbaines.net> In-Reply-To: <87o8m39bnc.fsf@cbaines.net> Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 19 Sep 2020 10:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43494 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43494@debbugs.gnu.org Received: via spool by 43494-submit@debbugs.gnu.org id=B43494.160051026427608 (code B ref 43494); Sat, 19 Sep 2020 10:12:01 +0000 Received: (at 43494) by debbugs.gnu.org; 19 Sep 2020 10:11:04 +0000 Received: from localhost ([127.0.0.1]:45699 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpf-0007Ay-6f for submit@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:03 -0400 Received: from mira.cbaines.net ([212.71.252.8]:57764) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpd-0007Aa-Rp for 43494@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:02 -0400 Received: from localhost (unknown [46.237.174.246]) by mira.cbaines.net (Postfix) with ESMTPSA id 507D927BBE7 for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 11:11:01 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id e31c660e for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 10:10:59 +0000 (UTC) From: Christopher Baines Date: Sat, 19 Sep 2020 11:10:56 +0100 Message-Id: <20200919101059.8930-1-mail@cbaines.net> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/packages/package-management.scm (guix-build-coordinator): New variable. --- gnu/packages/package-management.scm | 74 +++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index ec87226197..afa45d6789 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -50,6 +50,7 @@ #:use-module (gnu packages cpio) #:use-module (gnu packages crypto) #:use-module (gnu packages curl) + #:use-module (gnu packages databases) #:use-module (gnu packages dbm) #:use-module (gnu packages docbook) #:use-module (gnu packages file) @@ -981,6 +982,79 @@ environments.") ;; and the fonts included in this package are licensed OFL1.1. (license (list license:gpl3+ license:agpl3+ license:silofl1.1)))) +(define-public guix-build-coordinator + (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d") + (revision "0")) + (package + (name "guix-build-coordinator") + (version (git-version "0" revision commit)) + (source (origin + (method git-fetch) + (uri (git-reference + (url "https://git.cbaines.net/git/guix/build-coordinator") + (commit commit))) + (sha256 + (base32 + "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b")) + (file-name (string-append name "-" version "-checkout")))) + (build-system gnu-build-system) + (arguments + `(#:modules (((guix build guile-build-system) + #:select (target-guile-effective-version)) + ,@%gnu-build-system-modules) + #:imported-modules ((guix build guile-build-system) + ,@%gnu-build-system-modules) + #:phases + (modify-phases %standard-phases + (add-before 'build 'set-GUILE_AUTO_COMPILE + (lambda _ + ;; To avoid warnings relating to 'guild'. + (setenv "GUILE_AUTO_COMPILE" "0") + #t)) + (add-after 'install 'wrap-executable + (lambda* (#:key inputs outputs #:allow-other-keys) + (let* ((out (assoc-ref outputs "out")) + (bin (string-append out "/bin")) + (guile (assoc-ref inputs "guile")) + (version (target-guile-effective-version)) + (scm (string-append out "/share/guile/site/" version)) + (go (string-append out "/lib/guile/" version "/site-ccache"))) + (for-each + (lambda (file) + (simple-format (current-error-port) "wrapping: ~A\n" file) + (wrap-program file + `("PATH" ":" prefix + (,bin ,(assoc-ref inputs "sqitch"))) + `("GUILE_LOAD_PATH" ":" prefix + (,scm ,(getenv "GUILE_LOAD_PATH"))) + `("GUILE_LOAD_COMPILED_PATH" ":" prefix + (,go ,(getenv "GUILE_LOAD_COMPILED_PATH"))))) + (find-files bin))) + #t)) + (delete 'strip)))) ; As the .go files aren't compatible + (native-inputs + `(("pkg-config" ,pkg-config) + ("autoconf" ,autoconf) + ("automake" ,automake))) + (inputs + `(("guile" ,@(assoc-ref (package-native-inputs guix) "guile")) + ("guile-fibers" ,guile-fibers) + ("guile-prometheus" ,guile-prometheus) + ("guile-gcrypt" ,guile-gcrypt) + ("guile-json" ,guile-json-3) + ("guile-lzlib" ,guile-lzlib) + ("guile-sqlite3" ,guile-sqlite3) + ("guix" ,guix) + ("sqlite" ,sqlite) + ("sqitch" ,sqitch))) + (home-page "https://git.cbaines.net/guix/build-coordinator/") + (synopsis "Tool to help build derivations") + (description + "The Guix Build Coordinator helps with performing lots of builds across +potentially many machines, and with doing something with the results and +outputs of those builds.") + (license license:gpl3+)))) + (define-public guix-jupyter (package (name "guix-jupyter") From patchwork Sat Sep 19 10:10:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 24218 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 67FCF27BBE8; Sat, 19 Sep 2020 11:12:26 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, NORMAL_HTTP_TO_IP,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, UNPARSEABLE_RELAY,URIBL_BLOCKED,WEIRD_PORT autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6281E27BBE7 for ; Sat, 19 Sep 2020 11:12:24 +0100 (BST) Received: from localhost ([::1]:53980 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kJZqx-0002u6-I6 for patchwork@mira.cbaines.net; Sat, 19 Sep 2020 06:12:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49716) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJZqd-0002tC-4F for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34162) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kJZqc-0008Te-RG for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kJZqc-0007D1-Nb for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43494] [PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 19 Sep 2020 10:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43494 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43494@debbugs.gnu.org Received: via spool by 43494-submit@debbugs.gnu.org id=B43494.160051026827627 (code B ref 43494); Sat, 19 Sep 2020 10:12:02 +0000 Received: (at 43494) by debbugs.gnu.org; 19 Sep 2020 10:11:08 +0000 Received: from localhost ([127.0.0.1]:45703 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpj-0007BQ-GH for submit@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:08 -0400 Received: from mira.cbaines.net ([212.71.252.8]:57766) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpd-0007Ac-UH for 43494@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:03 -0400 Received: from localhost (unknown [46.237.174.246]) by mira.cbaines.net (Postfix) with ESMTPSA id 7EDCC27BBE8 for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 11:11:01 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id ca682f73 for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 10:10:59 +0000 (UTC) From: Christopher Baines Date: Sat, 19 Sep 2020 11:10:57 +0100 Message-Id: <20200919101059.8930-2-mail@cbaines.net> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200919101059.8930-1-mail@cbaines.net> References: <20200919101059.8930-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/services/guix.scm (): New record type. (guix-build-coordinator-configuration, guix-build-coordinator-configuration?, guix-build-coordinator-configuration-package, guix-build-coordinator-configuration-user, guix-build-coordinator-configuration-group, guix-build-coordinator-configuration-datastore-uri-string, guix-build-coordinator-configuration-agent-communication-uri-string, guix-build-coordinator-configuration-client-communication-uri-string, guix-build-coordinator-configuration-allocation-strategy, guix-build-coordinator-configuration-hooks, guix-build-coordinator-configuration-guile, make-guix-build-coordinator-start-script, guix-build-coordinator-shepherd-services, guix-build-coordinator-activation, guix-build-coordinator-account): New procedures. (guix-build-coordinator-service-type): New variable. * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable. * doc/guix.texi (Guix Services): Document it. --- doc/guix.texi | 67 ++++++++++++++ gnu/services/guix.scm | 200 +++++++++++++++++++++++++++++++++++++++++- gnu/tests/guix.scm | 75 +++++++++++++++- 3 files changed, 338 insertions(+), 4 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index f7e2204b53..4f3f0e851f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27487,6 +27487,73 @@ The complete list of possible options can be found in the man page for @node Guix Services @subsection Guix Services +@subsubheading Guix Build Coordinator +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build +Coordinator} aids in distributing derivation builds among machines +running an @dfn{agent}. The build daemon is still used to build the +derivations, but the Guix Build Coordinator manages allocating builds +and working with the results. + +@quotation Note +This service is considered experimental. Configuration options may be +changed in a backwards-incompatible manner, and not all features have +been thorougly tested. +@end quotation + +The Guix Build Coordinator consists of one @dfn{coordinator}, and one or +more connected @dfn{agent} processes. The coordinator process handles +clients submitting builds, and allocating builds to agents. The agent +processes talk to a build daemon to actually perform the builds, then +send the results back to the coordinator. + +There is a script to run the coordinator component of the Guix Build +Coordinator, but the Guix service uses a custom Guile script instead, to +provide better integration with G-expressions used in the configuration. + +@defvar {Scheme Variable} guix-build-coordinator-service-type +Service type for the Guix Build Coordinator. Its value must be a +@code{guix-build-coordinator-configuration} object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-configuration +Data type representing the configuration of the Guix Build Coordinator. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator"}) +The system user to run the service as. + +@item @code{group} (default: @code{"guix-build-coordinator"}) +The system group to run the service as. + +@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"}) +The URI to use for the database. + +@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"}) +The URI describing how to listen to requests from agent processes. + +@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"}) +The URI describing how to listen to requests from clients. The client +API allows submitting builds and currently isn't authenticated, so take +care when configuring this value. + +@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy}) +A G-expression for the allocation strategy to be used. This is a +procedure that takes the datastore as an argument and populates the +allocation plan in the database. + +@item @code{hooks} (default: @var{'()}) +An association list of hooks. These provide a way to execute arbitrary +code upon certian events, like a build result being processed. + +@item @code{guile} (default: @code{guile-3.0-latest}) +The Guile package with which to run the Guix Build Coordinator. + +@end table +@end deftp + @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information about diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index 10a8581a62..1bacd61190 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -17,20 +17,40 @@ ;;; along with GNU Guix. If not, see . (define-module (gnu services guix) + #:use-module (srfi srfi-1) #:use-module (ice-9 match) #:use-module (guix gexp) #:use-module (guix records) + #:use-module (guix packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages admin) + #:use-module (gnu packages databases) #:use-module (gnu packages web) + #:use-module (gnu packages guile) + #:use-module (gnu packages guile-xyz) + #:use-module (gnu packages package-management) #:use-module (gnu services) #:use-module (gnu services base) #:use-module (gnu services admin) #:use-module (gnu services shepherd) #:use-module (gnu services getmail) #:use-module (gnu system shadow) - #:export ( + #:export (guix-build-coordinator-configuration + guix-build-coordinator-configuration? + guix-build-coordinator-configuration-package + guix-build-coordinator-configuration-user + guix-build-coordinator-configuration-group + guix-build-coordinator-configuration-datastore-uri-string + guix-build-coordinator-configuration-agent-communication-uri-string + guix-build-coordinator-configuration-client-communication-uri-string + guix-build-coordinator-configuration-allocation-strategy + guix-build-coordinator-configuration-hooks + guix-build-coordinator-configuration-guile + + guix-build-coordinator-service-type + + guix-data-service-configuration guix-data-service-configuration? guix-data-service-package @@ -45,11 +65,185 @@ ;;;; Commentary: ;;; -;;; This module implements a service that to run instances of the Guix Data -;;; Service, which provides data about Guix over time. +;;; Services specifically related to GNU Guix. ;;; ;;;; Code: +(define-record-type* + guix-build-coordinator-configuration make-guix-build-coordinator-configuration + guix-build-coordinator-configuration? + (package guix-build-coordinator-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-configuration-user + (default "guix-build-coordinator")) + (group guix-build-coordinator-configuration-group + (default "guix-build-coordinator")) + (database-uri-string + guix-build-coordinator-configuration-datastore-uri-string + (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db")) + (agent-communication-uri-string + guix-build-coordinator-configuration-agent-communication-uri-string + (default "http://0.0.0.0:8745")) + (client-communication-uri-string + guix-build-coordinator-configuration-client-communication-uri-string + (default "http://127.0.0.1:8746")) + (allocation-strategy + guix-build-coordinator-configuration-allocation-strategy + (default #~basic-build-allocation-strategy)) + (hooks guix-build-coordinator-configuration-hooks + (default '())) + (guile guix-build-coordinator-configuration-guile + (default guile-3.0-latest))) + +(define* (make-guix-build-coordinator-start-script database-uri-string + allocation-strategy + pid-file + guix-build-coordinator-package + #:key + agent-communication-uri-string + client-communication-uri-string + (hooks '()) + (guile guile-3.0)) + (program-file + "start-guix-build-coordinator" + (with-extensions (cons guix-build-coordinator-package + ;; This is a poorly constructed Guile load path, + ;; since it contains things that aren't Guile + ;; libraries, but it means that the Guile libraries + ;; needed for the Guix Build Coordinator don't need + ;; to be individually specified here. + (map second (package-inputs + guix-build-coordinator-package))) + #~(begin + (use-modules (srfi srfi-1) + (ice-9 match) + (web uri) + (prometheus) + (guix-build-coordinator hooks) + (guix-build-coordinator datastore) + (guix-build-coordinator build-allocator) + (guix-build-coordinator coordinator)) + + (let* ((metrics-registry (make-metrics-registry + #:namespace + "guixbuildcoordinator_")) + (datastore (database-uri->datastore + #$database-uri-string + #:metrics-registry metrics-registry)) + (hooks + (list #$@(map (match-lambda + ((name . hook-gexp) + #~(cons name #$hook-gexp))) + hooks))) + (hooks-with-defaults + `(,@hooks + ,@(remove (match-lambda + ((name . _) (assq-ref hooks name))) + %default-hooks))) + (build-coordinator (make-build-coordinator + #:datastore datastore + #:hooks hooks-with-defaults + #:metrics-registry metrics-registry + #:allocation-strategy #$allocation-strategy))) + + (run-coordinator-service + build-coordinator + #:update-datastore? #t + #:pid-file #$pid-file + #:agent-communication-uri (string->uri + #$agent-communication-uri-string) + #:client-communication-uri (string->uri + #$client-communication-uri-string))))) + #:guile guile)) + +(define (guix-build-coordinator-shepherd-services config) + (match-record config + (package user group database-uri-string + agent-communication-uri-string + client-communication-uri-string + allocation-strategy + hooks + guile) + (list + (shepherd-service + (documentation "Guix Build Coordinator") + (provision '(guix-build-coordinator)) + (requirement '(networking)) + (start #~(make-forkexec-constructor + (list #$(make-guix-build-coordinator-start-script + database-uri-string + allocation-strategy + "/var/run/guix-build-coordinator/pid" + package + #:agent-communication-uri-string + agent-communication-uri-string + #:client-communication-uri-string + client-communication-uri-string + #:hooks hooks + #:guile guile)) + #:user #$user + #:group #$group + #:pid-file "/var/run/guix-build-coordinator/pid" + ;; Allow time for migrations to run + #:pid-file-timeout 60 + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/coordinator.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-activation config) + #~(begin + (use-modules (guix build utils)) + + (define %user (getpw "guix-build-coordinator")) + + (chmod "/var/lib/guix-build-coordinator" #o755) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator") + (chown "/var/run/guix-build-coordinator" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-account config) + (match-record config + (user group) + (list (user-group + (name group) + (system? #t)) + (user-account + (name user) + (group group) + (system? #t) + (comment "Guix Build Coordinator user") + (home-directory "/var/lib/guix-build-coordinator") + (shell (file-append shadow "/sbin/nologin")))))) + +(define guix-build-coordinator-service-type + (service-type + (name 'guix-build-coordinator) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-activation) + (service-extension account-service-type + guix-build-coordinator-account))) + (default-value + (guix-build-coordinator-configuration)) + (description + "Run an instance of the Guix Build Coordinator."))) + + +;;; +;;; Guix Data Service +;;; + (define-record-type* guix-data-service-configuration make-guix-data-service-configuration guix-data-service-configuration? diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm index 6139e31cf0..20b67d55d3 100644 --- a/gnu/tests/guix.scm +++ b/gnu/tests/guix.scm @@ -35,7 +35,80 @@ #:use-module (guix store) #:use-module (guix utils) #:use-module (ice-9 match) - #:export (%test-guix-data-service)) + #:export (%test-guix-build-coordinator + %test-guix-data-service)) + +;;; +;;; Guix Build Coordinator +;;; + +(define %guix-build-coordinator-os + (simple-operating-system + (service dhcp-client-service-type) + (service guix-build-coordinator-service-type))) + +(define (run-guix-build-coordinator-test) + (define os + (marionette-operating-system + %guix-build-coordinator-os + #:imported-modules '((gnu services herd) + (guix combinators)))) + + (define forwarded-port 8745) + + (define vm + (virtual-machine + (operating-system os) + (memory-size 1024) + (port-forwardings `((,forwarded-port . 8745))))) + + (define test + (with-imported-modules '((gnu build marionette)) + #~(begin + (use-modules (srfi srfi-11) (srfi srfi-64) + (gnu build marionette) + (web uri) + (web client) + (web response)) + + (define marionette + (make-marionette (list #$vm))) + + (mkdir #$output) + (chdir #$output) + + (test-begin "guix-build-coordinator") + + (test-assert "service running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (match (start-service 'guix-build-coordinator) + (#f #f) + (('service response-parts ...) + (match (assq-ref response-parts 'running) + ((pid) (number? pid)))))) + marionette)) + + (test-equal "http-get" + 200 + (let-values + (((response text) + (http-get #$(simple-format + #f "http://localhost:~A/metrics" forwarded-port) + #:decode-body? #t))) + (response-code response))) + + (test-end) + (exit (= (test-runner-fail-count (test-runner-current)) 0))))) + + (gexp->derivation "guix-build-coordinator-test" test)) + +(define %test-guix-build-coordinator + (system-test + (name "guix-build-coordinator") + (description "Connect to a running Guix Build Coordinator.") + (value (run-guix-build-coordinator-test)))) ;;; From patchwork Sat Sep 19 10:10:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 24220 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2F34E27BBE8; Sat, 19 Sep 2020 11:12:34 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id DC82D27BBE7 for ; Sat, 19 Sep 2020 11:12:32 +0100 (BST) Received: from localhost ([::1]:54152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kJZr6-0002ye-4a for patchwork@mira.cbaines.net; Sat, 19 Sep 2020 06:12:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49714) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJZqc-0002t3-NP for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34161) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kJZqc-0008Ta-EP for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kJZqc-0007Cu-AR for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43494] [PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 19 Sep 2020 10:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43494 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43494@debbugs.gnu.org Received: via spool by 43494-submit@debbugs.gnu.org id=B43494.160051026727619 (code B ref 43494); Sat, 19 Sep 2020 10:12:02 +0000 Received: (at 43494) by debbugs.gnu.org; 19 Sep 2020 10:11:07 +0000 Received: from localhost ([127.0.0.1]:45701 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpj-0007BN-0V for submit@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:07 -0400 Received: from mira.cbaines.net ([212.71.252.8]:57768) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpe-0007Ae-2j for 43494@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:03 -0400 Received: from localhost (unknown [46.237.174.246]) by mira.cbaines.net (Postfix) with ESMTPSA id 95E4927BBE9 for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 11:11:01 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 67c71c7c for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 10:10:59 +0000 (UTC) From: Christopher Baines Date: Sat, 19 Sep 2020 11:10:58 +0100 Message-Id: <20200919101059.8930-3-mail@cbaines.net> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200919101059.8930-1-mail@cbaines.net> References: <20200919101059.8930-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/services/guix.scm (): New record type. (guix-build-coordinator-agent-configuration, guix-build-coordinator-agent-configuration?, guix-build-coordinator-agent-configuration-package, guix-build-coordinator-agent-configuration-user, guix-build-coordinator-agent-configuration-coordinator, guix-build-coordinator-agent-configuration-uuid), guix-build-coordinator-agent-configuration-password, guix-build-coordinator-agent-configuration-password-file, guix-build-coordinator-agent-configuration-systems, guix-build-coordinator-agent-configuration-max-parallel-builds, guix-build-coordinator-agent-configuration-derivation-substitute-urls, guix-build-coordinator-agent-configuration-non-derivation-substitute-urls, guix-build-coordinator-agent-shepherd-services, guix-build-coordinator-agent-activation, guix-build-coordinator-agent-account): New procedures. (guix-build-coordinator-agent-service-type): New variable. * doc/guix.texi (Guix Services): Document it. --- doc/guix.texi | 50 +++++++++++++++++ gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index 4f3f0e851f..b2dde30ddb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27554,6 +27554,56 @@ The Guile package with which to run the Guix Build Coordinator. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type +Service type for a Guix Build Coordinator agent. Its value must be a +@code{guix-build-coordinator-agent-configuration} object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-agent-configuration +Data type representing the configuration a Guix Build Coordinator agent. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator-agent"}) +The system user to run the service as. + +@item @code{coordinator} (default: @code{"http://localhost:8745"}) +The URI to use when connecting to the coordinator. + +@item @code{uuid} +The UUID of the agent. This should be generated by the coordinator +process, stored in the coordinator database, and used by the intended +agent. + +@item @code{password} (default: @code{#f}) +The password to use when connecting to the coordinator. A file to read +the password from can also be specified, and this is more secure. + +@item @code{password-file} (default: @code{#f}) +A file containing the password to use when connecting to the +coordinator. + +@item @code{systems} (default: @var{#f}) +The systems for which this agent should fetch builds. The agent process +will use the current system it's running on as the default. + +@item @code{max-parallel-builds} (default: @code{1}) +The number of builds to perform in parallel. + +@item @code{derivation-substitute-urls} (default: @code{1}) +URLs from which to attempt to fetch substitutes for derivations, if the +derivations aren't already available. + +@item @code{non-derivation-substitute-urls} (default: @code{1}) +URLs from which to attempt to fetch substitutes for build inputs, if the +input store items aren't already available. + +@end table +@end deftp + + @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information about diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index 1bacd61190..e4f5cf57b9 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -50,6 +50,21 @@ guix-build-coordinator-service-type + guix-build-coordinator-agent-configuration + guix-build-coordinator-agent-configuration? + guix-build-coordinator-agent-configuration-package + guix-build-coordinator-agent-configuration-user + guix-build-coordinator-agent-configuration-coordinator + guix-build-coordinator-agent-configuration-uuid + guix-build-coordinator-agent-configuration-password + guix-build-coordinator-agent-configuration-password-file + guix-build-coordinator-agent-configuration-systems + guix-build-coordinator-agent-configuration-max-parallel-builds + guix-build-coordinator-agent-configuration-derivation-substitute-urls + guix-build-coordinator-agent-configuration-non-derivation-substitute-urls + + guix-build-coordinator-agent-service-type + guix-data-service-configuration guix-data-service-configuration? @@ -95,6 +110,33 @@ (guile guix-build-coordinator-configuration-guile (default guile-3.0-latest))) +(define-record-type* + guix-build-coordinator-agent-configuration + make-guix-build-coordinator-agent-configuration + guix-build-coordinator-agent-configuration? + (package guix-build-coordinator-agent-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-agent-configuration-user + (default "guix-build-coordinator-agent")) + (coordinator guix-build-coordinator-agent-configuration-coordinator + (default "http://localhost:8745")) + (uuid guix-build-coordinator-agent-configuration-uuid) + (password guix-build-coordinator-agent-configuration-password + (default #f)) + (password-file guix-build-coordinator-agent-configuration-password-file + (default #f)) + (systems guix-build-coordinator-agent-configuration-systems + (default #f)) + (max-parallel-builds + guix-build-coordinator-agent-configuration-max-parallel-builds + (default 1)) + (derivation-substitute-urls + guix-build-coordinator-agent-configuration-derivation-substitute-urls + (default #f)) + (non-derivation-substitute-urls + guix-build-coordinator-agent-configuration-non-derivation-substitute-urls + (default #f))) + (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file @@ -239,6 +281,85 @@ (description "Run an instance of the Guix Build Coordinator."))) +(define (guix-build-coordinator-agent-shepherd-services config) + (match-record config + (package user coordinator uuid password password-file max-parallel-builds + derivation-substitute-urls non-derivation-substitute-urls + systems) + (list + (shepherd-service + (documentation "Guix Build Coordinator Agent") + (provision '(guix-build-coordinator-agent)) + (requirement '(networking)) + (start #~(make-forkexec-constructor + (list #$(file-append package "/bin/guix-build-coordinator-agent") + #$(string-append "--coordinator=" coordinator) + #$(string-append "--uuid=" uuid) + #$@(if password + #~(#$(string-append "--password=" password)) + #~()) + #$@(if password-file + #~(#$(string-append "--password-file=" password-file)) + #~()) + #$(simple-format #f "--max-parallel-builds=~A" + max-parallel-builds) + #$@(if derivation-substitute-urls + #~(#$(string-append + "--derivation-substitute-urls=" + (string-join derivation-substitute-urls " "))) + #~()) + #$@(if non-derivation-substitute-urls + #~(#$(string-append + "--non-derivation-substitute-urls=" + (string-join derivation-substitute-urls " "))) + #~()) + #$@(map (lambda (system) + (string-append "--system=" system)) + (or systems '()))) + #:user #$user + #:pid-file "/var/run/guix-build-coordinator-agent/pid" + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/agent.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-agent-activation config) + #~(begin + (use-modules (guix build utils)) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator-agent") + (chown "/var/run/guix-build-coordinator-agent" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-agent-account config) + (list (user-account + (name (guix-build-coordinator-agent-configuration-user config)) + (group "nogroup") + (system? #t) + (comment "Guix Build Coordinator agent user") + (home-directory "/var/empty") + (shell (file-append shadow "/sbin/nologin"))))) + +(define guix-build-coordinator-agent-service-type + (service-type + (name 'guix-build-coordinator-agent) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-agent-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-agent-activation) + (service-extension account-service-type + guix-build-coordinator-agent-account))) + (description + "Run a Guix Build Coordinator agent."))) + ;;; ;;; Guix Data Service From patchwork Sat Sep 19 10:10:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 24219 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 1340E27BBE8; Sat, 19 Sep 2020 11:12:31 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,UNPARSEABLE_RELAY,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 006F027BBE7 for ; Sat, 19 Sep 2020 11:12:30 +0100 (BST) Received: from localhost ([::1]:54040 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kJZr3-0002vu-6A for patchwork@mira.cbaines.net; Sat, 19 Sep 2020 06:12:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49718) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJZqd-0002tT-Gt for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34163) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kJZqd-0008Ti-7I for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kJZqd-0007D9-3R for guix-patches@gnu.org; Sat, 19 Sep 2020 06:12:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 19 Sep 2020 10:12:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43494 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43494@debbugs.gnu.org Received: via spool by 43494-submit@debbugs.gnu.org id=B43494.160051026827633 (code B ref 43494); Sat, 19 Sep 2020 10:12:03 +0000 Received: (at 43494) by debbugs.gnu.org; 19 Sep 2020 10:11:08 +0000 Received: from localhost ([127.0.0.1]:45705 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpk-0007BY-9y for submit@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:08 -0400 Received: from mira.cbaines.net ([212.71.252.8]:57770) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJZpe-0007An-7U for 43494@debbugs.gnu.org; Sat, 19 Sep 2020 06:11:03 -0400 Received: from localhost (unknown [46.237.174.246]) by mira.cbaines.net (Postfix) with ESMTPSA id BBC6D27BBEA for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 11:11:01 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id be83c506 for <43494@debbugs.gnu.org>; Sat, 19 Sep 2020 10:10:59 +0000 (UTC) From: Christopher Baines Date: Sat, 19 Sep 2020 11:10:59 +0100 Message-Id: <20200919101059.8930-4-mail@cbaines.net> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200919101059.8930-1-mail@cbaines.net> References: <20200919101059.8930-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/services/guix.scm (): New record type. (guix-build-coordinator-queue-builds-configuration, guix-build-coordinator-queue-builds-configuration?, guix-build-coordinator-queue-builds-configuration-package, guix-build-coordinator-queue-builds-configuration-user, guix-build-coordinator-queue-builds-coordinator, guix-build-coordinator-queue-builds-configuration-systems, guix-build-coordinator-queue-builds-configuration-system-and-targets, guix-build-coordinator-queue-builds-configuration-guix-data-service, guix-build-coordinator-queue-builds-configuration-processed-commits-file, guix-build-coordinator-queue-builds-shepherd-services, guix-build-coordinator-queue-builds-activation, guix-build-coordinator-queue-builds-account): New procedures. (guix-build-coordinator-queue-builds-service-type): New variable. --- doc/guix.texi | 38 +++++++++++++++ gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index b2dde30ddb..767c99c5cb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27603,6 +27603,44 @@ input store items aren't already available. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type +Service type for the +guix-build-coordinator-queue-builds-from-guix-data-service script. Its +value must be a @code{guix-build-coordinator-queue-builds-configuration} +object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-queue-builds-configuration +Data type representing the options to the queue builds from guix data +service script. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"}) +The system user to run the service as. + +@item @code{coordinator} (default: @code{"http://localhost:8745"}) +The URI to use when connecting to the coordinator. + +@item @code{systems} (default: @code{#f}) +The systems for which to fetch derivations to build. + +@item @code{systems-and-targets} (default: @code{#f}) +An association list of system and target pairs for which to fetch +derivations to build. + +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"}) +The Guix Data Service instance from which to query to find out about +derivations to build. + +@item @code{processed-commits-file} (default: @code{"/var/cache/guix-build-coordinator-queue-builds/processed-commits"}) +A file to record which commits have been processed, to avoid needlessly +processing them again if the service is restarted. + +@end table +@end deftp @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index e4f5cf57b9..1493a24be5 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -65,6 +65,18 @@ guix-build-coordinator-agent-service-type + guix-build-coordinator-queue-builds-configuration + guix-build-coordinator-queue-builds-configuration? + guix-build-coordinator-queue-builds-configuration-package + guix-build-coordinator-queue-builds-configuration-user + guix-build-coordinator-queue-builds-coordinator + guix-build-coordinator-queue-builds-configuration-systems + guix-build-coordinator-queue-builds-configuration-system-and-targets + guix-build-coordinator-queue-builds-configuration-guix-data-service + guix-build-coordinator-queue-builds-configuration-processed-commits-file + + guix-build-coordinator-queue-builds-service-type + guix-data-service-configuration guix-data-service-configuration? @@ -137,6 +149,28 @@ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls (default #f))) +(define-record-type* + guix-build-coordinator-queue-builds-configuration + make-guix-build-coordinator-queue-builds-configuration + guix-build-coordinator-queue-builds-configuration? + (package guix-build-coordinator-queue-builds-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-queue-builds-configuration-user + (default "guix-build-coordinator-queue-builds")) + (coordinator guix-build-coordinator-queue-builds-coordinator + (default "http://localhost:8745")) + (systems guix-build-coordinator-queue-builds-configuration-systems + (default #f)) + (systems-and-targets + guix-build-coordinator-queue-builds-configuration-system-and-targets + (default #f)) + (guix-data-service + guix-build-coordinator-queue-builds-configuration-guix-data-service + (default "https://data.guix.gnu.org")) + (processed-commits-file + guix-build-coordinator-queue-builds-configuration-processed-commits-file + (default "/var/cache/guix-build-coordinator-queue-builds/processed-commits"))) + (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file @@ -360,6 +394,80 @@ (description "Run a Guix Build Coordinator agent."))) +(define (guix-build-coordinator-queue-builds-shepherd-services config) + (match-record config + (package user coordinator systems systems-and-targets + guix-data-service processed-commits-file) + (list + (shepherd-service + (documentation "Guix Build Coordinator queue builds from Guix Data Service") + (provision '(guix-build-coordinator-queue-builds)) + (requirement '(networking)) + (start + #~(make-forkexec-constructor + (list + #$(file-append + package + "/bin/guix-build-coordinator-queue-builds-from-guix-data-service") + #$(string-append "--coordinator=" coordinator) + #$@(map (lambda (system) + (string-append "--system=" system)) + (or systems '())) + #$@(map (match-lambda + ((system . target) + (string-append "--system-and-target=" system "=" target))) + (or systems-and-targets '())) + #$@(if guix-data-service + #~(#$(string-append "--guix-data-service=" guix-data-service)) + #~()) + #$@(if processed-commits-file + #~(#$(string-append "--processed-commits-file=" + processed-commits-file)) + #~())) + #:user #$user + #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid" + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/queue-builds.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-queue-builds-activation config) + #~(begin + (use-modules (guix build utils)) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator-queue-builds") + (chown "/var/run/guix-build-coordinator-queue-builds" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-queue-builds-account config) + (list (user-account + (name (guix-build-coordinator-queue-builds-configuration-user config)) + (group "nogroup") + (system? #t) + (comment "Guix Build Coordinator queue-builds user") + (home-directory "/var/empty") + (shell (file-append shadow "/sbin/nologin"))))) + +(define guix-build-coordinator-queue-builds-service-type + (service-type + (name 'guix-build-coordinator-queue-builds) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-queue-builds-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-queue-builds-activation) + (service-extension account-service-type + guix-build-coordinator-queue-builds-account))) + (description + "Run the Guix Build Coordinator queue builds script."))) + ;;; ;;; Guix Data Service