| Message ID | 20200918184042.22660-4-mail@cbaines.net |
|---|---|
| State | Accepted |
| Headers | show |
| Series | Add package and services for the Guix Build Coordinator | expand |
| Context | Check | Description |
|---|---|---|
| cbaines/comparison | success | View comparision |
| cbaines/git branch | success | View Git branch |
| cbaines/applying patch | success | View Laminar job |
Christopher Baines <mail@cbaines.net> skribis: > * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>): > New record type. > (guix-build-coordinator-queue-builds-configuration, > guix-build-coordinator-queue-builds-configuration?, > guix-build-coordinator-queue-builds-configuration-package, > guix-build-coordinator-queue-builds-configuration-user, > guix-build-coordinator-queue-builds-coordinator, > guix-build-coordinator-queue-builds-configuration-systems, > guix-build-coordinator-queue-builds-configuration-system-and-targets, > guix-build-coordinator-queue-builds-configuration-guix-data-service, > guix-build-coordinator-queue-builds-configuration-processed-commits-file, > guix-build-coordinator-queue-builds-shepherd-services, > guix-build-coordinator-queue-builds-activation, > guix-build-coordinator-queue-builds-account): New procedures. > (guix-build-coordinator-queue-builds-service-type): New variable. [...] > +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type > +Service type for the > +guix-build-coordinator-queue-builds-from-guix-data-service script. Its Oh! :-) ‘guix-build-queue’ maybe? Because after all, it can queue builds coming from anywhere, not just from the Data Service, right? It’d be nice to add a sentence or two explaining how the “queue” fits in the big picture (I’m not quite sure :-)). > +@item @code{systems-and-targets} (default: @code{#f}) > +An association list of system and target pairs for which to fetch > +derivations to build. Cross-compilation target triplets are a notion that’s not available at the derivation level; it only exists for packages. Are we mixing things here? > +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"}) > +The Guix Data Service instance from which to query to find out about > +derivations to build. Ah so the queue pulls from the Data Service. Got it! > +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"}) > +A file to record which commits have been processed, to avoid needlessly > +processing them again if the service is restarted. Maybe in /var/cache by default, no? > + (processed-commits-file > + guix-build-coordinator-queue-builds-configuration-processed-commits-file > + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits"))) Maybe “state-file”? In addition to saving space :-), it’d leave room for additional bits of state. That’s all, thank you! Ludo’.
Ludovic Courtès <ludo@gnu.org> writes: > Christopher Baines <mail@cbaines.net> skribis: > >> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>): >> New record type. >> (guix-build-coordinator-queue-builds-configuration, >> guix-build-coordinator-queue-builds-configuration?, >> guix-build-coordinator-queue-builds-configuration-package, >> guix-build-coordinator-queue-builds-configuration-user, >> guix-build-coordinator-queue-builds-coordinator, >> guix-build-coordinator-queue-builds-configuration-systems, >> guix-build-coordinator-queue-builds-configuration-system-and-targets, >> guix-build-coordinator-queue-builds-configuration-guix-data-service, >> guix-build-coordinator-queue-builds-configuration-processed-commits-file, >> guix-build-coordinator-queue-builds-shepherd-services, >> guix-build-coordinator-queue-builds-activation, >> guix-build-coordinator-queue-builds-account): New procedures. >> (guix-build-coordinator-queue-builds-service-type): New variable. > > [...] > >> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type >> +Service type for the >> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its > > Oh! :-) > > ‘guix-build-queue’ maybe? Because after all, it can queue builds coming > from anywhere, not just from the Data Service, right? > > It’d be nice to add a sentence or two explaining how the “queue” fits in > the big picture (I’m not quite sure :-)). So, this got me thinking a bit, maybe this script should actually exist outside of the guix-build-coordinator repository... You can run `guix-build-coordinator build ...` to build a derivation, and what this does is just make the relevant HTTP request to the coordinator. What this script does is just fetch derivation names from an instance of the Guix Data Service, and ask the Guix Build Coordinator to build them. >> +@item @code{systems-and-targets} (default: @code{#f}) >> +An association list of system and target pairs for which to fetch >> +derivations to build. > > Cross-compilation target triplets are a notion that’s not available at > the derivation level; it only exists for packages. > > Are we mixing things here? This setting is basically configuing the query parameters to use for this page for example [1]. You're right that it's not available at the derivation level, but you can ask the Guix Data Service for derivations for packages which were computed for a specific system and target. 1: http://data.guix.gnu.org/revision/70ef8b24550c54cc8e9f20026bfd24b8680499b4/package-derivations >> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"}) >> +The Guix Data Service instance from which to query to find out about >> +derivations to build. > > Ah so the queue pulls from the Data Service. Got it! I wouldn't read too much in to "queue" in the name here, "submit" might be a better word to use, since that's the term used on the Guix Build Coordinator side. There's also nothing resemling a queue anywhere... >> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"}) >> +A file to record which commits have been processed, to avoid needlessly >> +processing them again if the service is restarted. > > Maybe in /var/cache by default, no? Ah, yeah, I've changed it to be in /var/cache, as that's fine. >> + (processed-commits-file >> + guix-build-coordinator-queue-builds-configuration-processed-commits-file >> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits"))) > > Maybe “state-file”? In addition to saving space :-), it’d leave room > for additional bits of state. Currently, it's just a file with each commit on its own line. Given that, if there's a need to store more state, it might just end up in different files. If that happens, this setting could become a directory, rather than a specific file to avoid adding extra configuration options.
diff --git a/doc/guix.texi b/doc/guix.texi index db91831197..1b47a01217 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27590,6 +27590,44 @@ input store items aren't already available. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type +Service type for the +guix-build-coordinator-queue-builds-from-guix-data-service script. Its +value must be a @code{guix-build-coordinator-queue-builds-configuration} +object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-queue-builds-configuration +Data type representing the options to the queue builds from guix data +service script. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"}) +The system user to run the service as. + +@item @code{coordinator} (default: @code{"http://localhost:8745"}) +The URI to use when connecting to the coordinator. + +@item @code{systems} (default: @code{#f}) +The systems for which to fetch derivations to build. + +@item @code{systems-and-targets} (default: @code{#f}) +An association list of system and target pairs for which to fetch +derivations to build. + +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"}) +The Guix Data Service instance from which to query to find out about +derivations to build. + +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"}) +A file to record which commits have been processed, to avoid needlessly +processing them again if the service is restarted. + +@end table +@end deftp @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index 71e58237da..975febebd1 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -65,6 +65,18 @@ guix-build-coordinator-agent-service-type + guix-build-coordinator-queue-builds-configuration + guix-build-coordinator-queue-builds-configuration? + guix-build-coordinator-queue-builds-configuration-package + guix-build-coordinator-queue-builds-configuration-user + guix-build-coordinator-queue-builds-coordinator + guix-build-coordinator-queue-builds-configuration-systems + guix-build-coordinator-queue-builds-configuration-system-and-targets + guix-build-coordinator-queue-builds-configuration-guix-data-service + guix-build-coordinator-queue-builds-configuration-processed-commits-file + + guix-build-coordinator-queue-builds-service-type + <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration? @@ -137,6 +149,28 @@ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls (default #f))) +(define-record-type* <guix-build-coordinator-queue-builds-configuration> + guix-build-coordinator-queue-builds-configuration + make-guix-build-coordinator-queue-builds-configuration + guix-build-coordinator-queue-builds-configuration? + (package guix-build-coordinator-queue-builds-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-queue-builds-configuration-user + (default "guix-build-coordinator-queue-builds")) + (coordinator guix-build-coordinator-queue-builds-coordinator + (default "http://localhost:8745")) + (systems guix-build-coordinator-queue-builds-configuration-systems + (default #f)) + (systems-and-targets + guix-build-coordinator-queue-builds-configuration-system-and-targets + (default #f)) + (guix-data-service + guix-build-coordinator-queue-builds-configuration-guix-data-service + (default "https://data.guix.gnu.org")) + (processed-commits-file + guix-build-coordinator-queue-builds-configuration-processed-commits-file + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits"))) + (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file @@ -364,6 +398,80 @@ (description "Run an instance of the Guix Build Coordinator."))) +(define (guix-build-coordinator-queue-builds-shepherd-services config) + (match-record config <guix-build-coordinator-queue-builds-configuration> + (package user coordinator systems systems-and-targets + guix-data-service processed-commits-file) + (list + (shepherd-service + (documentation "Guix Build Coordinator queue builds from Guix Data Service") + (provision '(guix-build-coordinator-queue-builds)) + (requirement '(networking)) + (start + #~(make-forkexec-constructor + (list + #$(file-append + package + "/bin/guix-build-coordinator-queue-builds-from-guix-data-service") + #$(string-append "--coordinator=" coordinator) + #$@(map (lambda (system) + (string-append "--system=" system)) + (or systems '())) + #$@(map (match-lambda + ((system . target) + (string-append "--system-and-target=" system "=" target))) + (or systems-and-targets '())) + #$@(if guix-data-service + #~(#$(string-append "--guix-data-service=" guix-data-service)) + #~()) + #$@(if processed-commits-file + #~(#$(string-append "--processed-commits-file=" + processed-commits-file)) + #~())) + #:user #$user + #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid" + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/queue-builds.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-queue-builds-activation config) + #~(begin + (use-modules (guix build utils)) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator-queue-builds") + (chown "/var/run/guix-build-coordinator-queue-builds" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-queue-builds-account config) + (list (user-account + (name (guix-build-coordinator-queue-builds-configuration-user config)) + (group "nogroup") + (system? #t) + (comment "Guix Build Coordinator queue-builds user") + (home-directory "/var/empty") + (shell (file-append shadow "/sbin/nologin"))))) + +(define guix-build-coordinator-queue-builds-service-type + (service-type + (name 'guix-build-coordinator-queue-builds) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-queue-builds-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-queue-builds-activation) + (service-extension account-service-type + guix-build-coordinator-queue-builds-account))) + (description + "Run the Guix Build Coordinator queue builds script."))) + ;;; ;;; Guix Data Service