From patchwork Sun Sep 22 12:03:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 15436 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 7745717408; Sun, 22 Sep 2019 13:04:19 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id CCFF117406 for ; Sun, 22 Sep 2019 13:04:18 +0100 (BST) Received: from localhost ([::1]:46548 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iC0bC-0005SE-Dv for patchwork@mira.cbaines.net; Sun, 22 Sep 2019 08:04:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55500) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iC0b2-0005Q2-Lj for guix-patches@gnu.org; Sun, 22 Sep 2019 08:04:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iC0aw-0005uY-Ut for guix-patches@gnu.org; Sun, 22 Sep 2019 08:04:08 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:52862) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iC0aw-0005u6-AN for guix-patches@gnu.org; Sun, 22 Sep 2019 08:04:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1iC0aw-0002N9-3D for guix-patches@gnu.org; Sun, 22 Sep 2019 08:04:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#37412] [PATCH 2/2] services: Add the Guix Data Service. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 22 Sep 2019 12:04:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 37412 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 37412@debbugs.gnu.org Received: via spool by 37412-submit@debbugs.gnu.org id=B37412.15691538229079 (code B ref 37412); Sun, 22 Sep 2019 12:04:02 +0000 Received: (at 37412) by debbugs.gnu.org; 22 Sep 2019 12:03:42 +0000 Received: from localhost ([127.0.0.1]:33448 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iC0ab-0002MM-5w for submit@debbugs.gnu.org; Sun, 22 Sep 2019 08:03:41 -0400 Received: from mira.cbaines.net ([212.71.252.8]:35898) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iC0aY-0002M7-Vh for 37412@debbugs.gnu.org; Sun, 22 Sep 2019 08:03:40 -0400 Received: from localhost (220.185.93.209.dyn.plus.net [209.93.185.220]) by mira.cbaines.net (Postfix) with ESMTPSA id 265C917407 for <37412@debbugs.gnu.org>; Sun, 22 Sep 2019 13:03:38 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b34b6487 for <37412@debbugs.gnu.org>; Sun, 22 Sep 2019 12:03:37 +0000 (UTC) From: Christopher Baines Date: Sun, 22 Sep 2019 13:03:37 +0100 Message-Id: <20190922120337.3533-2-mail@cbaines.net> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190922120337.3533-1-mail@cbaines.net> References: <87v9tt1l0h.fsf@cbaines.net> <20190922120337.3533-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/services/guix.scm: New file. * gnu/tests/guix.scm: New file. * gnu/local.mk (GNU_SYSTEM_MODULES): Add both new files. * doc/guix.texi (Guix Services): New section documenting the Guix Data Service. --- doc/guix.texi | 52 +++++++++++ gnu/local.mk | 2 + gnu/services/guix.scm | 212 ++++++++++++++++++++++++++++++++++++++++++ gnu/tests/guix.scm | 173 ++++++++++++++++++++++++++++++++++ 4 files changed, 439 insertions(+) create mode 100644 gnu/services/guix.scm create mode 100644 gnu/tests/guix.scm diff --git a/doc/guix.texi b/doc/guix.texi index 39d4b865f6..24da18ea13 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11657,6 +11657,7 @@ declaration. * Virtualization Services:: Virtualization services. * Version Control Services:: Providing remote access to Git repositories. * Game Services:: Game servers. +* Guix Services:: Services relating specifically to Guix. * Miscellaneous Services:: Other services. @end menu @@ -24167,6 +24168,57 @@ The port to bind the server to. @end table @end deftp + +@node Guix Services +@subsection Guix Services + +@subsubheading Guix Data Service +The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores +and provides data about GNU Guix. This includes information about +packages, derivations and lint warnings. + +The data is stored in a PostgreSQL database, and available through a web +interface. + +@defvar {Scheme Variable} guix-data-service-type +Service type for the Guix Data Service. Its value must be a +@code{guix-data-service-configuration} object. The service optionally +extends the getmail service, as the guix-commits mailing list is used to +find out about changes in the Guix git repository. +@end defvar + +@deftp {Data Type} guix-data-service-configuration +Data type representing the configuration of the Guix Data Service. + +@table @asis +@item @code{package} (default: @code{guix-data-service}) +The Guix Data Service package to use. + +@item @code{user} (default: @code{"guix-data-service"}) +The system user to run the service as. + +@item @code{group} (default: @code{"guix-data-service"}) +The system group to run the service as. + +@item @code{port} (default: @code{8765}) +The port to bind the web service to. + +@item @code{host} (default: @code{"127.0.0.1"}) +The host to bind the web service to. + +@item @code{getmail-idle-mailboxes} (default: @code{#f}) +If set, this is the list of mailboxes that the getmail service will be +configured to listen to. + +@item @code{commits-getmail-retriever-configuration} (default: @code{#f}) +If set, this is the @code{getmail-retriever-configuration} object with +which to configure getmail to fetch mail from the guix-commits mailing +list. + +@end table +@end deftp + + @node Miscellaneous Services @subsection Miscellaneous Services diff --git a/gnu/local.mk b/gnu/local.mk index 3f32b9cbf2..b30afdd585 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -532,6 +532,7 @@ GNU_SYSTEM_MODULES = \ %D%/services/authentication.scm \ %D%/services/games.scm \ %D%/services/getmail.scm \ + %D%/services/guix.scm \ %D%/services/kerberos.scm \ %D%/services/lirc.scm \ %D%/services/virtualization.scm \ @@ -596,6 +597,7 @@ GNU_SYSTEM_MODULES = \ %D%/tests/desktop.scm \ %D%/tests/dict.scm \ %D%/tests/docker.scm \ + %D%/tests/guix.scm \ %D%/tests/monitoring.scm \ %D%/tests/nfs.scm \ %D%/tests/install.scm \ diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm new file mode 100644 index 0000000000..0f0fad39b0 --- /dev/null +++ b/gnu/services/guix.scm @@ -0,0 +1,212 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2019 Christopher Baines +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify +;;; it under the terms of the GNU General Public License as published by +;;; the Free Software Foundation, either version 3 of the License, or +;;; (at your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, +;;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu services guix) + #:use-module (ice-9 match) + #:use-module (guix gexp) + #:use-module (guix records) + #:use-module ((gnu packages base) + #:select (glibc-utf8-locales)) + #:use-module (gnu packages admin) + #:use-module (gnu packages web) + #:use-module (gnu services) + #:use-module (gnu services base) + #:use-module (gnu services admin) + #:use-module (gnu services shepherd) + #:use-module (gnu services getmail) + #:use-module (gnu system shadow) + #:export ( + guix-data-service-configuration + guix-data-service-configuration? + guix-data-service-package + guix-data-service-user + guix-data-service-group + guix-data-service-port + guix-data-service-host + guix-data-service-getmail-idle-mailboxes + guix-data-service-commits-getmail-retriever-configuration + + guix-data-service-type)) + +;;;; Commentary: +;;; +;;; This module implements a service that to run instances of the Guix Data +;;; Service, which provides data about Guix over time. +;;; +;;;; Code: + +(define-record-type* + guix-data-service-configuration make-guix-data-service-configuration + guix-data-service-configuration? + (package guix-data-service-package + (default guix-data-service)) + (user guix-data-service-configuration-user + (default "guix-data-service")) + (group guix-data-service-configuration-group + (default "guix-data-service")) + (port guix-data-service-port + (default 8765)) + (host guix-data-service-host + (default "127.0.0.1")) + (getmail-idle-mailboxes + guix-data-service-getmail-idle-mailboxes + (default #f)) + (commits-getmail-retriever-configuration + guix-data-service-commits-getmail-retriever-configuration + (default #f))) + +(define (guix-data-service-profile-packages config) + "Return the guix-data-service package, this will populate the +ca-certificates.crt file in the system profile." + (list + (guix-data-service-package config))) + +(define (guix-data-service-shepherd-services config) + (match-record config + (package user group port host) + (list + (shepherd-service + (documentation "Guix Data Service web server") + (provision '(guix-data-service)) + (requirement '(postgres networking)) + (start #~(make-forkexec-constructor + (list #$(file-append package + "/bin/guix-data-service") + "--pid-file=/var/run/guix-data-service/pid" + #$(string-append "--port=" (number->string port)) + #$(string-append "--host=" host) + ;; Perform any database migrations when the + ;; service is started + "--update-database") + + #:user #$user + #:group #$group + #:pid-file "/var/run/guix-data-service/pid" + ;; Allow time for migrations to run + #:pid-file-timeout 60 + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-data-service/web.log")) + (stop #~(make-kill-destructor))) + + (shepherd-service + (documentation "Guix Data Service process jobs") + (provision '(guix-data-service-process-jobs)) + (requirement '(postgres + networking + ;; Require guix-data-service, as that the database + ;; migrations are handled through this service + guix-data-service)) + (start #~(make-forkexec-constructor + (list + #$(file-append package + "/bin/guix-data-service-process-jobs")) + #:user #$user + #:group #$group + #:environment-variables + `("HOME=/var/lib/guix-data-service" + "GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt" + ,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-data-service/process-jobs.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-data-service-activation config) + #~(begin + (use-modules (guix build utils)) + + (define %user (getpw "guix-data-service")) + + (chmod "/var/lib/guix-data-service" #o755) + + (mkdir-p "/var/log/guix-data-service") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-data-service") + (chown "/var/run/guix-data-service" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-data-service-account config) + (match-record config + (user group) + (list (user-group + (name group) + (system? #t)) + (user-account + (name user) + (group group) + (system? #t) + (comment "Guix Data Service user") + (home-directory "/var/lib/guix-data-service") + (shell (file-append shadow "/sbin/nologin")))))) + +(define (guix-data-service-getmail-configuration config) + (match config + (($ package user group + port host + #f #f) + '()) + (($ package user group + port host + getmail-idle-mailboxes + commits-getmail-retriever-configuration) + (list + (getmail-configuration + (name 'guix-data-service) + (user user) + (group group) + (directory "/var/lib/getmail/guix-data-service") + (rcfile + (getmail-configuration-file + (retriever commits-getmail-retriever-configuration) + (destination + (getmail-destination-configuration + (type "MDA_external") + (path (file-append + package + "/bin/guix-data-service-process-branch-updated-email")))) + (options + (getmail-options-configuration + (read-all #f) + (delivered-to #f) + (received #f))))) + (idle getmail-idle-mailboxes)))))) + +(define guix-data-service-type + (service-type + (name 'guix-data-service) + (extensions + (list + (service-extension profile-service-type + guix-data-service-profile-packages) + (service-extension shepherd-root-service-type + guix-data-service-shepherd-services) + (service-extension activation-service-type + guix-data-service-activation) + (service-extension account-service-type + guix-data-service-account) + (service-extension getmail-service-type + guix-data-service-getmail-configuration))) + (default-value + (guix-data-service-configuration)) + (description + "Run an instance of the Guix Data Service."))) diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm new file mode 100644 index 0000000000..6139e31cf0 --- /dev/null +++ b/gnu/tests/guix.scm @@ -0,0 +1,173 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2019 Christopher Baines +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu tests guix) + #:use-module (gnu tests) + #:use-module (gnu system) + #:use-module (gnu system file-systems) + #:use-module (gnu system shadow) + #:use-module (gnu system vm) + #:use-module (gnu services) + #:use-module (gnu services guix) + #:use-module (gnu services databases) + #:use-module (gnu services shepherd) + #:use-module (gnu services networking) + #:use-module (gnu packages databases) + #:use-module (guix packages) + #:use-module (guix modules) + #:use-module (guix records) + #:use-module (guix gexp) + #:use-module (guix store) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:export (%test-guix-data-service)) + + +;;; +;;; Guix Data Service +;;; + +(define guix-data-service-initial-database-setup-service + (let ((user "guix_data_service") + (name "guix_data_service")) + (define start-gexp + #~(lambda () + (let ((pid (primitive-fork)) + (postgres (getpwnam "postgres"))) + (if (eq? pid 0) + (dynamic-wind + (const #t) + (lambda () + (setgid (passwd:gid postgres)) + (setuid (passwd:uid postgres)) + (primitive-exit + (if (and + (zero? + (system* #$(file-append postgresql "/bin/createuser") + #$user)) + (zero? + (system* #$(file-append postgresql "/bin/createdb") + "-O" #$user #$name))) + 0 + 1))) + (lambda () + (primitive-exit 1))) + (zero? (cdr (waitpid pid))))))) + + (shepherd-service + (requirement '(postgres)) + (provision '(guix-data-service-initial-database-setup)) + (start start-gexp) + (stop #~(const #f)) + (respawn? #f) + (one-shot? #t) + (documentation "Setup Guix Data Service database.")))) + +(define %guix-data-service-os + (simple-operating-system + (service dhcp-client-service-type) + (service postgresql-service-type + (postgresql-configuration + (config-file + (postgresql-config-file + (hba-file + (plain-file "pg_hba.conf" + " +local all all trust +host all all 127.0.0.1/32 trust +host all all ::1/128 trust")))))) + (service guix-data-service-type + (guix-data-service-configuration + (host "0.0.0.0"))) + (simple-service 'guix-data-service-database-setup + shepherd-root-service-type + (list guix-data-service-initial-database-setup-service)))) + +(define (run-guix-data-service-test) + (define os + (marionette-operating-system + %guix-data-service-os + #:imported-modules '((gnu services herd) + (guix combinators)))) + + (define forwarded-port 8080) + + (define vm + (virtual-machine + (operating-system os) + (memory-size 1024) + (port-forwardings `((,forwarded-port . 8765))))) + + (define test + (with-imported-modules '((gnu build marionette)) + #~(begin + (use-modules (srfi srfi-11) (srfi srfi-64) + (gnu build marionette) + (web uri) + (web client) + (web response)) + + (define marionette + (make-marionette (list #$vm))) + + (mkdir #$output) + (chdir #$output) + + (test-begin "guix-data-service") + + (test-assert "service running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (match (start-service 'guix-data-service) + (#f #f) + (('service response-parts ...) + (match (assq-ref response-parts 'running) + ((pid) (number? pid)))))) + marionette)) + + (test-assert "process jobs service running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (match (start-service 'guix-data-service-process-jobs) + (#f #f) + (('service response-parts ...) + (match (assq-ref response-parts 'running) + ((pid) (number? pid)))))) + marionette)) + + (test-equal "http-get" + 200 + (let-values + (((response text) + (http-get #$(simple-format + #f "http://localhost:~A/healthcheck" forwarded-port) + #:decode-body? #t))) + (response-code response))) + + (test-end) + (exit (= (test-runner-fail-count (test-runner-current)) 0))))) + + (gexp->derivation "guix-data-service-test" test)) + +(define %test-guix-data-service + (system-test + (name "guix-data-service") + (description "Connect to a running Guix Data Service.") + (value (run-guix-data-service-test))))