From patchwork Tue Jul 2 08:56:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14455 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 752BD17180; Tue, 2 Jul 2019 09:57:52 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 141771715A for ; Tue, 2 Jul 2019 09:57:52 +0100 (BST) Received: from localhost ([::1]:50818 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hiEbn-0000nM-On for patchwork@mira.cbaines.net; Tue, 02 Jul 2019 04:57:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59427) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hiEb2-0008LT-9f for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hiEb0-00029I-M0 for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37732) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hiEb0-000298-Gq for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hiEb0-0004ee-Dc for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#36469] [PATCH 1/2] pack: 'docker' backend records the profile's search paths. References: <20190702083731.10354-1-ludo@gnu.org> In-Reply-To: <20190702083731.10354-1-ludo@gnu.org> Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 02 Jul 2019 08:57:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 36469 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 36469@debbugs.gnu.org Received: via spool by 36469-submit@debbugs.gnu.org id=B36469.156205778017833 (code B ref 36469); Tue, 02 Jul 2019 08:57:02 +0000 Received: (at 36469) by debbugs.gnu.org; 2 Jul 2019 08:56:20 +0000 Received: from localhost ([127.0.0.1]:46551 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hiEaK-0004dY-0i for submit@debbugs.gnu.org; Tue, 02 Jul 2019 04:56:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:46886) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hiEaI-0004dE-Gf for 36469@debbugs.gnu.org; Tue, 02 Jul 2019 04:56:19 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:45015) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hiEaC-0001mZ-Ju; Tue, 02 Jul 2019 04:56:12 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=50386 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hiEaB-0006bp-19; Tue, 02 Jul 2019 04:56:11 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 2 Jul 2019 10:56:00 +0200 Message-Id: <20190702085601.10865-1-ludo@gnu.org> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Ludovic Courtès * guix/docker.scm (config): Add #:environment parameter and honor it. (build-docker-image): Likewise, and pass it to 'config'. * guix/scripts/pack.scm (docker-image): Import (guix profiles) and (guix search-paths). Call 'profile-search-paths' and pass #:environment to 'build-docker-image'. * gnu/tests/docker.scm (run-docker-test)["Load docker image and run it"]: Add example that expects (json) to be available. * gnu/tests/docker.scm (build-tarball&run-docker-test): Replace %BOOTSTRAP-GUILE by GUILE-2.2 and GUILE-JSON in the environment. --- gnu/tests/docker.scm | 16 ++++++++++------ guix/docker.scm | 17 +++++++++++++---- guix/scripts/pack.scm | 23 +++++++++++++++++++---- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index f2674cdbe8..27fde49e75 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -27,7 +27,6 @@ #:use-module (gnu services networking) #:use-module (gnu services docker) #:use-module (gnu services desktop) - #:use-module (gnu packages bootstrap) ; %bootstrap-guile #:use-module (gnu packages docker) #:use-module (gnu packages guile) #:use-module (guix gexp) @@ -101,7 +100,7 @@ inside %DOCKER-OS." marionette)) (test-equal "Load docker image and run it" - '("hello world" "hi!") + '("hello world" "hi!" "JSON!") (marionette-eval `(begin (define slurp @@ -125,8 +124,13 @@ inside %DOCKER-OS." (response2 (slurp ;default entry point ,(string-append #$docker-cli "/bin/docker") "run" repository&tag - "-c" "(display \"hi!\")"))) - (list response1 response2))) + "-c" "(display \"hi!\")")) + (response3 (slurp ;default entry point + environment + ,(string-append #$docker-cli "/bin/docker") + "run" repository&tag + "-c" "(use-modules (json)) + (display (json-string->scm (scm->json-string \"JSON!\")))"))) + (list response1 response2 response3))) marionette)) (test-end) @@ -144,7 +148,7 @@ inside %DOCKER-OS." (version "0") (source #f) (build-system trivial-build-system) - (arguments `(#:guile ,%bootstrap-guile + (arguments `(#:guile ,guile-2.2 #:builder (let ((out (assoc-ref %outputs "out"))) (mkdir out) @@ -158,7 +162,7 @@ standard output device and then enters a new line.") (home-page #f) (license license:public-domain))) (profile (profile-derivation (packages->manifest - (list %bootstrap-guile + (list guile-2.2 guile-json guest-script-package)) #:hooks '() #:locales? #f)) diff --git a/guix/docker.scm b/guix/docker.scm index 7fe83d9797..b1bd226fa1 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -73,7 +73,7 @@ `((,(generate-tag path) . ((latest . ,id))))) ;; See https://github.com/opencontainers/image-spec/blob/master/config.md -(define* (config layer time arch #:key entry-point) +(define* (config layer time arch #:key entry-point (environment '())) "Generate a minimal image configuration for the given LAYER file." ;; "architecture" must be values matching "platform.arch" in the ;; runtime-spec at @@ -81,9 +81,13 @@ `((architecture . ,arch) (comment . "Generated by GNU Guix") (created . ,time) - (config . ,(if entry-point - `((entrypoint . ,entry-point)) - #nil)) + (config . ,`((env . ,(map (match-lambda + ((name . value) + (string-append name "=" value))) + environment)) + ,@(if entry-point + `((entrypoint . ,entry-point)) + '()))) (container_config . #nil) (os . "linux") (rootfs . ((type . "layers") @@ -113,6 +117,7 @@ return \"a\"." (system (utsname:machine (uname))) database entry-point + (environment '()) compressor (creation-time (current-time time-utc))) "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX @@ -124,6 +129,9 @@ When DATABASE is true, copy it to /var/guix/db in the image and create When ENTRY-POINT is true, it must be a list of strings; it is stored as the entry point in the Docker image JSON structure. +ENVIRONMENT must be a list of name/value pairs. It specifies the environment +variables that must be defined in the resulting image. + SYMLINKS must be a list of (SOURCE -> TARGET) tuples describing symlinks to be created in the image, where each TARGET is relative to PREFIX. TRANSFORMATIONS must be a list of (OLD -> NEW) tuples describing how to @@ -234,6 +242,7 @@ SRFI-19 time-utc object, as the creation time in metadata." (lambda () (scm->json (config (string-append id "/layer.tar") time arch + #:environment environment #:entry-point entry-point)))) (with-output-to-file "manifest.json" (lambda () diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index c90b777222..bb6a8cda1a 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -27,6 +27,7 @@ #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) + #:use-module ((guix self) #:select (make-config.scm)) #:use-module (guix grafts) #:autoload (guix inferior) (inferior-package?) #:use-module (guix monads) @@ -440,11 +441,24 @@ the image." (define build ;; Guile-JSON and Guile-Gcrypt are required by (guix docker). (with-extensions (list guile-json guile-gcrypt) - (with-imported-modules (source-module-closure '((guix docker) - (guix build store-copy)) - #:select? not-config?) + (with-imported-modules `(((guix config) => ,(make-config.scm)) + ,@(source-module-closure + `((guix docker) + (guix build store-copy) + (guix profiles) + (guix search-paths)) + #:select? not-config?)) #~(begin - (use-modules (guix docker) (srfi srfi-19) (guix build store-copy)) + (use-modules (guix docker) (guix build store-copy) + (guix profiles) (guix search-paths) + (srfi srfi-19) (ice-9 match)) + + (define environment + (map (match-lambda + ((spec . value) + (cons (search-path-specification-variable spec) + value))) + (profile-search-paths #$profile))) (setenv "PATH" (string-append #$archiver "/bin")) @@ -455,6 +469,7 @@ the image." #$profile #:database #+database #:system (or #$target (utsname:machine (uname))) + #:environment environment #:entry-point #$(and entry-point #~(string-append #$profile "/" #$entry-point)) From patchwork Tue Jul 2 08:56:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 14454 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4A85717180; Tue, 2 Jul 2019 09:57:12 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id F1F601715A for ; Tue, 2 Jul 2019 09:57:11 +0100 (BST) Received: from localhost ([::1]:50808 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hiEb9-0008Pr-EV for patchwork@mira.cbaines.net; Tue, 02 Jul 2019 04:57:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59425) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hiEb2-0008LQ-8x for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hiEb1-00029T-26 for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37733) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hiEb0-00029P-V7 for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hiEb0-0004el-TV for guix-patches@gnu.org; Tue, 02 Jul 2019 04:57:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#36469] [PATCH 2/2] pack: 'squashfs' backend records the profile's search paths. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 02 Jul 2019 08:57:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 36469 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 36469@debbugs.gnu.org Received: via spool by 36469-submit@debbugs.gnu.org id=B36469.156205778117840 (code B ref 36469); Tue, 02 Jul 2019 08:57:02 +0000 Received: (at 36469) by debbugs.gnu.org; 2 Jul 2019 08:56:21 +0000 Received: from localhost ([127.0.0.1]:46553 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hiEaK-0004da-Fa for submit@debbugs.gnu.org; Tue, 02 Jul 2019 04:56:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:46890) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hiEaI-0004dF-Oq for 36469@debbugs.gnu.org; Tue, 02 Jul 2019 04:56:19 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:45016) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hiEaD-0001mz-Gt; Tue, 02 Jul 2019 04:56:13 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=50386 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hiEaC-0006bp-O3; Tue, 02 Jul 2019 04:56:13 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 2 Jul 2019 10:56:01 +0200 Message-Id: <20190702085601.10865-2-ludo@gnu.org> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190702085601.10865-1-ludo@gnu.org> References: <20190702085601.10865-1-ludo@gnu.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Ludovic Courtès * guix/scripts/pack.scm (singularity-environment-file): New procedure. (squashfs-image): Use it, and create /.singularity/env/90-environment.sh. * gnu/tests/singularity.scm (run-singularity-test)["singularity run, with environment"]: New test, currently skipped. * gnu/tests/singularity.scm (build-tarball&run-singularity-test): Add GUILE-JSON to the profile. --- gnu/tests/singularity.scm | 18 ++++++++++++++++- guix/scripts/pack.scm | 41 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/gnu/tests/singularity.scm b/gnu/tests/singularity.scm index 668043a0bc..2f3a6f289d 100644 --- a/gnu/tests/singularity.scm +++ b/gnu/tests/singularity.scm @@ -111,6 +111,21 @@ "run" #$image "-c" "(exit 42)")) marionette)) + ;; FIXME: Singularity 2.x doesn't directly honor + ;; /.singularity.d/env/*.sh. Instead, you have to load those files + ;; manually, which we don't do. Remove 'test-skip' call once we've + ;; switch to Singularity 3.x. + (test-skip 1) + (test-equal "singularity run, with environment" + 0 + (marionette-eval + ;; Check whether GUILE_LOAD_PATH is properly set, allowing us to + ;; find the (json) module. + `(status:exit-val + (system* #$(file-append singularity "/bin/singularity") + "--debug" "run" #$image "-c" "(use-modules (json))")) + marionette)) + (test-end) (exit (= (test-runner-fail-count (test-runner-current)) 0))))) @@ -122,7 +137,8 @@ (guile (set-guile-for-build (default-guile))) ;; 'singularity exec' insists on having /bin/sh in the image. (profile (profile-derivation (packages->manifest - (list bash-minimal guile-2.2)) + (list bash-minimal + guile-2.2 guile-json)) #:hooks '() #:locales? #f)) (tarball (squashfs-image "singularity-pack" profile diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index bb6a8cda1a..4ac5dfc896 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -286,6 +286,32 @@ added to the pack." build #:references-graphs `(("profile" ,profile)))) +(define (singularity-environment-file profile) + "Return a shell script that defines the environment variables corresponding +to the search paths of PROFILE." + (define build + (with-extensions (list guile-gcrypt) + (with-imported-modules `(((guix config) => ,(make-config.scm)) + ,@(source-module-closure + `((guix profiles) + (guix search-paths)) + #:select? not-config?)) + #~(begin + (use-modules (guix profiles) (guix search-paths) + (ice-9 match)) + + (call-with-output-file #$output + (lambda (port) + (for-each (match-lambda + ((spec . value) + (format port "~a=~a~%export ~a~%" + (search-path-specification-variable spec) + value + (search-path-specification-variable spec)))) + (profile-search-paths #$profile)))))))) + + (computed-file "singularity-environment.sh" build)) + (define* (squashfs-image name profile #:key target (profile-name "guix-profile") @@ -305,6 +331,9 @@ added to the pack." (file-append (store-database (list profile)) "/db/db.sqlite"))) + (define environment + (singularity-environment-file profile)) + (define build (with-imported-modules (source-module-closure '((guix build utils) @@ -339,6 +368,7 @@ added to the pack." `(,@(map store-info-item (call-with-input-file "profile" read-reference-graph)) + #$environment ,#$output ;; Do not perform duplicate checking because we @@ -379,10 +409,19 @@ added to the pack." target))))))) '#$symlinks) + "-p" "/.singularity.d d 555 0 0" + + ;; Create the environment file. + "-p" "/.singularity.d/env d 555 0 0" + "-p" ,(string-append + "/.singularity.d/env/90-environment.sh s 777 0 0 " + (relative-file-name "/.singularity.d/env" + #$environment)) + ;; Create /.singularity.d/actions, and optionally the 'run' ;; script, used by 'singularity run'. - "-p" "/.singularity.d d 555 0 0" "-p" "/.singularity.d/actions d 555 0 0" + ,@(if entry-point `(;; This one if for Singularity 2.x. "-p"