From patchwork Mon Jun 21 06:11:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 30560 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 7241527BC81; Mon, 21 Jun 2021 07:13:16 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B06B127BC78 for ; Mon, 21 Jun 2021 07:13:15 +0100 (BST) Received: from localhost ([::1]:38446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lvDBK-0007ua-RM for patchwork@mira.cbaines.net; Mon, 21 Jun 2021 02:13:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47924) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvDB8-0007u9-Vi for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:51892) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvDB8-0001ra-Hu for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lvDB8-0006cI-Be for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49149] [PATCH 1/7] pack: Extract builder code from self-contained-tarball. References: <20210621061039.31557-1-maxim.cournoyer@gmail.com> In-Reply-To: <20210621061039.31557-1-maxim.cournoyer@gmail.com> Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 21 Jun 2021 06:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49149 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49149@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 49149-submit@debbugs.gnu.org id=B49149.162425594925297 (code B ref 49149); Mon, 21 Jun 2021 06:13:02 +0000 Received: (at 49149) by debbugs.gnu.org; 21 Jun 2021 06:12:29 +0000 Received: from localhost ([127.0.0.1]:35184 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAa-0006Zx-K8 for submit@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:29 -0400 Received: from mail-qk1-f177.google.com ([209.85.222.177]:41699) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAZ-0006Zk-5c for 49149@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:27 -0400 Received: by mail-qk1-f177.google.com with SMTP id 22so6580452qkv.8 for <49149@debbugs.gnu.org>; Sun, 20 Jun 2021 23:12:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=nK+XZhiFPUZF48SwwcEJOQPOaSHkTwT9yOejmrHORIk=; b=Mc+L4hmSt+uB/hrshFoqOiD3TqO+oSEvXW/8hVSjpNvBEcXwbxvQqpyMwY7qDvDq7c laXqBLmkRrZ6xAQdpFBBktb6NccbBokNP9bLWw3RUoYtiZUjEzZQDWA5IQxsb2I0u8TM NuOabm0KajiJLpTddnIEfHDpGQcNmInSsXYn/oSkSVF9RLICT73x3xJuaIGsjmXEpLgY 5PpD9V/NiPxn5zQpWb8a07Gi8CnZjq3OcDxNCJKfRbu1/CfeNq4p2bHDbPSM1Wft9vaF 2rzWtHJdwT2PB4zXl95HFsizCmSEXBWZOQGwNwZ47TF8nSOfDrtFjck01o76T3PLk/L2 YSYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=nK+XZhiFPUZF48SwwcEJOQPOaSHkTwT9yOejmrHORIk=; b=lGqtK8vpYBiHs2USjAA3Xu8D3j0dMmmJI4XY2xEJavz6f1xTikYsK9NDLp4vOqn1Q7 VupMNMKPhaOlcUlJ6/Q1bThvwtSVj1PmetPqUrTVDViIIWzmlbrZKHf4c6E6j33yfZF1 52ZJribbCwOL1OrqgA214PzDC0D1rcIyuK4e0hPEiZb9SO8tYSYpj4VqKivp89+xl2Gq CoeOwE0VM0RjpblCOFhS/PmkahqibtF82yi1NRsYVnr+RZO8SJL1hF1PvhKiuNDIwu1k BhJLpHuHPRo3sjXTQ5/jyRZxzYlegOHGQBUQ8wpYqz6fhIrmImWKiBwlu+Gb3iLSUx/J UY9Q== X-Gm-Message-State: AOAM532eB/BdE0NwDHlfVqYQOf6cTym6WmnroNGFIvlmbSfmLUfDKe1d /AdX+Xj1sUpZX/V8fE0ic8EsVogFtmwzhA== X-Google-Smtp-Source: ABdhPJzBl8oxZ0tsoN9oIDp8kGyrqOXE52cFFinhjwwAkToF7DEMkeln88AB5Y5sKJyBU5cITlur9A== X-Received: by 2002:a05:620a:218e:: with SMTP id g14mr21312418qka.290.1624255941431; Sun, 20 Jun 2021 23:12:21 -0700 (PDT) Received: from localhost.localdomain (dsl-148-219.b2b2c.ca. [66.158.148.219]) by smtp.gmail.com with ESMTPSA id i11sm8478663qke.74.2021.06.20.23.12.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Jun 2021 23:12:21 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Jun 2021 02:11:58 -0400 Message-Id: <20210621061205.31878-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches This is made to allow reusing it for the debian-archive pack format, added in a subsequent commit. * guix/scripts/pack.scm (self-contained-tarball/builder): New procedure, containing the build code extracted from self-contained-tarball. (self-contained-tarball): Use the above procedure. --- guix/scripts/pack.scm | 270 ++++++++++++++++++++++-------------------- 1 file changed, 141 insertions(+), 129 deletions(-) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 8cb4e6d2cc..ac477850e6 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -172,22 +172,17 @@ dependencies are registered." (computed-file "store-database" build #:options `(#:references-graphs ,(zip labels items)))) -(define* (self-contained-tarball name profile - #:key target - (profile-name "guix-profile") - deduplicate? - entry-point - (compressor (first %compressors)) - localstatedir? - (symlinks '()) - (archiver tar)) - "Return a self-contained tarball containing a store initialized with the -closure of PROFILE, a derivation. The tarball contains /gnu/store; if -LOCALSTATEDIR? is true, it also contains /var/guix, including /var/guix/db -with a properly initialized store database. - -SYMLINKS must be a list of (SOURCE -> TARGET) tuples denoting symlinks to be -added to the pack." + +;;; +;;; Tarball format. +;;; +(define* (self-contained-tarball/builder profile + #:key (profile-name "guix-profile") + (compressor (first %compressors)) + localstatedir? + (symlinks '()) + (archiver tar)) + "Return the G-Expression of the builder used for self-contained-tarball." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -209,125 +204,142 @@ added to the pack." (and (not-config? module) (not (equal? '(guix store deduplication) module)))) - (define build - (with-imported-modules (source-module-closure - `((guix build utils) - (guix build union) - (gnu build install)) - #:select? import-module?) - #~(begin - (use-modules (guix build utils) - ((guix build union) #:select (relative-file-name)) - (gnu build install) - (srfi srfi-1) - (srfi srfi-26) - (ice-9 match)) + (with-imported-modules (source-module-closure + `((guix build utils) + (guix build union) + (gnu build install)) + #:select? import-module?) + #~(begin + (use-modules (guix build utils) + ((guix build union) #:select (relative-file-name)) + (gnu build install) + (srfi srfi-1) + (srfi srfi-26) + (ice-9 match)) + + (define %root "root") + + (define symlink->directives + ;; Return "populate directives" to make the given symlink and its + ;; parent directories. + (match-lambda + ((source '-> target) + (let ((target (string-append #$profile "/" target)) + (parent (dirname source))) + ;; Never add a 'directory' directive for "/" so as to + ;; preserve its ownnership when extracting the archive (see + ;; below), and also because this would lead to adding the + ;; same entries twice in the tarball. + `(,@(if (string=? parent "/") + '() + `((directory ,parent))) + (,source + -> ,(relative-file-name parent target))))))) + + (define directives + ;; Fully-qualified symlinks. + (append-map symlink->directives '#$symlinks)) + + ;; The --sort option was added to GNU tar in version 1.28, released + ;; 2014-07-28. For testing, we use the bootstrap tar, which is + ;; older and doesn't support it. + (define tar-supports-sort? + (zero? (system* (string-append #+archiver "/bin/tar") + "cf" "/dev/null" "--files-from=/dev/null" + "--sort=name"))) + + ;; Make sure non-ASCII file names are properly handled. + #+set-utf8-locale + + ;; Add 'tar' to the search path. + (setenv "PATH" #+(file-append archiver "/bin")) + + ;; Note: there is not much to gain here with deduplication and there + ;; is the overhead of the '.links' directory, so turn it off. + ;; Furthermore GNU tar < 1.30 sometimes fails to extract tarballs + ;; with hard links: + ;; . + (populate-single-profile-directory %root + #:profile #$profile + #:profile-name #$profile-name + #:closure "profile" + #:database #+database) + + ;; Create SYMLINKS. + (for-each (cut evaluate-populate-directive <> %root) + directives) + + ;; Create the tarball. Use GNU format so there's no file name + ;; length limitation. + (with-directory-excursion %root + (apply invoke "tar" + #+@(if (compressor-command compressor) + #~("-I" + (string-join + '#+(compressor-command compressor))) + #~()) + "--format=gnu" + ;; Avoid non-determinism in the archive. + ;; Use mtime = 1, not zero, because that is what the daemon + ;; does for files in the store (see the 'mtimeStore' constant + ;; in local-store.cc.) + (if tar-supports-sort? "--sort=name" "--mtime=@1") + "--owner=root:0" + "--group=root:0" + "--check-links" + "-cvf" #$output + ;; Avoid adding / and /var to the tarball, so + ;; that the ownership and permissions of those + ;; directories will not be overwritten when + ;; extracting the archive. Do not include /root + ;; because the root account might have a + ;; different home directory. + #$@(if localstatedir? + '("./var/guix") + '()) + + (string-append "." (%store-directory)) + + (delete-duplicates + (filter-map (match-lambda + (('directory directory) + (string-append "." directory)) + ((source '-> _) + (string-append "." source)) + (_ #f)) + directives))))))) - (define %root "root") - - (define symlink->directives - ;; Return "populate directives" to make the given symlink and its - ;; parent directories. - (match-lambda - ((source '-> target) - (let ((target (string-append #$profile "/" target)) - (parent (dirname source))) - ;; Never add a 'directory' directive for "/" so as to - ;; preserve its ownnership when extracting the archive (see - ;; below), and also because this would lead to adding the - ;; same entries twice in the tarball. - `(,@(if (string=? parent "/") - '() - `((directory ,parent))) - (,source - -> ,(relative-file-name parent target))))))) - - (define directives - ;; Fully-qualified symlinks. - (append-map symlink->directives '#$symlinks)) - - ;; The --sort option was added to GNU tar in version 1.28, released - ;; 2014-07-28. For testing, we use the bootstrap tar, which is - ;; older and doesn't support it. - (define tar-supports-sort? - (zero? (system* (string-append #+archiver "/bin/tar") - "cf" "/dev/null" "--files-from=/dev/null" - "--sort=name"))) - - ;; Make sure non-ASCII file names are properly handled. - #+set-utf8-locale - - ;; Add 'tar' to the search path. - (setenv "PATH" #+(file-append archiver "/bin")) - - ;; Note: there is not much to gain here with deduplication and there - ;; is the overhead of the '.links' directory, so turn it off. - ;; Furthermore GNU tar < 1.30 sometimes fails to extract tarballs - ;; with hard links: - ;; . - (populate-single-profile-directory %root - #:profile #$profile - #:profile-name #$profile-name - #:closure "profile" - #:database #+database) - - ;; Create SYMLINKS. - (for-each (cut evaluate-populate-directive <> %root) - directives) - - ;; Create the tarball. Use GNU format so there's no file name - ;; length limitation. - (with-directory-excursion %root - (exit - (zero? (apply system* "tar" - #+@(if (compressor-command compressor) - #~("-I" - (string-join - '#+(compressor-command compressor))) - #~()) - "--format=gnu" - - ;; Avoid non-determinism in the archive. Use - ;; mtime = 1, not zero, because that is what the - ;; daemon does for files in the store (see the - ;; 'mtimeStore' constant in local-store.cc.) - (if tar-supports-sort? "--sort=name" "--mtime=@1") - "--mtime=@1" ;for files in /var/guix - "--owner=root:0" - "--group=root:0" - - "--check-links" - "-cvf" #$output - ;; Avoid adding / and /var to the tarball, so - ;; that the ownership and permissions of those - ;; directories will not be overwritten when - ;; extracting the archive. Do not include /root - ;; because the root account might have a - ;; different home directory. - #$@(if localstatedir? - '("./var/guix") - '()) - - (string-append "." (%store-directory)) - - (delete-duplicates - (filter-map (match-lambda - (('directory directory) - (string-append "." directory)) - ((source '-> _) - (string-append "." source)) - (_ #f)) - directives))))))))) +(define* (self-contained-tarball name profile + #:key target + (profile-name "guix-profile") + deduplicate? + entry-point + (compressor (first %compressors)) + localstatedir? + (symlinks '()) + (archiver tar)) + "Return a self-contained tarball containing a store initialized with the +closure of PROFILE, a derivation. The tarball contains /gnu/store; if +LOCALSTATEDIR? is true, it also contains /var/guix, including /var/guix/db +with a properly initialized store database. +SYMLINKS must be a list of (SOURCE -> TARGET) tuples denoting symlinks to be +added to the pack." (when entry-point (warning (G_ "entry point not supported in the '~a' format~%") 'tarball)) - (gexp->derivation (string-append name ".tar" - (compressor-extension compressor)) - build - #:target target - #:references-graphs `(("profile" ,profile)))) + (gexp->derivation + (string-append name ".tar" + (compressor-extension compressor)) + (self-contained-tarball/builder profile + #:profile-name profile-name + #:compressor compressor + #:localstatedir? localstatedir? + #:symlinks symlinks + #:archiver archiver) + #:target target + #:references-graphs `(("profile" ,profile)))) (define (singularity-environment-file profile) "Return a shell script that defines the environment variables corresponding From patchwork Mon Jun 21 06:12:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 30561 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 5668F27BC81; Mon, 21 Jun 2021 07:13:18 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 76B3F27BC78 for ; Mon, 21 Jun 2021 07:13:17 +0100 (BST) Received: from localhost ([::1]:38484 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lvDBM-0007wd-Hi for patchwork@mira.cbaines.net; Mon, 21 Jun 2021 02:13:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47928) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvDB9-0007uP-Hn for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:51894) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvDB9-0001s6-9z for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lvDB9-0006cY-6q for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49149] [PATCH 2/7] pack: Factorize base tar options. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 21 Jun 2021 06:13:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49149 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49149@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 49149-submit@debbugs.gnu.org id=B49149.162425596925359 (code B ref 49149); Mon, 21 Jun 2021 06:13:03 +0000 Received: (at 49149) by debbugs.gnu.org; 21 Jun 2021 06:12:49 +0000 Received: from localhost ([127.0.0.1]:35191 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAu-0006ax-OR for submit@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:49 -0400 Received: from mail-qt1-f169.google.com ([209.85.160.169]:42567) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAt-0006aO-17 for 49149@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:47 -0400 Received: by mail-qt1-f169.google.com with SMTP id x21so3454528qtq.9 for <49149@debbugs.gnu.org>; Sun, 20 Jun 2021 23:12:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Gc0uudKTiV1MlAuoJBX9RmBclzpzaZS/bhTN4i4HSAc=; b=VMtNAKLThCS/r7L6ITtGefRp+lUh6+YKa6HNuA4eUkWgQfC9yS8b+aB3/c8pV2m88v HP+bGaLadeTHWwOMuabJZCte3ZZgjJ9wU5LiVB5eObHTELeqo1iLr7flYKgMdktZN/lz Z5HToijy6S7slBGekstuSpda4ncm6x5rxW8t1T+nXSKJZfJJuOoTJm0Qx5biAZRy5aWQ 9AE+AJzhWnAo9Jz9EhytqlRx898xw7/Rfbch8rAIzIPtXacK2v0OV7grP5eZjUs8Q04E vqbfhRSxdSRNNr3ukSH6phFP13zyezayRtJUkF4ehXE+MATdauOsD0IZpVJAl7PdRmXR VcFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Gc0uudKTiV1MlAuoJBX9RmBclzpzaZS/bhTN4i4HSAc=; b=ixixVNOj8pam9bbTWSRxmu0k/87qdycjtjBRRGs8Zh4cvVNtkGOsXxe58f/2b9+jJt QaUrE4SwJp6IxFB8WBEWrDMQeUOv9piL78NBw2fX3CMP7XN2Gude4Q1DqRuzIF3ZnPy4 u0bgEaWhmQaWiSB026nTkUMf+3g3K1M61xbgb0SyuCjo6JYR3owsChi/uy8PldAEuMZo 9LcvPk90UC848eavsxWrBRufky0dECan4ASoX6pbne6S3PW2qwsH29IlgkadClLP8ksn e1iU1QBEYc4jVlpVQdw3U8PHjBQcmmkVoLpFGIA6jrCtVF0ftOLpN6hX+HToo8i1dJKf 38IQ== X-Gm-Message-State: AOAM530uV017XAIROqlvHbE29ruecc4zdLRtOgPMEOdLNNr8CCWLZHzV 69U1mTR8+WXJAtlX875vM+LxrSRXAiNAvQ== X-Google-Smtp-Source: ABdhPJzSKFErKl79zhc2Cp1FpeRgLiShJJ6JCLbBq47fbd+MYSIQyKBLdLc1/6GhB0rjeMPNmWuBnA== X-Received: by 2002:ac8:6a09:: with SMTP id t9mr3903074qtr.118.1624255961494; Sun, 20 Jun 2021 23:12:41 -0700 (PDT) Received: from localhost.localdomain (dsl-148-219.b2b2c.ca. [66.158.148.219]) by smtp.gmail.com with ESMTPSA id i11sm8478663qke.74.2021.06.20.23.12.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Jun 2021 23:12:41 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Jun 2021 02:12:00 -0400 Message-Id: <20210621061205.31878-3-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210621061205.31878-1-maxim.cournoyer@gmail.com> References: <20210621061205.31878-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/docker.scm (%tar-determinism-options): Move to a new module and rename to `tar-base-options'. Adjust references accordingly. * guix/build/pack.scm: New file. * Makefile.am (MODULES): Register it. * guix/scripts/pack.scm (self-contained-tarball/builder): Use it. --- Makefile.am | 1 + guix/build/pack.scm | 52 +++++++++++++++++++++++++++ guix/docker.scm | 20 ++--------- guix/scripts/pack.scm | 81 +++++++++++++++++-------------------------- 4 files changed, 87 insertions(+), 67 deletions(-) create mode 100644 guix/build/pack.scm diff --git a/Makefile.am b/Makefile.am index aa21b5383b..9c4b33c77a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -220,6 +220,7 @@ MODULES = \ guix/build/linux-module-build-system.scm \ guix/build/store-copy.scm \ guix/build/json.scm \ + guix/build/pack.scm \ guix/build/utils.scm \ guix/build/union.scm \ guix/build/profiles.scm \ diff --git a/guix/build/pack.scm b/guix/build/pack.scm new file mode 100644 index 0000000000..05c7a3c594 --- /dev/null +++ b/guix/build/pack.scm @@ -0,0 +1,52 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Maxim Cournoyer +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix build pack) + #:use-module (guix build utils) + #:export (tar-base-options)) + +(define* (tar-base-options #:key tar compressor) + "Return the base GNU tar options required to produce deterministic archives +deterministically. When TAR, a GNU tar command file name, is provided, the +`--sort' option is used only if supported. When COMPRESSOR, a command such as +'(\"gzip\" \"-9n\"), is provided, the compressor is explicitly specified via +the `-I' option." + (define (tar-supports-sort? tar) + (zero? (system* tar "cf" "/dev/null" "--files-from=/dev/null" + "--sort=name"))) + + `(,@(if compressor + (list "-I" (string-join compressor)) + '()) + ;; The --sort option was added to GNU tar in version 1.28, released + ;; 2014-07-28. For testing, we use the bootstrap tar, which is older + ;; and doesn't support it. + ,@(if (and=> tar tar-supports-sort?) + '("--sort=name") + '()) + ;; Use GNU format so there's no file name length limitation. + "--format=gnu" + "--mtime=@1" + "--owner=root:0" + "--group=root:0" + ;; The 'nlink' of the store item files leads tar to store hard links + ;; instead of actual copies. However, the 'nlink' count depends on + ;; deduplication in the store; it's an "implicit input" to the build + ;; process. Use '--hard-dereference' to eliminate it. + "--hard-dereference" + "--check-links")) diff --git a/guix/docker.scm b/guix/docker.scm index 889aaeacb5..bd952e45ec 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -21,6 +21,7 @@ (define-module (guix docker) #:use-module (gcrypt hash) #:use-module (guix base16) + #:use-module (guix build pack) #:use-module ((guix build utils) #:select (mkdir-p delete-file-recursively @@ -110,18 +111,6 @@ Return a version of TAG that follows these rules." (rootfs . ((type . "layers") (diff_ids . #(,(layer-diff-id layer))))))) -(define %tar-determinism-options - ;; GNU tar options to produce archives deterministically. - '("--sort=name" "--mtime=@1" - "--owner=root:0" "--group=root:0" - - ;; When 'build-docker-image' is passed store items, the 'nlink' of the - ;; files therein leads tar to store hard links instead of actual copies. - ;; However, the 'nlink' count depends on deduplication in the store; it's - ;; an "implicit input" to the build process. '--hard-dereference' - ;; eliminates it. - "--hard-dereference")) - (define directive-file ;; Return the file or directory created by a 'evaluate-populate-directive' ;; directive. @@ -238,7 +227,7 @@ SRFI-19 time-utc object, as the creation time in metadata." (apply invoke "tar" "-cf" "../layer.tar" `(,@transformation-options - ,@%tar-determinism-options + ,@(tar-base-options) ,@paths ,@(scandir "." (lambda (file) @@ -273,9 +262,6 @@ SRFI-19 time-utc object, as the creation time in metadata." (scm->json (repositories prefix id repository))))) (apply invoke "tar" "-cf" image "-C" directory - `(,@%tar-determinism-options - ,@(if compressor - (list "-I" (string-join compressor)) - '()) + `(,@(tar-base-options #:compressor compressor) ".")) (delete-file-recursively directory))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index ac477850e6..d11f498925 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -205,12 +205,14 @@ dependencies are registered." (not (equal? '(guix store deduplication) module)))) (with-imported-modules (source-module-closure - `((guix build utils) + `((guix build pack) + (guix build utils) (guix build union) (gnu build install)) #:select? import-module?) #~(begin - (use-modules (guix build utils) + (use-modules (guix build pack) + (guix build utils) ((guix build union) #:select (relative-file-name)) (gnu build install) (srfi srfi-1) @@ -240,19 +242,10 @@ dependencies are registered." ;; Fully-qualified symlinks. (append-map symlink->directives '#$symlinks)) - ;; The --sort option was added to GNU tar in version 1.28, released - ;; 2014-07-28. For testing, we use the bootstrap tar, which is - ;; older and doesn't support it. - (define tar-supports-sort? - (zero? (system* (string-append #+archiver "/bin/tar") - "cf" "/dev/null" "--files-from=/dev/null" - "--sort=name"))) - ;; Make sure non-ASCII file names are properly handled. #+set-utf8-locale - ;; Add 'tar' to the search path. - (setenv "PATH" #+(file-append archiver "/bin")) + (define tar #+(file-append archiver "/bin/tar")) ;; Note: there is not much to gain here with deduplication and there ;; is the overhead of the '.links' directory, so turn it off. @@ -269,45 +262,33 @@ dependencies are registered." (for-each (cut evaluate-populate-directive <> %root) directives) - ;; Create the tarball. Use GNU format so there's no file name - ;; length limitation. + ;; Create the tarball. (with-directory-excursion %root - (apply invoke "tar" - #+@(if (compressor-command compressor) - #~("-I" - (string-join - '#+(compressor-command compressor))) - #~()) - "--format=gnu" - ;; Avoid non-determinism in the archive. - ;; Use mtime = 1, not zero, because that is what the daemon - ;; does for files in the store (see the 'mtimeStore' constant - ;; in local-store.cc.) - (if tar-supports-sort? "--sort=name" "--mtime=@1") - "--owner=root:0" - "--group=root:0" - "--check-links" - "-cvf" #$output - ;; Avoid adding / and /var to the tarball, so - ;; that the ownership and permissions of those - ;; directories will not be overwritten when - ;; extracting the archive. Do not include /root - ;; because the root account might have a - ;; different home directory. - #$@(if localstatedir? - '("./var/guix") - '()) - - (string-append "." (%store-directory)) - - (delete-duplicates - (filter-map (match-lambda - (('directory directory) - (string-append "." directory)) - ((source '-> _) - (string-append "." source)) - (_ #f)) - directives))))))) + (apply invoke tar + `(,@(tar-base-options + #:tar tar + #:compressor '#+(and=> compressor compressor-command)) + "-cvf" ,#$output + ;; Avoid adding / and /var to the tarball, so + ;; that the ownership and permissions of those + ;; directories will not be overwritten when + ;; extracting the archive. Do not include /root + ;; because the root account might have a + ;; different home directory. + ,#$@(if localstatedir? + '("./var/guix") + '()) + + ,(string-append "." (%store-directory)) + + ,@(delete-duplicates + (filter-map (match-lambda + (('directory directory) + (string-append "." directory)) + ((source '-> _) + (string-append "." source)) + (_ #f)) + directives)))))))) (define* (self-contained-tarball name profile #:key target From patchwork Mon Jun 21 06:12:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 30562 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id F043327BC81; Mon, 21 Jun 2021 07:13:20 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6BF4527BC78 for ; Mon, 21 Jun 2021 07:13:20 +0100 (BST) Received: from localhost ([::1]:38608 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lvDBP-00081p-Gj for patchwork@mira.cbaines.net; Mon, 21 Jun 2021 02:13:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47944) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvDBA-0007w9-Pw for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:51897) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvDBA-0001tp-HG for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lvDBA-0006cw-DJ for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:04 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49149] [PATCH 5/7] pack: Prevent duplicate files in tar archives. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 21 Jun 2021 06:13:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49149 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49149@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 49149-submit@debbugs.gnu.org id=B49149.162425597225399 (code B ref 49149); Mon, 21 Jun 2021 06:13:04 +0000 Received: (at 49149) by debbugs.gnu.org; 21 Jun 2021 06:12:52 +0000 Received: from localhost ([127.0.0.1]:35200 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAx-0006bU-RU for submit@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:52 -0400 Received: from mail-qv1-f41.google.com ([209.85.219.41]:36371) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAw-0006aY-B0 for 49149@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:50 -0400 Received: by mail-qv1-f41.google.com with SMTP id im10so6893973qvb.3 for <49149@debbugs.gnu.org>; Sun, 20 Jun 2021 23:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9sYSuoxh/TXX0vpZJIGqEGuZnOV29nWz3iCoA1cYbX4=; b=jIeO8o2AdhRHLnGOzTu6AGjp0/kTxZ4lzWCmSNYQ3X/7LL+3v0WSaq9x6FZ3T8x7x5 5o/TImIr9ZXSRqnW3UTXZ9Vw4yf8RysFt1kGT8a9b5bLXE9gtz5+EytNTFJZpY6HLnHN JCObUfKTG2lrO4Ai3FjR+TUI7089/LRAHVUF5WecUf7sBfBmcQTV2qneAG6NDNUYdACh Iz4TyNCvAcd8IzpipIp/HYbUgYGQP9O0eCbtEvEsplyJlyMKAEhGDjW/Bc575fYy/g0J 4mvCHxrax2zZo6WS/O4x5v/ar8Uu31YaFitZWaBTq249/6A19Ka+/F+c+twJpEPjgmzC t5rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9sYSuoxh/TXX0vpZJIGqEGuZnOV29nWz3iCoA1cYbX4=; b=KEo0e1WysN2ez6pFhzgTi3PT4zTnpR5C9EghqVjpdE9KYFouoDS+SVKrxaiT1lsBHn RMyMtK+30qhS4eQO4m4ZHTeVgF9le2QI81U01YaFig+ZIvtBQnSv3lvmNGx3fFeVngfI FUXW2D3xDO2OMYAosgvDK+EFdNJ3I47t/Z77ERckZIoQz1jPbMEGPymK/skGOeHSSfN3 DIG28Pe0xbUrlhzHF0zszTTEWaZFQFFF+Cp9S8IGNFvUXt5ckBKI/Cu2apX/q06Yc7XM i0IwSROnKSrU/oX1r1ksZeORYEqIKYftHnmbU/nvLVWGKm9u3Zhx7HqThusntkkCap+K Prxg== X-Gm-Message-State: AOAM533w14lXECziEQjjPi1RUeJdpRpPRdqSZvHExOSmT9V4QH8+9guS TfNuFoQgzG6qELFPegeAwwst3+fKtE02RA== X-Google-Smtp-Source: ABdhPJwKELnPF5QJFXzuk1HcuLong5Lm4ltkdpl7qGX0ryWnzz/b0dHEZr6hdC7un+BNsGY+lURLhg== X-Received: by 2002:a05:6214:b6c:: with SMTP id ey12mr18374648qvb.48.1624255964788; Sun, 20 Jun 2021 23:12:44 -0700 (PDT) Received: from localhost.localdomain (dsl-148-219.b2b2c.ca. [66.158.148.219]) by smtp.gmail.com with ESMTPSA id i11sm8478663qke.74.2021.06.20.23.12.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Jun 2021 23:12:44 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Jun 2021 02:12:03 -0400 Message-Id: <20210621061205.31878-6-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210621061205.31878-1-maxim.cournoyer@gmail.com> References: <20210621061205.31878-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Tar translate duplicate files in the archive into hard links. These can cause problems, as not every tool support them; for example dpkg doesn't. * gnu/system/file-systems.scm (reduce-directories): New procedure. (file-prefix?): Lift the restriction on file prefix. The procedure can be useful for comparing relative file names. Adjust doc. (file-name-depth): New procedure, extracted from ... (btrfs-store-subvolume-file-name): ... here. * guix/scripts/pack.scm (self-contained-tarball/builder): Use reduce-directories. * tests/file-systems.scm ("reduce-directories"): New test. --- gnu/system/file-systems.scm | 56 +++++++++++++++++++++++++------------ guix/scripts/pack.scm | 6 ++-- tests/file-systems.scm | 7 ++++- 3 files changed, 48 insertions(+), 21 deletions(-) diff --git a/gnu/system/file-systems.scm b/gnu/system/file-systems.scm index 464e87cb18..fb87bfc85b 100644 --- a/gnu/system/file-systems.scm +++ b/gnu/system/file-systems.scm @@ -55,6 +55,7 @@ file-system-dependencies file-system-location + reduce-directories file-system-type-predicate btrfs-subvolume? btrfs-store-subvolume-file-name @@ -231,8 +232,8 @@ (char-set-complement (char-set #\/))) (define (file-prefix? file1 file2) - "Return #t if FILE1 denotes the name of a file that is a parent of FILE2, -where both FILE1 and FILE2 are absolute file name. For example: + "Return #t if FILE1 denotes the name of a file that is a parent of FILE2. +For example: (file-prefix? \"/gnu\" \"/gnu/store\") => #t @@ -240,19 +241,41 @@ where both FILE1 and FILE2 are absolute file name. For example: (file-prefix? \"/gn\" \"/gnu/store\") => #f " - (and (string-prefix? "/" file1) - (string-prefix? "/" file2) - (let loop ((file1 (string-tokenize file1 %not-slash)) - (file2 (string-tokenize file2 %not-slash))) - (match file1 - (() - #t) - ((head1 tail1 ...) - (match file2 - ((head2 tail2 ...) - (and (string=? head1 head2) (loop tail1 tail2))) - (() - #f))))))) + (let loop ((file1 (string-tokenize file1 %not-slash)) + (file2 (string-tokenize file2 %not-slash))) + (match file1 + (() + #t) + ((head1 tail1 ...) + (match file2 + ((head2 tail2 ...) + (and (string=? head1 head2) (loop tail1 tail2))) + (() + #f)))))) + +(define (file-name-depth file-name) + (length (string-tokenize file-name %not-slash))) + +(define (reduce-directories file-names) + "Eliminate entries in FILE-NAMES that are children of other entries in +FILE-NAMES. This is for example useful when passing a list of files to GNU +tar, which would otherwise descend into each directory passed and archive the +duplicate files as hard links, which can be undesirable." + (let* ((file-names/sorted + ;; Ascending sort by file hierarchy depth, then by file name length. + (stable-sort (delete-duplicates file-names) + (lambda (f1 f2) + (let ((depth1 (file-name-depth f1)) + (depth2 (file-name-depth f2))) + (if (= depth1 depth2) + (string< f1 f2) + (< depth1 depth2))))))) + (reverse (fold (lambda (file-name results) + (if (find (cut file-prefix? <> file-name) results) + results ;parent found -- skipping + (cons file-name results))) + '() + file-names/sorted)))) (define* (file-system-device->string device #:key uuid-type) "Return the string representations of the DEVICE field of a @@ -624,9 +647,6 @@ store is located, else #f." s (string-append "/" s))) - (define (file-name-depth file-name) - (length (string-tokenize file-name %not-slash))) - (and-let* ((btrfs-subvolume-fs (filter btrfs-subvolume? file-systems)) (btrfs-subvolume-fs* (sort btrfs-subvolume-fs diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 9d4bb9f497..8a108b7a1a 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -225,13 +225,15 @@ dependencies are registered." `((guix build pack) (guix build utils) (guix build union) - (gnu build install)) + (gnu build install) + (gnu system file-systems)) #:select? import-module?) #~(begin (use-modules (guix build pack) (guix build utils) ((guix build union) #:select (relative-file-name)) (gnu build install) + ((gnu system file-systems) #:select (reduce-directories)) (srfi srfi-1) (srfi srfi-26) (ice-9 match)) @@ -298,7 +300,7 @@ dependencies are registered." ,(string-append "." (%store-directory)) - ,@(delete-duplicates + ,@(reduce-directories (filter-map (match-lambda (('directory directory) (string-append "." directory)) diff --git a/tests/file-systems.scm b/tests/file-systems.scm index 7f7c373884..80acb6d5b9 100644 --- a/tests/file-systems.scm +++ b/tests/file-systems.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ludovic Courtès -;;; Copyright © 2020 Maxim Cournoyer +;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -50,6 +50,11 @@ (device "/foo") (flags '(bind-mount read-only))))))))) +(test-equal "reduce-directories" + '("./opt/gnu/" "./opt/gnuism" "a/b/c") + (reduce-directories '("./opt/gnu/etc" "./opt/gnu/" "./opt/gnu/bin" + "./opt/gnu/lib/debug" "./opt/gnuism" "a/b/c" "a/b/c"))) + (test-assert "does not pull (guix config)" ;; This module is meant both for the host side and "build side", so make ;; sure it doesn't pull in (guix config), which depends on the user's