From patchwork Tue Jul 6 21:11:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 31207 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 92B4627BC84; Tue, 6 Jul 2021 22:13:14 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,SPF_HELO_PASS, T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id DA38527BC81 for ; Tue, 6 Jul 2021 22:13:12 +0100 (BST) Received: from localhost ([::1]:34814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m0sNT-0008I0-Rw for patchwork@mira.cbaines.net; Tue, 06 Jul 2021 17:13:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55804) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m0sNL-0008HR-2d for guix-patches@gnu.org; Tue, 06 Jul 2021 17:13:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:38806) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m0sNK-0004Ji-Rn for guix-patches@gnu.org; Tue, 06 Jul 2021 17:13:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m0sNK-0000vb-OT for guix-patches@gnu.org; Tue, 06 Jul 2021 17:13:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49348] [PATCH v2 3/4] pack: Streamline how files are included in tarballs. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 06 Jul 2021 21:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49348 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49348@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 49348-submit@debbugs.gnu.org id=B49348.16256059443495 (code B ref 49348); Tue, 06 Jul 2021 21:13:02 +0000 Received: (at 49348) by debbugs.gnu.org; 6 Jul 2021 21:12:24 +0000 Received: from localhost ([127.0.0.1]:50346 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m0sMh-0000uI-TJ for submit@debbugs.gnu.org; Tue, 06 Jul 2021 17:12:24 -0400 Received: from mail-qk1-f180.google.com ([209.85.222.180]:34701) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m0sMg-0000tt-7e for 49348@debbugs.gnu.org; Tue, 06 Jul 2021 17:12:22 -0400 Received: by mail-qk1-f180.google.com with SMTP id g4so21680247qkl.1 for <49348@debbugs.gnu.org>; Tue, 06 Jul 2021 14:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+E/D2n4BtSsH4nDSrF7X05HXdKWSrJuummj3isbdtJs=; b=LlD+MWAAsuqTjux/4A8eTCrMNmkJpzJhgT6v1CBbpGnoEiHa5mowwF+JJESJq8omsM RLo8WyHq6R/cU8SGEkDZmxFYx/2LcWUVj+vTRthXHT2Do5RXCOBA93PpOiHqbMKwqQFi gGsF3Y4Pn2eLKaIqeJnPRV/n7B4+vX90fAEVc13V/PeFf/hRDiAHcw4I1/h7gQDTBjMJ Azq3wJTJy51ZeQGdrrG5EncqsBSww0b3doRxpvwsTOLHlPTZ75Mtj0v2QBwIcJRlkfVU LDSiJwnKr/53Hr53PL5MoORuwgaQbdVeNtNYACpDAnx/zKH7RVYvh+iajUoq4z7Kg+mc zjWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+E/D2n4BtSsH4nDSrF7X05HXdKWSrJuummj3isbdtJs=; b=Vjb/N9i/i0OYiUeRS2SY0FRKiMpTkl3yl4DHl9o+YChW8TYlRg01n2OspYLZQxsN1l U3omrkOo6kZIRmX6hStUFx20/W7fenLxaRkyISm5rkmY16wlCfh4JJ2Mpsb6Vn//m7ih LT0WJtcdS5Wx68p+DAz7ziNaNurGeKCClbN6Qb09c/FXqjBEsKGLdZoPvMLDDXIrv6hc 6CzVyFIPSZ9XzwacJGn22/xZyCanJN8NSIjB7o27uyNy00W5gLTjjwMyh4K+3iFHPn2W xc4roKcUUzIP0i5su9Mo/+bJV4ELZoWsn6AZ0MCRGArVz/5jWGp7pkO9w33UhM/cxa8Q /IDg== X-Gm-Message-State: AOAM533tn3XXet3KRDaFGSiqWs/uB0w8z4/NsNwE54ANmyFQM48d+zxW 1Vz1YOFE02ISlXHG96Ef9G5EhrCVTM4zEQ== X-Google-Smtp-Source: ABdhPJxl32FbsIIaDbgKCB5oGDqqucE9pw+yDPgpHoCF5QFBM+cjBceDDgsN1yXRSCpkZacnGrvbmQ== X-Received: by 2002:a37:64c1:: with SMTP id y184mr15294947qkb.273.1625605936727; Tue, 06 Jul 2021 14:12:16 -0700 (PDT) Received: from localhost.localdomain (dsl-205-233-125-111.b2b2c.ca. [205.233.125.111]) by smtp.gmail.com with ESMTPSA id s19sm7240662qks.77.2021.07.06.14.12.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jul 2021 14:12:16 -0700 (PDT) From: Maxim Cournoyer Date: Tue, 6 Jul 2021 17:11:38 -0400 Message-Id: <20210706211139.2806-3-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210706211139.2806-1-maxim.cournoyer@gmail.com> References: <20210703060642.2424-1-maxim.cournoyer@gmail.com> <20210706211139.2806-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Thanks to Guillem Jover on the OFTC's #debian-dpkg channel for helping with troubleshooting. Letting GNU Tar recursively walk the complete files hierarchy side-steps the risks associated with providing a list of file names: 1. Duplicated files in the archive (recorded as hard links by GNU Tar) 2. Missing parent directories. The above would cause dpkg to malfunction, for example by aborting early and skipping triggers when there are missing parent directories. * guix/scripts/pack.scm (self-contained-tarball/builder): Do not call POPULATE-SINGLE-PROFILE-DIRECTORY, which creates extraneous files such as /root. Instead, call POPULATE-STORE and INSTALL-DATABASE-AND-GC-ROOTS individually to more precisely generate the file system. Replace the list of files by the current directory, "." and streamline the way options are passed. * gnu/system/file-systems.scm (reduce-directories): Remove procedure. * tests/file-systems.scm ("reduce-directories"): Remove test. --- gnu/system/file-systems.scm | 22 ----------------- guix/scripts/pack.scm | 49 ++++++++++++------------------------- tests/file-systems.scm | 7 +----- 3 files changed, 17 insertions(+), 61 deletions(-) diff --git a/gnu/system/file-systems.scm b/gnu/system/file-systems.scm index 4a3c1fe008..b9eda80958 100644 --- a/gnu/system/file-systems.scm +++ b/gnu/system/file-systems.scm @@ -55,7 +55,6 @@ file-system-dependencies file-system-location - reduce-directories file-system-type-predicate btrfs-subvolume? btrfs-store-subvolume-file-name @@ -266,27 +265,6 @@ For example: (define (file-name-depth file-name) (length (string-tokenize file-name %not-slash))) -(define (reduce-directories file-names) - "Eliminate entries in FILE-NAMES that are children of other entries in -FILE-NAMES. This is for example useful when passing a list of files to GNU -tar, which would otherwise descend into each directory passed and archive the -duplicate files as hard links, which can be undesirable." - (let* ((file-names/sorted - ;; Ascending sort by file hierarchy depth, then by file name length. - (stable-sort (delete-duplicates file-names) - (lambda (f1 f2) - (let ((depth1 (file-name-depth f1)) - (depth2 (file-name-depth f2))) - (if (= depth1 depth2) - (string< f1 f2) - (< depth1 depth2))))))) - (reverse (fold (lambda (file-name results) - (if (find (cut file-prefix? <> file-name) results) - results ;parent found -- skipping - (cons file-name results))) - '() - file-names/sorted)))) - (define* (file-system-device->string device #:key uuid-type) "Return the string representations of the DEVICE field of a record. When the device is a UUID, its representation is chosen depending on diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 78201d6f5f..9e1f270dfb 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -231,17 +231,17 @@ its source property." (with-imported-modules (source-module-closure `((guix build pack) + (guix build store-copy) (guix build utils) (guix build union) - (gnu build install) - (gnu system file-systems)) + (gnu build install)) #:select? import-module?) #~(begin (use-modules (guix build pack) + (guix build store-copy) (guix build utils) ((guix build union) #:select (relative-file-name)) (gnu build install) - ((gnu system file-systems) #:select (reduce-directories)) (srfi srfi-1) (srfi srfi-26) (ice-9 match)) @@ -279,11 +279,11 @@ its source property." ;; Furthermore GNU tar < 1.30 sometimes fails to extract tarballs ;; with hard links: ;; . - (populate-single-profile-directory %root - #:profile #$profile - #:profile-name #$profile-name - #:closure "profile" - #:database #+database) + (populate-store (list "profile") %root #:deduplicate? #f) + + (when #+localstatedir? + (install-database-and-gc-roots %root #+database #$profile + #:profile-name #$profile-name)) ;; Create SYMLINKS. (for-each (cut evaluate-populate-directive <> %root) @@ -291,31 +291,14 @@ its source property." ;; Create the tarball. (with-directory-excursion %root - (apply invoke tar - `(,@(tar-base-options - #:tar tar - #:compressor '#+(and=> compressor compressor-command)) - "-cvf" ,#$output - ;; Avoid adding / and /var to the tarball, so - ;; that the ownership and permissions of those - ;; directories will not be overwritten when - ;; extracting the archive. Do not include /root - ;; because the root account might have a - ;; different home directory. - ,#$@(if localstatedir? - '("./var/guix") - '()) - - ,(string-append "." (%store-directory)) - - ,@(reduce-directories - (filter-map (match-lambda - (('directory directory) - (string-append "." directory)) - ((source '-> _) - (string-append "." source)) - (_ #f)) - directives)))))))) + ;; GNU Tar recurses directories by default. Simply add the whole + ;; current directory, which contains all the generated files so far. + ;; This avoids creating duplicate files in the archives that would + ;; be stored as hard links by GNU Tar. + (apply invoke tar "-cvf" #$output "." + (tar-base-options + #:tar tar + #:compressor '#+(and=> compressor compressor-command))))))) (define* (self-contained-tarball name profile #:key target diff --git a/tests/file-systems.scm b/tests/file-systems.scm index 80acb6d5b9..7f7c373884 100644 --- a/tests/file-systems.scm +++ b/tests/file-systems.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ludovic Courtès -;;; Copyright © 2020, 2021 Maxim Cournoyer +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -50,11 +50,6 @@ (device "/foo") (flags '(bind-mount read-only))))))))) -(test-equal "reduce-directories" - '("./opt/gnu/" "./opt/gnuism" "a/b/c") - (reduce-directories '("./opt/gnu/etc" "./opt/gnu/" "./opt/gnu/bin" - "./opt/gnu/lib/debug" "./opt/gnuism" "a/b/c" "a/b/c"))) - (test-assert "does not pull (guix config)" ;; This module is meant both for the host side and "build side", so make ;; sure it doesn't pull in (guix config), which depends on the user's