From patchwork Thu Sep 8 15:30:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Mathieu Othacehe X-Patchwork-Id: 42338 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id C531627BBEA; Thu, 8 Sep 2022 16:32:26 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 0AE0827BBE9 for ; Thu, 8 Sep 2022 16:32:26 +0100 (BST) Received: from localhost ([::1]:49368 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oWJVx-0006i1-4W for patchwork@mira.cbaines.net; Thu, 08 Sep 2022 11:32:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60200) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oWJVa-0006gu-Kr for guix-patches@gnu.org; Thu, 08 Sep 2022 11:32:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:42986) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oWJVa-00087i-CU for guix-patches@gnu.org; Thu, 08 Sep 2022 11:32:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oWJVa-0006R5-7Z for guix-patches@gnu.org; Thu, 08 Sep 2022 11:32:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#57680] [PATCH 1/2] guix: Add compression module. References: <20220908152512.6589-1-othacehe@gnu.org> In-Reply-To: <20220908152512.6589-1-othacehe@gnu.org> Resent-From: Mathieu Othacehe Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 08 Sep 2022 15:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 57680 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 57680@debbugs.gnu.org Cc: Mathieu Othacehe Received: via spool by 57680-submit@debbugs.gnu.org id=B57680.166265107524656 (code B ref 57680); Thu, 08 Sep 2022 15:32:02 +0000 Received: (at 57680) by debbugs.gnu.org; 8 Sep 2022 15:31:15 +0000 Received: from localhost ([127.0.0.1]:59915 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oWJUo-0006Pb-Dk for submit@debbugs.gnu.org; Thu, 08 Sep 2022 11:31:14 -0400 Received: from eggs.gnu.org ([209.51.188.92]:46396) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oWJUj-0006P8-AN for 57680@debbugs.gnu.org; Thu, 08 Sep 2022 11:31:12 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42176) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oWJUe-0007wb-3U for 57680@debbugs.gnu.org; Thu, 08 Sep 2022 11:31:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=kL2mDF8+8fA5BLqmtsWWs+y2eqH9PlgTQmqNqdlg4lU=; b=BxwPoxrTY5r7e2 bVBSuM+15qodKncOCcxqFW+OvbuZK32ywpICTmwWYNTtIu1DNuRB/hcouVmr8FLHmerNWzLfr3LZZ JLSSLU89J8827SFZ6omuZ7mDtfuwzZI2aJLys99MTatPoSVq1+kBIpymivVjFQdbEHSIJm4p3pjbK zcbftd/ShkbmHqCOxtwV06U4hyYZYvi4rRAU+h4bo+52toh4ETLOjdGoh5081EoyuPMkgG/xkVdE/ a/V7eByKm7hiyqwsXZe00JN8melZNvCTFisoAfE+TtXuYHYCSuSR097DSbp2svs6MSVjvs0e0DndO T/azk2u8GdvJ2mx9DRjA==; Received: from [2a02:587:ac08:6f00:6fbe:4f6e:9429:a4d] (port=58946 helo=meije.lan) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oWJUK-000270-0M; Thu, 08 Sep 2022 11:30:52 -0400 From: Mathieu Othacehe Date: Thu, 8 Sep 2022 17:30:32 +0200 Message-Id: <20220908153033.7016-1-othacehe@gnu.org> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Move the compression record to a dedicated module so that it can be used outside (guix scripts pack) module. * guix/scripts/pack.scm (, %compressors, lookup-compressor): Move it to ... * guix/compression.scm: ... this new file. * Makefile.am (MODULES): Add it. --- Makefile.am | 1 + guix/compression.scm | 69 +++++++++++++++++++++++++++++++++++++++++++ guix/scripts/pack.scm | 46 ++--------------------------- 3 files changed, 72 insertions(+), 44 deletions(-) create mode 100644 guix/compression.scm diff --git a/Makefile.am b/Makefile.am index 22dcc43f99..65b2ec4612 100644 --- a/Makefile.am +++ b/Makefile.am @@ -80,6 +80,7 @@ MODULES = \ guix/base32.scm \ guix/base64.scm \ guix/ci.scm \ + guix/compression.scm \ guix/cpio.scm \ guix/cpu.scm \ guix/deprecation.scm \ diff --git a/guix/compression.scm b/guix/compression.scm new file mode 100644 index 0000000000..10ec4a7cda --- /dev/null +++ b/guix/compression.scm @@ -0,0 +1,69 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 Mathieu Othacehe +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix compression) + #:use-module (guix gexp) + #:use-module (guix ui) + #:use-module ((gnu packages compression) #:hide (zip)) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) + #:use-module (ice-9 match) + #:export (compressor + compressor? + compressor-name + compressor-extension + compressor-command + %compressors + lookup-compressor)) + +;; Type of a compression tool. +(define-record-type + (compressor name extension command) + compressor? + (name compressor-name) ;string (e.g., "gzip") + (extension compressor-extension) ;string (e.g., ".lz") + (command compressor-command)) ;gexp (e.g., #~(list "/gnu/store/…/gzip" + ; "-9n" )) + +(define %compressors + ;; Available compression tools. + (list (compressor "gzip" ".gz" + #~(list #+(file-append gzip "/bin/gzip") "-9n")) + (compressor "lzip" ".lz" + #~(list #+(file-append lzip "/bin/lzip") "-9")) + (compressor "xz" ".xz" + #~(append (list #+(file-append xz "/bin/xz") + "-e") + (%xz-parallel-args))) + (compressor "bzip2" ".bz2" + #~(list #+(file-append bzip2 "/bin/bzip2") "-9")) + (compressor "zstd" ".zst" + ;; The default level 3 compresses better than gzip in a + ;; fraction of the time, while the highest level 19 + ;; (de)compresses more slowly and worse than xz. + #~(list #+(file-append zstd "/bin/zstd") "-3")) + (compressor "none" "" #f))) + +(define (lookup-compressor name) + "Return the compressor object called NAME. Error out if it could not be +found." + (or (find (match-lambda + (($ name*) + (string=? name* name))) + %compressors) + (leave (G_ "~a: compressor not found~%") name))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index d3ee69840c..0331ec7b04 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -27,6 +27,7 @@ (define-module (guix scripts pack) #:use-module (guix scripts) #:use-module (guix ui) + #:use-module (guix compression) #:use-module (guix gexp) #:use-module ((guix build utils) #:select (%xz-parallel-args)) #:use-module (guix utils) @@ -61,13 +62,7 @@ (define-module (guix scripts pack) #:use-module (srfi srfi-26) #:use-module (srfi srfi-37) #:use-module (ice-9 match) - #:export (compressor? - compressor-name - compressor-extension - compressor-command - %compressors - lookup-compressor - self-contained-tarball + #:export (self-contained-tarball debian-archive docker-image squashfs-image @@ -75,34 +70,6 @@ (define-module (guix scripts pack) %formats guix-pack)) -;; Type of a compression tool. -(define-record-type - (compressor name extension command) - compressor? - (name compressor-name) ;string (e.g., "gzip") - (extension compressor-extension) ;string (e.g., ".lz") - (command compressor-command)) ;gexp (e.g., #~(list "/gnu/store/…/gzip" - ; "-9n" )) - -(define %compressors - ;; Available compression tools. - (list (compressor "gzip" ".gz" - #~(list #+(file-append gzip "/bin/gzip") "-9n")) - (compressor "lzip" ".lz" - #~(list #+(file-append lzip "/bin/lzip") "-9")) - (compressor "xz" ".xz" - #~(append (list #+(file-append xz "/bin/xz") - "-e") - (%xz-parallel-args))) - (compressor "bzip2" ".bz2" - #~(list #+(file-append bzip2 "/bin/bzip2") "-9")) - (compressor "zstd" ".zst" - ;; The default level 3 compresses better than gzip in a - ;; fraction of the time, while the highest level 19 - ;; (de)compresses more slowly and worse than xz. - #~(list #+(file-append zstd "/bin/zstd") "-3")) - (compressor "none" "" #f))) - ;; This one is only for use in this module, so don't put it in %compressors. (define bootstrap-xz (compressor "bootstrap-xz" ".xz" @@ -110,15 +77,6 @@ (define bootstrap-xz "-e") (%xz-parallel-args)))) -(define (lookup-compressor name) - "Return the compressor object called NAME. Error out if it could not be -found." - (or (find (match-lambda - (($ name*) - (string=? name* name))) - %compressors) - (leave (G_ "~a: compressor not found~%") name))) - (define not-config? ;; Select (guix …) and (gnu …) modules, except (guix config). (match-lambda From patchwork Thu Sep 8 15:30:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Mathieu Othacehe X-Patchwork-Id: 42339 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 1DB8227BBEA; Thu, 8 Sep 2022 16:32:34 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5F0B827BBE9 for ; Thu, 8 Sep 2022 16:32:33 +0100 (BST) Received: from localhost ([::1]:39760 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oWJW4-00073T-Iy for patchwork@mira.cbaines.net; Thu, 08 Sep 2022 11:32:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60202) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oWJVb-0006h6-5r for guix-patches@gnu.org; Thu, 08 Sep 2022 11:32:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:42989) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oWJVa-00087r-TW for guix-patches@gnu.org; Thu, 08 Sep 2022 11:32:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oWJVa-0006RE-Nc for guix-patches@gnu.org; Thu, 08 Sep 2022 11:32:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#57680] [PATCH 2/2] system: image: Add tarball support. Resent-From: Mathieu Othacehe Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 08 Sep 2022 15:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 57680 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 57680@debbugs.gnu.org Cc: Mathieu Othacehe , Alex Griffin Received: via spool by 57680-submit@debbugs.gnu.org id=B57680.166265107724666 (code B ref 57680); Thu, 08 Sep 2022 15:32:02 +0000 Received: (at 57680) by debbugs.gnu.org; 8 Sep 2022 15:31:17 +0000 Received: from localhost ([127.0.0.1]:59917 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oWJUq-0006Pl-UF for submit@debbugs.gnu.org; Thu, 08 Sep 2022 11:31:17 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42428) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oWJUn-0006PL-AB for 57680@debbugs.gnu.org; Thu, 08 Sep 2022 11:31:13 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42178) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oWJUh-0007yT-1A; Thu, 08 Sep 2022 11:31:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=/JKZue/VSSZwV+d8CeDbv9nR3c79ckqdxhRu0gwWZA0=; b=oR4USJjSV7QT22TLpM+6 njZkyrnUPVOwawlje0RVcN/ekhevMedG3CcQEPkcFmMzMQmWNeM6wdNcROMqBZVhPCoamPuW70Yrh IXaxYmFm7wpenWQSCb+2LWg1LVvqvCKQr+IO2YNHmGTZfL0j9o38vKltOuw4iHoswNpZIssE5tiIL /lXWeCMtf2bZxxPgoIL0rlfcZR3tp6E47osPOlvRV9FtOhiaPQED/Z9DVNzQkdyWI+gOtnrKbjJdV wiUMmZI1klSyEeS3jl65sMbgxk9fm3lxTwUpGHWlHzGn5qy/sjpV1qZnnmxkyIBM6Ge8q0KALv19J J6Ph+73OvJB+DQ==; Received: from [2a02:587:ac08:6f00:6fbe:4f6e:9429:a4d] (port=58946 helo=meije.lan) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oWJUe-000270-Du; Thu, 08 Sep 2022 11:31:05 -0400 From: Mathieu Othacehe Date: Thu, 8 Sep 2022 17:30:33 +0200 Message-Id: <20220908153033.7016-2-othacehe@gnu.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220908153033.7016-1-othacehe@gnu.org> References: <20220908153033.7016-1-othacehe@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Alex Griffin * gnu/image.scm ()[fields]: Add tarball to the supported formats. * gnu/system/image.scm (tarball-image, tarball-image-type): New variables. (system-tarball-image): New procedure. (image->root-file-system): Add tarball image support. (system-image): Ditto. Signed-off-by: Mathieu Othacehe --- gnu/image.scm | 2 +- gnu/system/image.scm | 82 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/gnu/image.scm b/gnu/image.scm index 4a0068934e..18e24d3cac 100644 --- a/gnu/image.scm +++ b/gnu/image.scm @@ -152,7 +152,7 @@ (define-with-syntax-properties (name (value properties)) ;; The supported image formats. (define-set-sanitizer validate-image-format format - (disk-image compressed-qcow2 docker iso9660)) + (disk-image compressed-qcow2 docker iso9660 tarball)) ;; The supported partition table types. (define-set-sanitizer validate-partition-table-type partition-table-type diff --git a/gnu/system/image.scm b/gnu/system/image.scm index a04363a130..5e50210523 100644 --- a/gnu/system/image.scm +++ b/gnu/system/image.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen ;;; Copyright © 2022 Pavel Shlyak ;;; Copyright © 2022 Denis 'GNUtoo' Carikli +;;; Copyright © 2022 Alex Griffin ;;; ;;; This file is part of GNU Guix. ;;; @@ -20,6 +21,7 @@ ;;; along with GNU Guix. If not, see . (define-module (gnu system image) + #:use-module (guix compression) #:use-module (guix diagnostics) #:use-module (guix discovery) #:use-module (guix gexp) @@ -73,6 +75,7 @@ (define-module (gnu system image) efi-disk-image iso9660-image docker-image + tarball-image raw-with-offset-disk-image image-with-os @@ -82,6 +85,7 @@ (define-module (gnu system image) iso-image-type uncompressed-iso-image-type docker-image-type + tarball-image-type raw-with-offset-image-type image-with-label @@ -149,6 +153,10 @@ (define docker-image (image (format 'docker))) +(define tarball-image + (image + (format 'tarball))) + (define* (raw-with-offset-disk-image #:optional (offset root-offset)) (image (format 'disk-image) @@ -211,6 +219,11 @@ (define docker-image-type (name 'docker) (constructor (cut image-with-os docker-image <>)))) +(define tarball-image-type + (image-type + (name 'tarball) + (constructor (cut image-with-os tarball-image <>)))) + (define raw-with-offset-image-type (image-type (name 'raw-with-offset) @@ -681,6 +694,71 @@ (define builder #:options `(#:references-graphs ((,graph ,os)) #:substitutable? ,substitutable?)))) + +;; +;; Tarball image. +;; + +(define* (system-tarball-image image + #:key + (name "image") + (compressor (srfi-1:first %compressors))) + "Build a tarball of IMAGE. NAME is the base name to use for the +output file." + (let* ((os (image-operating-system image)) + (substitutable? (image-substitutable? image)) + (schema (local-file (search-path %load-path + "guix/store/schema.sql"))) + (name (string-append name ".tar" (compressor-extension compressor))) + (graph "system-graph")) + (define builder + (with-extensions gcrypt-sqlite3&co ;for (guix store database) + (with-imported-modules `(,@(source-module-closure + '((guix build pack) + (guix build store-copy) + (guix build utils) + (guix store database) + (gnu build image)) + #:select? not-config?) + ((guix config) => ,(make-config.scm))) + #~(begin + (use-modules (guix build pack) + (guix build store-copy) + (guix build utils) + (guix store database) + (gnu build image)) + + ;; Set the SQL schema location. + (sql-schema #$schema) + + ;; Allow non-ASCII file names--e.g., 'nss-certs'--to be decoded. + (setenv "GUIX_LOCPATH" + #+(file-append glibc-utf8-locales "/lib/locale")) + (setlocale LC_ALL "en_US.utf8") + + (let ((image-root (string-append (getcwd) "/tmp-root")) + (tar #+(file-append tar "/bin/tar"))) + + (mkdir-p image-root) + (initialize-root-partition image-root + #:references-graphs '(#$graph) + #:deduplicate? #f + #:system-directory #$os) + + (with-directory-excursion image-root + (apply invoke tar "-cvf" #$output "." + (tar-base-options + #:tar tar + #:compressor + #+(and=> compressor compressor-command))))))))) + + (computed-file name builder + ;; Allow offloading so that this I/O-intensive process + ;; doesn't run on the build farm's head node. + #:local-build? #f + #:options `(#:references-graphs ((,graph ,os)) + #:substitutable? ,substitutable?)))) + ;; ;; Image creation. @@ -690,7 +768,7 @@ (define (image->root-file-system image) "Return the IMAGE root partition file-system type." (case (image-format image) ((iso9660) "iso9660") - ((docker) "dummy") + ((docker tarball) "dummy") (else (partition-file-system (find-root-partition image))))) @@ -827,6 +905,8 @@ (define target (cond ("bootcfg" ,bootcfg)))) ((memq image-format '(docker)) (system-docker-image image*)) + ((memq image-format '(tarball)) + (system-tarball-image image*)) ((memq image-format '(iso9660)) (system-iso9660-image image*