From patchwork Tue Dec 26 02:18:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleg Pykhalov X-Patchwork-Id: 58075 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 551D227BBEB; Tue, 26 Dec 2023 02:21:27 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI, SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 8B5DA27BBE9 for ; Tue, 26 Dec 2023 02:21:26 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rHx41-00070k-Ru; Mon, 25 Dec 2023 21:21:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rHx3z-0006zF-Rf for guix-patches@gnu.org; Mon, 25 Dec 2023 21:20:59 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rHx3z-0007i8-JE; Mon, 25 Dec 2023 21:20:59 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rHx44-0000n6-3L; Mon, 25 Dec 2023 21:21:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#62153] [PATCH 4/5] guix: pack: Build layered images. Resent-From: Oleg Pykhalov Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 26 Dec 2023 02:21:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62153 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 62153@debbugs.gnu.org Cc: Oleg Pykhalov , Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 62153-submit@debbugs.gnu.org id=B62153.17035572102932 (code B ref 62153); Tue, 26 Dec 2023 02:21:04 +0000 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:10 +0000 Received: from localhost ([127.0.0.1]:55618 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx3B-0000l8-FY for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:10 -0500 Received: from mail-lf1-x136.google.com ([2a00:1450:4864:20::136]:38077) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx34-0000jp-Nf for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:05 -0500 Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-50e7a0d07a0so122988e87.0 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557189; x=1704161989; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BhVNkDMX7JP5NGW3OEcnZMSwtVpydjEjv3znzIK2QTY=; b=IAD3+8Wopa8r9AmKhjSvQWV92KOV2fGIw0PoJrm4Jgk7d16QMI84WtOjTHL48hJBVw uGPJReZzFGb8TggXZbFZrwv/SO7sc1SjY2erhEOwDePrasSSFffWngVtb8Xh4s9C5ydW v4LPMv52x3ocvwtDrs169+BukvJvK4jcM2V5lA9QmbDHciOpjO7bLp4sHDR1/0gJWTMr eFakZpRC65hakwv+dBls3N0WaELOJLQviVY3HO9U+RlJKpS7B74kf1FJPOJFVAGc+bZ9 MmnS+OtETtesZ3RM8+cvW3Xjs2vjV6cnJwzHtLWFkC1Jnr97usFoA67N3Ih7u4GRdi4T NT9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557189; x=1704161989; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BhVNkDMX7JP5NGW3OEcnZMSwtVpydjEjv3znzIK2QTY=; b=tmar6f+NSnk7ZENlbReUxUDLrmtUY5vc5zo/KbuTfLbpWZAQnvTLo3wnhQU8p8Q5UF ZcvNiYRifcoG0Wy88NLKVvIuFiOAsnD6xSOO3qJiCQw4seoNKOqOWiSNKgpxaFbaFMtH rTONWBQTLGKKy9h0diEYla4Iu3MxGCvssBbZviSyFteYG1plMobG7S48VqydxHHOPWCp /c+dZwAevjO7YMd0U97laXDDoQ02cjEv1O+tXXKwma1f7FBeAC6d64nTYRnIDopaiVsR y6+mra7q0DSbFSoGViV3lzvDJI+317rc+G4w6cq2E+kPyB440dzy0ZdSXJSNWYBiEnav Nmcg== X-Gm-Message-State: AOJu0YzOOQ0YeTWFfYN+l9VcZaW8tle8W4lxnubrt8l/1VmgmNBF6cvg BRfYnIXzU8RZ8b7Ww/8tnavAqO9rlHk= X-Google-Smtp-Source: AGHT+IH7eZs1bzPjO9KpFR9WnVJAvA8I3VrH9Vm3u7m+R9ydzyw1a1Fa81eaIPjQWV2ELzKWCpRh8A== X-Received: by 2002:a2e:9dd2:0:b0:2ca:143:a52f with SMTP id x18-20020a2e9dd2000000b002ca0143a52fmr5071748ljj.2.1703557189003; Mon, 25 Dec 2023 18:19:49 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:48 -0800 (PST) From: Oleg Pykhalov Date: Tue, 26 Dec 2023 05:18:56 +0300 Message-ID: <77bae2565cb5506342028896e1a45c757f3bad51.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/scripts/pack.scm (docker-image, guix-pack, %default-options, %docker-format-options, show-docker-format-options/detailed): Handle '--max-layers' option. * doc/guix.texi (Invoking guix pack): Document this. Change-Id: I90660b2421fcdde891f003469fe2e2edaac7da41 --- doc/guix.texi | 26 ++++++++++++++++++++++- guix/scripts/pack.scm | 38 ++++++++++++++++++++++++++------- tests/pack.scm | 49 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 9 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index cca250dc31..d21048405a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -56,7 +56,7 @@ Copyright @copyright{} 2017, 2018, 2019, 2020, 2023 Arun Isaac@* Copyright @copyright{} 2017 nee@* Copyright @copyright{} 2018 Rutger Helling@* -Copyright @copyright{} 2018, 2021 Oleg Pykhalov@* +Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* @@ -7441,6 +7441,30 @@ Invoking guix pack guix pack -f docker --entry-point=bin/guile --entry-point-argument="--help" guile @end example +@cindex maximum layers argument, for docker images +@item --max-layers=@code{n} +Specifies the maximum number of Docker image layers allowed when +building an image. + +@example +guix pack -f docker --max-layers=100 guile +@end example + +This option allows you to limit the number of layers in a Docker image. +Docker images are comprised of multiple layers, and each layer adds to +the overall size and complexity of the image. By setting a maximum +number of layers, you can control the following effects: + +@itemize +@item Disk Usage: +Increasing the number of layers can help optimize the disk space +required to store multiple images built with a similar package graph. + +@item Pulling: +When transferring images between different nodes or systems, having more +layers can reduce the time required to pull the image. +@end itemize + @item --expression=@var{expr} @itemx -e @var{expr} Consider the package @var{expr} evaluates to. diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 4c0a602eb1..22f0dd6061 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -9,6 +9,7 @@ ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin ;;; Copyright © 2023 Graham James Addis +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -48,6 +49,7 @@ (define-module (guix scripts pack) #:use-module (guix scripts build) #:use-module (guix transformations) #:use-module ((guix self) #:select (make-config.scm)) + #:use-module ((guix docker) #:select (%docker-image-max-layers)) #:use-module (gnu compression) #:use-module (gnu packages) #:use-module (gnu packages bootstrap) @@ -517,12 +519,15 @@ (define* (docker-image name profile localstatedir? (symlinks '()) (archiver tar) - (extra-options '())) + (extra-options '()) + max-layers) "Return a derivation to construct a Docker image of PROFILE. The image is a tarball conforming to the Docker Image Specification, compressed with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a be a GNU triplet and it is used to derive the architecture metadata in -the image. EXTRA-OPTIONS may contain the IMAGE-TAG keyword argument." +the image. EXTRA-OPTIONS may contain the IMAGE-TAG keyword argument. +If MAX-LAYERS is not false, the image will with many of the store paths being +on their own layer to improve sharing between images." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -583,11 +588,17 @@ (define* (docker-image name profile (cons* (string-append prefix "/" entry-point) entry-point-argument)))) - (setenv "PATH" #+(file-append archiver "/bin")) + (setenv "PATH" + (string-join `(#+(file-append archiver "/bin") + #+@(if max-layers + (list (file-append gzip "/bin")) + '())) + ":")) (let-keywords '#$extra-options #f ((image-tag #f) - (entry-point-argument #f)) + (entry-point-argument #f) + (max-layers #f)) (build-docker-image #$output (map store-info-item @@ -609,7 +620,8 @@ (define* (docker-image name profile #:compressor #+(compressor-command compressor) #:creation-time - (make-time time-utc 0 1))))))) + (make-time time-utc 0 1) + #:max-layers max-layers)))))) (gexp->derivation (string-append name ".tar" (compressor-extension compressor)) @@ -1287,6 +1299,7 @@ (define %default-options (verbosity . 1) (symlinks . ()) (entry-point-argument . ()) + (max-layers . ,%docker-image-max-layers) (compressor . ,(first %compressors)))) (define %formats @@ -1324,7 +1337,11 @@ (define (required-option symbol) (define %docker-format-options (list (required-option 'image-tag) (option '(#\A "entry-point-argument") #t #f - entry-point-argument-spec-option-parser))) + entry-point-argument-spec-option-parser) + (option '("max-layers") #t #f + (lambda (opt name arg result) + (alist-cons 'max-layers (string->number* arg) + result))))) (define (show-docker-format-options) (display (G_ " @@ -1338,7 +1355,10 @@ (define (show-docker-format-options/detailed) -A, --entry-point-argument=COMMAND/PARAMETER Value(s) to use for the Docker EntryPoint arguments. Multiple instances are accepted. This is only valid - in conjunction with the --entry-point option")) + in conjunction with the --entry-point option + + --max-layers=N + Number of image layers")) (newline) (exit 0)) @@ -1651,7 +1671,9 @@ (define-command (guix-pack . args) (list #:image-tag (assoc-ref opts 'image-tag) #:entry-point-argument - (assoc-ref opts 'entry-point-argument))) + (assoc-ref opts 'entry-point-argument) + #:max-layers + (assoc-ref opts 'max-layers))) ('deb (list #:control-file (process-file-arg opts 'control-file) diff --git a/tests/pack.scm b/tests/pack.scm index ac78817a70..fda4dc04c6 100644 --- a/tests/pack.scm +++ b/tests/pack.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017-2021, 2023 Ludovic Courtès ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2021, 2023 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (test-pack) #:use-module (guix gexp) #:use-module (guix modules) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (%store-directory)) #:use-module (gnu packages) #:use-module ((gnu packages base) #:select (libc-utf8-locales-for-target)) #:use-module (gnu packages bootstrap) @@ -250,6 +252,53 @@ (define rpm-for-tests (mkdir #$output))))))) (built-derivations (list check)))) + (unless store (test-skip 1)) + (test-assertm "docker-layered-image + localstatedir" + (mlet* %store-monad + ((guile (set-guile-for-build (default-guile))) + (profile -> (profile + (content (packages->manifest (list %bootstrap-guile))) + (hooks '()) + (locales? #f))) + (tarball (docker-image "docker-pack" profile + #:symlinks '(("/bin/Guile" -> "bin/guile")) + #:localstatedir? #t + #:max-layers 100)) + (check (gexp->derivation "check-tarball" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define bin + (string-append "." #$profile "/bin")) + + (define store + (string-append "." #$(%store-directory))) + + (setenv "PATH" (string-append #$%tar-bootstrap "/bin")) + (mkdir "base") + (with-directory-excursion "base" + (invoke "tar" "xvf" #$tarball)) + + (match (find-files "base" "layer.tar") + ((layers ...) + (for-each (lambda (layer) + (invoke "tar" "xvf" layer) + (invoke "chmod" "--recursive" "u+w" store)) + layers))) + + (when + (and (file-exists? (string-append bin "/guile")) + (file-exists? "var/guix/db/db.sqlite") + (file-is-directory? "tmp") + (string=? (string-append #$%bootstrap-guile "/bin") + (pk 'binlink (readlink bin))) + (string=? (string-append #$profile "/bin/guile") + (pk 'guilelink (readlink "bin/Guile")))) + (mkdir #$output))))))) + (built-derivations (list check)))) + (unless store (test-skip 1)) (test-assertm "squashfs-image + localstatedir" (mlet* %store-monad