From patchwork Fri Sep 25 15:50:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 24294 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8616527BBE9; Fri, 25 Sep 2020 17:02:48 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 18D1227BBE8 for ; Fri, 25 Sep 2020 17:02:48 +0100 (BST) Received: from localhost ([::1]:36598 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kLqBL-0006uj-2e for patchwork@mira.cbaines.net; Fri, 25 Sep 2020 12:02:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38990) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kLq0w-0005Xo-3D for guix-patches@gnu.org; Fri, 25 Sep 2020 11:52:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:33578) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kLq0v-00017O-Nf for guix-patches@gnu.org; Fri, 25 Sep 2020 11:52:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kLq0v-0002Nj-LT for guix-patches@gnu.org; Fri, 25 Sep 2020 11:52:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43614] [PATCH 1/2] guix build: Record package transformations in manifest entries. References: <20200925154612.28330-1-ludo@gnu.org> In-Reply-To: <20200925154612.28330-1-ludo@gnu.org> Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 25 Sep 2020 15:52:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43614 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43614@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 43614-submit@debbugs.gnu.org id=B43614.16010490889104 (code B ref 43614); Fri, 25 Sep 2020 15:52:01 +0000 Received: (at 43614) by debbugs.gnu.org; 25 Sep 2020 15:51:28 +0000 Received: from localhost ([127.0.0.1]:45122 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kLq0O-0002Ml-8k for submit@debbugs.gnu.org; Fri, 25 Sep 2020 11:51:28 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60268) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kLq0D-0002MP-C2 for 43614@debbugs.gnu.org; Fri, 25 Sep 2020 11:51:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39146) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kLq06-00013j-Nl; Fri, 25 Sep 2020 11:51:12 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33080 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kLq04-0001ig-EL; Fri, 25 Sep 2020 11:51:10 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 25 Sep 2020 17:50:59 +0200 Message-Id: <20200925155100.28555-1-ludo@gnu.org> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches With this change, package transformation options used while building a manifest are saved in the metadata of the manifest entries. * guix/scripts/build.scm (transformation-procedure): New procedure. (options->transformation)[applicable]: Use it. Change to a list of key/value/proc tuples instead of key/proc pairs. [package-with-transformation-properties, tagged-object]: New procedures. Use them. (package-transformations, manifest-entry-with-transformations): New procedures. * guix/scripts/pack.scm (guix-pack)[with-transformations]: New procedure. Use it. * guix/scripts/package.scm (process-actions)[transform-entry]: Use it. * tests/guix-package-aliases.sh: Add test. --- guix/scripts/build.scm | 80 +++++++++++++++++++++++++++-------- guix/scripts/pack.scm | 29 +++++++------ guix/scripts/package.scm | 13 +++--- tests/guix-package-aliases.sh | 6 +++ 4 files changed, 93 insertions(+), 35 deletions(-) diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 38e0516c95..c8529856f7 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -61,6 +61,7 @@ %transformation-options options->transformation + manifest-entry-with-transformations show-transformation-options-help guix-build @@ -405,6 +406,14 @@ a checkout of the Git repository at the given URL." (with-commit . ,transform-package-source-commit) (with-git-url . ,transform-package-source-git-url))) +(define (transformation-procedure key) + "Return the transformation procedure associated with KEY, a symbol such as +'with-source', or #f if there is none." + (any (match-lambda + ((k . proc) + (and (eq? k key) proc))) + %transformations)) + (define %transformation-options ;; The command-line interface to the above transformations. (let ((parser (lambda (symbol) @@ -454,32 +463,69 @@ derivation, etc.), applies the transformations specified by OPTS." ;; order in which they appear on the command line. (filter-map (match-lambda ((key . value) - (match (any (match-lambda - ((k . proc) - (and (eq? k key) proc))) - %transformations) + (match (transformation-procedure key) (#f #f) (transform ;; XXX: We used to pass TRANSFORM a list of several ;; arguments, but we now pass only one, assuming that ;; transform composes well. - (cons key (transform (list value))))))) + (list key value (transform (list value))))))) (reverse opts))) + (define (package-with-transformation-properties p) + (package/inherit p + (properties `((transformations + . ,(map (match-lambda + ((key value _) + (cons key value))) + applicable)) + ,@(package-properties p))))) + (lambda (store obj) - (fold (match-lambda* - (((name . transform) obj) - (let ((new (transform store obj))) - (when (eq? new obj) - (warning (G_ "transformation '~a' had no effect on ~a~%") - name - (if (package? obj) - (package-full-name obj) - obj))) - new))) - obj - applicable))) + (define (tagged-object new) + (if (and (not (eq? obj new)) + (package? new) (not (null? applicable))) + (package-with-transformation-properties new) + new)) + + (tagged-object + (fold (match-lambda* + (((name value transform) obj) + (let ((new (transform store obj))) + (when (eq? new obj) + (warning (G_ "transformation '~a' had no effect on ~a~%") + name + (if (package? obj) + (package-full-name obj) + obj))) + new))) + obj + applicable)))) + +(define (package-transformations package) + "Return the transformations applied to PACKAGE according to its properties." + (match (assq-ref (package-properties package) 'transformations) + (#f '()) + (transformations transformations))) + +(define (manifest-entry-with-transformations entry) + "Return ENTRY with an additional 'transformations' property if it's not +already there." + (let ((properties (manifest-entry-properties entry))) + (if (assq 'transformations properties) + entry + (let ((item (manifest-entry-item entry))) + (manifest-entry + (inherit entry) + (properties + (match (and (package? item) + (package-transformations item)) + ((or #f '()) + properties) + (transformations + `((transformations . ,transformations) + ,@properties))))))))) ;;; diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 379e6a3ac6..39bbb55eaf 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -1134,19 +1134,24 @@ Create a bundle of PACKAGE.\n")) manifest)) identity)) + (define (with-transformations manifest) + (map-manifest-entries manifest-entry-with-transformations + manifest)) + (with-provenance - (cond - ((and (not (null? manifests)) (not (null? packages))) - (leave (G_ "both a manifest and a package list were given~%"))) - ((not (null? manifests)) - (concatenate-manifests - (map (lambda (file) - (let ((user-module (make-user-module - '((guix profiles) (gnu))))) - (load* file user-module))) - manifests))) - (else - (packages->manifest packages)))))) + (with-transformations + (cond + ((and (not (null? manifests)) (not (null? packages))) + (leave (G_ "both a manifest and a package list were given~%"))) + ((not (null? manifests)) + (concatenate-manifests + (map (lambda (file) + (let ((user-module (make-user-module + '((guix profiles) (gnu))))) + (load* file user-module))) + manifests))) + (else + (packages->manifest packages))))))) (with-error-handling (with-store store diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm index 7e7c37eac4..83f8c123d9 100644 --- a/guix/scripts/package.scm +++ b/guix/scripts/package.scm @@ -864,12 +864,13 @@ processed, #f otherwise." (define (transform-entry entry) (let ((item (transform store (manifest-entry-item entry)))) - (manifest-entry - (inherit entry) - (item item) - (version (if (package? item) - (package-version item) - (manifest-entry-version entry)))))) + (manifest-entry-with-transformations + (manifest-entry + (inherit entry) + (item item) + (version (if (package? item) + (package-version item) + (manifest-entry-version entry))))))) (when (equal? profile %current-profile) ;; Normally the daemon created %CURRENT-PROFILE when we connected, unless diff --git a/tests/guix-package-aliases.sh b/tests/guix-package-aliases.sh index e24bff3a56..5ad65830d7 100644 --- a/tests/guix-package-aliases.sh +++ b/tests/guix-package-aliases.sh @@ -40,6 +40,12 @@ if guix install -r guile-bootstrap -p "$profile" --bootstrap then false; else true; fi test -x "$profile/bin/guile" +# Use a package transformation option and make sure it's recorded. +guix install --bootstrap guile-bootstrap -p "$profile" \ + --with-input=libreoffice=inkscape +test -x "$profile/bin/guile" +grep "libreoffice=inkscape" "$profile/manifest" + guix upgrade --version guix upgrade -n guix upgrade gui.e -n From patchwork Fri Sep 25 15:51:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 24297 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2608727BBE9; Fri, 25 Sep 2020 17:20:08 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 9F37827BBE8 for ; Fri, 25 Sep 2020 17:20:07 +0100 (BST) Received: from localhost ([::1]:53448 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kLqS6-0001Oc-M2 for patchwork@mira.cbaines.net; Fri, 25 Sep 2020 12:20:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38992) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kLq0w-0005YU-Dq for guix-patches@gnu.org; Fri, 25 Sep 2020 11:52:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:33579) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kLq0w-00017U-1r for guix-patches@gnu.org; Fri, 25 Sep 2020 11:52:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kLq0w-0002Nr-1R for guix-patches@gnu.org; Fri, 25 Sep 2020 11:52:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43614] [PATCH 2/2] guix package: Re-apply package transformation when upgrading. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 25 Sep 2020 15:52:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43614 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43614@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 43614-submit@debbugs.gnu.org id=B43614.16010490939121 (code B ref 43614); Fri, 25 Sep 2020 15:52:01 +0000 Received: (at 43614) by debbugs.gnu.org; 25 Sep 2020 15:51:33 +0000 Received: from localhost ([127.0.0.1]:45124 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kLq0S-0002N2-T8 for submit@debbugs.gnu.org; Fri, 25 Sep 2020 11:51:33 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60272) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kLq0D-0002MQ-BO for 43614@debbugs.gnu.org; Fri, 25 Sep 2020 11:51:31 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39147) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kLq07-00013l-Fp; Fri, 25 Sep 2020 11:51:12 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33080 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kLq07-0001ig-1R; Fri, 25 Sep 2020 11:51:11 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Fri, 25 Sep 2020 17:51:00 +0200 Message-Id: <20200925155100.28555-2-ludo@gnu.org> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200925155100.28555-1-ludo@gnu.org> References: <20200925155100.28555-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/scripts/package.scm (transaction-upgrade-entry)[upgrade]: Add 'transform' parameter. Pass PKG through it. Use 'manifest-entry-with-transformations'. Call 'options->transformation' to get the transformation procedure. * tests/guix-package.sh: Add 'guix package -u' test. * tests/packages.scm ("transaction-upgrade-entry, transformation options preserved"): New test. * doc/guix.texi (Invoking guix package): Mention that transformations are preserved across upgrades. (Package Transformation Options): Likewise. --- doc/guix.texi | 27 +++++++++++++++++++++++++++ guix/scripts/package.scm | 20 +++++++++++++++----- tests/guix-package.sh | 17 ++++++++++++++++- tests/packages.scm | 23 +++++++++++++++++++++++ 4 files changed, 81 insertions(+), 6 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 6b2c749bc7..81dbe95e9f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -3082,6 +3082,29 @@ in the distribution currently installed. To update your distribution, you should regularly run @command{guix pull} (@pxref{Invoking guix pull}). +@cindex package transformations, upgrades +When upgrading, package transformations that were originally applied +when creating the profile are automatically re-applied (@pxref{Package +Transformation Options}). For example, assume you first installed Emacs +from the tip of its development branch with: + +@example +guix install emacs-next --with-branch=emacs-next=master +@end example + +Next time you run @command{guix upgrade}, Guix will again pull the tip +of the Emacs development branch and build @code{emacs-next} from that +checkout. + +Note that transformation options such as @option{--with-branch} and +@option{--with-source} depend on external state; it is up to you to +ensure that they work as expected. You can also discard a +transformations that apply to a package by running: + +@example +guix install @var{package} +@end example + @item --do-not-upgrade[=@var{regexp} @dots{}] When used together with the @option{--upgrade} option, do @emph{not} upgrade any packages whose name matches a @var{regexp}. For example, to @@ -9135,6 +9158,10 @@ This is a convenient way to create customized packages on the fly without having to type in the definitions of package variants (@pxref{Defining Packages}). +Package transformation options are preserved across upgrades: +@command{guix upgrade} attempts to apply transformation options +initially used when creating the profile to the upgraded packages. + @table @code @item --with-source=@var{source} diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm index 83f8c123d9..2f04652634 100644 --- a/guix/scripts/package.scm +++ b/guix/scripts/package.scm @@ -218,12 +218,13 @@ non-zero relevance score." (output (manifest-entry-output old))) transaction))) - (define (upgrade entry) + (define (upgrade entry transform) (match entry (($ name version output (? string? path)) (match (find-best-packages-by-name name #f) ((pkg . rest) - (let ((candidate-version (package-version pkg))) + (let* ((pkg (transform store pkg)) + (candidate-version (package-version pkg))) (match (package-superseded pkg) ((? package? new) (supersede entry new)) @@ -231,12 +232,14 @@ non-zero relevance score." (case (version-compare candidate-version version) ((>) (manifest-transaction-install-entry - (package->manifest-entry* pkg output) + (manifest-entry-with-transformations + (package->manifest-entry* pkg output)) transaction)) ((<) transaction) ((=) - (let* ((new (package->manifest-entry* pkg output))) + (let* ((new (manifest-entry-with-transformations + (package->manifest-entry* pkg output)))) ;; Here we want to determine whether the NEW actually ;; differs from ENTRY, but we need to intercept ;; 'build-things' calls because they would prevent us from @@ -255,7 +258,14 @@ non-zero relevance score." (if (manifest-transaction-removal-candidate? entry transaction) transaction - (upgrade entry))) + + ;; Upgrade ENTRY, preserving transformation options listed in its + ;; properties. + (let ((transform (options->transformation + (or (assq-ref (manifest-entry-properties entry) + 'transformations) + '())))) + (upgrade entry transform)))) ;;; diff --git a/tests/guix-package.sh b/tests/guix-package.sh index 1f955257be..c7ef07c89a 100644 --- a/tests/guix-package.sh +++ b/tests/guix-package.sh @@ -1,5 +1,5 @@ # GNU Guix --- Functional package management for GNU -# Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès +# Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès # Copyright © 2013 Nikita Karetnikov # # This file is part of GNU Guix. @@ -193,6 +193,21 @@ grep -E 'emacs[[:blank:]]+42\.5\.9rc7' "$tmpfile" rm "$emacs_tarball" "$tmpfile" rmdir "$module_dir" +# Install with package transformations. +guix install --bootstrap -p "$profile" sed --with-input=sed=guile-bootstrap +grep "sed=guile-bootstrap" "$profile/manifest" +test "$(readlink -f "$profile/bin/guile")" \ + = "$(guix build guile-bootstrap)/bin/guile" +test ! -f "$profile/bin/sed" + +# Make sure the package transformation is preserved. +guix package --bootstrap -p "$profile" -u +grep "sed=guile-bootstrap" "$profile/manifest" +test "$(readlink -f "$profile/bin/guile")" \ + = "$(guix build guile-bootstrap)/bin/guile" +test ! -f "$profile/bin/sed" +rm "$profile" "$profile"-[0-9]-link + # Profiles with a relative file name. Make sure we don't create dangling # symlinks--see bug report at # . diff --git a/tests/packages.scm b/tests/packages.scm index cbd0503733..b1885f70d2 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -185,6 +185,29 @@ (string=? (manifest-pattern-version pattern) "1") (string=? (manifest-pattern-output pattern) "out"))))))) +(test-equal "transaction-upgrade-entry, transformation options preserved" + (derivation-file-name (package-derivation %store grep)) + + (let* ((old (dummy-package "sed" (version "1"))) + (props '((transformations . ((with-input . "sed=grep"))))) + (tx (transaction-upgrade-entry + %store + (manifest-entry + (inherit (package->manifest-entry old)) + (properties props) + (item (string-append (%store-prefix) "/" + (make-string 32 #\e) "-foo-1"))) + (manifest-transaction)))) + (match (manifest-transaction-install tx) + (((? manifest-entry? entry)) + (and (string=? (manifest-entry-version entry) + (package-version grep)) + (string=? (manifest-entry-name entry) + (package-name grep)) + (equal? (manifest-entry-properties entry) props) + (derivation-file-name + (package-derivation %store (manifest-entry-item entry)))))))) + (test-assert "transaction-upgrade-entry, grafts" ;; Ensure that, when grafts are enabled, 'transaction-upgrade-entry' doesn't ;; try to build stuff.