From patchwork Tue Jan 4 15:09:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35934 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 27B8A27BBEA; Tue, 4 Jan 2022 15:10:43 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7499327BBE9 for ; Tue, 4 Jan 2022 15:10:42 +0000 (GMT) Received: from localhost ([::1]:58370 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n4lST-0006TA-Kx for patchwork@mira.cbaines.net; Tue, 04 Jan 2022 10:10:41 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55722) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4lRq-0006NX-H7 for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:55137) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n4lRq-0000Hc-6E for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n4lRq-0004Gl-1r for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v3 1/4] guix hash: Extract file hashing procedures. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 04 Jan 2022 15:10:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Sarah Morgensen , Maxime Devos Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164130898916358 (code B ref 50072); Tue, 04 Jan 2022 15:10:01 +0000 Received: (at 50072) by debbugs.gnu.org; 4 Jan 2022 15:09:49 +0000 Received: from localhost ([127.0.0.1]:38442 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRZ-0004FW-6c for submit@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:48 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:40110) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRX-0004F4-6y for 50072@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:44 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id ef9g2600W4UW6Th06f9hSi; Tue, 04 Jan 2022 16:09:41 +0100 From: Maxime Devos Date: Tue, 4 Jan 2022 15:09:34 +0000 Message-Id: <20220104150937.35872-2-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220104150937.35872-1-maximedevos@telenet.be> References: <20220104150937.35872-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641308981; bh=i9yb7nh1nBUy1tiYhhqR0CWJ3JVd+uJc5NmKjxoYf0I=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Ezw8ODvHeDPC0Va5vpyaelFrqXX/yzfl/dsUSXx1RESlaYzobhmMaOyDlLE5uvqVp KFJvcWUrOXeMbcPD8QlMqw93WL7OnZOTIgNXpCIuLZjpKnD92deHYogurThlc7RbyJ sjnbvHTnQnPkcqOMHOyqUhu6CoFwaL7bwCzpZnIp/wW5ApOG8g0ZSh8nM4E2GVbE7F qlcyNuHpGCF5O+XdghyeXgS/+jbEMzqgUEWU3ok3XjDcWKbki5MJKs0LslnzKncuGA 60+/wgkAB5XZLQ8yRUWqeOwAybefrZmLrK9WAmALrwxqz2fUAQ9VYx/SvJs7vp1XV7 FGx9U0VqYUSSQ== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen * guix/scripts/hash.scm (guix-hash)[vcs-file?] (nar-hash, default-hash): Extract hashing logic to... * guix/hash.scm (vcs-file?, file-hash*): ... these new procedures in this new file. Modified-by: Maxime Devos --- Makefile.am | 1 + guix/hash.scm | 68 +++++++++++++++++++++++++++++++++++++++++++ guix/scripts/hash.scm | 22 +++----------- 3 files changed, 73 insertions(+), 18 deletions(-) create mode 100644 guix/hash.scm diff --git a/Makefile.am b/Makefile.am index 8c5682a1c6..bc3d0087d0 100644 --- a/Makefile.am +++ b/Makefile.am @@ -99,6 +99,7 @@ MODULES = \ guix/extracting-download.scm \ guix/git-download.scm \ guix/hg-download.scm \ + guix/hash.scm \ guix/swh.scm \ guix/monads.scm \ guix/monad-repl.scm \ diff --git a/guix/hash.scm b/guix/hash.scm new file mode 100644 index 0000000000..19cbc41ad1 --- /dev/null +++ b/guix/hash.scm @@ -0,0 +1,68 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2022 Maxime Devos +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix hash) + #:use-module (gcrypt hash) + #:use-module (guix serialization) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:export (vcs-file? + file-hash*)) + +(define (vcs-file? file stat) + "Returns true if FILE is a version control system file." + (case (stat:type stat) + ((directory) + (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) + ((regular) + ;; Git sub-modules have a '.git' file that is a regular text file. + (string=? (basename file) ".git")) + (else + #f))) + +(define* (file-hash* file #:key + (algorithm (hash-algorithm sha256)) + (recursive? 'auto) + (select? (negate vcs-file?))) + "Compute the hash of FILE with ALGORITHM. If RECURSIVE? is #true or 'auto', +recurse into subdirectories of FILE, computing the combined hash (nar hash) of +all files for which (SELECT? FILE STAT) returns true. + +Symbolic links are not dereferenced unless RECURSIVE? is false. + +This procedure must only be used under controlled circumstances; +the detection of symbolic links in FILE is racy. + +Keep in mind that the hash of a regular file depends on RECURSIVE?: +if the recursive hash is desired, it must be set to #true. Otherwise, it must +be set to #false or 'auto'. In most situations, the non-recursive hash is desired +for regular files." + (if (or (eq? recursive? #true) + (and (eq? recursive? 'auto) + ;; Don't change this to (eq? 'directory ...), because otherwise + ;; if 'file' denotes a symbolic link, the 'file-hash' below + ;; would dereference it -- dereferencing symbolic links would + ;; open an avoidable can of potential worms. + (not (eq? 'regular (stat:type (lstat file)))))) + (let-values (((port get-hash) + (open-hash-port algorithm))) + (write-file file port #:select? select?) + (force-output port) + (get-hash)) + (file-hash algorithm file))) diff --git a/guix/scripts/hash.scm b/guix/scripts/hash.scm index d73e3d13dd..28d587b944 100644 --- a/guix/scripts/hash.scm +++ b/guix/scripts/hash.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2016 Jan Nieuwenhuizen ;;; Copyright © 2018 Tim Gesthuizen ;;; Copyright © 2021 Simon Tournier +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,6 +25,7 @@ #:use-module (gcrypt hash) #:use-module (guix serialization) #:use-module (guix ui) + #:use-module (guix hash) #:use-module (guix scripts) #:use-module (guix base16) #:use-module (guix base32) @@ -46,20 +48,14 @@ (define* (nar-hash file #:optional (algorithm (assoc-ref %default-options 'hash-algorithm)) select?) - (let-values (((port get-hash) - (open-hash-port algorithm))) - (write-file file port #:select? select?) - (force-output port) - (get-hash))) + (file-hash* file #:algorithm algorithm #:select? select? #:recursive? #true)) (define* (default-hash file #:optional (algorithm (assoc-ref %default-options 'hash-algorithm)) select?) (match file ("-" (port-hash algorithm (current-input-port))) - (_ - (call-with-input-file file - (cute port-hash algorithm <>))))) + (_ (file-hash* file #:algorithm algorithm #:recursive? #false)))) (define* (git-hash file #:optional (algorithm (assoc-ref %default-options 'hash-algorithm)) @@ -181,16 +177,6 @@ use '--serializer' instead~%")) (parse-command-line args %options (list %default-options) #:build-options? #f)) - (define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - (let* ((opts (parse-options)) (args (filter-map (match-lambda (('argument . value) From patchwork Tue Jan 4 15:09:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35936 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 1E84D27BBEA; Tue, 4 Jan 2022 15:10:54 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 4FD9D27BBE9 for ; Tue, 4 Jan 2022 15:10:53 +0000 (GMT) Received: from localhost ([::1]:59280 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n4lSe-00075W-ET for patchwork@mira.cbaines.net; Tue, 04 Jan 2022 10:10:52 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55730) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4lRq-0006Oe-VO for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:55138) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n4lRq-0000J8-KJ for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n4lRq-0004Gs-FY for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v3 2/4] import: Factorize file hashing. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 04 Jan 2022 15:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Sarah Morgensen Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164130899316369 (code B ref 50072); Tue, 04 Jan 2022 15:10:02 +0000 Received: (at 50072) by debbugs.gnu.org; 4 Jan 2022 15:09:53 +0000 Received: from localhost ([127.0.0.1]:38444 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRc-0004Fn-Sv for submit@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:53 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:40120) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRX-0004F6-6v for 50072@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:44 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id ef9g2600W4UW6Th06f9hSr; Tue, 04 Jan 2022 16:09:41 +0100 From: Maxime Devos Date: Tue, 4 Jan 2022 15:09:35 +0000 Message-Id: <20220104150937.35872-3-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220104150937.35872-1-maximedevos@telenet.be> References: <20220104150937.35872-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641308981; bh=pToVcY6tyNJdA8to9CMlcbffGWezAHQB9+uOz7gou4c=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LdBMHrkP11N1kcxdHNDpgjSYrgRACy3Y7S6qN/DfTmxrUARMx/YkSKohhNGM54J88 XV2gtXFfvmh0K4NBA3QLOigqYIYunxT+hTlVrfQnFD/I18COA9cPoGziYpAxDdVnWi 5N6dI0NS6XEr0msU6IxqAHlPJtR1EAEEUM0dJTIU0E2YBoUvzZbcfyI8TR4N7FXbiT Xykf1fw7y+DU/8LQJGIMvovBgo8NYbfNjOHlLVZ3ER2Ds4egAq2azkX6K7fITv30Xr yPGMXI/vdanKqEKOeefrvU6w45DcuGoQg5nd/aPiPo3emUaFmIFZScoG4n1HOz1/kc J9Lj8VRT3YJbQ== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen * guix/import/cran.scm (vcs-file?, file-hash): Remove procedures. (description->package): Use 'file-hash*' instead. * guix/import/elpa.scm (vcs-file?, file-hash): Remove procedures. (git-repository->origin, elpa-package->sexp): Use 'file-hash* instead'. * guix/import/go.scm (vcs-file?, file-hash): Remove procedures. (git-checkout-hash): Use 'file-hash*' instead. * guix/import/minetest.scm (file-hash): Remove procedure. (make-minetest-sexp): Use 'file-hash*' instead. --- guix/import/cran.scm | 32 +++----------------------------- guix/import/elpa.scm | 29 +++++------------------------ guix/import/go.scm | 25 +++---------------------- guix/import/minetest.scm | 19 ++++++++----------- 4 files changed, 19 insertions(+), 86 deletions(-) diff --git a/guix/import/cran.scm b/guix/import/cran.scm index 1389576cad..b61402078d 100644 --- a/guix/import/cran.scm +++ b/guix/import/cran.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2015, 2016, 2017, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2020 Martin Becze +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -35,10 +36,9 @@ #:use-module (guix memoization) #:use-module (guix http-client) #:use-module (guix diagnostics) + #:use-module (guix hash) #:use-module (guix i18n) - #:use-module (gcrypt hash) #:use-module (guix store) - #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix base32) #:use-module ((guix download) #:select (download-to-store)) #:use-module (guix import utils) @@ -196,17 +196,6 @@ bioconductor package NAME, or #F if the package is unknown." (bioconductor-packages-list type)) (cut assoc-ref <> "Version"))) -;; XXX taken from (guix scripts hash) -(define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - ;; Little helper to download URLs only once. (define download (memoize @@ -464,16 +453,6 @@ reference the pkg-config tool." (define (needs-knitr? meta) (member "knitr" (listify meta "VignetteBuilder"))) -;; XXX adapted from (guix scripts hash) -(define (file-hash file select? recursive?) - ;; Compute the hash of FILE. - (if recursive? - (let-values (((port get-hash) (open-sha256-port))) - (write-file file port #:select? select?) - (force-output port) - (get-hash)) - (call-with-input-file file port-sha256))) - (define (description->package repository meta) "Return the `package' s-expression for an R package published on REPOSITORY from the alist META, which was derived from the R package's DESCRIPTION file." @@ -571,12 +550,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (sha256 (base32 ,(bytevector->nix-base32-string - (case repository - ((git) - (file-hash source (negate vcs-file?) #t)) - ((hg) - (file-hash source (negate vcs-file?) #t)) - (else (file-sha256 source)))))))) + (file-hash* source #:recursive? (or git? hg?))))))) ,@(if (not (and git? hg? (equal? (string-append "r-" name) (cran-guix-name name)))) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm index edabb88b7a..c5167eacb5 100644 --- a/guix/import/elpa.scm +++ b/guix/import/elpa.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2020 Martin Becze ;;; Copyright © 2020 Ricardo Wurmus ;;; Copyright © 2021 Xinglu Chen +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -37,10 +38,10 @@ #:use-module (guix import utils) #:use-module (guix http-client) #:use-module (guix git) + #:use-module (guix hash) #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix store) #:use-module (guix ui) - #:use-module (gcrypt hash) #:use-module (guix base32) #:use-module (guix upstream) #:use-module (guix packages) @@ -229,27 +230,6 @@ keywords to values." (close-port port) (data->recipe (cons ':name data)))) -;; XXX adapted from (guix scripts hash) -(define (file-hash file select? recursive?) - ;; Compute the hash of FILE. - (if recursive? - (let-values (((port get-hash) (open-sha256-port))) - (write-file file port #:select? select?) - (force-output port) - (get-hash)) - (call-with-input-file file port-sha256))) - -;; XXX taken from (guix scripts hash) -(define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - (define (git-repository->origin recipe url) "Fetch origin details from the Git repository at URL for the provided MELPA RECIPE." @@ -271,7 +251,7 @@ RECIPE." (sha256 (base32 ,(bytevector->nix-base32-string - (file-hash directory (negate vcs-file?) #t))))))) + (file-hash* directory #:recursive? #true))))))) (define* (melpa-recipe->origin recipe) "Fetch origin details from the MELPA recipe and associated repository for @@ -380,7 +360,8 @@ type ''." (sha256 (base32 ,(if tarball - (bytevector->nix-base32-string (file-sha256 tarball)) + (bytevector->nix-base32-string + (file-hash* tarball #:recursive? #false)) "failed to download package"))))))) (build-system emacs-build-system) ,@(maybe-inputs 'propagated-inputs dependencies) diff --git a/guix/import/go.scm b/guix/import/go.scm index 26dbc34b63..c7673e6a1a 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -26,6 +26,7 @@ (define-module (guix import go) #:use-module (guix build-system go) #:use-module (guix git) + #:use-module (guix hash) #:use-module (guix i18n) #:use-module (guix diagnostics) #:use-module (guix import utils) @@ -36,11 +37,10 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix memoization) #:autoload (htmlprag) (html->sxml) ;from Guile-Lib - #:autoload (guix git) (update-cached-checkout) - #:autoload (gcrypt hash) (open-hash-port hash-algorithm sha256) #:autoload (guix serialization) (write-file) #:autoload (guix base32) (bytevector->nix-base32-string) #:autoload (guix build utils) (mkdir-p) + #:autoload (gcrypt hash) (hash-algorithm sha256) #:use-module (ice-9 match) #:use-module (ice-9 peg) #:use-module (ice-9 rdelim) @@ -499,25 +499,6 @@ source." goproxy (module-meta-repo-root meta-data))) -;; XXX: Copied from (guix scripts hash). -(define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - -;; XXX: Adapted from 'file-hash' in (guix scripts hash). -(define* (file-hash file #:optional (algorithm (hash-algorithm sha256))) - ;; Compute the hash of FILE. - (let-values (((port get-hash) (open-hash-port algorithm))) - (write-file file port #:select? (negate vcs-file?)) - (force-output port) - (get-hash))) - (define* (git-checkout-hash url reference algorithm) "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or tag." @@ -536,7 +517,7 @@ tag." (update-cached-checkout url #:ref `(tag-or-commit . ,reference))))) - (file-hash checkout algorithm))) + (file-hash* checkout #:algorithm algorithm #:recursive? #true))) (define (vcs->origin vcs-type vcs-repo-url version) "Generate the `origin' block of a package depending on what type of source diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm index abddd885ee..a7bdbfebca 100644 --- a/guix/import/minetest.scm +++ b/guix/import/minetest.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2021 Maxime Devos +;;; Copyright © 2021, 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -39,6 +39,7 @@ #:use-module (guix base32) #:use-module (guix git) #:use-module ((guix git-download) #:prefix download:) + #:use-module (guix hash) #:use-module (guix store) #:export (%default-sort-key %contentdb-api @@ -286,14 +287,6 @@ results. The return value is a list of records." (with-store store (latest-repository-commit store url #:ref ref))) -;; XXX adapted from (guix scripts hash) -(define (file-hash file) - "Compute the hash of FILE." - (let-values (((port get-hash) (open-sha256-port))) - (write-file file port) - (force-output port) - (get-hash))) - (define (make-minetest-sexp author/name version repository commit inputs home-page synopsis description media-license license) @@ -314,9 +307,13 @@ MEDIA-LICENSE and LICENSE." ;; The git commit is not always available. ,(and commit (bytevector->nix-base32-string - (file-hash + (file-hash* (download-git-repository repository - `(commit . ,commit))))))) + `(commit . ,commit)) + ;; 'download-git-repository' already filtered out the '.git' + ;; directory. + #:select? (const #true) + #:recursive? #true))))) (file-name (git-file-name name version)))) (build-system minetest-mod-build-system) ,@(maybe-propagated-inputs (map contentdb->package-name inputs)) From patchwork Tue Jan 4 15:09:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35937 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 23FF227BBEA; Tue, 4 Jan 2022 15:11:03 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id D780427BBE9 for ; Tue, 4 Jan 2022 15:11:02 +0000 (GMT) Received: from localhost ([::1]:59970 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n4lSo-0007Y4-0g for patchwork@mira.cbaines.net; Tue, 04 Jan 2022 10:11:02 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55742) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4lRr-0006Qw-PP for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:55140) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n4lRr-0000QN-FQ for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n4lRr-0004H7-Au for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v3 3/4] refresh: Support non-tarball sources. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 04 Jan 2022 15:10:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Sarah Morgensen Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164130899416385 (code B ref 50072); Tue, 04 Jan 2022 15:10:03 +0000 Received: (at 50072) by debbugs.gnu.org; 4 Jan 2022 15:09:54 +0000 Received: from localhost ([127.0.0.1]:38448 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRi-0004G7-6y for submit@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:54 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:40132) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRX-0004F7-Ty for 50072@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:45 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id ef9g2600W4UW6Th06f9hSu; Tue, 04 Jan 2022 16:09:41 +0100 From: Maxime Devos Date: Tue, 4 Jan 2022 15:09:36 +0000 Message-Id: <20220104150937.35872-4-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220104150937.35872-1-maximedevos@telenet.be> References: <20220104150937.35872-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641308981; bh=1C9ahprOwcy10Tnk5guGYz+QxZ/hL2vALCXNKEI79wE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Bp2w2GFiG4Ts/sYFXSxseb16fN1VXWRBBJ5iqbHaonnOB4iQ52YwCnrtiuGH268Od fuxyU8tWjByVV7X9hzygwlYCcPrr1RphKonEp2q5a3hFgmflcTLljpQEaL8x+h4NXH 3uOiK1TlLKvbgJbnr4yezImTgec4gnlxCQltxJYAFxOLsCtnux1s7eY07mRm1x/4tA hWmY5hooMpWh8j5wMCTq1Dj96hGQV0B662g39PCQdUxuz8gM3yCeE/PSnr7Mq+XOqu BOPM/rAQhUm+pvMARRM9DBfv6kpuXQJVj/+/Pjw92dIXQKwOCo3VSg2NAx051kiy5f daqa2+xf9JZJw== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen * guix/scripts/refresh.scm (update-package): Use 'file-hash*' instead of 'port-sha256'. Rename TARBALL to OUTPUT. --- guix/scripts/refresh.scm | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm index 8806f0f740..68bb9040d8 100644 --- a/guix/scripts/refresh.scm +++ b/guix/scripts/refresh.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2018 Efraim Flashner ;;; Copyright © 2019 Ricardo Wurmus ;;; Copyright © 2020 Simon Tournier +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,7 +27,6 @@ (define-module (guix scripts refresh) #:use-module (guix ui) - #:use-module (gcrypt hash) #:use-module (guix scripts) #:use-module ((guix scripts build) #:select (%standard-build-options)) #:use-module (guix store) @@ -38,6 +38,7 @@ #:use-module (guix scripts graph) #:use-module (guix monads) #:use-module (guix gnupg) + #:use-module (guix hash) #:use-module (gnu packages) #:use-module ((gnu packages commencement) #:select (%final-inputs)) #:use-module (ice-9 match) @@ -314,14 +315,14 @@ KEY-DOWNLOAD specifies a download policy for missing OpenPGP keys; allowed values: 'interactive' (default), 'always', and 'never'. When WARN? is true, warn about packages that have no matching updater." (if (lookup-updater package updaters) - (let-values (((version tarball source) + (let-values (((version output source) (package-update store package updaters #:key-download key-download)) ((loc) (or (package-field-location package 'version) (package-location package)))) (when version - (if (and=> tarball file-exists?) + (if (and=> output file-exists?) (begin (info loc (G_ "~a: updating from version ~a to version ~a...~%") @@ -363,8 +364,7 @@ warn about packages that have no matching updater." (info loc (G_ "~a: consider removing this propagated input: ~a~%") name change-name)))) (upstream-source-input-changes source)) - (let ((hash (call-with-input-file tarball - port-sha256))) + (let ((hash (file-hash* output))) (update-package-source package source hash))) (warning (G_ "~a: version ~a could not be \ downloaded and authenticated; not updating~%") From patchwork Tue Jan 4 15:09:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35935 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B419627BBEA; Tue, 4 Jan 2022 15:10:44 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id A606227BBE9 for ; Tue, 4 Jan 2022 15:10:43 +0000 (GMT) Received: from localhost ([::1]:58444 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n4lSU-0006Wo-S8 for patchwork@mira.cbaines.net; Tue, 04 Jan 2022 10:10:42 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55744) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n4lRs-0006Rn-7D for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:55141) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n4lRr-0000TZ-SE for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n4lRr-0004HE-Oi for guix-patches@gnu.org; Tue, 04 Jan 2022 10:10:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v3 4/4] upstream: Support updating and fetching 'git-fetch' origins. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 04 Jan 2022 15:10:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Sarah Morgensen , Maxime Devos Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164130899516392 (code B ref 50072); Tue, 04 Jan 2022 15:10:03 +0000 Received: (at 50072) by debbugs.gnu.org; 4 Jan 2022 15:09:55 +0000 Received: from localhost ([127.0.0.1]:38450 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRi-0004GE-H8 for submit@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:55 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:40150) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n4lRX-0004F8-Ty for 50072@debbugs.gnu.org; Tue, 04 Jan 2022 10:09:48 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id ef9g2600W4UW6Th06f9hSy; Tue, 04 Jan 2022 16:09:41 +0100 From: Maxime Devos Date: Tue, 4 Jan 2022 15:09:37 +0000 Message-Id: <20220104150937.35872-5-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220104150937.35872-1-maximedevos@telenet.be> References: <20220104150937.35872-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641308982; bh=QiD3exeARFKTO8aRSuAVLT6rafXwAxA/gqtoRI66V1o=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=JZC8R5URCP5ZP+bpAOhrcs8nDNsielIGDDEdRerK/AKJE1bdPzrrztkQSwa/8BSc5 tUEOXAys2Kuk5Io67N4DL5zJ4JDRpIUru+hQ3Mu8Zkpj/AAFW6IOXVFYX/2YrJdWOz qQyGakHiVBVanAeP9BnE3dLXB61dnegRRoKkw3UxWtXBMc3BEp5A/PLG454xJ7w7+3 EB92ubN+O/GaFXYJur6DT1obwGvl5M4gh/nzSPll7LoAzBNYh2TlP3FLJLrNtAqtbv /VOijX/VEceLrMDxmdYJGDO2wAYZo10rUtc9dzLf/OWHDkGRcFQny26oMyV5dmBZRS QqX28hz7kFZRA== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen Updaters need to be modified to return 'git-reference' objects. This patch modifies the 'generic-git' and 'minetest' updater, but others might need to be modified as well. * guix/git.scm (git-reference->git-checkout): New procedure. * guix/upstream.scm (package-update/git-fetch): New procedure. ()[urls]: Document it can be a 'git-reference'. (%method-updates): Add 'git-fetch' mapping. (update-package-source): Support 'git-reference' sources. (upstream-source-compiler/url-fetch): Split off from ... (upstream-source-compiler): ... this, and call ... (upstream-source-compiler/git-fetch): ... this new procedure if the URL field contains a 'git-reference'. * guix/import/git.scm (latest-git-tag-version): Always return two values and document that the tag is returned as well. (latest-git-release)[urls]: Use the 'git-reference' instead of the repository URL. * guix/import/minetest.scm (latest-minetest-release)[urls]: Don't wrap the 'git-reference' in a list. * tests/minetest.scm (upstream-source->sexp): Adjust to new convention. Co-authored-by: Maxime Devos --- guix/git.scm | 14 ++++++++- guix/import/git.scm | 22 +++++++------ guix/import/minetest.scm | 6 ++-- guix/upstream.scm | 68 +++++++++++++++++++++++++++++++++++----- tests/minetest.scm | 7 ++--- 5 files changed, 93 insertions(+), 24 deletions(-) diff --git a/guix/git.scm b/guix/git.scm index dc2ca1be84..43e85a5026 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2021 Kyle Meyer ;;; Copyright © 2021 Marius Bakke +;;; Copyright © 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -33,6 +34,8 @@ #:use-module (guix utils) #:use-module (guix records) #:use-module (guix gexp) + #:autoload (guix git-download) + (git-reference-url git-reference-commit git-reference-recursive?) #:use-module (guix sets) #:use-module ((guix diagnostics) #:select (leave warning)) #:use-module (guix progress) @@ -65,7 +68,9 @@ git-checkout-url git-checkout-branch git-checkout-commit - git-checkout-recursive?)) + git-checkout-recursive? + + git-reference->git-checkout)) (define %repository-cache-directory (make-parameter (string-append (cache-directory #:ensure? #f) @@ -672,6 +677,13 @@ is true, limit to only refs/tags." (commit git-checkout-commit (default #f)) ;#f | tag | commit (recursive? git-checkout-recursive? (default #f))) +(define (git-reference->git-checkout reference) + "Convert the REFERENCE to an equivalent ." + (git-checkout + (url (git-reference-url reference)) + (commit (git-reference-commit reference)) + (recursive? (git-reference-recursive? reference)))) + (define* (latest-repository-commit* url #:key ref recursive? log-port) ;; Monadic variant of 'latest-repository-commit'. (lambda (store) diff --git a/guix/import/git.scm b/guix/import/git.scm index 1eb219f3fe..4cf404677c 100644 --- a/guix/import/git.scm +++ b/guix/import/git.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2021 Xinglu Chen ;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -34,6 +35,7 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) #:export (%generic-git-updater ;; For tests. @@ -172,21 +174,21 @@ repository at URL." (values version tag))))))) (define (latest-git-tag-version package) - "Given a PACKAGE, return the latest version of it, or #f if the latest version -could not be determined." + "Given a PACKAGE, return the latest version of it and the corresponding git +tag, or #false and #false if the latest version could not be determined." (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) (warning (or (package-field-location package 'source) (package-location package)) (G_ "~a for ~a~%") (condition-message c) (package-name package)) - #f) + (values #f #f)) ((eq? (exception-kind c) 'git-error) (warning (or (package-field-location package 'source) (package-location package)) (G_ "failed to fetch Git repository for ~a~%") (package-name package)) - #f)) + (values #f #f))) (let* ((source (package-source package)) (url (git-reference-url (origin-uri source))) (property (cute assq-ref (package-properties package) <>))) @@ -208,14 +210,16 @@ could not be determined." "Return an for the latest release of PACKAGE." (let* ((name (package-name package)) (old-version (package-version package)) - (url (git-reference-url (origin-uri (package-source package)))) - (new-version (latest-git-tag-version package))) - - (and new-version + (old-reference (origin-uri (package-source package))) + (new-version new-version-tag (latest-git-tag-version package))) + (and new-version new-version-tag (upstream-source (package name) (version new-version) - (urls (list url)))))) + (urls (git-reference + (url (git-reference-url old-reference)) + (commit new-version-tag) + (recursive? (git-reference-recursive? old-reference)))))))) (define %generic-git-updater (upstream-updater diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm index a7bdbfebca..3b2cdcdcac 100644 --- a/guix/import/minetest.scm +++ b/guix/import/minetest.scm @@ -504,9 +504,9 @@ or #false if the latest release couldn't be determined." (upstream-source (package (package:package-name pkg)) (version (release-version release)) - (urls (list (download:git-reference - (url (package-repository contentdb-package)) - (commit (release-commit release)))))))) + (urls (download:git-reference + (url (package-repository contentdb-package)) + (commit (release-commit release))))))) (define %minetest-updater (upstream-updater diff --git a/guix/upstream.scm b/guix/upstream.scm index 632e9ebc4f..bb6db2cedb 100644 --- a/guix/upstream.scm +++ b/guix/upstream.scm @@ -2,6 +2,8 @@ ;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2015 Alex Kost ;;; Copyright © 2019 Ricardo Wurmus +;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2021, 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,12 +26,14 @@ #:use-module (guix discovery) #:use-module ((guix download) #:select (download-to-store url-fetch)) + #:use-module (guix git-download) #:use-module (guix gnupg) #:use-module (guix packages) #:use-module (guix diagnostics) #:use-module (guix ui) #:use-module (guix base32) #:use-module (guix gexp) + #:use-module (guix git) #:use-module (guix store) #:use-module ((guix derivations) #:select (built-derivations derivation->output-path)) #:autoload (gcrypt hash) (port-sha256) @@ -93,7 +97,7 @@ upstream-source? (package upstream-source-package) ;string (version upstream-source-version) ;string - (urls upstream-source-urls) ;list of strings + (urls upstream-source-urls) ;list of strings|git-reference (signature-urls upstream-source-signature-urls ;#f | list of strings (default #f)) (input-changes upstream-source-input-changes @@ -357,10 +361,20 @@ values: 'interactive' (default), 'always', and 'never'." data url) #f))))))) -(define-gexp-compiler (upstream-source-compiler (source ) - system target) - "Download SOURCE from its first URL and lower it as a fixed-output -derivation that would fetch it." +(define (upstream-source-compiler/git-fetch source system target) + "Lower SOURCE, an using git." + ;; TODO: it would be nice to support provenance tracking, as + ;; in 'upstream-source-compiler/url-fetch'. + ;; + ;; TODO: this causes + ;; + ;; ‘Wrong type to apply: #< url: "https://github.com/minetest-mods/unified_inventory" branch: #f commit: "d6688872c84417d2f61d6f5e607aea39d78920aa" recursive?: #f>’? + ;; (Another error results if it is wrapped in a 'return'.) + (git-reference->git-checkout (upstream-source-urls source))) + +(define (upstream-source-compiler/url-fetch source system target) + "Lower SOURCE, an pointing to a tarball, as a +fixed-output derivation that would fetch it, and verify its authenticity." (mlet* %store-monad ((url -> (first (upstream-source-urls source))) (signature -> (and=> (upstream-source-signature-urls source) @@ -378,6 +392,15 @@ derivation that would fetch it." (url-fetch url 'sha256 hash (store-path-package-name tarball) #:system system)))) +(define-gexp-compiler (upstream-source-compiler (source ) + system target) + "Download SOURCE and verify its authenticity if possible. When feasible, +lower it as a fixed-output derivation that would fetch it, to improve +provenance tracking." + (if (git-reference? (upstream-source-urls source)) + (upstream-source-compiler/git-fetch source system target) + (upstream-source-compiler/url-fetch source system target))) + (define (find2 pred lst1 lst2) "Like 'find', but operate on items from both LST1 and LST2. Return two values: the item from LST1 and the item from LST2 that match PRED." @@ -430,9 +453,24 @@ SOURCE, an ." #:key-download key-download))) (values version tarball source)))))) +(define* (package-update/git-fetch store package source #:key key-download) + "Return the version, checkout, and SOURCE, to update PACKAGE to +SOURCE, an ." + ;; TODO: it would be nice to authenticate commits, e.g. with + ;; "guix git authenticate" or a list of permitted signing keys. + (define ref (upstream-source-urls source)) ; a + (values (upstream-source-version source) + (latest-repository-commit + store + (git-reference-url ref) + #:ref `(tag-or-commit . ,(git-reference-commit ref)) + #:recursive? (git-reference-recursive? ref)) + source)) + (define %method-updates ;; Mapping of origin methods to source update procedures. - `((,url-fetch . ,package-update/url-fetch))) + `((,url-fetch . ,package-update/url-fetch) + (,git-fetch . ,package-update/git-fetch))) (define* (package-update store package #:optional (updaters (force %updaters)) @@ -492,9 +530,22 @@ new version string if an update was made, and #f otherwise." (origin-hash (package-source package)))) (old-url (match (origin-uri (package-source package)) ((? string? url) url) + ((? git-reference? ref) + (git-reference-url ref)) (_ #f))) (new-url (match (upstream-source-urls source) - ((first _ ...) first))) + ((first _ ...) first) + ((? git-reference? ref) + (git-reference-url ref)) + (_ #f))) + (old-commit (match (origin-uri (package-source package)) + ((? git-reference? ref) + (git-reference-commit ref)) + (_ #f))) + (new-commit (match (upstream-source-urls source) + ((? git-reference? ref) + (git-reference-commit ref)) + (_ #f))) (file (and=> (location-file loc) (cut search-path %load-path <>)))) (if file @@ -508,6 +559,9 @@ new version string if an update was made, and #f otherwise." 'filename file)) (replacements `((,old-version . ,version) (,old-hash . ,hash) + ,@(if (and old-commit new-commit) + `((,old-commit . ,new-commit)) + '()) ,@(if (and old-url new-url) `((,(dirname old-url) . ,(dirname new-url))) diff --git a/tests/minetest.scm b/tests/minetest.scm index 77b9aa928f..cbb9e83889 100644 --- a/tests/minetest.scm +++ b/tests/minetest.scm @@ -387,10 +387,9 @@ during a dynamic extent where that package is available on ContentDB." ;; Update detection (define (upstream-source->sexp upstream-source) - (define urls (upstream-source-urls upstream-source)) - (unless (= 1 (length urls)) - (error "only a single URL is expected")) - (define url (first urls)) + (define url (upstream-source-urls upstream-source)) + (unless (git-reference? url) + (error "a is expected")) `(,(upstream-source-package upstream-source) ,(upstream-source-version upstream-source) ,(git-reference-url url)