From patchwork Wed Jan 5 14:07:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35985 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0A3EB27BBEA; Wed, 5 Jan 2022 15:14:23 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5FAD927BBE9 for ; Wed, 5 Jan 2022 15:14:22 +0000 (GMT) Received: from localhost ([::1]:47588 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n57zZ-0004g5-J6 for patchwork@mira.cbaines.net; Wed, 05 Jan 2022 10:14:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54486) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n56xT-0007JN-MH for guix-patches@gnu.org; Wed, 05 Jan 2022 09:08:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56471) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n56xO-0006be-GN for guix-patches@gnu.org; Wed, 05 Jan 2022 09:08:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n56xO-0004WZ-Cb for guix-patches@gnu.org; Wed, 05 Jan 2022 09:08:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v5 1/4] guix hash: Extract file hashing procedures. References: In-Reply-To: Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 05 Jan 2022 14:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Sarah Morgensen , Maxime Devos Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164139167917356 (code B ref 50072); Wed, 05 Jan 2022 14:08:02 +0000 Received: (at 50072) by debbugs.gnu.org; 5 Jan 2022 14:07:59 +0000 Received: from localhost ([127.0.0.1]:39777 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xH-0004Vj-Hy for submit@debbugs.gnu.org; Wed, 05 Jan 2022 09:07:59 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:58248) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xE-0004UX-Vt for 50072@debbugs.gnu.org; Wed, 05 Jan 2022 09:07:54 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id f27s2600F4UW6Th0627sN9; Wed, 05 Jan 2022 15:07:52 +0100 From: Maxime Devos Date: Wed, 5 Jan 2022 14:07:47 +0000 Message-Id: <20220105140750.18214-1-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641391672; bh=NbrokZoVPvuKyRtEizbkjq1LE4lpGqaE5iJPt+N60jY=; h=From:To:Cc:Subject:Date; b=I8uIGPGF8KvuthIZQpGDtlV3tveyZEoPVpVo7uEAP8BFk1/GyvyNc7aRbMsMn+vLK Pt2iI/fET/TEq5IVbRRxMWogSaniKqeJO6D1tBVbmQXHmcBP0HlBrRLp6/Quua9sr4 sXhe1Bvu0kUFsvHZFIoqySZEFFaBqXd4gF2g+Vg3CcJoJ1SO7spyG6y+i/oE4vJh1R s4GDq2viNx3886+hp+PFv0tqC6PnR37nNLtpPrsRAULt8OILIC6l9DufLjFJe9bM5L YSoz6tcNqlL3y3keMOJByQv1k3CYqCiP3XDDGk9F87a1bP7rJDQJ1tDHhocckqokNk f7RfdXdwkveNw== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen * guix/scripts/hash.scm (guix-hash)[vcs-file?] (nar-hash, default-hash): Extract hashing logic to... * guix/hash.scm (vcs-file?, file-hash*): ... these new procedures in this new file. Modified-by: Maxime Devos --- Makefile.am | 1 + guix/hash.scm | 73 +++++++++++++++++++++++++++++++++++++++++++ guix/scripts/hash.scm | 22 +++---------- 3 files changed, 78 insertions(+), 18 deletions(-) create mode 100644 guix/hash.scm base-commit: 9708681f1a9f221ae6cad64625ba8309b6742653 diff --git a/Makefile.am b/Makefile.am index 8c5682a1c6..bc3d0087d0 100644 --- a/Makefile.am +++ b/Makefile.am @@ -99,6 +99,7 @@ MODULES = \ guix/extracting-download.scm \ guix/git-download.scm \ guix/hg-download.scm \ + guix/hash.scm \ guix/swh.scm \ guix/monads.scm \ guix/monad-repl.scm \ diff --git a/guix/hash.scm b/guix/hash.scm new file mode 100644 index 0000000000..3cb68e5c44 --- /dev/null +++ b/guix/hash.scm @@ -0,0 +1,73 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2022 Maxime Devos +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix hash) + #:use-module (gcrypt hash) + #:use-module (guix serialization) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:export (vcs-file? + file-hash*)) + +(define (vcs-file? file stat) + "Returns true if FILE is a version control system file." + (case (stat:type stat) + ((directory) + (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) + ((regular) + ;; Git sub-modules have a '.git' file that is a regular text file. + (string=? (basename file) ".git")) + (else + #f))) + +(define* (file-hash* file #:key + (algorithm (hash-algorithm sha256)) + (recursive? 'auto) + (select? (negate vcs-file?))) + "Compute the hash of FILE with ALGORITHM. + +Symbolic links are only dereferenced if RECURSIVE? is false. +Directories are only supported if RECURSIVE? is #true or 'auto'. +The executable bit is only recorded if RECURSIVE? is #true. +If FILE is a symbolic link, it is only followed if RECURSIVE? is false. + +For regular files, there are two different hashes when the executable +hash isn't recorded: the regular hash and the nar hash. In most situations, +the regular hash is desired and setting RECURSIVE? to 'auto' does the right +thing for both regular files and directories. + +This procedure must only be used under controlled circumstances; +the detection of symbolic links in FILE is racy. + +When FILE is a directory, the procedure SELECT? called as (SELECT? FILE STAT) +decides which files to include. By default, version control files are +excluded. To include everything, SELECT? can be set to (const #true)." + (if (or (eq? recursive? #true) + (and (eq? recursive? 'auto) + ;; Don't change this to (eq? 'directory ...), because otherwise + ;; if 'file' denotes a symbolic link, the 'file-hash' below + ;; would dereference it -- dereferencing symbolic links would + ;; open an avoidable can of potential worms. + (not (eq? 'regular (stat:type (lstat file)))))) + (let-values (((port get-hash) + (open-hash-port algorithm))) + (write-file file port #:select? select?) + (force-output port) + (get-hash)) + (file-hash algorithm file))) diff --git a/guix/scripts/hash.scm b/guix/scripts/hash.scm index d73e3d13dd..28d587b944 100644 --- a/guix/scripts/hash.scm +++ b/guix/scripts/hash.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2016 Jan Nieuwenhuizen ;;; Copyright © 2018 Tim Gesthuizen ;;; Copyright © 2021 Simon Tournier +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,6 +25,7 @@ #:use-module (gcrypt hash) #:use-module (guix serialization) #:use-module (guix ui) + #:use-module (guix hash) #:use-module (guix scripts) #:use-module (guix base16) #:use-module (guix base32) @@ -46,20 +48,14 @@ (define* (nar-hash file #:optional (algorithm (assoc-ref %default-options 'hash-algorithm)) select?) - (let-values (((port get-hash) - (open-hash-port algorithm))) - (write-file file port #:select? select?) - (force-output port) - (get-hash))) + (file-hash* file #:algorithm algorithm #:select? select? #:recursive? #true)) (define* (default-hash file #:optional (algorithm (assoc-ref %default-options 'hash-algorithm)) select?) (match file ("-" (port-hash algorithm (current-input-port))) - (_ - (call-with-input-file file - (cute port-hash algorithm <>))))) + (_ (file-hash* file #:algorithm algorithm #:recursive? #false)))) (define* (git-hash file #:optional (algorithm (assoc-ref %default-options 'hash-algorithm)) @@ -181,16 +177,6 @@ use '--serializer' instead~%")) (parse-command-line args %options (list %default-options) #:build-options? #f)) - (define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - (let* ((opts (parse-options)) (args (filter-map (match-lambda (('argument . value) From patchwork Wed Jan 5 14:07:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35983 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6356A27BBEA; Wed, 5 Jan 2022 14:33:14 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 8B1E227BBE9 for ; Wed, 5 Jan 2022 14:33:13 +0000 (GMT) Received: from localhost ([::1]:44870 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n57Lk-00027M-Lm for patchwork@mira.cbaines.net; Wed, 05 Jan 2022 09:33:12 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54680) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n56yM-0000Mp-Qj for guix-patches@gnu.org; Wed, 05 Jan 2022 09:09:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56478) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n56yM-0006hL-Hu for guix-patches@gnu.org; Wed, 05 Jan 2022 09:09:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n56yM-0004Yk-EL for guix-patches@gnu.org; Wed, 05 Jan 2022 09:09:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v5 2/4] import: Factorize file hashing. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 05 Jan 2022 14:09:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Sarah Morgensen Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164139168817436 (code B ref 50072); Wed, 05 Jan 2022 14:09:02 +0000 Received: (at 50072) by debbugs.gnu.org; 5 Jan 2022 14:08:08 +0000 Received: from localhost ([127.0.0.1]:39788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xQ-0004X1-Eb for submit@debbugs.gnu.org; Wed, 05 Jan 2022 09:08:08 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:59578) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xF-0004VQ-FM for 50072@debbugs.gnu.org; Wed, 05 Jan 2022 09:08:00 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id f27s2600F4UW6Th0627sND; Wed, 05 Jan 2022 15:07:52 +0100 From: Maxime Devos Date: Wed, 5 Jan 2022 14:07:48 +0000 Message-Id: <20220105140750.18214-2-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220105140750.18214-1-maximedevos@telenet.be> References: <20220105140750.18214-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641391672; bh=pToVcY6tyNJdA8to9CMlcbffGWezAHQB9+uOz7gou4c=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=X4cUK+OA6rL9+/XyeJwMgaiTL8CkW4TWCbkR3c4CVt6LBtSoegx7gZBuK/FLnjoPn CzSSnx5wX6NaReSuhjubESFeQWDRLEukwLLoMf12FHgCMOp+esmuU5gXW+MGfZr6pR YavuolMXcAMs/hpA/HZbGdvJsWp15cNMSUIrV9NM00fsn0JzAKJcwioBWxXGy4s/oO nQ/KqFmOtSItOwONraenqYi1Wepfkqh7s2oJ3r5UGM1A5W4ujlWsMWZufqGmMN+/gl hZGbSXszcdVkALk2usYmtt+z80fbirrft9s42wZ/B1YcpiuAnaDr7YJRtI8O2GZrJT XHw0ZEq/3sbMg== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen * guix/import/cran.scm (vcs-file?, file-hash): Remove procedures. (description->package): Use 'file-hash*' instead. * guix/import/elpa.scm (vcs-file?, file-hash): Remove procedures. (git-repository->origin, elpa-package->sexp): Use 'file-hash* instead'. * guix/import/go.scm (vcs-file?, file-hash): Remove procedures. (git-checkout-hash): Use 'file-hash*' instead. * guix/import/minetest.scm (file-hash): Remove procedure. (make-minetest-sexp): Use 'file-hash*' instead. --- guix/import/cran.scm | 32 +++----------------------------- guix/import/elpa.scm | 29 +++++------------------------ guix/import/go.scm | 25 +++---------------------- guix/import/minetest.scm | 19 ++++++++----------- 4 files changed, 19 insertions(+), 86 deletions(-) diff --git a/guix/import/cran.scm b/guix/import/cran.scm index 1389576cad..b61402078d 100644 --- a/guix/import/cran.scm +++ b/guix/import/cran.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2015, 2016, 2017, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2020 Martin Becze +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -35,10 +36,9 @@ #:use-module (guix memoization) #:use-module (guix http-client) #:use-module (guix diagnostics) + #:use-module (guix hash) #:use-module (guix i18n) - #:use-module (gcrypt hash) #:use-module (guix store) - #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix base32) #:use-module ((guix download) #:select (download-to-store)) #:use-module (guix import utils) @@ -196,17 +196,6 @@ bioconductor package NAME, or #F if the package is unknown." (bioconductor-packages-list type)) (cut assoc-ref <> "Version"))) -;; XXX taken from (guix scripts hash) -(define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - ;; Little helper to download URLs only once. (define download (memoize @@ -464,16 +453,6 @@ reference the pkg-config tool." (define (needs-knitr? meta) (member "knitr" (listify meta "VignetteBuilder"))) -;; XXX adapted from (guix scripts hash) -(define (file-hash file select? recursive?) - ;; Compute the hash of FILE. - (if recursive? - (let-values (((port get-hash) (open-sha256-port))) - (write-file file port #:select? select?) - (force-output port) - (get-hash)) - (call-with-input-file file port-sha256))) - (define (description->package repository meta) "Return the `package' s-expression for an R package published on REPOSITORY from the alist META, which was derived from the R package's DESCRIPTION file." @@ -571,12 +550,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (sha256 (base32 ,(bytevector->nix-base32-string - (case repository - ((git) - (file-hash source (negate vcs-file?) #t)) - ((hg) - (file-hash source (negate vcs-file?) #t)) - (else (file-sha256 source)))))))) + (file-hash* source #:recursive? (or git? hg?))))))) ,@(if (not (and git? hg? (equal? (string-append "r-" name) (cran-guix-name name)))) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm index edabb88b7a..c5167eacb5 100644 --- a/guix/import/elpa.scm +++ b/guix/import/elpa.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2020 Martin Becze ;;; Copyright © 2020 Ricardo Wurmus ;;; Copyright © 2021 Xinglu Chen +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -37,10 +38,10 @@ #:use-module (guix import utils) #:use-module (guix http-client) #:use-module (guix git) + #:use-module (guix hash) #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix store) #:use-module (guix ui) - #:use-module (gcrypt hash) #:use-module (guix base32) #:use-module (guix upstream) #:use-module (guix packages) @@ -229,27 +230,6 @@ keywords to values." (close-port port) (data->recipe (cons ':name data)))) -;; XXX adapted from (guix scripts hash) -(define (file-hash file select? recursive?) - ;; Compute the hash of FILE. - (if recursive? - (let-values (((port get-hash) (open-sha256-port))) - (write-file file port #:select? select?) - (force-output port) - (get-hash)) - (call-with-input-file file port-sha256))) - -;; XXX taken from (guix scripts hash) -(define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - (define (git-repository->origin recipe url) "Fetch origin details from the Git repository at URL for the provided MELPA RECIPE." @@ -271,7 +251,7 @@ RECIPE." (sha256 (base32 ,(bytevector->nix-base32-string - (file-hash directory (negate vcs-file?) #t))))))) + (file-hash* directory #:recursive? #true))))))) (define* (melpa-recipe->origin recipe) "Fetch origin details from the MELPA recipe and associated repository for @@ -380,7 +360,8 @@ type ''." (sha256 (base32 ,(if tarball - (bytevector->nix-base32-string (file-sha256 tarball)) + (bytevector->nix-base32-string + (file-hash* tarball #:recursive? #false)) "failed to download package"))))))) (build-system emacs-build-system) ,@(maybe-inputs 'propagated-inputs dependencies) diff --git a/guix/import/go.scm b/guix/import/go.scm index 26dbc34b63..c7673e6a1a 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -26,6 +26,7 @@ (define-module (guix import go) #:use-module (guix build-system go) #:use-module (guix git) + #:use-module (guix hash) #:use-module (guix i18n) #:use-module (guix diagnostics) #:use-module (guix import utils) @@ -36,11 +37,10 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix memoization) #:autoload (htmlprag) (html->sxml) ;from Guile-Lib - #:autoload (guix git) (update-cached-checkout) - #:autoload (gcrypt hash) (open-hash-port hash-algorithm sha256) #:autoload (guix serialization) (write-file) #:autoload (guix base32) (bytevector->nix-base32-string) #:autoload (guix build utils) (mkdir-p) + #:autoload (gcrypt hash) (hash-algorithm sha256) #:use-module (ice-9 match) #:use-module (ice-9 peg) #:use-module (ice-9 rdelim) @@ -499,25 +499,6 @@ source." goproxy (module-meta-repo-root meta-data))) -;; XXX: Copied from (guix scripts hash). -(define (vcs-file? file stat) - (case (stat:type stat) - ((directory) - (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS"))) - ((regular) - ;; Git sub-modules have a '.git' file that is a regular text file. - (string=? (basename file) ".git")) - (else - #f))) - -;; XXX: Adapted from 'file-hash' in (guix scripts hash). -(define* (file-hash file #:optional (algorithm (hash-algorithm sha256))) - ;; Compute the hash of FILE. - (let-values (((port get-hash) (open-hash-port algorithm))) - (write-file file port #:select? (negate vcs-file?)) - (force-output port) - (get-hash))) - (define* (git-checkout-hash url reference algorithm) "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or tag." @@ -536,7 +517,7 @@ tag." (update-cached-checkout url #:ref `(tag-or-commit . ,reference))))) - (file-hash checkout algorithm))) + (file-hash* checkout #:algorithm algorithm #:recursive? #true))) (define (vcs->origin vcs-type vcs-repo-url version) "Generate the `origin' block of a package depending on what type of source diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm index abddd885ee..a7bdbfebca 100644 --- a/guix/import/minetest.scm +++ b/guix/import/minetest.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2021 Maxime Devos +;;; Copyright © 2021, 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -39,6 +39,7 @@ #:use-module (guix base32) #:use-module (guix git) #:use-module ((guix git-download) #:prefix download:) + #:use-module (guix hash) #:use-module (guix store) #:export (%default-sort-key %contentdb-api @@ -286,14 +287,6 @@ results. The return value is a list of records." (with-store store (latest-repository-commit store url #:ref ref))) -;; XXX adapted from (guix scripts hash) -(define (file-hash file) - "Compute the hash of FILE." - (let-values (((port get-hash) (open-sha256-port))) - (write-file file port) - (force-output port) - (get-hash))) - (define (make-minetest-sexp author/name version repository commit inputs home-page synopsis description media-license license) @@ -314,9 +307,13 @@ MEDIA-LICENSE and LICENSE." ;; The git commit is not always available. ,(and commit (bytevector->nix-base32-string - (file-hash + (file-hash* (download-git-repository repository - `(commit . ,commit))))))) + `(commit . ,commit)) + ;; 'download-git-repository' already filtered out the '.git' + ;; directory. + #:select? (const #true) + #:recursive? #true))))) (file-name (git-file-name name version)))) (build-system minetest-mod-build-system) ,@(maybe-propagated-inputs (map contentdb->package-name inputs)) From patchwork Wed Jan 5 14:07:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35984 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 966DA27BBEA; Wed, 5 Jan 2022 15:14:10 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5459627BBE9 for ; Wed, 5 Jan 2022 15:14:10 +0000 (GMT) Received: from localhost ([::1]:47346 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n57zN-0004Vc-4t for patchwork@mira.cbaines.net; Wed, 05 Jan 2022 10:14:09 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54482) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n56xT-0007JH-LE for guix-patches@gnu.org; Wed, 05 Jan 2022 09:08:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56472) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n56xO-0006bf-Rs for guix-patches@gnu.org; Wed, 05 Jan 2022 09:08:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n56xO-0004Wg-ON for guix-patches@gnu.org; Wed, 05 Jan 2022 09:08:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v5 3/4] refresh: Support non-tarball sources. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 05 Jan 2022 14:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Sarah Morgensen Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164139167917364 (code B ref 50072); Wed, 05 Jan 2022 14:08:02 +0000 Received: (at 50072) by debbugs.gnu.org; 5 Jan 2022 14:07:59 +0000 Received: from localhost ([127.0.0.1]:39779 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xL-0004Vt-GA for submit@debbugs.gnu.org; Wed, 05 Jan 2022 09:07:59 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:59590) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xF-0004VS-JH for 50072@debbugs.gnu.org; Wed, 05 Jan 2022 09:07:54 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id f27s2600F4UW6Th0627sNF; Wed, 05 Jan 2022 15:07:53 +0100 From: Maxime Devos Date: Wed, 5 Jan 2022 14:07:49 +0000 Message-Id: <20220105140750.18214-3-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220105140750.18214-1-maximedevos@telenet.be> References: <20220105140750.18214-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641391673; bh=1C9ahprOwcy10Tnk5guGYz+QxZ/hL2vALCXNKEI79wE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=mrNU9OuZ6bLKjlioVstON2ONFyvngww/oILFAg9qjXdqMpcgq6NseQ0QToYaV/uKj bcAuVl0RtC279jvU/+8aHKWObepkx1hm4r6BN5gKCtDna8SVeixIfeaMk0TMd1tXHf 5ba7ISKRdityFjNsRzT4rKVOCCAPOaq6kRM96CXJ5P/L4yWbsCfirxeer2HPlUMkXR J/et30vHGZJVkdylY1wvxAP1v8B7kMgoVypad69TpwpRNZ31S4nKBuKoDUzG6cR+ci h4a/wFqjKHNKthQNGQy/FQ4OFpj7gJY9hc69FhtlwGzk4CoD0b6VnNflljhLxXQ2Uf 0DN55QE0SI/Bg== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen * guix/scripts/refresh.scm (update-package): Use 'file-hash*' instead of 'port-sha256'. Rename TARBALL to OUTPUT. --- guix/scripts/refresh.scm | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm index 8806f0f740..68bb9040d8 100644 --- a/guix/scripts/refresh.scm +++ b/guix/scripts/refresh.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2018 Efraim Flashner ;;; Copyright © 2019 Ricardo Wurmus ;;; Copyright © 2020 Simon Tournier +;;; Copyright © 2021 Sarah Morgensen ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,7 +27,6 @@ (define-module (guix scripts refresh) #:use-module (guix ui) - #:use-module (gcrypt hash) #:use-module (guix scripts) #:use-module ((guix scripts build) #:select (%standard-build-options)) #:use-module (guix store) @@ -38,6 +38,7 @@ #:use-module (guix scripts graph) #:use-module (guix monads) #:use-module (guix gnupg) + #:use-module (guix hash) #:use-module (gnu packages) #:use-module ((gnu packages commencement) #:select (%final-inputs)) #:use-module (ice-9 match) @@ -314,14 +315,14 @@ KEY-DOWNLOAD specifies a download policy for missing OpenPGP keys; allowed values: 'interactive' (default), 'always', and 'never'. When WARN? is true, warn about packages that have no matching updater." (if (lookup-updater package updaters) - (let-values (((version tarball source) + (let-values (((version output source) (package-update store package updaters #:key-download key-download)) ((loc) (or (package-field-location package 'version) (package-location package)))) (when version - (if (and=> tarball file-exists?) + (if (and=> output file-exists?) (begin (info loc (G_ "~a: updating from version ~a to version ~a...~%") @@ -363,8 +364,7 @@ warn about packages that have no matching updater." (info loc (G_ "~a: consider removing this propagated input: ~a~%") name change-name)))) (upstream-source-input-changes source)) - (let ((hash (call-with-input-file tarball - port-sha256))) + (let ((hash (file-hash* output))) (update-package-source package source hash))) (warning (G_ "~a: version ~a could not be \ downloaded and authenticated; not updating~%") From patchwork Wed Jan 5 14:07:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 35982 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 02AD027BBEA; Wed, 5 Jan 2022 14:33:00 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 87E9C27BBE9 for ; Wed, 5 Jan 2022 14:32:58 +0000 (GMT) Received: from localhost ([::1]:43932 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n57LT-0001S1-Qq for patchwork@mira.cbaines.net; Wed, 05 Jan 2022 09:32:57 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54670) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n56yM-0000Ly-GJ for guix-patches@gnu.org; Wed, 05 Jan 2022 09:09:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56477) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n56yM-0006h6-5J for guix-patches@gnu.org; Wed, 05 Jan 2022 09:09:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n56yM-0004Yd-1X for guix-patches@gnu.org; Wed, 05 Jan 2022 09:09:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH v5 4/4] upstream: Support updating and fetching 'git-fetch' origins. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 05 Jan 2022 14:09:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50072@debbugs.gnu.org Cc: Sarah Morgensen , Maxime Devos Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.164139168417426 (code B ref 50072); Wed, 05 Jan 2022 14:09:01 +0000 Received: (at 50072) by debbugs.gnu.org; 5 Jan 2022 14:08:04 +0000 Received: from localhost ([127.0.0.1]:39781 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xL-0004W1-Ro for submit@debbugs.gnu.org; Wed, 05 Jan 2022 09:08:04 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:59606) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n56xF-0004VT-OP for 50072@debbugs.gnu.org; Wed, 05 Jan 2022 09:07:59 -0500 Received: from localhost.localdomain ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by michel.telenet-ops.be with bizsmtp id f27s2600F4UW6Th0627tNK; Wed, 05 Jan 2022 15:07:53 +0100 From: Maxime Devos Date: Wed, 5 Jan 2022 14:07:50 +0000 Message-Id: <20220105140750.18214-4-maximedevos@telenet.be> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220105140750.18214-1-maximedevos@telenet.be> References: <20220105140750.18214-1-maximedevos@telenet.be> MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1641391673; bh=OcXbP9H75mnRWI/rumIqeKoSasXir76HNnl6Eq+S86Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=S6BqnNYB7/40K5ktPL/ePIhMeXkNrDQz+7fVQ9OMch5SRyiD+Clb3z1XQuAqv0Mcj U4Zv7JL64hOzebW4sxHDMyf9y7hiH6SPkBFRoBIgVnACWPn0cg6RscFGxJsml5WXke 185+OutBmcCDjmA4lH97S88LCIrVBEE9VUlcd8zd5RFVAMo2EYd/041AewjZ9wGwJc vvse2C9qbgQH3+1OigFDSwT6GHl4qxcDNPkj1f4NiCIfMB8iWWpITBoNUtZjIVttyQ vsdWXf/1psUjv6k/7VMG8POi1rWpbMrC+iXeaTUuBFGN0sejZnbfTmpkSquVF39AWM oFTtPX47S4I1w== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From: Sarah Morgensen Updaters need to be modified to return 'git-reference' objects. This patch modifies the 'generic-git' and 'minetest' updater, but others might need to be modified as well. * guix/git.scm (git-reference->git-checkout): New procedure. * guix/upstream.scm (package-update/git-fetch): New procedure. ()[urls]: Document it can be a 'git-reference'. (%method-updates): Add 'git-fetch' mapping. (update-package-source): Support 'git-reference' sources. (upstream-source-compiler/url-fetch): Split off from ... (upstream-source-compiler): ... this, and call ... (upstream-source-compiler/git-fetch): ... this new procedure if the URL field contains a 'git-reference'. * guix/import/git.scm (latest-git-tag-version): Always return two values and document that the tag is returned as well. (latest-git-release)[urls]: Use the 'git-reference' instead of the repository URL. * guix/import/minetest.scm (latest-minetest-release)[urls]: Don't wrap the 'git-reference' in a list. * tests/minetest.scm (upstream-source->sexp): Adjust to new convention. Co-authored-by: Maxime Devos --- guix/git.scm | 14 +++++++- guix/import/git.scm | 22 +++++++----- guix/import/minetest.scm | 6 ++-- guix/upstream.scm | 73 ++++++++++++++++++++++++++++++++++++---- tests/minetest.scm | 7 ++-- 5 files changed, 98 insertions(+), 24 deletions(-) diff --git a/guix/git.scm b/guix/git.scm index dc2ca1be84..43e85a5026 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2021 Kyle Meyer ;;; Copyright © 2021 Marius Bakke +;;; Copyright © 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -33,6 +34,8 @@ #:use-module (guix utils) #:use-module (guix records) #:use-module (guix gexp) + #:autoload (guix git-download) + (git-reference-url git-reference-commit git-reference-recursive?) #:use-module (guix sets) #:use-module ((guix diagnostics) #:select (leave warning)) #:use-module (guix progress) @@ -65,7 +68,9 @@ git-checkout-url git-checkout-branch git-checkout-commit - git-checkout-recursive?)) + git-checkout-recursive? + + git-reference->git-checkout)) (define %repository-cache-directory (make-parameter (string-append (cache-directory #:ensure? #f) @@ -672,6 +677,13 @@ is true, limit to only refs/tags." (commit git-checkout-commit (default #f)) ;#f | tag | commit (recursive? git-checkout-recursive? (default #f))) +(define (git-reference->git-checkout reference) + "Convert the REFERENCE to an equivalent ." + (git-checkout + (url (git-reference-url reference)) + (commit (git-reference-commit reference)) + (recursive? (git-reference-recursive? reference)))) + (define* (latest-repository-commit* url #:key ref recursive? log-port) ;; Monadic variant of 'latest-repository-commit'. (lambda (store) diff --git a/guix/import/git.scm b/guix/import/git.scm index 1eb219f3fe..4cf404677c 100644 --- a/guix/import/git.scm +++ b/guix/import/git.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2021 Xinglu Chen ;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -34,6 +35,7 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) #:export (%generic-git-updater ;; For tests. @@ -172,21 +174,21 @@ repository at URL." (values version tag))))))) (define (latest-git-tag-version package) - "Given a PACKAGE, return the latest version of it, or #f if the latest version -could not be determined." + "Given a PACKAGE, return the latest version of it and the corresponding git +tag, or #false and #false if the latest version could not be determined." (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) (warning (or (package-field-location package 'source) (package-location package)) (G_ "~a for ~a~%") (condition-message c) (package-name package)) - #f) + (values #f #f)) ((eq? (exception-kind c) 'git-error) (warning (or (package-field-location package 'source) (package-location package)) (G_ "failed to fetch Git repository for ~a~%") (package-name package)) - #f)) + (values #f #f))) (let* ((source (package-source package)) (url (git-reference-url (origin-uri source))) (property (cute assq-ref (package-properties package) <>))) @@ -208,14 +210,16 @@ could not be determined." "Return an for the latest release of PACKAGE." (let* ((name (package-name package)) (old-version (package-version package)) - (url (git-reference-url (origin-uri (package-source package)))) - (new-version (latest-git-tag-version package))) - - (and new-version + (old-reference (origin-uri (package-source package))) + (new-version new-version-tag (latest-git-tag-version package))) + (and new-version new-version-tag (upstream-source (package name) (version new-version) - (urls (list url)))))) + (urls (git-reference + (url (git-reference-url old-reference)) + (commit new-version-tag) + (recursive? (git-reference-recursive? old-reference)))))))) (define %generic-git-updater (upstream-updater diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm index a7bdbfebca..3b2cdcdcac 100644 --- a/guix/import/minetest.scm +++ b/guix/import/minetest.scm @@ -504,9 +504,9 @@ or #false if the latest release couldn't be determined." (upstream-source (package (package:package-name pkg)) (version (release-version release)) - (urls (list (download:git-reference - (url (package-repository contentdb-package)) - (commit (release-commit release)))))))) + (urls (download:git-reference + (url (package-repository contentdb-package)) + (commit (release-commit release))))))) (define %minetest-updater (upstream-updater diff --git a/guix/upstream.scm b/guix/upstream.scm index 632e9ebc4f..1fe996ef3d 100644 --- a/guix/upstream.scm +++ b/guix/upstream.scm @@ -2,6 +2,8 @@ ;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2015 Alex Kost ;;; Copyright © 2019 Ricardo Wurmus +;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2021, 2022 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,12 +26,15 @@ #:use-module (guix discovery) #:use-module ((guix download) #:select (download-to-store url-fetch)) + #:use-module (guix git-download) #:use-module (guix gnupg) #:use-module (guix packages) #:use-module (guix diagnostics) #:use-module (guix ui) #:use-module (guix base32) #:use-module (guix gexp) + #:autoload (guix git) (latest-repository-commit git-reference->git-checkout) + #:use-module (guix hash) #:use-module (guix store) #:use-module ((guix derivations) #:select (built-derivations derivation->output-path)) #:autoload (gcrypt hash) (port-sha256) @@ -93,7 +98,7 @@ upstream-source? (package upstream-source-package) ;string (version upstream-source-version) ;string - (urls upstream-source-urls) ;list of strings + (urls upstream-source-urls) ;list of strings|git-reference (signature-urls upstream-source-signature-urls ;#f | list of strings (default #f)) (input-changes upstream-source-input-changes @@ -357,10 +362,9 @@ values: 'interactive' (default), 'always', and 'never'." data url) #f))))))) -(define-gexp-compiler (upstream-source-compiler (source ) - system target) - "Download SOURCE from its first URL and lower it as a fixed-output -derivation that would fetch it." +(define (upstream-source-compiler/url-fetch source system) + "Lower SOURCE, an pointing to a tarball, as a +fixed-output derivation that would fetch it, and verify its authenticity." (mlet* %store-monad ((url -> (first (upstream-source-urls source))) (signature -> (and=> (upstream-source-signature-urls source) @@ -378,6 +382,30 @@ derivation that would fetch it." (url-fetch url 'sha256 hash (store-path-package-name tarball) #:system system)))) +(define (upstream-source-compiler/git-fetch source system) + "Lower SOURCE, an using git, as a fixed-output +derivation that would fetch it." + (mlet* %store-monad ((reference -> (upstream-source-urls source)) + (checkout + (lower-object + (git-reference->git-checkout reference) + system))) + ;; Like in 'upstream-source-compiler/url-fetch', return a fixed-output + ;; derivation instead of CHECKOUT. + (git-fetch reference 'sha256 + (file-hash* checkout #:recursive? #true #:select? (const #true)) + (git-file-name (upstream-source-package source) + (upstream-source-version source)) + #:system system))) + +(define-gexp-compiler (upstream-source-compiler (source ) + system target) + "Download SOURCE, lower it as a fixed-output derivation that would fetch it, +and verify its authenticity if possible." + (if (git-reference? (upstream-source-urls source)) + (upstream-source-compiler/git-fetch source system) + (upstream-source-compiler/url-fetch source system))) + (define (find2 pred lst1 lst2) "Like 'find', but operate on items from both LST1 and LST2. Return two values: the item from LST1 and the item from LST2 that match PRED." @@ -430,9 +458,24 @@ SOURCE, an ." #:key-download key-download))) (values version tarball source)))))) +(define* (package-update/git-fetch store package source #:key key-download) + "Return the version, checkout, and SOURCE, to update PACKAGE to +SOURCE, an ." + ;; TODO: it would be nice to authenticate commits, e.g. with + ;; "guix git authenticate" or a list of permitted signing keys. + (define ref (upstream-source-urls source)) ; a + (values (upstream-source-version source) + (latest-repository-commit + store + (git-reference-url ref) + #:ref `(tag-or-commit . ,(git-reference-commit ref)) + #:recursive? (git-reference-recursive? ref)) + source)) + (define %method-updates ;; Mapping of origin methods to source update procedures. - `((,url-fetch . ,package-update/url-fetch))) + `((,url-fetch . ,package-update/url-fetch) + (,git-fetch . ,package-update/git-fetch))) (define* (package-update store package #:optional (updaters (force %updaters)) @@ -492,9 +535,22 @@ new version string if an update was made, and #f otherwise." (origin-hash (package-source package)))) (old-url (match (origin-uri (package-source package)) ((? string? url) url) + ((? git-reference? ref) + (git-reference-url ref)) (_ #f))) (new-url (match (upstream-source-urls source) - ((first _ ...) first))) + ((first _ ...) first) + ((? git-reference? ref) + (git-reference-url ref)) + (_ #f))) + (old-commit (match (origin-uri (package-source package)) + ((? git-reference? ref) + (git-reference-commit ref)) + (_ #f))) + (new-commit (match (upstream-source-urls source) + ((? git-reference? ref) + (git-reference-commit ref)) + (_ #f))) (file (and=> (location-file loc) (cut search-path %load-path <>)))) (if file @@ -508,6 +564,9 @@ new version string if an update was made, and #f otherwise." 'filename file)) (replacements `((,old-version . ,version) (,old-hash . ,hash) + ,@(if (and old-commit new-commit) + `((,old-commit . ,new-commit)) + '()) ,@(if (and old-url new-url) `((,(dirname old-url) . ,(dirname new-url))) diff --git a/tests/minetest.scm b/tests/minetest.scm index 77b9aa928f..cbb9e83889 100644 --- a/tests/minetest.scm +++ b/tests/minetest.scm @@ -387,10 +387,9 @@ during a dynamic extent where that package is available on ContentDB." ;; Update detection (define (upstream-source->sexp upstream-source) - (define urls (upstream-source-urls upstream-source)) - (unless (= 1 (length urls)) - (error "only a single URL is expected")) - (define url (first urls)) + (define url (upstream-source-urls upstream-source)) + (unless (git-reference? url) + (error "a is expected")) `(,(upstream-source-package upstream-source) ,(upstream-source-version upstream-source) ,(git-reference-url url)