From patchwork Tue Dec 15 09:57:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25794 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4401C27BC05; Tue, 15 Dec 2020 09:59:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id D198327BC04 for ; Tue, 15 Dec 2020 09:59:19 +0000 (GMT) Received: from localhost ([::1]:53932 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp771-0000Tc-1m for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:59:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35878) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75o-0007ie-AB for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44131) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75m-0004Xj-LA for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75m-0007BJ-FJ for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 1/6] tests: Check the build trace for hash mismatches on substitutes. References: <20201215093830.10322-1-ludo@gnu.org> In-Reply-To: <20201215093830.10322-1-ludo@gnu.org> Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802626427528 (code B ref 45253); Tue, 15 Dec 2020 09:58:02 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:44 +0000 Received: from localhost ([127.0.0.1]:55664 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75U-00079p-8L for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:44 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42652) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75S-00079U-V4 for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:43 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48635) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75N-0004Q3-00; Tue, 15 Dec 2020 04:57:37 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75M-0001j8-FA; Tue, 15 Dec 2020 04:57:36 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:25 +0100 Message-Id: <20201215095730.10954-1-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * tests/store.scm ("substitute, corrupt output hash, build trace"): New test. --- tests/store.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/tests/store.scm b/tests/store.scm index 38051bf5e5..7f1ec51875 100644 --- a/tests/store.scm +++ b/tests/store.scm @@ -787,6 +787,61 @@ (build-derivations s (list d)) #f)))))) +(test-assert "substitute, corrupt output hash, build trace" + ;; Likewise, and check the build trace. + (with-store s + (let* ((c "hello, world") ; contents of the output + (d (build-expression->derivation + s "corrupt-substitute" + `(mkdir %output) + #:guile-for-build + (package-derivation s %bootstrap-guile (%current-system)))) + (o (derivation->output-path d))) + ;; Make sure we use 'guix substitute'. + (set-build-options s + #:print-build-trace #t + #:use-substitutes? #t + #:fallback? #f + #:substitute-urls (%test-substitute-urls)) + + (with-derivation-substitute d c + (sha256 => (make-bytevector 32 0)) ;select a hash that doesn't match C + + (define output + (call-with-output-string + (lambda (port) + (parameterize ((current-build-output-port port)) + (guard (c ((store-protocol-error? c) #t)) + (build-derivations s (list d)) + #f))))) + + (define actual-hash + (let-values (((port get-hash) + (gcrypt:open-hash-port + (gcrypt:hash-algorithm gcrypt:sha256)))) + (write-file-tree "foo" port + #:file-type+size + (lambda _ + (values 'regular (string-length c))) + #:file-port + (lambda _ + (open-input-string c))) + (close-port port) + (bytevector->nix-base32-string (get-hash)))) + + (define expected-hash + (bytevector->nix-base32-string (make-bytevector 32 0))) + + (define mismatch + (string-append "@ hash-mismatch " o " sha256 " + expected-hash " " actual-hash "\n")) + + (define failure + (string-append "@ substituter-failed " o)) + + (and (string-contains output mismatch) + (string-contains output failure)))))) + (test-assert "substitute --fallback" (with-store s (let* ((t (random-text)) ; contents of the output From patchwork Tue Dec 15 09:57:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25793 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E9B6B27BC05; Tue, 15 Dec 2020 09:59:12 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7481127BC04 for ; Tue, 15 Dec 2020 09:59:11 +0000 (GMT) Received: from localhost ([::1]:53486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp76s-0000Hm-Lh for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:59:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35872) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75o-0007iY-7U for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44132) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75m-0004YA-S2 for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75m-0007BQ-RJ for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 2/6] daemon: Let 'guix substitute' perform hash checks. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802626827556 (code B ref 45253); Tue, 15 Dec 2020 09:58:02 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:48 +0000 Received: from localhost ([127.0.0.1]:55669 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75X-0007AI-Mt for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42656) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75T-00079V-4S for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:43 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48636) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75N-0004QF-T9; Tue, 15 Dec 2020 04:57:37 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75N-0001j8-9q; Tue, 15 Dec 2020 04:57:37 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:26 +0100 Message-Id: <20201215095730.10954-2-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201215095730.10954-1-ludo@gnu.org> References: <20201215095730.10954-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches This way, the hash of the store item can be computed as it is restored, thereby avoiding an additional file tree traversal ('hashPath' call) later on in the daemon. Consequently, it should reduce latency between subsequent substitute downloads. This is a followup to 5ff521452b9ec2aae9ed8e4bb7bdc250a581f203. * guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New procedure. (process-substitution): Wrap INPUT into a hash input port, 'hashed', and read from it. Compare the actual and expected hashes, and print a "hash-mismatch" status line when they differ. When they match, print not just "success" but also the nar hash and size. * nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]: Remove. (SubstitutionGoal::finished): Tokenize 'status'. Parse it and handle "success" and "hash-mismatch" accordingly. Call 'hashPath' only when the returned hash is not SHA256. (SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr' handling. * tests/substitute.scm ("substitute, invalid hash"): Rename to... ("substitute, invalid narinfo hash"): ... this. ("substitute, invalid hash"): New test. --- guix/scripts/substitute.scm | 45 +++++++++++++++++++---- nix/libstore/build.cc | 73 ++++++++++++++++++++----------------- tests/substitute.scm | 52 ++++++++++++++++++++++++-- 3 files changed, 125 insertions(+), 45 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 25075eedff..17d0002b9f 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -26,6 +26,8 @@ #:use-module (guix combinators) #:use-module (guix config) #:use-module (guix records) + #:use-module (guix diagnostics) + #:use-module (guix i18n) #:use-module ((guix serialization) #:select (restore-file)) #:autoload (guix scripts discover) (read-substitute-urls) #:use-module (gcrypt hash) @@ -256,6 +258,18 @@ connection (typically PORT) is kept open once data has been fetched from URI." ;; for more information. (contents narinfo-contents)) +(define (narinfo-hash-algorithm+value narinfo) + "Return two values: the hash algorithm used by NARINFO and its value as a +bytevector." + (match (string-tokenize (narinfo-hash narinfo) + (char-set-complement (char-set #\:))) + ((algorithm base32) + (values (lookup-hash-algorithm (string->symbol algorithm)) + (nix-base32-string->bytevector base32))) + (_ + (raise (formatted-message + (G_ "invalid narinfo hash: ~s") (narinfo-hash narinfo)))))) + (define (narinfo-hash->sha256 hash) "If the string HASH denotes a sha256 hash, return it as a bytevector. Otherwise return #f." @@ -1033,7 +1047,9 @@ one. Return #f if URI's scheme is 'file' or #f." (define* (process-substitution store-item destination #:key cache-urls acl print-build-trace?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to -DESTINATION as a nar file. Verify the substitute against ACL." +DESTINATION as a nar file. Verify the substitute against ACL, and verify its +hash against what appears in the narinfo. Print a status line on the current +output port." (define narinfo (lookup-narinfo cache-urls store-item (cut valid-narinfo? <> acl))) @@ -1044,9 +1060,6 @@ DESTINATION as a nar file. Verify the substitute against ACL." (let-values (((uri compression file-size) (narinfo-best-uri narinfo))) - ;; Tell the daemon what the expected hash of the Nar itself is. - (format #t "~a~%" (narinfo-hash narinfo)) - (unless print-build-trace? (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) @@ -1079,9 +1092,16 @@ DESTINATION as a nar file. Verify the substitute against ACL." ;; closed here, while the child process doing the ;; reporting will close it upon exit. (decompressed-port (string->symbol compression) - progress))) + progress)) + + ;; Compute the actual nar hash as we read it. + ((algorithm expected) + (narinfo-hash-algorithm+value narinfo)) + ((hashed get-hash) + (open-hash-input-port algorithm input))) ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file input destination) + (restore-file hashed destination) + (close-port hashed) (close-port input) ;; Wait for the reporter to finish. @@ -1091,8 +1111,17 @@ DESTINATION as a nar file. Verify the substitute against ACL." ;; one to visually separate substitutions. (display "\n\n" (current-error-port)) - ;; Tell the daemon that we're done. - (display "success\n" (current-output-port))))) + ;; Check whether we got the data announced in NARINFO. + (let ((actual (get-hash))) + (if (bytevector=? actual expected) + ;; Tell the daemon that we're done. + (format (current-output-port) "success ~a ~a~%" + (narinfo-hash narinfo) (narinfo-size narinfo)) + ;; The actual data has a different hash than that in NARINFO. + (format (current-output-port) "hash-mismatch ~a ~a ~a~%" + (hash-algorithm-name algorithm) + (bytevector->nix-base32-string expected) + (bytevector->nix-base32-string actual))))))) ;;; diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index b5551b87ae..b19471a68f 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -2790,10 +2790,6 @@ private: /* The substituter. */ std::shared_ptr substituter; - /* Either the empty string, or the expected hash as returned by the - substituter. */ - string expectedHashStr; - /* Either the empty string, or the status phrase returned by the substituter. */ string status; @@ -3032,36 +3028,47 @@ void SubstitutionGoal::finished() /* Check the exit status and the build result. */ HashResult hash; try { - - if (status != "success") - throw SubstError(format("fetching path `%1%' (status: '%2%')") - % storePath % status); - - if (!pathExists(destPath)) - throw SubstError(format("substitute did not produce path `%1%'") % destPath); - - if (expectedHashStr == "") - throw SubstError(format("substituter did not communicate hash for `%1'") % storePath); - - hash = hashPath(htSHA256, destPath); - - /* Verify the expected hash we got from the substituer. */ - size_t n = expectedHashStr.find(':'); - if (n == string::npos) - throw Error(format("bad hash from substituter: %1%") % expectedHashStr); - HashType hashType = parseHashType(string(expectedHashStr, 0, n)); - if (hashType == htUnknown) - throw Error(format("unknown hash algorithm in `%1%'") % expectedHashStr); - Hash expectedHash = parseHash16or32(hashType, string(expectedHashStr, n + 1)); - Hash actualHash = hashType == htSHA256 ? hash.first : hashPath(hashType, destPath).first; - if (expectedHash != actualHash) { - if (settings.printBuildTrace) + auto statusList = tokenizeString >(status); + + if (statusList.empty()) { + throw SubstError(format("fetching path `%1%' (empty status: '%2%')") + % storePath % status); + } else if (statusList[0] == "hash-mismatch") { + if (settings.printBuildTrace) { + auto hashType = statusList[1]; + auto expectedHash = statusList[2]; + auto actualHash = statusList[3]; printMsg(lvlError, format("@ hash-mismatch %1% %2% %3% %4%") - % storePath % "sha256" - % printHash16or32(expectedHash) - % printHash16or32(actualHash)); + % storePath + % hashType % expectedHash % actualHash); + } throw SubstError(format("hash mismatch for substituted item `%1%'") % storePath); + } else if (statusList[0] == "success") { + if (!pathExists(destPath)) + throw SubstError(format("substitute did not produce path `%1%'") % destPath); + + std::string hashStr = statusList[1]; + size_t n = hashStr.find(':'); + if (n == string::npos) + throw Error(format("bad hash from substituter: %1%") % hashStr); + + HashType hashType = parseHashType(string(hashStr, 0, n)); + switch (hashType) { + case htUnknown: + throw Error(format("unknown hash algorithm in `%1%'") % hashStr); + case htSHA256: + hash.first = parseHash16or32(hashType, string(hashStr, n + 1)); + hash.second = std::atoi(statusList[2].c_str()); + break; + default: + /* The database only stores SHA256 hashes, so compute it. */ + hash = hashPath(htSHA256, destPath); + break; + } } + else + throw SubstError(format("fetching path `%1%' (status: '%2%')") + % storePath % status); } catch (SubstError & e) { @@ -3123,9 +3130,7 @@ void SubstitutionGoal::handleChildOutput(int fd, const string & data) string trimmed = (end != string::npos) ? input.substr(0, end) : input; /* Update the goal's state accordingly. */ - if (expectedHashStr == "") { - expectedHashStr = trimmed; - } else if (status == "") { + if (status == "") { status = trimmed; worker.wakeUp(shared_from_this()); } else { diff --git a/tests/substitute.scm b/tests/substitute.scm index b86ce09425..241c55a1f8 100644 --- a/tests/substitute.scm +++ b/tests/substitute.scm @@ -28,7 +28,9 @@ #:use-module (guix base32) #:use-module ((guix store) #:select (%store-prefix)) #:use-module ((guix ui) #:select (guix-warning-port)) - #:use-module ((guix utils) #:select (call-with-compressed-output-port)) + #:use-module ((guix utils) + #:select (call-with-temporary-directory + call-with-compressed-output-port)) #:use-module ((guix build utils) #:select (mkdir-p delete-file-recursively dump-port)) #:use-module (guix tests http) @@ -36,6 +38,7 @@ #:use-module (rnrs io ports) #:use-module (web uri) #:use-module (ice-9 regex) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -304,7 +307,7 @@ System: mips64el-linux\n") (lambda () (guix-substitute "--substitute"))))) -(test-quit "substitute, invalid hash" +(test-quit "substitute, invalid narinfo hash" "no valid substitute" ;; The hash in the signature differs from the hash of %NARINFO. (with-narinfo (string-append %narinfo "Signature: " @@ -317,6 +320,49 @@ System: mips64el-linux\n") (lambda () (guix-substitute "--substitute"))))) +(test-equal "substitute, invalid hash" + (string-append "hash-mismatch sha256 " + (bytevector->nix-base32-string (sha256 #vu8())) " " + (let-values (((port get-hash) + (open-hash-port (hash-algorithm sha256))) + ((content) + "Substitutable data.")) + (write-file-tree "foo" port + #:file-type+size + (lambda _ + (values 'regular + (string-length content))) + #:file-port + (lambda _ + (open-input-string content))) + (close-port port) + (bytevector->nix-base32-string (get-hash))) + "\n") + + ;; Arrange so the actual data hash does not match the 'NarHash' field in the + ;; narinfo. + (with-output-to-string + (lambda () + (let ((narinfo (string-append "StorePath: " (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-wrong-hash +URL: example.nar +Compression: none +NarHash: sha256:" (bytevector->nix-base32-string (sha256 #vu8())) " +NarSize: 42 +References: +Deriver: " (%store-prefix) "/foo.drv +System: mips64el-linux\n"))) + (with-narinfo (string-append narinfo "Signature: " + (signature-field narinfo) "\n") + (call-with-temporary-directory + (lambda (directory) + (with-input-from-string (string-append + "substitute " (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-wrong-hash " + directory "/wrong-hash\n") + (lambda () + (guix-substitute "--substitute")))))))))) + (test-quit "substitute, unauthorized key" "no valid substitute" (with-narinfo (string-append %narinfo "Signature: " @@ -326,7 +372,7 @@ System: mips64el-linux\n") "\n") (with-input-from-string (string-append "substitute " (%store-prefix) - "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo" + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-wrong-hash" " foo\n") (lambda () (guix-substitute "--substitute"))))) From patchwork Tue Dec 15 09:57:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25791 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8845327BC05; Tue, 15 Dec 2020 09:58:18 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 24E4927BC04 for ; Tue, 15 Dec 2020 09:58:18 +0000 (GMT) Received: from localhost ([::1]:51144 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp761-0007jq-5Y for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:58:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35874) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75o-0007iZ-7t for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44133) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75n-0004YT-7j for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75n-0007BX-6w for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 3/6] tests: Check the mtime and permissions of substituted items. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802626827563 (code B ref 45253); Tue, 15 Dec 2020 09:58:03 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:48 +0000 Received: from localhost ([127.0.0.1]:55671 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75Y-0007AP-CS for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42660) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75U-00079X-5i for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:44 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48637) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75O-0004R4-UC; Tue, 15 Dec 2020 04:57:38 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75O-0001j8-6i; Tue, 15 Dec 2020 04:57:38 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:27 +0100 Message-Id: <20201215095730.10954-3-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201215095730.10954-1-ludo@gnu.org> References: <20201215095730.10954-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * tests/store.scm ("substitute") ("substitute + build-things with output path") ("substitute + build-things with specific output"): Call 'canonical-file?'. * tests/substitute.scm ("substitute, authorized key"): Check the mtime and permissions of "substitute-retrieved". --- tests/store.scm | 3 +++ tests/substitute.scm | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/store.scm b/tests/store.scm index 7f1ec51875..4dc125bcb9 100644 --- a/tests/store.scm +++ b/tests/store.scm @@ -715,6 +715,7 @@ #:substitute-urls (%test-substitute-urls)) (and (has-substitutes? s o) (build-derivations s (list d)) + (canonical-file? o) (equal? c (call-with-input-file o get-string-all))))))) (test-assert "substitute + build-things with output path" @@ -735,6 +736,7 @@ (and (has-substitutes? s o) (build-things s (list o)) ;give the output path (valid-path? s o) + (canonical-file? o) (equal? c (call-with-input-file o get-string-all))))))) (test-assert "substitute + build-things with specific output" @@ -755,6 +757,7 @@ (build-things s `((,(derivation-file-name d) . "out"))) (valid-path? s o) + (canonical-file? o) (equal? c (call-with-input-file o get-string-all))))))) (test-assert "substitute, corrupt output hash" diff --git a/tests/substitute.scm b/tests/substitute.scm index 241c55a1f8..4f55a14957 100644 --- a/tests/substitute.scm +++ b/tests/substitute.scm @@ -378,7 +378,7 @@ System: mips64el-linux\n"))) (guix-substitute "--substitute"))))) (test-equal "substitute, authorized key" - "Substitutable data." + '("Substitutable data." 1 #o444) (with-narinfo (string-append %narinfo "Signature: " (signature-field %narinfo)) (dynamic-wind @@ -387,7 +387,9 @@ System: mips64el-linux\n"))) (request-substitution (string-append (%store-prefix) "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") "substitute-retrieved") - (call-with-input-file "substitute-retrieved" get-string-all)) + (list (call-with-input-file "substitute-retrieved" get-string-all) + (stat:mtime (lstat "substitute-retrieved")) + (stat:perms (lstat "substitute-retrieved")))) (lambda () (false-if-exception (delete-file "substitute-retrieved")))))) From patchwork Tue Dec 15 09:57:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25795 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0936827BC05; Tue, 15 Dec 2020 09:59:22 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B007F27BC04 for ; Tue, 15 Dec 2020 09:59:21 +0000 (GMT) Received: from localhost ([::1]:54078 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp772-0000XH-Tw for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:59:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35876) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75o-0007ib-8i for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44134) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75n-0004YU-JP for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75n-0007Bf-JB for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 4/6] daemon: Do not reset timestamps and permissions on substituted items. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802626927571 (code B ref 45253); Tue, 15 Dec 2020 09:58:03 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:49 +0000 Received: from localhost ([127.0.0.1]:55673 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75Y-0007AW-Mp for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42666) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75V-00079Z-Mu for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:45 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48638) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75Q-0004RD-00; Tue, 15 Dec 2020 04:57:40 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75P-0001j8-7j; Tue, 15 Dec 2020 04:57:39 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:28 +0100 Message-Id: <20201215095730.10954-4-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201215095730.10954-1-ludo@gnu.org> References: <20201215095730.10954-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches 'guix substitute' now takes care of it via 'restore-file'. * nix/libstore/build.cc (SubstitutionGoal::finished): Remove call to 'canonicalisePathMetaData'. --- nix/libstore/build.cc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index b19471a68f..ea809c6971 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -3085,7 +3085,8 @@ void SubstitutionGoal::finished() if (repair) replaceValidPath(storePath, destPath); - canonicalisePathMetaData(storePath, -1); + /* Note: 'guix substitute' takes care of resetting timestamps and + permissions on 'destPath', so no need to do it here. */ worker.store.optimisePath(storePath); // FIXME: combine with hashPath() From patchwork Tue Dec 15 09:57:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25796 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 9966827BC05; Tue, 15 Dec 2020 09:59:25 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 4735127BC04 for ; Tue, 15 Dec 2020 09:59:25 +0000 (GMT) Received: from localhost ([::1]:54358 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp776-0000eI-GP for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:59:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35880) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75o-0007ih-Aj for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44135) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75o-0004YV-27 for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75n-0007Bm-W3 for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 5/6] tests: Make sure substituted items are deduplicated. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802626927577 (code B ref 45253); Tue, 15 Dec 2020 09:58:03 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:49 +0000 Received: from localhost ([127.0.0.1]:55675 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75Z-0007Ae-0W for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42670) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75W-00079b-Dv for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:46 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48640) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75R-0004RM-27; Tue, 15 Dec 2020 04:57:41 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75Q-0001j8-9t; Tue, 15 Dec 2020 04:57:40 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:29 +0100 Message-Id: <20201215095730.10954-5-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201215095730.10954-1-ludo@gnu.org> References: <20201215095730.10954-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * tests/store.scm ("substitute, deduplication"): New test. --- tests/store.scm | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tests/store.scm b/tests/store.scm index 4dc125bcb9..c9a08ac690 100644 --- a/tests/store.scm +++ b/tests/store.scm @@ -718,6 +718,30 @@ (canonical-file? o) (equal? c (call-with-input-file o get-string-all))))))) +(test-assert "substitute, deduplication" + (with-store s + (let* ((c (random-text)) ; contents of the output + (g (package-derivation s %bootstrap-guile)) + (d1 (build-expression->derivation s "substitute-me" + `(begin ,c (exit 1)) + #:guile-for-build g)) + (d2 (build-expression->derivation s "build-me" + `(call-with-output-file %output + (lambda (p) + (display ,c p))) + #:guile-for-build g)) + (o1 (derivation->output-path d1)) + (o2 (derivation->output-path d2))) + (with-derivation-substitute d1 c + (set-build-options s #:use-substitutes? #t + #:substitute-urls (%test-substitute-urls)) + (and (has-substitutes? s o1) + (build-derivations s (list d2)) ;build + (build-derivations s (list d1)) ;substitute + (canonical-file? o1) + (equal? c (call-with-input-file o1 get-string-all)) + (= (stat:ino (stat o1)) (stat:ino (stat o2)))))))) + (test-assert "substitute + build-things with output path" (with-store s (let* ((c (random-text)) ;contents of the output From patchwork Tue Dec 15 09:57:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25792 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 9AA2327BC05; Tue, 15 Dec 2020 09:58:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id DC18727BC04 for ; Tue, 15 Dec 2020 09:58:19 +0000 (GMT) Received: from localhost ([::1]:51232 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp763-0007mC-3a for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:58:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35882) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75q-0007ja-4A for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44136) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75p-0004Yn-Gb for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75o-0007Bt-Cr for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 6/6] daemon: Delegate deduplication to 'guix substitute'. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802627527592 (code B ref 45253); Tue, 15 Dec 2020 09:58:04 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:55 +0000 Received: from localhost ([127.0.0.1]:55677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75f-0007Ax-BC for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:55 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42678) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75X-00079d-7m for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:47 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48641) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75R-0004SG-Vs; Tue, 15 Dec 2020 04:57:41 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75R-0001j8-Bt; Tue, 15 Dec 2020 04:57:41 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:30 +0100 Message-Id: <20201215095730.10954-6-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201215095730.10954-1-ludo@gnu.org> References: <20201215095730.10954-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches This removes the main source of latency between subsequent downloads. * nix/libstore/build.cc (SubstitutionGoal::tryToRun): Add a "deduplicate" key to ENV. (SubstitutionGoal::finished): Remove call to 'optimisePath'. * guix/scripts/substitute.scm (process-substitution)[destination-in-store?] [dump-file/deduplicate*]: New variables. Pass #:dump-file to 'restore-file'. * guix/scripts/substitute.scm (guix-substitute)[deduplicate?]: New variable. Pass #:deduplicate? to 'process-substitution'. * guix/serialization.scm (dump-file): Export and augment 'dump-file'. --- guix/scripts/substitute.scm | 31 ++++++++++++++++++++++++++----- guix/serialization.scm | 8 ++++++-- nix/libstore/build.cc | 13 ++++++++----- 3 files changed, 40 insertions(+), 12 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 17d0002b9f..38702d0c4b 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -28,7 +28,8 @@ #:use-module (guix records) #:use-module (guix diagnostics) #:use-module (guix i18n) - #:use-module ((guix serialization) #:select (restore-file)) + #:use-module ((guix serialization) #:select (restore-file dump-file)) + #:autoload (guix store deduplication) (dump-file/deduplicate) #:autoload (guix scripts discover) (read-substitute-urls) #:use-module (gcrypt hash) #:use-module (guix base32) @@ -1045,15 +1046,27 @@ one. Return #f if URI's scheme is 'file' or #f." (call-with-cached-connection uri (lambda (port) exp ...))) (define* (process-substitution store-item destination - #:key cache-urls acl print-build-trace?) + #:key cache-urls acl + deduplicate? print-build-trace?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL, and verify its -hash against what appears in the narinfo. Print a status line on the current -output port." +hash against what appears in the narinfo. When DEDUPLICATE? is true, and if +DESTINATION is in the store, deduplicate its files. Print a status line on +the current output port." (define narinfo (lookup-narinfo cache-urls store-item (cut valid-narinfo? <> acl))) + (define destination-in-store? + (string-prefix? (string-append (%store-prefix) "/") + destination)) + + (define (dump-file/deduplicate* . args) + ;; Make sure deduplication looks at the right store (necessary in test + ;; environments). + (apply dump-file/deduplicate + (append args (list #:store (%store-prefix))))) + (unless narinfo (leave (G_ "no valid substitute for '~a'~%") store-item)) @@ -1100,7 +1113,11 @@ output port." ((hashed get-hash) (open-hash-input-port algorithm input))) ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file hashed destination) + (restore-file hashed destination + #:dump-file (if (and destination-in-store? + deduplicate?) + dump-file/deduplicate* + dump-file)) (close-port hashed) (close-port input) @@ -1248,6 +1265,9 @@ default value." ((= string->number number) (> number 0)) (_ #f))) + (define deduplicate? + (find-daemon-option "deduplicate")) + ;; The daemon's agent code opens file descriptor 4 for us and this is where ;; stderr should go. (parameterize ((current-error-port (if (%error-to-file-descriptor-4?) @@ -1307,6 +1327,7 @@ default value." (process-substitution store-path destination #:cache-urls (substitute-urls) #:acl (current-acl) + #:deduplicate? deduplicate? #:print-build-trace? print-build-trace?) (loop)))))) diff --git a/guix/serialization.scm b/guix/serialization.scm index 9e2dce8bb0..59cd93fb18 100644 --- a/guix/serialization.scm +++ b/guix/serialization.scm @@ -51,7 +51,8 @@ write-file write-file-tree fold-archive - restore-file)) + restore-file + dump-file)) ;;; Comment: ;;; @@ -458,7 +459,10 @@ depends on TYPE." (&nar-read-error (port port) (file file) (token x))))))))) (define (dump-file file input size type) - "Dump SIZE bytes from INPUT to FILE." + "Dump SIZE bytes from INPUT to FILE. + +This procedure is suitable for use as the #:dump-file argument to +'restore-file'." (call-with-output-file file (lambda (output) (dump input output size)))) diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index ea809c6971..20d83fea4a 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -2984,7 +2984,12 @@ void SubstitutionGoal::tryToRun() if (!worker.substituter) { const Strings args = { "substitute", "--substitute" }; - const std::map env = { { "_NIX_OPTIONS", settings.pack() } }; + const std::map env = { + { "_NIX_OPTIONS", + settings.pack() + "deduplicate=" + + (settings.autoOptimiseStore ? "yes" : "no") + } + }; worker.substituter = std::make_shared(settings.guixProgram, args, env); } @@ -3085,10 +3090,8 @@ void SubstitutionGoal::finished() if (repair) replaceValidPath(storePath, destPath); - /* Note: 'guix substitute' takes care of resetting timestamps and - permissions on 'destPath', so no need to do it here. */ - - worker.store.optimisePath(storePath); // FIXME: combine with hashPath() + /* Note: 'guix substitute' takes care of resetting timestamps and of + deduplicating 'destPath', so no need to do it here. */ ValidPathInfo info2; info2.path = storePath;