From patchwork Tue Dec 15 09:57:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 25792 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 9AA2327BC05; Tue, 15 Dec 2020 09:58:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id DC18727BC04 for ; Tue, 15 Dec 2020 09:58:19 +0000 (GMT) Received: from localhost ([::1]:51232 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp763-0007mC-3a for patchwork@mira.cbaines.net; Tue, 15 Dec 2020 04:58:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35882) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp75q-0007ja-4A for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44136) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kp75p-0004Yn-Gb for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kp75o-0007Bt-Cr for guix-patches@gnu.org; Tue, 15 Dec 2020 04:58:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45253] [PATCH 6/6] daemon: Delegate deduplication to 'guix substitute'. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 15 Dec 2020 09:58:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45253 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45253@debbugs.gnu.org Received: via spool by 45253-submit@debbugs.gnu.org id=B45253.160802627527592 (code B ref 45253); Tue, 15 Dec 2020 09:58:04 +0000 Received: (at 45253) by debbugs.gnu.org; 15 Dec 2020 09:57:55 +0000 Received: from localhost ([127.0.0.1]:55677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75f-0007Ax-BC for submit@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:55 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42678) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kp75X-00079d-7m for 45253@debbugs.gnu.org; Tue, 15 Dec 2020 04:57:47 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48641) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp75R-0004SG-Vs; Tue, 15 Dec 2020 04:57:41 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44330 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kp75R-0001j8-Bt; Tue, 15 Dec 2020 04:57:41 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 15 Dec 2020 10:57:30 +0100 Message-Id: <20201215095730.10954-6-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201215095730.10954-1-ludo@gnu.org> References: <20201215095730.10954-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches This removes the main source of latency between subsequent downloads. * nix/libstore/build.cc (SubstitutionGoal::tryToRun): Add a "deduplicate" key to ENV. (SubstitutionGoal::finished): Remove call to 'optimisePath'. * guix/scripts/substitute.scm (process-substitution)[destination-in-store?] [dump-file/deduplicate*]: New variables. Pass #:dump-file to 'restore-file'. * guix/scripts/substitute.scm (guix-substitute)[deduplicate?]: New variable. Pass #:deduplicate? to 'process-substitution'. * guix/serialization.scm (dump-file): Export and augment 'dump-file'. --- guix/scripts/substitute.scm | 31 ++++++++++++++++++++++++++----- guix/serialization.scm | 8 ++++++-- nix/libstore/build.cc | 13 ++++++++----- 3 files changed, 40 insertions(+), 12 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 17d0002b9f..38702d0c4b 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -28,7 +28,8 @@ #:use-module (guix records) #:use-module (guix diagnostics) #:use-module (guix i18n) - #:use-module ((guix serialization) #:select (restore-file)) + #:use-module ((guix serialization) #:select (restore-file dump-file)) + #:autoload (guix store deduplication) (dump-file/deduplicate) #:autoload (guix scripts discover) (read-substitute-urls) #:use-module (gcrypt hash) #:use-module (guix base32) @@ -1045,15 +1046,27 @@ one. Return #f if URI's scheme is 'file' or #f." (call-with-cached-connection uri (lambda (port) exp ...))) (define* (process-substitution store-item destination - #:key cache-urls acl print-build-trace?) + #:key cache-urls acl + deduplicate? print-build-trace?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL, and verify its -hash against what appears in the narinfo. Print a status line on the current -output port." +hash against what appears in the narinfo. When DEDUPLICATE? is true, and if +DESTINATION is in the store, deduplicate its files. Print a status line on +the current output port." (define narinfo (lookup-narinfo cache-urls store-item (cut valid-narinfo? <> acl))) + (define destination-in-store? + (string-prefix? (string-append (%store-prefix) "/") + destination)) + + (define (dump-file/deduplicate* . args) + ;; Make sure deduplication looks at the right store (necessary in test + ;; environments). + (apply dump-file/deduplicate + (append args (list #:store (%store-prefix))))) + (unless narinfo (leave (G_ "no valid substitute for '~a'~%") store-item)) @@ -1100,7 +1113,11 @@ output port." ((hashed get-hash) (open-hash-input-port algorithm input))) ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file hashed destination) + (restore-file hashed destination + #:dump-file (if (and destination-in-store? + deduplicate?) + dump-file/deduplicate* + dump-file)) (close-port hashed) (close-port input) @@ -1248,6 +1265,9 @@ default value." ((= string->number number) (> number 0)) (_ #f))) + (define deduplicate? + (find-daemon-option "deduplicate")) + ;; The daemon's agent code opens file descriptor 4 for us and this is where ;; stderr should go. (parameterize ((current-error-port (if (%error-to-file-descriptor-4?) @@ -1307,6 +1327,7 @@ default value." (process-substitution store-path destination #:cache-urls (substitute-urls) #:acl (current-acl) + #:deduplicate? deduplicate? #:print-build-trace? print-build-trace?) (loop)))))) diff --git a/guix/serialization.scm b/guix/serialization.scm index 9e2dce8bb0..59cd93fb18 100644 --- a/guix/serialization.scm +++ b/guix/serialization.scm @@ -51,7 +51,8 @@ write-file write-file-tree fold-archive - restore-file)) + restore-file + dump-file)) ;;; Comment: ;;; @@ -458,7 +459,10 @@ depends on TYPE." (&nar-read-error (port port) (file file) (token x))))))))) (define (dump-file file input size type) - "Dump SIZE bytes from INPUT to FILE." + "Dump SIZE bytes from INPUT to FILE. + +This procedure is suitable for use as the #:dump-file argument to +'restore-file'." (call-with-output-file file (lambda (output) (dump input output size)))) diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index ea809c6971..20d83fea4a 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -2984,7 +2984,12 @@ void SubstitutionGoal::tryToRun() if (!worker.substituter) { const Strings args = { "substitute", "--substitute" }; - const std::map env = { { "_NIX_OPTIONS", settings.pack() } }; + const std::map env = { + { "_NIX_OPTIONS", + settings.pack() + "deduplicate=" + + (settings.autoOptimiseStore ? "yes" : "no") + } + }; worker.substituter = std::make_shared(settings.guixProgram, args, env); } @@ -3085,10 +3090,8 @@ void SubstitutionGoal::finished() if (repair) replaceValidPath(storePath, destPath); - /* Note: 'guix substitute' takes care of resetting timestamps and - permissions on 'destPath', so no need to do it here. */ - - worker.store.optimisePath(storePath); // FIXME: combine with hashPath() + /* Note: 'guix substitute' takes care of resetting timestamps and of + deduplicating 'destPath', so no need to do it here. */ ValidPathInfo info2; info2.path = storePath;