From patchwork Thu Dec 28 09:40:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 58132 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id DC8AE27BBEA; Thu, 28 Dec 2023 09:41:53 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 863C927BBE2 for ; Thu, 28 Dec 2023 09:41:52 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rImt5-0007go-1I; Thu, 28 Dec 2023 04:41:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rImt1-0007d4-2M for guix-patches@gnu.org; Thu, 28 Dec 2023 04:41:07 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rImt0-000833-Kb; Thu, 28 Dec 2023 04:41:06 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rImsy-0005li-DE; Thu, 28 Dec 2023 04:41:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v4 6/7] substitute: Decode substitutes using ERIS. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Thu, 28 Dec 2023 09:41:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ludo@gnu.org, maximedevos@telenet.be, Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.170375645022089 (code B ref 52555); Thu, 28 Dec 2023 09:41:04 +0000 Received: (at 52555) by debbugs.gnu.org; 28 Dec 2023 09:40:50 +0000 Received: from localhost ([127.0.0.1]:38526 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rImsj-0005kC-MQ for submit@debbugs.gnu.org; Thu, 28 Dec 2023 04:40:50 -0500 Received: from mout01.posteo.de ([185.67.36.65]:38393) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rImsW-0005iV-TB for 52555@debbugs.gnu.org; Thu, 28 Dec 2023 04:40:37 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 194E124002B for <52555@debbugs.gnu.org>; Thu, 28 Dec 2023 10:40:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703756432; bh=obLvOQtF5pnpzyIIgP+dO7Dj0TpeECFmi/zdZjg9Wu8=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=gPxVCDii+qC1pOBD8e9XRILtkrUjS2GmPTnEKwXdcMaCWRppCj4SpmrDn/dFJVR+a cLDpKAZT6zEFLTXoD2k+cLdT07Ttkvrafc1oaVLa89Y1SnJFrWge85Jf7nC4qZF7qX qmu+KxYWYrBOJcde+FqO3Ep7z8LCK0Psb3u2h/Y02cBTwdV6JTjK6ltY2YKkeceLbt CGfkf1ikY3LrpwnWpo6Gt6PrKOWXhlHKDiBTsvHLGAIr0wd3u/eMbgG6oe2/XWv4Q9 Hc9xZ0eUMPQoQg4ffKTdvwHwW7EVHrJ92WPcHqOB8QqCWZe0vCC+Hrru9khkXWmEC0 eFKhL4DOrZCcw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4T13Ql1R7mz6tyZ; Thu, 28 Dec 2023 10:40:31 +0100 (CET) From: pukkamustard Date: Thu, 28 Dec 2023 09:40:09 +0000 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/scripts/substitute.scm: Decode substitutes using ERIS. * guix/eris.scm (eris-decode-store-item): New function. * nix/nix-daemon/guix-daemon.cc (options): Add eris-store-url option. --- guix/eris.scm | 45 +++++++++++++++++---- guix/scripts/substitute.scm | 76 +++++++++++++++++++++++++++-------- nix/nix-daemon/guix-daemon.cc | 5 +++ 3 files changed, 102 insertions(+), 24 deletions(-) diff --git a/guix/eris.scm b/guix/eris.scm index 3fbedd0cb7..1aa52e69dd 100644 --- a/guix/eris.scm +++ b/guix/eris.scm @@ -21,6 +21,7 @@ (define-module (guix eris) #:use-module (eris) #:use-module (eris fs) #:use-module (eris sqlite) + #:use-module (sqlite3) #:use-module (eris coap) #:use-module (eris read-capability) @@ -31,7 +32,8 @@ (define-module (guix eris) #:use-module (srfi srfi-171) #:export (%eris-store-url - eris-encode-store-item)) + eris-encode-store-item + eris-decode-store-item)) (define %eris-store-url (make-parameter @@ -83,8 +85,7 @@ (define (guix-eris-block-reducer) (() (ecbr)) ((conn ref-block) (ecbr conn ref-block)) ((conn) - (ecbr conn) - (close-port conn))))) + (ecbr conn))))) ;; TODO ;; ('coap+tcp #f) @@ -97,10 +98,40 @@ (define (guix-eris-block-reducer) ;; SRFI-171 that counts the number of blocks. rcount))) +(define (call-with-guix-eris-block-ref proc) + (let ((store-url (%eris-store-url))) + (if (uri? store-url) + (match (uri-scheme store-url) + + ('sqlite + (let ((db (eris-sqlite-open (uri-path store-url)))) + (proc (lambda (ref) (eris-sqlite-ref db ref))) + (sqlite-close db))) + + ('coap+unix + (let ((conn (open-coap-unix-socket (uri-path store-url))) + (req-uri (build-uri 'coap #:path ".well-known/eris"))) + (proc + (lambda (ref) + (eris-coap-block-ref req-uri ref #:connection conn))) + (close-port conn))) + + (_ (error "Don't know how to handle ERIS store URL " + (uri->string (%eris-store-url))))) + + (error "No ERIS store to get blocks.")))) + (define* (eris-encode-store-item item) "Encodes the store item ITEM using ERIS and returns the read capability as string." - (eris-read-capability->string - (eris-fs-encode item - #:convergence-secret (%guix-eris-convergence-secret) - #:block-reducer (guix-eris-block-reducer)))) + (eris-fs-encode item + #:convergence-secret (%guix-eris-convergence-secret) + #:block-reducer (guix-eris-block-reducer))) + +(define* (eris-decode-store-item eris-urn destination) + "Decode a store item with read-capability ERIS-URN to DESTINATION." + (call-with-guix-eris-block-ref + (lambda (block-ref) + (eris-fs-decode eris-urn destination + #:block-ref block-ref) + #t))) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 37cd08e289..3c060f1c89 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -45,6 +45,8 @@ (define-module (guix scripts substitute) . guix:open-connection-for-uri))) #:autoload (gnutls) (error/invalid-session error/again error/interrupted) #:use-module (guix progress) + #:use-module (guix eris) + #:use-module (fibers) #:use-module ((guix build syscalls) #:select (set-thread-name)) #:use-module (ice-9 rdelim) @@ -656,9 +658,11 @@ (define* (process-substitution/fallback port narinfo destination (() (loop rest))))))) + (define* (process-substitution port store-item destination #:key cache-urls acl - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + (eris? #f)) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL, and verify its hash against what appears in the narinfo. When DEDUPLICATE? is true, and if @@ -674,20 +678,56 @@ (define* (process-substitution port store-item destination (leave (G_ "no valid substitute for '~a'~%") store-item)) - (guard (c ((network-error? c) - (format (current-error-port) - (G_ "retrying download of '~a' with other substitute URLs...~%") - store-item) - (process-substitution/fallback port narinfo destination - #:cache-urls cache-urls - #:acl acl - #:deduplicate? deduplicate? - #:print-build-trace? - print-build-trace?))) - (download-nar narinfo destination - #:status-port port - #:deduplicate? deduplicate? - #:print-build-trace? print-build-trace?))) + (if (and eris? + (%eris-store-url) + (narinfo-eris-urn narinfo)) + + (unless + + ;; Attempt to fetch substitute via ERIS + (let ((eris-urn (narinfo-eris-urn narinfo))) + (format (current-error-port) + (G_ "Downloading ~a...~%") (uri->string eris-urn)) + (run-fibers + (lambda () + (guard + (c (else + (format (current-error-port) + (G_ "failed to decode substitute from ERIS URN ~a: ~a" + (uri->string eris-urn) + c)) + #f)) + (eris-decode-store-item eris-urn destination) + ;; Tell the daemon that we're done. + (format port "success ~a ~a~%" + (narinfo-hash narinfo) (narinfo-size narinfo)))))) + + ;; Retry without ERIS on failure. + (process-substitution port store-item destination + #:cache-urls cache-urls + #:acl acl + #:deduplicate? deduplicate? + #:print-build-trace? print-build-trace? + #:eris? #f)) + + (guard (c ((network-error? c) + (format (current-error-port) + (G_ "retrying download of '~a' with other substitute URLs...~%") + store-item) + (process-substitution/fallback port narinfo destination + #:cache-urls cache-urls + #:acl acl + #:deduplicate? deduplicate? + #:print-build-trace? + print-build-trace?))) + (download-nar narinfo destination + #:status-port port + #:deduplicate? deduplicate? + #:print-build-trace? print-build-trace?)))) + + + + ;;; @@ -876,7 +916,8 @@ (define-command (guix-substitute . args) ;; Download STORE-PATH and store it as a Nar in file DESTINATION. ;; Specify the number of columns of the terminal so the progress ;; report displays nicely. - (parameterize ((current-terminal-columns (client-terminal-columns))) + (parameterize ((current-terminal-columns (client-terminal-columns)) + (%eris-store-url (find-daemon-option "eris-store-url"))) (let loop () (match (read-line) ((? eof-object?) @@ -887,7 +928,8 @@ (define-command (guix-substitute . args) #:acl (current-acl) #:deduplicate? deduplicate? #:print-build-trace? - print-build-trace?) + print-build-trace? + #:eris? #t) (loop)))))) (opts (leave (G_ "~a: unrecognized options~%") opts)))))) diff --git a/nix/nix-daemon/guix-daemon.cc b/nix/nix-daemon/guix-daemon.cc index d7ab9c5e64..d6b054bc6c 100644 --- a/nix/nix-daemon/guix-daemon.cc +++ b/nix/nix-daemon/guix-daemon.cc @@ -90,6 +90,7 @@ builds derivations on behalf of its clients."); #define GUIX_OPT_MAX_SILENT_TIME 19 #define GUIX_OPT_LOG_COMPRESSION 20 #define GUIX_OPT_DISCOVER 21 +#define GUIX_OPT_ERIS_STORE_URL 22 static const struct argp_option options[] = { @@ -132,6 +133,8 @@ static const struct argp_option options[] = n_("use the specified compression type for build logs") }, { "discover", GUIX_OPT_DISCOVER, "yes/no", OPTION_ARG_OPTIONAL, n_("use substitute servers discovered on the local network") }, + { "eris-store-url", GUIX_OPT_ERIS_STORE_URL, n_("URL"), 0, + n_("use URL to retrieve blocks of ERIS encoded substitutes") }, /* '--disable-deduplication' was known as '--disable-store-optimization' up to Guix 0.7 included, so keep the alias around. */ @@ -270,6 +273,8 @@ parse_opt (int key, char *arg, struct argp_state *state) useDiscover = string_to_bool (arg); settings.set ("discover", useDiscover ? "true" : "false"); break; + case GUIX_OPT_ERIS_STORE_URL: + settings.set ("eris-store-url", arg); case GUIX_OPT_DEBUG: verbosity = lvlDebug; break;