From patchwork Tue Dec 29 09:59:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: M X-Patchwork-Id: 26109 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B974F27BC05; Tue, 29 Dec 2020 15:04:55 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id F3A3027BC04 for ; Tue, 29 Dec 2020 15:04:46 +0000 (GMT) Received: from localhost ([::1]:33092 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kuGYI-0007ct-1n for patchwork@mira.cbaines.net; Tue, 29 Dec 2020 10:04:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:43988) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kuFhK-00089i-80 for guix-patches@gnu.org; Tue, 29 Dec 2020 09:10:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:54747) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kuFhJ-0000G4-Tb for guix-patches@gnu.org; Tue, 29 Dec 2020 09:10:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kuFhJ-0004nP-Nf for guix-patches@gnu.org; Tue, 29 Dec 2020 09:10:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#33899] Ludo's patch rebased on master References: <20181228231205.8068-1-ludo@gnu.org> In-Reply-To: <20181228231205.8068-1-ludo@gnu.org> Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 29 Dec 2020 14:10:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 33899 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 33899@debbugs.gnu.org Received: via spool by 33899-submit@debbugs.gnu.org id=B33899.160925094318350 (code B ref 33899); Tue, 29 Dec 2020 14:10:01 +0000 Received: (at 33899) by debbugs.gnu.org; 29 Dec 2020 14:09:03 +0000 Received: from localhost ([127.0.0.1]:38058 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kuFgK-0004lZ-JG for submit@debbugs.gnu.org; Tue, 29 Dec 2020 09:09:03 -0500 Received: from baptiste.telenet-ops.be ([195.130.132.51]:37598) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kuBmx-0004EJ-00 for 33899@debbugs.gnu.org; Tue, 29 Dec 2020 04:59:37 -0500 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by baptiste.telenet-ops.be with bizsmtp id A9zQ2400K0mfAB4019zZRB; Tue, 29 Dec 2020 10:59:33 +0100 Message-ID: From: Maxime Devos Date: Tue, 29 Dec 2020 10:59:13 +0100 User-Agent: Evolution 3.34.2 MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 29 Dec 2020 09:08:59 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Mailman-Approved-At: Tue, 29 Dec 2020 10:04:37 -0500 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi Guix, I've rebased Ludovic's patch on master (08d8c2d3c08e4f35325553e75abc76da40630334), resolving merge conflicts. Make and make check succeed, except for tests/cve.scm and tests/swh.scm. For completeness, I've attached the logs of the failing tests. I don't think they rare related to the changes in the patch, though. I most likely won't have time to test and complete this patch in the near future. On an unrelated note, I've changed e-mail addresses due to excessive spam-filtering From d300bd6b37680f26fbc9b339264476fcc35e1787 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= Date: Fri, 28 Dec 2018 18:40:06 +0100 Subject: [PATCH 5/5] DRAFT substitute: Add IPFS support. Missing: - documentation - command-line options - progress report when downloading over IPFS - fallback when we fail to fetch from IPFS * guix/scripts/substitute.scm ()[ipfs]: New field. (read-narinfo): Read "IPFS". (process-substitution/http): New procedure, with code formerly in 'process-substitution'. (process-substitution): Check for IPFS and call 'ipfs:restore-file-tree' when IPFS is true. --- guix/scripts/substitute.scm | 112 ++++++++++++++++++++---------------- 1 file changed, 63 insertions(+), 49 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index feae2df9cb..8a888c5e01 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -43,6 +43,7 @@ #:use-module (guix progress) #:use-module ((guix build syscalls) #:select (set-thread-name)) + #:use-module ((guix ipfs) #:prefix ipfs:) #:use-module (ice-9 rdelim) #:use-module (ice-9 regex) #:use-module (ice-9 match) @@ -233,7 +234,7 @@ provide." (define-record-type (%make-narinfo path uri-base uris compressions file-sizes file-hashes nar-hash nar-size references deriver system - signature contents) + ipfs signature contents) narinfo? (path narinfo-path) (uri-base narinfo-uri-base) ;URI of the cache it originates from @@ -246,6 +247,7 @@ provide." (references narinfo-references) (deriver narinfo-deriver) (system narinfo-system) + (ipfs narinfo-ipfs) (signature narinfo-signature) ; canonical sexp ;; The original contents of a narinfo file. This field is needed because we ;; want to preserve the exact textual representation for verification purposes. @@ -288,7 +290,7 @@ s-expression: ~s~%") must contain the original contents of a narinfo file." (lambda (path urls compressions file-hashes file-sizes nar-hash nar-size references deriver system - signature) + ipfs signature) "Return a new object." (define len (length urls)) (%make-narinfo path cache-url @@ -312,6 +314,7 @@ must contain the original contents of a narinfo file." ((or #f "") #f) (_ deriver)) system + ipfs (false-if-exception (and=> signature narinfo-signature->canonical-sexp)) str))) @@ -330,7 +333,7 @@ No authentication and authorization checks are performed here!" (narinfo-maker str url) '("StorePath" "URL" "Compression" "FileHash" "FileSize" "NarHash" "NarSize" - "References" "Deriver" "System" + "References" "Deriver" "System" "IPFS" "Signature") '("URL" "Compression" "FileSize" "FileHash")))) @@ -962,6 +965,48 @@ the URI, its compression method (a string), and the compressed file size." (((uri compression file-size) _ ...) (values uri compression file-size)))) +(define* (process-substitution/http narinfo destination uri + compression + #:key print-build-trace?) + (unless print-build-trace? + (format (current-error-port) + (G_ "Downloading ~a...~%") (uri->string uri))) + (let*-values (((raw download-size) + ;; Note that Hydra currently generates Nars on the fly + ;; and doesn't specify a Content-Length, so + ;; DOWNLOAD-SIZE is #f in practice. + (fetch uri #:buffered? #f #:timeout? #f)) + ((progress) + (let* ((dl-size (or download-size + (and (equal? compression "none") + (narinfo-size narinfo)))) + (reporter (if print-build-trace? + (progress-reporter/trace + destination + (uri->string uri) dl-size + (current-error-port)) + (progress-reporter/file + (uri->string uri) dl-size + (current-error-port) + #:abbreviation nar-uri-abbreviation)))) + (progress-report-port reporter raw))) + ((input pids) + ;; NOTE: This 'progress' port of current process will be + ;; closed here, while the child process doing the + ;; reporting will close it upon exit. + (decompressed-port (string->symbol compression) + progress))) + ;; Unpack the Nar at INPUT into DESTINATION. + (restore-file input destination) + (close-port input) + + ;; Wait for the reporter to finish. + (every (compose zero? cdr waitpid) pids) + + ;; Skip a line after what 'progress-reporter/file' printed, and another + ;; one to visually separate substitutions. + (display "\n\n" (current-error-port)))) + (define* (process-substitution store-item destination #:key cache-urls acl print-build-trace?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to @@ -969,55 +1014,24 @@ DESTINATION as a nar file. Verify the substitute against ACL." (define narinfo (lookup-narinfo cache-urls store-item (cut valid-narinfo? <> acl))) - + (define ipfs (and=> narinfo narinfo-ipfs)) (unless narinfo (leave (G_ "no valid substitute for '~a'~%") store-item)) - - (let-values (((uri compression file-size) - (narinfo-best-uri narinfo))) - ;; Tell the daemon what the expected hash of the Nar itself is. - (format #t "~a~%" (narinfo-hash narinfo)) - - (unless print-build-trace? - (format (current-error-port) - (G_ "Downloading ~a...~%") (uri->string uri))) - - (let*-values (((raw download-size) - ;; Note that Hydra currently generates Nars on the fly - ;; and doesn't specify a Content-Length, so - ;; DOWNLOAD-SIZE is #f in practice. - (fetch uri #:buffered? #f #:timeout? #f)) - ((progress) - (let* ((dl-size (or download-size - (and (equal? compression "none") - (narinfo-size narinfo)))) - (reporter (if print-build-trace? - (progress-reporter/trace - destination - (uri->string uri) dl-size - (current-error-port)) - (progress-reporter/file - (uri->string uri) dl-size - (current-error-port) - #:abbreviation nar-uri-abbreviation)))) - (progress-report-port reporter raw))) - ((input pids) - ;; NOTE: This 'progress' port of current process will be - ;; closed here, while the child process doing the - ;; reporting will close it upon exit. - (decompressed-port (string->symbol compression) - progress))) - ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file input destination) - (close-port input) - - ;; Wait for the reporter to finish. - (every (compose zero? cdr waitpid) pids) - - ;; Skip a line after what 'progress-reporter/file' printed, and another - ;; one to visually separate substitutions. - (display "\n\n" (current-error-port))))) + ;; Tell the daemon what the expected hash of the Nar itself is. + (format #t "~a~%" (narinfo-hash narinfo)) + (if ipfs + (begin + (unless print-build-trace? + (format (current-error-port) + (G_ "Downloading from IPFS ~s...~%") ipfs)) + (ipfs:restore-file-tree ipfs destination)) + (let-values (((uri compression file-size) + (narinfo-best-uri narinfo))) + (process-substitution/http narinfo destination uri + compression + #:print-build-trace? + print-build-trace?)))) ;;; -- 2.29.2