From patchwork Fri Feb 26 17:41:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 27319 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A04AA27BC4D; Fri, 26 Feb 2021 17:43:26 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, FREEMAIL_FROM,MAILING_LIST_MULTI,PERCENT_RANDOM,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,T_DKIM_INVALID,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E6FFB27BC4C for ; Fri, 26 Feb 2021 17:43:23 +0000 (GMT) Received: from localhost ([::1]:42812 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lFh99-0008Oh-2C for patchwork@mira.cbaines.net; Fri, 26 Feb 2021 12:43:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55290) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lFh8o-0008Nc-Pw for guix-patches@gnu.org; Fri, 26 Feb 2021 12:43:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59003) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lFh8o-00061R-CO for guix-patches@gnu.org; Fri, 26 Feb 2021 12:43:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lFh8o-00055n-8N for guix-patches@gnu.org; Fri, 26 Feb 2021 12:43:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#46800] [PATCH] Allow defining multiple substituters Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 26 Feb 2021 17:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 46800 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46800@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161436133519508 (code B ref -1); Fri, 26 Feb 2021 17:43:02 +0000 Received: (at submit) by debbugs.gnu.org; 26 Feb 2021 17:42:15 +0000 Received: from localhost ([127.0.0.1]:42317 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lFh83-00054a-7J for submit@debbugs.gnu.org; Fri, 26 Feb 2021 12:42:15 -0500 Received: from lists.gnu.org ([209.51.188.17]:43888) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lFh82-00054T-0x for submit@debbugs.gnu.org; Fri, 26 Feb 2021 12:42:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:54760) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lFh7z-00083Z-R7 for guix-patches@gnu.org; Fri, 26 Feb 2021 12:42:13 -0500 Received: from laurent.telenet-ops.be ([2a02:1800:110:4::f00:19]:44168) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lFh7q-0005ep-BO for guix-patches@gnu.org; Fri, 26 Feb 2021 12:42:08 -0500 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by laurent.telenet-ops.be with bizsmtp id Zthx2401A0mfAB401thxrb; Fri, 26 Feb 2021 18:41:58 +0100 Message-ID: From: Maxime Devos Date: Fri, 26 Feb 2021 18:41:51 +0100 User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1614361318; bh=qm8b9Wy9o9C63ipY0uw120DrapTCge+3/T8Nllawi/k=; h=Subject:From:To:Cc:Date; b=hw3PDvLsmOlkidjIuhTko49kplx+05lXC0kIf4BEyXcfD27rdJ7fJGzRNnm4H0mw7 kamFPodjWkDxpYT1O7V0ci8y+tTeBd2KYxhBeYHiIBY6cddHI7ZvirMcYotucv4yoZ 8jfXTRDPAx/OnYGKo8exeL4cNABWRvzJ3LzrxvtOmtFoh+pkFXVNUN16mL7lvBViN8 roy2nvcKRu/k41ZMcbNTHWhi8UJRjGpSJCR57G+usLLtrg5EsWr7Znl5l9WjpkK34U Q5kAOvBR1K7t1e6yZStUqG7nSZmlJMuIc4Z12kaEkacmomtXYaiiMQrzuR7mK7mZex JmkdwfTKuZXRQ== Received-SPF: pass client-ip=2a02:1800:110:4::f00:19; envelope-from=maximedevos@telenet.be; helo=laurent.telenet-ops.be X-Spam_score_int: -9 X-Spam_score: -1.0 X-Spam_bar: - X-Spam_report: (-1.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PERCENT_RANDOM=1.838, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi Guix, This patch series is my suggestion for allowing multiple "substitution methods" or "substituters" as I call them. Currently, only a method for HTTP/S is defined, though I hope it will be a good basis for a common framework for substitutes over GNUnet and IPFS. Extending "guix-service-type" to allow configuration of substitution method is left for later. Any questions, remarks? Greetings, Maxime From b0b3f3e339ff834fd973f2f0f0bc7ad9be6ffd04 Mon Sep 17 00:00:00 2001 From: Maxime Devos Date: Fri, 26 Feb 2021 15:30:04 +0100 Subject: [PATCH 4/4] =?UTF-8?q?substitute:=20Unstub=20=E2=80=98verify-hash?= =?UTF-8?q?/unknown=E2=80=99.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This procedure is called if a substitution method returns 'unpacked'. While no method does that yet, it is expected the IPFS and GNUnet substituter will. * guix/scripts/substitute.scm (verify-hash/unknown): Unstub procedure. (process-substitution): When the substituter returns ‘unpacked’, verify whether we got the right substitute and canonicalize permissions and timestamps. * doc/substituters.texi (Defining Substituters): Document the absence of a requirement for substituters to normalize timestamps and file permissions. * tests/substitute.scm (write-string-as-nar): Define procedure, and use in test cases. Also test that the hash is verified when a substituter returns 'unpacked'. --- doc/substituters.texi | 4 +++ guix/scripts/substitute.scm | 43 +++++++++++++++++++---- tests/substitute.scm | 70 ++++++++++++++++++++++++++++++++----- 3 files changed, 102 insertions(+), 15 deletions(-) diff --git a/doc/substituters.texi b/doc/substituters.texi index 516e2c1eea..65b1a929b0 100644 --- a/doc/substituters.texi +++ b/doc/substituters.texi @@ -47,4 +47,8 @@ are correctly signed and have a correct hash; this is handled by @code{(guix scripts substitute)}. @var{nar-downloader} and @var{fetch-narinfos} can be @code{#f} if unimplemented by this substituter. + +Likewise, when returning @code{unpacked}, @var{nar-downloader} +does not need to normalize timestamps and file permissions. + @end deffn diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 78345dce8f..90fd7dd021 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -31,8 +31,10 @@ #:use-module (guix records) #:use-module (guix diagnostics) #:use-module (guix i18n) - #:use-module ((guix serialization) #:select (restore-file dump-file)) + #:use-module ((guix serialization) + #:select (restore-file write-file dump-file)) #:autoload (guix store deduplication) (dump-file/deduplicate) + #:autoload (guix store database) (reset-timestamps) #:autoload (guix scripts discover) (read-substitute-urls) #:autoload (guix scripts substitute http) (open-connection-for-uri/cached) #:use-module (gcrypt hash) @@ -49,6 +51,7 @@ #:use-module ((guix build syscalls) #:select (set-thread-name)) #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) #:use-module (ice-9 regex) #:use-module (ice-9 match) #:use-module (ice-9 format) @@ -531,11 +534,20 @@ to the narinfo." (bytevector->nix-base32-string expected) (bytevector->nix-base32-string actual)))) -(define (verify-hash/unknown . rest) - ;; Variant of verify-hash where the hash hasn't yet been computed. - ;; TODO: this will be implemented later in the patch series! - ;; (To be used by the IPFS and GNUnet substituter) - (leave (G_ "TODO verify-hash/unknown is unimplemented~%"))) +(define* (verify-hash/unknown file expected algorithm narinfo + #:key thunk) + "Check whether we got the data announced in the narinfo NARINFO. +FILE is the actual file we got and EXPECTED is the hash according +to the narinfo. Call THUNK after FILE was read, but before +the daemon is informed." + ;; Recreate the nar, hash it, and let verify-hash + ;; produce the 'success' or 'hash-mismatch' output. + (receive (hashed get-hash) + (open-hash-port algorithm) + (write-file file hashed) + (close hashed) + (thunk) + (verify-hash (get-hash) expected algorithm narinfo))) (define-syntax-rule (receive* kwargs exp exp* exp** ...) (call-with-values (lambda () exp) @@ -606,7 +618,24 @@ the current output port." (when after-input-close (after-input-close)) ;; Check whether we got the data announced in the NARINFO. - (verify-hash/unknown destination narinfo)) + (receive (algorithm expected) + (narinfo-hash-algorithm+value narinfo) + (verify-hash/unknown + destination expected algorithm narinfo + ;; Make sure the permissions and timestamps are canonical. + ;; + ;; This could theoretically be done somewhat more + ;; cache-friendly if done in the substitution method, + ;; by canonicalising a file right after it has been + ;; downloaded, but let's try for correctness first + ;; before efficiency. + ;; + ;; Also, this must be done *after* verifying the hash, + ;; in order to make the access time is set correctly. + ;; + ;; TODO it would be nice to deduplicate DESTINATION. + #:thunk + (lambda () (reset-timestamps destination))))) (else (format (current-error-port) "~s~%" input) (leave diff --git a/tests/substitute.scm b/tests/substitute.scm index 6c754f774d..1cb1a10402 100644 --- a/tests/substitute.scm +++ b/tests/substitute.scm @@ -324,6 +324,16 @@ System: mips64el-linux\n") (lambda () (guix-substitute "--substitute"))))) +(define (write-string-as-nar port content) + (write-file-tree "foo" port + #:file-type+size + (lambda _ + (values 'regular + (string-length content))) + #:file-port + (lambda _ + (open-input-string content)))) + (test-equal "substitute, invalid hash" (string-append "hash-mismatch sha256 " (bytevector->nix-base32-string (sha256 #vu8())) " " @@ -331,14 +341,7 @@ System: mips64el-linux\n") (open-hash-port (hash-algorithm sha256))) ((content) "Substitutable data.")) - (write-file-tree "foo" port - #:file-type+size - (lambda _ - (values 'regular - (string-length content))) - #:file-port - (lambda _ - (open-input-string content))) + (write-string-as-nar port content) (close-port port) (bytevector->nix-base32-string (get-hash))) "\n") @@ -367,6 +370,57 @@ System: mips64el-linux\n"))) (lambda () (guix-substitute "--substitute")))))))))) +(test-equal "substitute (unpacked), invalid hash" + (string-append "hash-mismatch sha256 " + (bytevector->nix-base32-string (sha256 #vu8())) " " + (let-values (((port get-hash) + (open-hash-port (hash-algorithm sha256))) + ((content) "Wrong data!")) + (write-string-as-nar port content) + (close-port port) + (bytevector->nix-base32-string (get-hash))) + "\n") + (with-output-to-string + (lambda () + ;; Arrange so the actual data hash does not match the 'NarHash' field in the + ;; narinfo. Use a substituter that does not produce a nar, but rather + ;; writes the item to the store by itself. + (define narinfo + ;; Pretend this hash and size actually correspond to + ;; some nar. + (string-append "StorePath: " (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-wrong-hash +URL: irrelevant +Compression: none +NarHash: sha256:" (bytevector->nix-base32-string (sha256 #vu8())) " +NarSize: 123 +References: +Deriver: " (%store-prefix) "/foo.drv +System: mips64el-linux\n")) + (parameterize ((substituters + (list + (make-substituter + 'test + (lambda (destination . rest) + (call-with-output-file destination + (cut display "Wrong data!" <>)) + 'unpacked) + (const + (list + (string->narinfo + (string-append narinfo "Signature: " + (signature-field narinfo) + "\n") + "test://"))) + '(test)))) + (substitute-urls '("test://"))) + (call-with-temporary-directory + (lambda (directory) + (request-substitution + (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + (string-append directory "/wrong-hash")))))))) + (test-quit "substitute, unauthorized key" "no valid substitute" (with-narinfo (string-append %narinfo "Signature: " -- 2.30.0