From patchwork Thu Dec 16 16:20:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 35280 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id DADB927BBE9; Thu, 16 Dec 2021 16:21:59 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 0748B27BBEA for ; Thu, 16 Dec 2021 16:21:59 +0000 (GMT) Received: from localhost ([::1]:44414 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxtW2-0005Iq-7O for patchwork@mira.cbaines.net; Thu, 16 Dec 2021 11:21:58 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56238) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtV8-0004SD-OX for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53195) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxtV8-0003Ob-Eg for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mxtV8-0003xx-3x for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 1/3] publish: Add ERIS URN to narinfo Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Dec 2021 16:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.163967165715196 (code B ref 52555); Thu, 16 Dec 2021 16:21:02 +0000 Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:20:57 +0000 Received: from localhost ([127.0.0.1]:36502 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV2-0003x1-HG for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:20:56 -0500 Received: from mout01.posteo.de ([185.67.36.65]:37321) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtUx-0003wb-RN for 52555@debbugs.gnu.org; Thu, 16 Dec 2021 11:20:54 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 43FC9240026 for <52555@debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1639671645; bh=c506nOniPbxfWxa5Datyw8oJED0OKeKWWVS7Qozcw10=; h=From:To:Cc:Subject:Date:From; b=GgY5+Cdof2C9QGoRmOaV/LbGZ9OSD/kJwQwxbMx1W50K/MnV3OyMltuBEOI60iAWg nR0Yn7SK+x0Z2b4Epyp3+0gYiVGKkR06lmq2UnI+KYz/gSSr3wwtUD7/f5MFdGxBt8 0BuI1Led8qyeypkADSsiR29ra02FPhd3oFu/TXE5/DiHS6iDjYUzmGpjINGoifaYmV bRtVXXWvp/4GUSzYGqVko3EgfwQnTPt4+tNpjlrhNCmgeoofH1OYeufxysrdBwsOOE zvNVgEOpGA2V9kjDQ+Lm9wSlZV5MdEgaBI0Mv92FojJy83vOrTqJsWZ9iWR+NUVIoB JpsinZnAntYtA== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JFHR01ll4z9rxT; Thu, 16 Dec 2021 17:20:43 +0100 (CET) From: pukkamustard Date: Thu, 16 Dec 2021 16:20:34 +0000 Message-Id: <20211216162036.694-1-pukkamustard@posteo.net> In-Reply-To: <20211216161724.547-1-pukkamustard@posteo.net> References: <20211216161724.547-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars. (narinfo-string): Add #:eris-urns parameter and honor it. (store-item->recutils): Add #:eris-urn parameter and honor it. * guix/scripts/narinfo.scm: ()[eris-urns]: New field. (narinfo-maker): Handle ERIS URN. * configure.ac: (HAVE_GUILE_ERIS): New conditional. * gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris. --- configure.ac | 5 +++++ gnu/packages/package-management.scm | 1 + guix/narinfo.scm | 10 ++++++---- guix/scripts/publish.scm | 31 +++++++++++++++++++++-------- 4 files changed, 35 insertions(+), 12 deletions(-) diff --git a/configure.ac b/configure.ac index 341cff8fbd..72396be8aa 100644 --- a/configure.ac +++ b/configure.ac @@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)]) AM_CONDITIONAL([HAVE_GUILE_AVAHI], [test "x$have_guile_avahi" = "xyes"]) +dnl Check for Guile-eris. +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)]) +AM_CONDITIONAL([HAVE_GUILE_ERIS], + [test "x$have_guile_eris" = "xyes"]) + dnl Guile-newt is used by the graphical installer. GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)]) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 9496499850..5c49167782 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -394,6 +394,7 @@ (define code ("guile-zstd" ,guile-zstd) ("guile-ssh" ,guile-ssh) ("guile-git" ,guile-git) + ("guile-eris" ,guile-eris) ;; XXX: Keep the development inputs here even though ;; they're unnecessary, just so that 'guix environment diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 4fc550aa6c..0972ede3c1 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -67,13 +67,14 @@ (define-module (guix narinfo) equivalent-narinfo?)) (define-record-type - (%make-narinfo path uri-base uris compressions file-sizes file-hashes - nar-hash nar-size references deriver system + (%make-narinfo path uri-base uris eris-urns compressions file-sizes + file-hashes nar-hash nar-size references deriver system signature contents) narinfo? (path narinfo-path) (uri-base narinfo-uri-base) ;URI of the cache it originates from (uris narinfo-uris) ;list of strings + (eris-urns narinfo-eris-urns) ;list of (strings | #f) (compressions narinfo-compressions) ;list of strings (file-sizes narinfo-file-sizes) ;list of (integers | #f) (file-hashes narinfo-file-hashes) @@ -134,7 +135,7 @@ (define (narinfo-signature->canonical-sexp str) (define (narinfo-maker str cache-url) "Return a narinfo constructor for narinfos originating from CACHE-URL. STR must contain the original contents of a narinfo file." - (lambda (path urls compressions file-hashes file-sizes + (lambda (path urls eris-urns compressions file-hashes file-sizes nar-hash nar-size references deriver system signature) "Return a new object." @@ -148,6 +149,7 @@ (define len (length urls)) (string-append cache-url url) (string-append cache-url "/" url))))) urls) + eris-urns compressions (match file-sizes (() (make-list len #f)) @@ -186,7 +188,7 @@ (define* (read-narinfo port #:optional url "FileHash" "FileSize" "NarHash" "NarSize" "References" "Deriver" "System" "Signature") - '("URL" "Compression" "FileSize" "FileHash")))) + '("URL" "ERIS" "Compression" "FileSize" "FileHash")))) (define (narinfo-sha256 narinfo) "Return the sha256 hash of NARINFO as a bytevector, or #f if NARINFO lacks a diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 6e2b4368da..8e4b90789b 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -58,6 +58,7 @@ (define-module (guix scripts publish) #:use-module (guix workers) #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) + #:use-module (eris) #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) @@ -308,7 +309,7 @@ (define* (store-item->recutils store-item #:key (nar-path "nar") (compression %no-compression) - file-size) + file-size eris-urn) "Return the 'Compression' and 'URL' fields of the narinfo for STORE-ITEM, with COMPRESSION, starting at NAR-PATH." (let ((url (encode-and-join-uri-path @@ -319,19 +320,22 @@ (define* (store-item->recutils store-item (($ type) (list (symbol->string type)))) ,(basename store-item))))) - (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]" - url (compression-type compression) file-size))) + (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]~@[ERIS: ~a~%~]" + url (compression-type compression) file-size eris-urn))) (define* (narinfo-string store store-path #:key (compressions (list %no-compression)) - (nar-path "nar") (file-sizes '())) + (nar-path "nar") (file-sizes '()) (eris-urns '())) "Generate a narinfo key/value string for STORE-PATH; an exception is raised if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs. Optionally, FILE-SIZES is a list of compression/integer pairs, where the integer is size in bytes of the compressed NAR; it informs the client of how -much needs to be downloaded." +much needs to be downloaded. + +Optionally, ERIS-URNS is a list of compression/string pairs, where the +string is the ERIS URN of the compressed NAR." (let* ((path-info (query-path-info store store-path)) (compressions (actual-compressions store-path compressions)) (hash (bytevector->nix-base32-string @@ -352,9 +356,12 @@ (define* (narinfo-string store store-path store-path (map (lambda (compression) (let ((size (assoc-ref file-sizes - compression))) + compression)) + (eris-urn (assoc-ref eris-urns + compression))) (store-item->recutils store-path #:file-size size + #:eris-urn eris-urn #:nar-path nar-path #:compression compression))) @@ -632,6 +639,12 @@ (define (compressed-nar-size compression) (and stat (cons compression (stat:size stat))))) + (define (compressed-eris-urn compression) + (let* ((nar (nar-cache-file cache item #:compression compression)) + (stat (stat nar #f))) + (and stat + (cons compression (call-with-input-file nar eris-encode->urn))))) + (let ((compression (actual-compressions item compressions))) (for-each (cut compress-nar cache item <>) compressions) @@ -646,11 +659,13 @@ (define (compressed-nar-size compression) ;; thread's connection to the store since we would end up sending ;; stuff concurrently on the same channel. (with-store store - (let ((sizes (filter-map compressed-nar-size compression))) + (let ((sizes (filter-map compressed-nar-size compression)) + (eris-urns (filter-map compressed-eris-urn compression))) (display (narinfo-string store item #:nar-path nar-path #:compressions compressions - #:file-sizes sizes) + #:file-sizes sizes + #:eris-urns eris-urns) port))) ;; Make the cached narinfo world-readable, contrary to what From patchwork Thu Dec 16 16:20:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 35278 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id CC29327BBEA; Thu, 16 Dec 2021 16:21:30 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 8778F27BBE9 for ; Thu, 16 Dec 2021 16:21:30 +0000 (GMT) Received: from localhost ([::1]:43122 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxtVZ-0004WE-Lr for patchwork@mira.cbaines.net; Thu, 16 Dec 2021 11:21:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56240) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtV9-0004Sz-5X for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53196) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxtV8-0003Ok-Kb for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mxtV8-0003y5-GU for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 2/3] WIP: gnu: guile-eris: Update to unreleased git version. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Dec 2021 16:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.163967165715202 (code B ref 52555); Thu, 16 Dec 2021 16:21:02 +0000 Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:20:57 +0000 Received: from localhost ([127.0.0.1]:36504 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV3-0003x3-0g for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:20:57 -0500 Received: from mout01.posteo.de ([185.67.36.65]:44809) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV0-0003wg-C9 for 52555@debbugs.gnu.org; Thu, 16 Dec 2021 11:20:55 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id C8A4A240027 for <52555@debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1639671648; bh=mzF6Dcz6gBsxGgNgXV2yX2To4wOq88FwLdvP0HAzkYo=; h=From:To:Cc:Subject:Date:From; b=n7IfKTN8HDGAo6xEuyFQh2PJ1Kld/Sj/lviOXqQSe/U0XMXzbw0JqTAzPeHhyH2W+ frlB6+zNnf2ug2LBhzBLm+wPFER2hxpmv2TO2DN+nSS6SoCHOTNFVWNu9VhFs5DY2J AMR650+puRcRgAGAQk59mNxHhwGW5FMrTpQoq549I2fZFmf7kIS+/fgapxwxKLIGu9 er1LA+mbm1F1JmBR/oz6HPjkZBFi8lMew85MZRatBy5ZMzqIa9N+W1Q0dnpbRiO8Qw KXAmAprHuQxwUlkG+3dJcIO9IZNW7mDpGjwhesz9C9uWq7x8K41KJwjdMN3vxddy/k 6YE9/vRp4Ep9Q== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JFHR400WSz9rxX; Thu, 16 Dec 2021 17:20:47 +0100 (CET) From: pukkamustard Date: Thu, 16 Dec 2021 16:20:35 +0000 Message-Id: <20211216162036.694-2-pukkamustard@posteo.net> In-Reply-To: <20211216162036.694-1-pukkamustard@posteo.net> References: <20211216161724.547-1-pukkamustard@posteo.net> <20211216162036.694-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version. [source]: Update source URI. [propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi. --- gnu/packages/guile-xyz.scm | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm index 8346d99996..5e6f31a6e2 100644 --- a/gnu/packages/guile-xyz.scm +++ b/gnu/packages/guile-xyz.scm @@ -4325,15 +4325,15 @@ (define-public guile-sodium (define-public guile-eris (package (name "guile-eris") - (version "0.2.0") + (version "f1e4dd87988f9a80b05a8051d7f5ba3daf79dcc1") (source (origin (method git-fetch) (uri (git-reference - (url "https://inqlab.net/git/eris.git") - (commit (string-append "v" version)))) + (url "https://inqlab.net/git/guile-eris.git") + (commit version))) (file-name (git-file-name name version)) - (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka")))) + (sha256 (base32 "0kgm4b4qn2s74wjvxy273gdi1l1m81i2k4kkk1zc6vlcg3np7p06")))) (build-system gnu-build-system) (arguments '()) (native-inputs @@ -4345,7 +4345,7 @@ (define-public guile-eris guile-srfi-180)) (inputs (list guile-3.0)) (propagated-inputs - (list guile-sodium)) + (list guile-sodium guile-json-4 guile-gdbm-ffi)) (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)") (description "Guile-ERIS is the reference implementation of the Encoding for Robust From patchwork Thu Dec 16 16:20:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 35279 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 87CF027BBEA; Thu, 16 Dec 2021 16:21:50 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id C43F827BBE9 for ; Thu, 16 Dec 2021 16:21:49 +0000 (GMT) Received: from localhost ([::1]:43918 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxtVs-0004xY-VH for patchwork@mira.cbaines.net; Thu, 16 Dec 2021 11:21:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56242) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtV9-0004TE-B6 for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53197) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxtV8-0003Oq-WE for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mxtV8-0003yC-SX for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 3/3] publish: Add IPFS support. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Dec 2021 16:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.163967166115218 (code B ref 52555); Thu, 16 Dec 2021 16:21:02 +0000 Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:21:01 +0000 Received: from localhost ([127.0.0.1]:36508 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV6-0003xO-Dp for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:21:00 -0500 Received: from mout01.posteo.de ([185.67.36.65]:34165) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV2-0003wj-VK for 52555@debbugs.gnu.org; Thu, 16 Dec 2021 11:20:57 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 66D91240026 for <52555@debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1639671651; bh=8FUfzH+H19v7PInjtcmZ9loRZ+gv+GSpvf6oOoA0eR4=; h=From:To:Cc:Subject:Date:From; b=q3677HhT4ire8v1On3rXXk2hHSHVeTfXOJJMYNfGarL65V+/ZMjqP0fAYBPzROczE eas83QDMVCfP6Cru45FGGM4PktcKU+vVuCpRh3MmCqINvg7z46nVn8nJVxhE3r3HrW 2KZZ2vrzt8gZYeVgqymH+gIqvJnG31N8yTe2L8JhgEe4TBxqd2gM4hjE9Re42GZpxx TTVrU15muau8PE+/Z3KNb3VoOrRor3jJvJ0j8aJRHpycDpvEaoOkmAy9S9E9Ho1LDL BWQY3CgIy8F7qnsaDG7ODIY1al8FAVlboSijotsprngtlipcaQeOfTo4blayBSJMgu 1nMOmikrWOjTA== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JFHR64dFHz9rxS; Thu, 16 Dec 2021 17:20:50 +0100 (CET) From: pukkamustard Date: Thu, 16 Dec 2021 16:20:36 +0000 Message-Id: <20211216162036.694-3-pukkamustard@posteo.net> In-Reply-To: <20211216162036.694-1-pukkamustard@posteo.net> References: <20211216161724.547-1-pukkamustard@posteo.net> <20211216162036.694-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/scripts/publish.scm: (show-help, %options): Add '--ipfs'. (render-narinfo/cached, bake-narinfo+nar, make-request-handler, run-publish-server): Add #:ipfs? and honor it. (guix-publish): Honor '--ipfs' and parameterize %ipfs-base-url. --- guix/scripts/publish.scm | 52 +++++++++++++++++++++++++++++++--------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 8e4b90789b..8e7fb47b9e 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -41,6 +41,8 @@ (define-module (guix scripts publish) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-37) + #:use-module (srfi srfi-71) + #:use-module (srfi srfi-171) #:use-module (web http) #:use-module (web request) #:use-module (web response) @@ -52,6 +54,7 @@ (define-module (guix scripts publish) #:use-module (guix base64) #:use-module (guix config) #:use-module (guix derivations) + #:use-module ((guix ipfs) #:prefix ipfs:) #:use-module (gcrypt hash) #:use-module (guix pki) #:use-module (gcrypt pk-crypto) @@ -59,6 +62,8 @@ (define-module (guix scripts publish) #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) #:use-module (eris) + #:use-module (eris read-capability) + #:use-module (eris blocks ipfs) #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) @@ -83,6 +88,7 @@ (define-module (guix scripts publish) run-publish-server guix-publish)) + (define (show-help) (format #t (G_ "Usage: guix publish [OPTION]... Publish ~a over HTTP.\n") %store-directory) @@ -102,6 +108,8 @@ (define (show-help) (display (G_ " --cache-bypass-threshold=SIZE serve store items below SIZE even when not cached")) + (display (G_ " + --ipfs[=GATEWAY] publish items over IPFS via GATEWAY")) (display (G_ " --workers=N use N workers to bake items")) (display (G_ " @@ -220,6 +228,10 @@ (define %options (lambda (opt name arg result) (alist-cons 'cache-bypass-threshold (size->number arg) result))) + (option '("ipfs") #f #t + (lambda (opt name arg result) + (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url)) + result))) (option '("workers") #t #f (lambda (opt name arg result) (alist-cons 'workers (string->number* arg) @@ -526,7 +538,7 @@ (define (bypass-cache? store item) (define* (render-narinfo/cached store request hash #:key ttl (compressions (list %no-compression)) (nar-path "nar") negative-ttl - cache pool) + cache pool ipfs?) "Respond to the narinfo request for REQUEST. If the narinfo is available in CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo requested using POOL." @@ -571,7 +583,8 @@ (define (delete-entry narinfo) (bake-narinfo+nar cache item #:ttl ttl #:compressions compressions - #:nar-path nar-path))) + #:nar-path nar-path + #:ipfs? ipfs?))) (when ttl (single-baker 'cache-cleanup @@ -631,7 +644,7 @@ (define (write-compressed-file call-with-compressed-output-port) (define* (bake-narinfo+nar cache item #:key ttl (compressions (list %no-compression)) - (nar-path "/nar")) + (nar-path "/nar") ipfs?) "Write the narinfo and nar for ITEM to CACHE." (define (compressed-nar-size compression) (let* ((nar (nar-cache-file cache item #:compression compression)) @@ -641,9 +654,19 @@ (define (compressed-nar-size compression) (define (compressed-eris-urn compression) (let* ((nar (nar-cache-file cache item #:compression compression)) - (stat (stat nar #f))) + (stat (stat nar #f)) + (block-reducer (if ipfs? + (eris-blocks-ipfs-reducer + #:ipfs-base-url (ipfs:%ipfs-base-url)) + rcount))) (and stat - (cons compression (call-with-input-file nar eris-encode->urn))))) + (cons compression + (call-with-input-file nar + (lambda (port) + (let ((read-cap _ + (eris-encode port #:block-reducer + block-reducer))) + (read-capability->string read-cap)))))))) (let ((compression (actual-compressions item compressions))) @@ -1115,7 +1138,8 @@ (define* (make-request-handler store cache pool narinfo-ttl narinfo-negative-ttl (nar-path "nar") - (compressions (list %no-compression))) + (compressions (list %no-compression)) + ipfs?) (define compression-type? string->compression-type) @@ -1147,7 +1171,8 @@ (define (handle request body) #:ttl narinfo-ttl #:negative-ttl narinfo-negative-ttl #:nar-path nar-path - #:compressions compressions) + #:compressions compressions + #:ipfs? ipfs?) (render-narinfo store request hash #:ttl narinfo-ttl #:negative-ttl narinfo-negative-ttl @@ -1218,7 +1243,7 @@ (define* (run-publish-server socket store advertise? port (compressions (list %no-compression)) (nar-path "nar") narinfo-ttl narinfo-negative-ttl - cache pool) + cache pool ipfs?) (when advertise? (let ((name (service-name))) ;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a @@ -1234,7 +1259,8 @@ (define* (run-publish-server socket store #:nar-path nar-path #:narinfo-ttl narinfo-ttl #:narinfo-negative-ttl narinfo-negative-ttl - #:compressions compressions) + #:compressions compressions + #:ipfs? ipfs?) concurrent-http-server `(#:socket ,socket))) @@ -1296,6 +1322,8 @@ (define-command (guix-publish . args) (repl-port (assoc-ref opts 'repl)) (cache (assoc-ref opts 'cache)) (workers (assoc-ref opts 'workers)) + (ipfs (assoc-ref opts 'ipfs)) + (ipfs? (if ipfs #t #f)) ;; Read the key right away so that (1) we fail early on if we can't ;; access them, and (2) we can then drop privileges. @@ -1315,7 +1343,8 @@ (define-command (guix-publish . args) (%private-key private-key) (cache-bypass-threshold (or (assoc-ref opts 'cache-bypass-threshold) - (cache-bypass-threshold)))) + (cache-bypass-threshold))) + (ipfs:%ipfs-base-url ipfs)) (info (G_ "publishing ~a on ~a, port ~d~%") %store-directory (inet-ntop (sockaddr:fam address) (sockaddr:addr address)) @@ -1344,7 +1373,8 @@ (define-command (guix-publish . args) #:nar-path nar-path #:compressions compressions #:narinfo-negative-ttl negative-ttl - #:narinfo-ttl ttl)))))) + #:narinfo-ttl ttl + #:ipfs? ipfs?)))))) ;;; Local Variables: ;;; eval: (put 'single-baker 'scheme-indent-function 1)