From patchwork Thu Dec 29 18:13:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45688 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A4D9827BBED; Thu, 29 Dec 2022 19:12:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id D574A27BBE9 for ; Thu, 29 Dec 2022 19:12:18 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQn-0000BP-W8; Thu, 29 Dec 2022 13:15:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQk-0000A0-W3 for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQk-00042u-2J for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQj-0007R7-UN for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 1/8] publish: Add ERIS URN to narinfo. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767028459 (code B ref 52555); Thu, 29 Dec 2022 18:15:01 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:30 +0000 Received: from localhost ([127.0.0.1]:32798 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQD-0007Og-BY for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:30 -0500 Received: from mout02.posteo.de ([185.67.36.66]:34393) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQB-0007OG-Mo for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:28 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 1F77C2401A9 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337662; bh=lrwWE98dyL4vG9CQTOKjY0FM2je3h7b72QM1GOagd7I=; h=From:To:Cc:Subject:Date:From; b=rY/0OHumeRqmQHm9ycrIyREYwhfVM4WFZHzDCG5fb7wY7E7FLI9NjGqFeCIRkxTFP 7opk0CVeyfn9FQc3ITWWnwdXqAe3yfNkL6CQTpxqzckPElDbPWni1mkx+ffjW7pWXo qL7tCFc7e7rjpyxh7ZVNn3iiUyMyhLJ+BgyhQecUCsVbSKWsvLLvecmC48xZjMa+Ma BboQRiziZ1Pqt/Kebv5WIKp8m+Yb0LmQ3Xet8DLxQx+p9/yL8j3Dj5wn8DkCc9zenx E9UAbBOqX1FQxCKW/5nkBesBl2uQ9t/kZxBIN7ZsHTa9+qQh1U+pfTI6PkYEG5doSJ OFPXpbZR0hu4A== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3d3ZBBz6tmY; Thu, 29 Dec 2022 19:14:21 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:20 +0000 Message-Id: <20221229181327.758-2-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars. (narinfo-string): Add #:eris-urn parameter and honor it. * guix/scripts/narinfo.scm: ()[eris-format,eris-urn]: New fields. (narinfo-maker): Handle ERIS URN and ERIS format. * configure.ac: (HAVE_GUILE_ERIS): New conditional. * gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris. --- configure.ac | 5 +++++ gnu/packages/package-management.scm | 1 + guix/narinfo.scm | 16 +++++++++----- guix/scripts/publish.scm | 34 ++++++++++++++++++++++++----- 4 files changed, 46 insertions(+), 10 deletions(-) diff --git a/configure.ac b/configure.ac index 92dede8014..52a37a6113 100644 --- a/configure.ac +++ b/configure.ac @@ -173,6 +173,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)]) AM_CONDITIONAL([HAVE_GUILE_AVAHI], [test "x$have_guile_avahi" = "xyes"]) +dnl Check for Guile-eris. +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)]) +AM_CONDITIONAL([HAVE_GUILE_ERIS], + [test "x$have_guile_eris" = "xyes"]) + dnl Guile-newt is used by the graphical installer. GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)]) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 2ffaa12247..3585f6c073 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -419,6 +419,7 @@ (define code ("guile-zstd" ,guile-zstd) ("guile-ssh" ,guile-ssh) ("guile-git" ,guile-git) + ("guile-eris" ,guile-eris) ;; XXX: Keep the development inputs here even though ;; they're unnecessary, just so that 'guix environment diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 741c7ad406..d7f8c88f8f 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -45,6 +45,8 @@ (define-module (guix narinfo) narinfo-file-sizes narinfo-hash narinfo-size + narinfo-eris-format + narinfo-eris-urn narinfo-references narinfo-deriver narinfo-system @@ -68,8 +70,8 @@ (define-module (guix narinfo) (define-record-type (%make-narinfo path uri-base uris compressions file-sizes file-hashes - nar-hash nar-size references deriver system - signature contents) + nar-hash nar-size eris-format eris-urn references deriver + system signature contents) narinfo? (path narinfo-path) (uri-base narinfo-uri-base) ;URI of the cache it originates from @@ -79,6 +81,8 @@ (define-record-type (file-hashes narinfo-file-hashes) (nar-hash narinfo-hash) (nar-size narinfo-size) + (eris-format narinfo-eris-format) + (eris-urn narinfo-eris-urn) (references narinfo-references) (deriver narinfo-deriver) (system narinfo-system) @@ -135,8 +139,8 @@ (define (narinfo-maker str cache-url) "Return a narinfo constructor for narinfos originating from CACHE-URL. STR must contain the original contents of a narinfo file." (lambda (path urls compressions file-hashes file-sizes - nar-hash nar-size references deriver system - signature) + nar-hash nar-size eris-format eris-urn references deriver + system signature) "Return a new object." (define len (length urls)) (%make-narinfo path cache-url @@ -157,6 +161,8 @@ (define len (length urls)) ((lst ...) (map string->number lst))) nar-hash (and=> nar-size string->number) + eris-format + (if eris-urn (string->uri eris-urn) #f) (string-tokenize references) (match deriver ((or #f "") #f) @@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url (narinfo-maker str url) '("StorePath" "URL" "Compression" "FileHash" "FileSize" "NarHash" "NarSize" - "References" "Deriver" "System" + "ERISFormat" "ERIS" "References" "Deriver" "System" "Signature") '("URL" "Compression" "FileSize" "FileHash")))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 3bf3bd9c7c..76b5a429f4 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -40,6 +40,7 @@ (define-module (guix scripts publish) #:use-module (srfi srfi-34) #:use-module (srfi srfi-37) #:use-module (srfi srfi-71) + #:use-module (srfi srfi-171) #:use-module (web http) #:use-module (web request) #:use-module (web response) @@ -57,6 +58,7 @@ (define-module (guix scripts publish) #:use-module (guix workers) #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) + #:use-module (eris) #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) @@ -145,6 +147,9 @@ (define %default-gzip-compression ;; Since we compress on the fly, default to fast compression. (compression 'gzip 3)) +(define %eris-zstd-compression + (compression 'zstd 19)) + (define (default-compression type) (compression type 3)) @@ -323,7 +328,8 @@ (define* (store-item->recutils store-item (define* (narinfo-string store store-path #:key (compressions (list %no-compression)) - (nar-path "nar") (file-sizes '())) + (nar-path "nar") (file-sizes '()) + eris-urn) "Generate a narinfo key/value string for STORE-PATH; an exception is raised if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs. @@ -345,10 +351,10 @@ (define* (narinfo-string store store-path "\ StorePath: ~a NarHash: sha256:~a -NarSize: ~d +NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~] References: ~a~%" store-path - hash size references)) + hash size eris-urn references)) ;; Do not render a "Deriver" line if we are rendering info for a ;; derivation. Also do not render a "System" line that would be ;; expensive to compute and is currently unused. @@ -633,6 +639,22 @@ (define (compressed-nar-size compression) (and stat (cons compression (stat:size stat))))) + (define (eris-encode-nar compressions) + (and (member %eris-zstd-compression compressions) + (let* ((nar (nar-cache-file cache item + #:compression %eris-zstd-compression)) + (stat (stat nar #f))) + (and stat + (call-with-input-file nar + (lambda (port) + (let ((eris-urn _ + (eris-encode port + #:block-size 'large + #:block-reducer rcount + #:convergence-secret + %null-convergence-secret))) + eris-urn))))))) + (let ((compression (actual-compressions item compressions))) (for-each (cut compress-nar cache item <>) compressions) @@ -640,7 +662,8 @@ (define (compressed-nar-size compression) (match compressions ((main others ...) (let ((narinfo (narinfo-cache-file cache item - #:compression main))) + #:compression main)) + (eris-urn (eris-encode-nar compressions))) (with-atomic-file-output narinfo (lambda (port) ;; Open a new connection to the store. We cannot reuse the main @@ -651,7 +674,8 @@ (define (compressed-nar-size compression) (display (narinfo-string store item #:nar-path nar-path #:compressions compressions - #:file-sizes sizes) + #:file-sizes sizes + #:eris-urn eris-urn) port))) ;; Make the cached narinfo world-readable, contrary to what From patchwork Thu Dec 29 18:13:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45682 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 25E7A27BBEB; Thu, 29 Dec 2022 18:49:03 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id AC57927BBE9 for ; Thu, 29 Dec 2022 18:49:00 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQl-0000AP-JM; Thu, 29 Dec 2022 13:15:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQk-0000A3-Om for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQk-00042z-Ft for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQk-0007RH-AF for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 2/8] publish: Store ERIS encoded blocks to a local block store. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767628487 (code B ref 52555); Thu, 29 Dec 2022 18:15:02 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:36 +0000 Received: from localhost ([127.0.0.1]:32804 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQJ-0007PO-MS for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:36 -0500 Received: from mout02.posteo.de ([185.67.36.66]:58729) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQB-0007OH-UO for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:29 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 6ECD92401BC for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337662; bh=uHwlg5kIjSso0T2R2trMFAAlm0ymvpLunCNwcfsNJy8=; h=From:To:Cc:Subject:Date:From; b=f38MZYaBgd5YNNv18miJEWnShK3EJs5B0CbbFOj9w4nqv9rTNVhBpExcFgZvrT6B0 UFY+xOV6pYBdM3e2WxrZCTdGbGSzi19ov2hs+JTD6XaakFlykMD2os6btN1z91ugsL J1QxQ10LrBCdpMrxVOXX3DJQ+Yby1Y70bupb6fkHixnSn+EbRZcoXj43lUuoictbew VPI/mOVnTMFAIUl62tzromgjAhLdlMO4wCpU/VYhzz196YXi2awwrXU6ZA744NwuH4 7W8P34QUq0e5jkAE2SNdzAdC25hV6W7hrxATWjgvToPQv5wzWOa5IMzAUDOiCPTgIj 7/Li8N4ZNeeBg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3d6jzwz6tmY; Thu, 29 Dec 2022 19:14:21 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:21 +0000 Message-Id: <20221229181327.758-3-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/eris.scm: New file. * guix/eris/fs-store.scm: New file. * Makefile.am (MODULES): Add new files. * guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer. --- Makefile.am | 2 ++ guix/eris.scm | 36 +++++++++++++++++++++ guix/eris/fs-store.scm | 67 ++++++++++++++++++++++++++++++++++++++++ guix/scripts/publish.scm | 14 +++++---- 4 files changed, 113 insertions(+), 6 deletions(-) create mode 100644 guix/eris.scm create mode 100644 guix/eris/fs-store.scm diff --git a/Makefile.am b/Makefile.am index b54288c0fc..c549fc8580 100644 --- a/Makefile.am +++ b/Makefile.am @@ -132,6 +132,8 @@ MODULES = \ guix/least-authority.scm \ guix/read-print.scm \ guix/ipfs.scm \ + guix/eris.scm \ + guix/eris/fs-store.scm \ guix/platform.scm \ guix/platforms/arm.scm \ guix/platforms/mips.scm \ diff --git a/guix/eris.scm b/guix/eris.scm new file mode 100644 index 0000000000..29d5e7b1db --- /dev/null +++ b/guix/eris.scm @@ -0,0 +1,36 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix eris) + #:use-module (eris) + + #:use-module (guix config) + #:use-module (guix eris fs-store) + + #:export (guix-eris-block-reducer + + %eris-block-store-directory)) + +(define %eris-block-store-directory + (make-parameter + (or (getenv "GUIX_ERIS_BLOCK_STORE_DIRECTORY") + (string-append %state-directory "/eris")))) + +(define (guix-eris-block-reducer) + "Returns a block reducer that stores blocks of ERIS encoded content." + (eris-fs-store-reducer (%eris-block-store-directory))) diff --git a/guix/eris/fs-store.scm b/guix/eris/fs-store.scm new file mode 100644 index 0000000000..2ef7607988 --- /dev/null +++ b/guix/eris/fs-store.scm @@ -0,0 +1,67 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix eris fs-store) + #:use-module (rnrs io ports) + #:use-module (guix build utils) ; for mkdir-p + #:use-module (eris utils base32) + + #:export (eris-fs-store-reducer + eris-fs-store-ref)) + +;;; Commentary: +;;; +;;; This module provides a file-system based store of ERIS encoded blocks. +;;; +;;; Code: + +(define (eris-fs-store-reducer store-directory) + (case-lambda + (() (mkdir-p store-directory)) + + ((result) result) + + ((_ ref-block) + (let* ((ref (car ref-block)) + (b32 (base32-encode ref)) + (pre (substring b32 0 2)) + (suf (substring b32 2)) + (pre-dir (string-append store-directory "/" pre)) + (path (string-append pre-dir "/" suf)) + (block (cdr ref-block))) + + (mkdir-p pre-dir) + + (unless (file-exists? path) + (call-with-output-file path + (lambda (port) (put-bytevector port block)) + #:binary #t)) + + #t)))) + +(define (eris-fs-store-ref store-directory) + (lambda (ref) + (let* ((b32 (base32-encode ref)) + (pre (substring b32 0 2)) + (suf (substring b32 2)) + (path (string-append store-directory "/" pre "/" suf))) + (if (file-exists? path) + (call-with-input-file path + (lambda (port) (get-bytevector-all port)) + #:binary #t) + #f)))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 76b5a429f4..7f14e4d4d4 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -52,6 +52,7 @@ (define-module (guix scripts publish) #:use-module (guix base64) #:use-module (guix config) #:use-module (guix derivations) + #:use-module (guix eris) #:use-module (gcrypt hash) #:use-module (guix pki) #:use-module (gcrypt pk-crypto) @@ -647,12 +648,13 @@ (define (eris-encode-nar compressions) (and stat (call-with-input-file nar (lambda (port) - (let ((eris-urn _ - (eris-encode port - #:block-size 'large - #:block-reducer rcount - #:convergence-secret - %null-convergence-secret))) + (let ((eris-urn + _ (eris-encode port + #:block-size 'large + #:block-reducer + (guix-eris-block-reducer) + #:convergence-secret + %null-convergence-secret))) eris-urn))))))) (let ((compression (actual-compressions item compressions))) From patchwork Thu Dec 29 18:13:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45685 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E57DF27BBED; Thu, 29 Dec 2022 18:54:32 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 8D3BB27BBE9 for ; Thu, 29 Dec 2022 18:54:31 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQp-0000CD-Fj; Thu, 29 Dec 2022 13:15:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQl-0000AH-6k for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQk-000460-TW for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQk-0007RQ-PA for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 3/8] publish: Add HTTP endpoint for resolving ERIS blocks. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767628495 (code B ref 52555); Thu, 29 Dec 2022 18:15:02 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:36 +0000 Received: from localhost ([127.0.0.1]:32806 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQK-0007PQ-AR for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:36 -0500 Received: from mout02.posteo.de ([185.67.36.66]:36003) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQC-0007OI-DL for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:29 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id E41B62401A9 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337662; bh=7ucG2aEORDt0EkiJ7tlZhrXCVtV20NuUyG5fyIXtlLw=; h=From:To:Cc:Subject:Date:From; b=UeM+QlMO0JxAbCTDMB9I+C5ReuvbFqqnir/JNRT7DwV0sNGJXBqtDEwbS4F+Ejvp2 StQ5lWzKwGOSYWuwl11l2mcaTnWpJRcS0vNrVlNJi7/gKZRcKx6p6b9lgQfkiFzQC1 EbFm3DEHSJU7ZVf32t2im+lw7mEFgET5VHHbLtcDf/DxD46x/ONvT5p6rvmgz7EgXX /Ra2FXjtFGRvR4B0j+OnUioR9Y0WLDJsSTMXf1k4q1KWFz8dIN8nkcIrj9eVm3wOyd RspPmI2m0JbPJHHoZ9epxs58wg7tJsgPNR5GAKJps04/tzaCRADM3t8s8DRoaya5cE Dy6OrJaPf+eww== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3f2lbTz6tmY; Thu, 29 Dec 2022 19:14:22 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:22 +0000 Message-Id: <20221229181327.758-4-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/eris/http.scm: New file. * Makefile.am (MODULES): Add it. * guix/scripts/publish.scm (make-request-handler): Add handler for RFC 2169 URN resolution endpoint. --- Makefile.am | 1 + guix/eris/http.scm | 40 ++++++++++++++++++++++++++++++++++++++++ guix/scripts/publish.scm | 20 ++++++++++++++++++++ 3 files changed, 61 insertions(+) create mode 100644 guix/eris/http.scm diff --git a/Makefile.am b/Makefile.am index c549fc8580..373f6b7c27 100644 --- a/Makefile.am +++ b/Makefile.am @@ -134,6 +134,7 @@ MODULES = \ guix/ipfs.scm \ guix/eris.scm \ guix/eris/fs-store.scm \ + guix/eris/http.scm \ guix/platform.scm \ guix/platforms/arm.scm \ guix/platforms/mips.scm \ diff --git a/guix/eris/http.scm b/guix/eris/http.scm new file mode 100644 index 0000000000..a8a9520197 --- /dev/null +++ b/guix/eris/http.scm @@ -0,0 +1,40 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix eris http) + #:use-module (eris utils base32) + #:use-module (web uri) + + #:use-module (srfi srfi-2) + + #:export (make-blake2b-urn-query-ref)) + +(define blake2b-uri-path-regexp + (make-regexp "^blake2b:" regexp/icase)) + +(define (make-blake2b-urn-query-ref block-ref) + (lambda (query) + (and-let* ((_ query) ; ensure query is not false + (urn (string->uri query)) + (_ (uri? urn)) + (_ (eqv? (uri-scheme urn) 'urn)) + (_ (regexp-exec blake2b-uri-path-regexp + (uri-path urn))) + (blake2b-ref (base32-decode + (string-drop (uri-path urn) 8)))) + (block-ref blake2b-ref)))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 7f14e4d4d4..15bdf02670 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -53,6 +53,8 @@ (define-module (guix scripts publish) #:use-module (guix config) #:use-module (guix derivations) #:use-module (guix eris) + #:use-module (guix eris fs-store) + #:use-module (guix eris http) #:use-module (gcrypt hash) #:use-module (guix pki) #:use-module (gcrypt pk-crypto) @@ -1083,6 +1085,14 @@ (define nar-path? (let ((expected (split-and-decode-uri-path nar-path))) (cut equal? expected <>))) + ;; Get ERIS blocks directly from the filesystem store. + (define eris-block-ref + (eris-fs-store-ref (%eris-block-store-directory))) + + ;; Create a handler for resolving blake2b URN queries. + (define blake2b-urn-query-ref + (make-blake2b-urn-query-ref eris-block-ref)) + (define (handle request body) (format #t "~a ~a~%" (request-method request) @@ -1125,6 +1135,16 @@ (define (handle request body) (("log" name) (render-log-file store request name)) + ;; /uri-res/N2R - RFC2169 URN resolution + (("uri-res" "N2R") + (let ((block (blake2b-urn-query-ref + (uri-query (request-uri request))))) + (if block + (values `((content-type . (application/octet-stream + (charset . "ISO-8859-1")))) + block) + (not-found request)))) + ;; Use different URLs depending on the compression type. This ;; guarantees that /nar URLs remain valid even when 'guix publish' ;; is restarted with different compression parameters. From patchwork Thu Dec 29 18:13:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45684 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4045227BBEB; Thu, 29 Dec 2022 18:52:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B91A727BBE9 for ; Thu, 29 Dec 2022 18:52:17 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQo-0000C2-BS; Thu, 29 Dec 2022 13:15:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQl-0000Ar-Vi for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQl-0004Cs-MZ for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQl-0007Rf-I4 for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 4/8] WIP: substitute: Fetch substitutes using ERIS. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767728510 (code B ref 52555); Thu, 29 Dec 2022 18:15:03 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:37 +0000 Received: from localhost ([127.0.0.1]:32810 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQL-0007Pf-6l for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:37 -0500 Received: from mout01.posteo.de ([185.67.36.65]:54713) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQD-0007OK-34 for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:30 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 54D32240149 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337663; bh=HALOp6BmLbWkF/uCoyDNMO6MNjP132bCXeHMxPGwckE=; h=From:To:Cc:Subject:Date:From; b=M9s4m7gqOxDtn1oXZWShSq/dECdvB4UuAcJivvUt4Lgh+/JAMhn0rPDOXrILXnR6E akjsOL1WZGk4ob7R6pHkfkjdh9UCiWAzbW88qZqoE7WVTDG2OutBQEivnJjdkbQ5ox de0LfvMGr2myvEMwb8G2e2E+XQiMUjYt3ronDb0K0yw/G+8FNcwYqSU3zDTce74FQ5 zC1Jz9xVH8Kg8zALAGZr6gdJZhe/awXoBnbYY135OLf/uSNzF9FgaqOkNX4FcwulIx N55XqyozUIgaRn2T6RCDyy27zxYldju+PIjb2HYxhRHyVJ2sHQMd2SxSXXBveShYJ6 xpyP/PHugameQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3f5tmNz6tn8; Thu, 29 Dec 2022 19:14:22 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:23 +0000 Message-Id: <20221229181327.758-5-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS. * guix/eris.scm (call-with-eris-block-ref): New procedure. TODO: - When to set prefer-eris? in scripts/substitute.scm? --- guix/eris.scm | 7 +++++++ guix/scripts/substitute.scm | 42 +++++++++++++++++++++++++++++-------- 2 files changed, 40 insertions(+), 9 deletions(-) diff --git a/guix/eris.scm b/guix/eris.scm index 29d5e7b1db..d9a0914b67 100644 --- a/guix/eris.scm +++ b/guix/eris.scm @@ -23,6 +23,7 @@ (define-module (guix eris) #:use-module (guix eris fs-store) #:export (guix-eris-block-reducer + call-with-eris-block-ref %eris-block-store-directory)) @@ -34,3 +35,9 @@ (define %eris-block-store-directory (define (guix-eris-block-reducer) "Returns a block reducer that stores blocks of ERIS encoded content." (eris-fs-store-reducer (%eris-block-store-directory))) + +(define (call-with-eris-block-ref f) + (let ((fs-store-block-ref + (eris-fs-store-ref + (%eris-block-store-directory)))) + (f fs-store-block-ref))) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 0efa61b0d7..8cf011d7e6 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -55,12 +55,16 @@ (define-module (guix scripts substitute) #:use-module (ice-9 ftw) #:use-module (rnrs bytevectors) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-2) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) #:use-module (srfi srfi-71) #:use-module (web uri) + #:use-module (eris) + #:use-module (eris read-capability) + #:use-module (guix eris) #:use-module (guix http-client) #:export (%allow-unauthenticated-substitutes? %reply-file-descriptor @@ -439,11 +443,12 @@ (define-syntax-rule (with-cached-connection uri port exp ...) (define* (download-nar narinfo destination #:key status-port - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + prefer-eris?) "Download the nar prescribed in NARINFO, which is assumed to be authentic and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and -if DESTINATION is in the store, deduplicate its files. Print a status line to -STATUS-PORT." +if DESTINATION is in the store, deduplicate its files. When PREFER-ERIS? is +true, attempt to ERIS to get the nar. Print a status line to STATUS-PORT." (define destination-in-store? (string-prefix? (string-append (%store-prefix) "/") destination)) @@ -474,14 +479,29 @@ (define (fetch uri) #:port port #:keep-alive? #t #:buffered? #f)))) + ((urn) + (let ((read-capability (->eris-read-capability uri))) + (if (eris-read-capability? read-capability) + (call-with-eris-block-ref + (lambda (block-ref) + (values (open-eris-input-port + read-capability + #:block-ref block-ref) + #f))) + (leave (G_ "unsupported substitute URI scheme: ~a~%") + (uri->string uri))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) (let ((uri compression file-size - (narinfo-best-uri narinfo - #:fast-decompression? - %prefer-fast-decompression?))) + (if (and prefer-eris? (narinfo-eris-urn narinfo)) + (values (narinfo-eris-urn narinfo) "zstd" #f) + (narinfo-best-uri narinfo + #:fast-decompression? + %prefer-fast-decompression?)))) + + (unless print-build-trace? (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) @@ -631,7 +651,8 @@ (define* (process-substitution/fallback port narinfo destination (define* (process-substitution port store-item destination #:key cache-urls acl - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + prefer-eris?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL, and verify its hash against what appears in the narinfo. When DEDUPLICATE? is true, and if @@ -660,7 +681,8 @@ (define narinfo (download-nar narinfo destination #:status-port port #:deduplicate? deduplicate? - #:print-build-trace? print-build-trace?))) + #:print-build-trace? print-build-trace? + #:prefer-eris? prefer-eris?))) ;;; @@ -858,7 +880,9 @@ (define reply-port #:acl (current-acl) #:deduplicate? deduplicate? #:print-build-trace? - print-build-trace?) + print-build-trace? + ;; TODO when to prefer ERIS? + #:prefer-eris? #t) (loop)))))) (opts (leave (G_ "~a: unrecognized options~%") opts)))))) From patchwork Thu Dec 29 18:13:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45683 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id AEB8727BBED; Thu, 29 Dec 2022 18:51:34 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id CD83927BBE9 for ; Thu, 29 Dec 2022 18:51:33 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQp-0000CB-9O; Thu, 29 Dec 2022 13:15:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQl-0000AX-Jq for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQl-00049B-9F for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQl-0007RX-4z for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 5/8] eris/http: Add HTTP block de-referencer. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767728502 (code B ref 52555); Thu, 29 Dec 2022 18:15:03 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:37 +0000 Received: from localhost ([127.0.0.1]:32808 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQK-0007PY-RI for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:37 -0500 Received: from mout02.posteo.de ([185.67.36.66]:50291) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQD-0007OL-CV for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:30 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id DB8E42401A9 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337663; bh=HxZoZL8oLhJfoeETpauNq/MdOkf5vrtFLoMKrJn87GI=; h=From:To:Cc:Subject:Date:From; b=ewumwp+ApQe2kiWg1LfwTDiMJ6mIVkX7HK4Aw3MjSp8a80vzihaVkq439No4puVsY 1nCHRn1Do7kHXHI9NACaWpRU6VSyLQW65QXWaqg/HVGEZy2SbTP9fWtnQw5Mr1DBLO MEVh2SV8t2RwjR/NIar3ET+KRNiHbShR1aWeOyyknF2OKJPYpe+7j+Hjp6ObkR5cge RTcKzGkZxG+WTief9biX5EROmfIKCUdZkTt1kluPUVUSQxxOKj+SaL4PgfwNGjamVg 7LHfeRFjZlHdknpeXWrtqe4C3oeJbMZ11E4xyFwYGfLhg7YM00nmQ/7FfPNFlN6VyX SHOYzuqMQO2tQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3g1lm5z6tn0; Thu, 29 Dec 2022 19:14:23 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:24 +0000 Message-Id: <20221229181327.758-6-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/eris/http.scm (eris-http-block-ref): New procedure. --- guix/eris/http.scm | 46 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git a/guix/eris/http.scm b/guix/eris/http.scm index a8a9520197..354d3be3be 100644 --- a/guix/eris/http.scm +++ b/guix/eris/http.scm @@ -18,11 +18,17 @@ (define-module (guix eris http) #:use-module (eris utils base32) + #:use-module (web uri) + #:use-module (web client) + #:use-module (web response) - #:use-module (srfi srfi-2) + #:use-module (rnrs base) + #:use-module (srfi srfi-2) ; and-let* + #:use-module (srfi srfi-71) ; extended let-syntax for multiple values - #:export (make-blake2b-urn-query-ref)) + #:export (make-blake2b-urn-query-ref + eris-http-block-ref)) (define blake2b-uri-path-regexp (make-regexp "^blake2b:" regexp/icase)) @@ -38,3 +44,39 @@ (define (make-blake2b-urn-query-ref block-ref) (blake2b-ref (base32-decode (string-drop (uri-path urn) 8)))) (block-ref blake2b-ref)))) + +(define (make-blake2b-urn ref) + (string-append "urn:blake2b:" (base32-encode ref))) + +(define (make-request-uri host ref) + (build-uri (uri-scheme host) + #:userinfo (uri-userinfo host) + #:host (uri-host host) + #:port (uri-port host) + #:path "/uri-res/N2R" + #:query (make-blake2b-urn ref))) + +(define* (eris-http-block-ref + ref + #:key host + (open-connection (lambda (host) (open-socket-for-uri host)))) + + (let* ((uri (make-request-uri host ref)) + (_ (format #t "URI: ~a\n" uri)) + (response body + (http-get uri + #:decode-body? #f + #:port (open-connection host) + #:keep-alive? #t))) + (if (eqv? (response-code response) 200) + body + #f))) + +;; (use-modules (eris) +;; (rnrs bytevectors)) + +;; (bytevector? +;; (eris-http-block-ref +;; (base32-decode +;; "4ACQM2Q5IB3DBHGKHM2WRTAXDBPMVZ7F6MG2TUXZ5QOJHW7P4N7Q") +;; #:host (string->uri "http://localhost:8081"))) From patchwork Thu Dec 29 18:13:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45689 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E411D27BBED; Thu, 29 Dec 2022 19:12:29 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 98FBB27BBE9 for ; Thu, 29 Dec 2022 19:12:28 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQt-0000Ei-AT; Thu, 29 Dec 2022 13:15:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQm-0000BQ-Vl for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQm-0004E9-KY for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQm-0007Rt-GA for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 6/8] WIP: eris: Use HTTP to get ERIS blocks. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767828524 (code B ref 52555); Thu, 29 Dec 2022 18:15:04 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:38 +0000 Received: from localhost ([127.0.0.1]:32814 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQM-0007Pu-2H for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:38 -0500 Received: from mout01.posteo.de ([185.67.36.65]:33369) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQE-0007ON-Br for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:31 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id D4569240149 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337664; bh=Gx1I1s+FDI7Xn3adSjvAKoHNcF9VxPncAf+3vjxJjPw=; h=From:To:Cc:Subject:Date:From; b=VLIn6U614l8dcsWWcXvdX0HzN0kjAU+NByfyD8z54eH9i+OMYHj8jvJc4N27EG2HG JQCb6Zz/w5tgR/x2UaeJhpk8FM3RqDpj6bqGeoCOxSxx0tMq4MfxvEdtRwfjis3vTE Uvj9ZoEaipWw7Hni65Z5fV0Xxg2VnfbUOU4Xr7eijyD7Em6vLTREdnaA8ApcrQMSFJ LMenOMtKfrEetYENYVlP/ngfEIZPBahQTzbP6ulaQtoI3qk5fWsGh0AP346SSO4p74 VP5C2A++caXplw+F3/HbLvfmfFl8NyUO4Rn0YUfCpAsdk3PQI3biLsZrohBM3gJ0o1 ds1obVD5zWwBg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3g5VGmz6tnF; Thu, 29 Dec 2022 19:14:23 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:25 +0000 Message-Id: <20221229181327.758-7-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches - guix/eris.scm (guix-eris-block-ref): Refactor from call-with-eris-block-ref and use eris-http-block-ref. - guix/eris/fs-store.scm (eris-fs-store-ref): Refactor to take keyword arguments. - guix/scripts/substitute.scm (download-nar): Use guix-eris-block-ref. - guix/scripts/publish.scm (make-request-handler): Use refactored eris-fs-store-ref. TODO: - default value for %eris-peers parameter in (eris) --- guix/eris.scm | 37 +++++++++++++++++++++++++++++++------ guix/eris/fs-store.scm | 21 ++++++++++----------- guix/scripts/publish.scm | 9 ++++----- guix/scripts/substitute.scm | 14 ++++++++------ 4 files changed, 53 insertions(+), 28 deletions(-) diff --git a/guix/eris.scm b/guix/eris.scm index d9a0914b67..4af17c2807 100644 --- a/guix/eris.scm +++ b/guix/eris.scm @@ -21,9 +21,13 @@ (define-module (guix eris) #:use-module (guix config) #:use-module (guix eris fs-store) + #:use-module (guix eris http) + + #:use-module (web uri) + #:use-module (ice-9 match) #:export (guix-eris-block-reducer - call-with-eris-block-ref + guix-eris-block-ref %eris-block-store-directory)) @@ -36,8 +40,29 @@ (define (guix-eris-block-reducer) "Returns a block reducer that stores blocks of ERIS encoded content." (eris-fs-store-reducer (%eris-block-store-directory))) -(define (call-with-eris-block-ref f) - (let ((fs-store-block-ref - (eris-fs-store-ref - (%eris-block-store-directory)))) - (f fs-store-block-ref))) +(define %eris-peers + (make-parameter + ;; TODO + (list (string->uri "http://localhost:8081")))) + +(define* (try-in-order ref #:key block-refs) + (match block-refs + ((block-ref . rest) + (let ((block (block-ref ref))) + (if block + block + (try-in-order ref #:block-refs rest)))) + (() #f))) + +(define* (guix-eris-block-ref ref #:key open-connection) + (try-in-order + ref + #:block-refs + (list + (lambda (ref) + (eris-fs-store-ref ref + #:store-directory (%eris-block-store-directory))) + (lambda (ref) + (eris-http-block-ref ref + #:host (string->uri "http://localhost:8081") + #:open-connection open-connection))))) diff --git a/guix/eris/fs-store.scm b/guix/eris/fs-store.scm index 2ef7607988..38f5926280 100644 --- a/guix/eris/fs-store.scm +++ b/guix/eris/fs-store.scm @@ -54,14 +54,13 @@ (define (eris-fs-store-reducer store-directory) #t)))) -(define (eris-fs-store-ref store-directory) - (lambda (ref) - (let* ((b32 (base32-encode ref)) - (pre (substring b32 0 2)) - (suf (substring b32 2)) - (path (string-append store-directory "/" pre "/" suf))) - (if (file-exists? path) - (call-with-input-file path - (lambda (port) (get-bytevector-all port)) - #:binary #t) - #f)))) +(define* (eris-fs-store-ref ref #:key store-directory) + (let* ((b32 (base32-encode ref)) + (pre (substring b32 0 2)) + (suf (substring b32 2)) + (path (string-append store-directory "/" pre "/" suf))) + (if (file-exists? path) + (call-with-input-file path + (lambda (port) (get-bytevector-all port)) + #:binary #t) + #f))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 15bdf02670..0ce50b2942 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -1085,13 +1085,12 @@ (define nar-path? (let ((expected (split-and-decode-uri-path nar-path))) (cut equal? expected <>))) - ;; Get ERIS blocks directly from the filesystem store. - (define eris-block-ref - (eris-fs-store-ref (%eris-block-store-directory))) - ;; Create a handler for resolving blake2b URN queries. (define blake2b-urn-query-ref - (make-blake2b-urn-query-ref eris-block-ref)) + (make-blake2b-urn-query-ref + (lambda (ref) (eris-fs-store-ref + ref + #:store-directory (%eris-block-store-directory))))) (define (handle request body) (format #t "~a ~a~%" diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 8cf011d7e6..14a21f6c37 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -482,12 +482,14 @@ (define (fetch uri) ((urn) (let ((read-capability (->eris-read-capability uri))) (if (eris-read-capability? read-capability) - (call-with-eris-block-ref - (lambda (block-ref) - (values (open-eris-input-port - read-capability - #:block-ref block-ref) - #f))) + (values + (open-eris-input-port + read-capability + #:block-ref (lambda (ref) + (guix-eris-block-ref + ref + #:open-connection open-connection-for-uri/cached))) + #f) (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) (else From patchwork Thu Dec 29 18:13:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45687 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0B42B27BBEB; Thu, 29 Dec 2022 19:02:53 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 1AC4D27BBE9 for ; Thu, 29 Dec 2022 19:02:51 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQt-0000Eh-AL; Thu, 29 Dec 2022 13:15:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQm-0000BK-D6 for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQm-0004Dz-3F for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQl-0007Rm-U2 for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 7/8] eris: Use parameterized %eris-peers when getting blocks. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767828517 (code B ref 52555); Thu, 29 Dec 2022 18:15:03 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:38 +0000 Received: from localhost ([127.0.0.1]:32812 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQL-0007Pm-No for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:38 -0500 Received: from mout02.posteo.de ([185.67.36.66]:48273) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQE-0007OO-Cy for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:31 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id DB00B2401E3 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337664; bh=0t3VJ7QgKBL8gGBqmmcrMOnEDnoXjqRl59gNRajS6t0=; h=From:To:Cc:Subject:Date:From; b=U/q3HYZOys4d1YBYzUtEZlljzp+hqeMAqPiQLTzdDE9pXf/uBTN03ug5BDGu8NGF2 Ldzki+Ii9Tfqp3ARCb8x/TG76OEhcPlVNONebJap9BSn1GGlZK0GEqd5KoPu34BDo8 QMPEvM0AbwRABqRGehvtRkXNbmMo04jaNUDvEWnWgjhZ/ed6qVPEZ4ETQPXl03jzHc noCHXXMwAQAMA+ksjxaeGfxqrmJMcREri1PxlNOkAqzZFG9HBTw/NrBuJWGN4xRCbu zzhB//HNFmwoAurcy60M3cSS/DstFAE30P+vn+l/4CcExmVxjj5vgRtY02JYNjsc9J 9ycM2TUuSHbLg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3h1JwQz6tnW; Thu, 29 Dec 2022 19:14:24 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:26 +0000 Message-Id: <20221229181327.758-8-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches --- guix/eris.scm | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/guix/eris.scm b/guix/eris.scm index 4af17c2807..d56643bec4 100644 --- a/guix/eris.scm +++ b/guix/eris.scm @@ -54,15 +54,30 @@ (define* (try-in-order ref #:key block-refs) (try-in-order ref #:block-refs rest)))) (() #f))) +(define* (peer->block-ref peer #:key open-connection) + (case (uri-scheme peer) + + ((http https) + (lambda (ref) + (eris-http-block-ref ref + #:host peer + #:open-connection open-connection))) + + ;; unsupported ERIS peer URL + (else (lambda (_) #f)))) + (define* (guix-eris-block-ref ref #:key open-connection) (try-in-order ref #:block-refs - (list + (cons + + ;; first try and get block from local block store (lambda (ref) (eris-fs-store-ref ref #:store-directory (%eris-block-store-directory))) - (lambda (ref) - (eris-http-block-ref ref - #:host (string->uri "http://localhost:8081") - #:open-connection open-connection))))) + + ;; then try peers + (map (lambda (peer) + (peer->block-ref peer #:open-connection open-connection)) + (%eris-peers))))) From patchwork Thu Dec 29 18:13:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 45681 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6C9FE27BBE9; Thu, 29 Dec 2022 18:24:18 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 31D0C27BBEB for ; Thu, 29 Dec 2022 18:24:15 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQs-0000Ea-SQ; Thu, 29 Dec 2022 13:15:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQn-0000Bl-Ni for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQn-0004EG-2P for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQm-0007S1-Tl for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 8/8] eris: Use IPFS to get ERIS blocks. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767928531 (code B ref 52555); Thu, 29 Dec 2022 18:15:04 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:39 +0000 Received: from localhost ([127.0.0.1]:32816 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQM-0007Q1-Hg for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:39 -0500 Received: from mout01.posteo.de ([185.67.36.65]:53869) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQE-0007OQ-Qi for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:31 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 5344A240088 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337665; bh=1ogSyEzUNPD6uxAMuJO0N6UJzwGcBVN8M95yEFq6iSg=; h=From:To:Cc:Subject:Date:From; b=PsqLYn5BSuPS81ye/XQCTbgoapQ1MIV5u3cQWqnJeRK3FCxPgTSmjPTsS2OUkhBhL esReGJOJ8q2ntO8AnA4mQEVRUKBR/0pnyA5I9jUlYA8MzSF7Qcccgek4xxupYz6xsN 0rREYoR6WZ77FmDIhlQ3Y4m9FG6AzHotCxxFPhgBJdlESKG9H+esB7R3yAsPgQfwvy wPTonuHO5GrtUSlQDoRGuiZfd0zEyuwUt5bI1KCEJixs2bdJQj9F3tSrFqnLdRtnuC FQfNlD+LEZbIYR1PsyJ2OW6FmS9NxXULFTtBKe8O/is2EKNNPvW89VqXpl5kQRC15C NhXDM3jf2vc8w== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3h5D1Mz6tml; Thu, 29 Dec 2022 19:14:24 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:27 +0000 Message-Id: <20221229181327.758-9-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/eris/ipfs.scm: New files. * Makefile.am (MODULES): Add it. * guix/eris.scm (%eris-peers): Add IPFS. (peer->block-ref): Handle IPFS peer. --- Makefile.am | 1 + guix/eris.scm | 32 ++++--- guix/eris/ipfs.scm | 214 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 237 insertions(+), 10 deletions(-) create mode 100644 guix/eris/ipfs.scm diff --git a/Makefile.am b/Makefile.am index 373f6b7c27..6f648a40a3 100644 --- a/Makefile.am +++ b/Makefile.am @@ -135,6 +135,7 @@ MODULES = \ guix/eris.scm \ guix/eris/fs-store.scm \ guix/eris/http.scm \ + guix/eris/ipfs.scm \ guix/platform.scm \ guix/platforms/arm.scm \ guix/platforms/mips.scm \ diff --git a/guix/eris.scm b/guix/eris.scm index d56643bec4..5b0c1ee36b 100644 --- a/guix/eris.scm +++ b/guix/eris.scm @@ -22,6 +22,7 @@ (define-module (guix eris) #:use-module (guix config) #:use-module (guix eris fs-store) #:use-module (guix eris http) + #:use-module (guix eris ipfs) #:use-module (web uri) #:use-module (ice-9 match) @@ -42,8 +43,10 @@ (define (guix-eris-block-reducer) (define %eris-peers (make-parameter - ;; TODO - (list (string->uri "http://localhost:8081")))) + ;; TODO: make ERIS peers configurable somewhere + (list + (string->uri "http://localhost:8081") + 'ipfs))) (define* (try-in-order ref #:key block-refs) (match block-refs @@ -55,18 +58,27 @@ (define* (try-in-order ref #:key block-refs) (() #f))) (define* (peer->block-ref peer #:key open-connection) - (case (uri-scheme peer) + (cond + ((uri? peer) (case (uri-scheme peer) - ((http https) - (lambda (ref) - (eris-http-block-ref ref - #:host peer - #:open-connection open-connection))) + ((http https) + (lambda (ref) + (eris-http-block-ref ref + #:host peer + #:open-connection open-connection))) - ;; unsupported ERIS peer URL - (else (lambda (_) #f)))) + ;; unsupported ERIS peer URL + (else (lambda (_) #f)))) + + ((eqv? 'ipfs peer) + (lambda (ref) + (eris-ipfs-ref ref #:open-connection open-connection))))) (define* (guix-eris-block-ref ref #:key open-connection) + "Attempts to dereference a block of some ERIS encoded content with reference +REF. First the local block store is checked, followed by remote peers as +configured in the parameter %eris-peers (in order). Returns #f if the block +could not be de-referenced." (try-in-order ref #:block-refs diff --git a/guix/eris/ipfs.scm b/guix/eris/ipfs.scm new file mode 100644 index 0000000000..9771414e7b --- /dev/null +++ b/guix/eris/ipfs.scm @@ -0,0 +1,214 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2018 Ludovic Courtès +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + + +;;; Commentary: +;;; +;;; This module provides an interface to the IPFS daemons HTTP API for storing +;;; and retrieving blocks. This can be used to store blocks of ERIS encoded +;;; content. +;;; +;;; See also the IPFS API documentation: +;;; https://docs.ipfs.io/reference/http/api/#api-v0-block-put + +(define-module (guix eris ipfs) + #:use-module (eris utils base32) + #:use-module (sodium generichash) + #:use-module (json) + #:use-module (web uri) + #:use-module (web client) + #:use-module (web response) + #:use-module (srfi srfi-71) + #:use-module (rnrs io ports) + #:use-module (rnrs bytevectors) + + #:use-module ((guix build download) + #:select ((open-connection-for-uri + . guix:open-connection-for-uri))) + #:export (%ipfs-base-url + + eris-ipfs-reducer + eris-ipfs-ref)) + + +;; CID encoding + +;; Multicodec codes +;; (https://github.com/multiformats/multicodec/blob/master/table.csv) +(define multicodec-raw-code #x55) +(define multicodec-blake2b-256-code #xb220) + +(define (blake2b-256->binary-cid hash) + "Encode a Blake2b-256 hash as binary CID" + (call-with-values + (lambda () (open-bytevector-output-port)) + (lambda (port get-bytevector) + ;; CID version + (put-u8 port 1) + ;; multicoded content-type + (put-u8 port multicodec-raw-code) + ;; set multihash to blake2b-256. This is the manually encoded varint of + ;; 0xb220 + (put-u8 port 160) (put-u8 port 228) (put-u8 port 2) + ;; set hash lenght + (put-u8 port 32) + ;; and finally the hash itself + (put-bytevector port hash) + + ;; finalize and get the bytevector + (get-bytevector)))) + +(define (binary-cid->cid bcid) + "Encode a binary CID as Base32 encoded CID" + ;; 'b' is the multibsae code for base32 + (string-append "b" + ;; the IPFS daemon uses lower-case, so to be consistent we + ;; also. + (string-downcase + ;; base32 encode the binary cid + (base32-encode bcid)))) + +(define blake2b-256->cid + (compose binary-cid->cid blake2b-256->binary-cid)) + + +;; IPFS API + +(define %ipfs-base-url + ;; URL of the IPFS gateway. + (make-parameter "http://localhost:5001")) + +(define* (call url decode + #:optional + (method http-post) + #:key port body (false-if-404? #t) (headers '()) + (keep-alive #t) + (open-connection guix:open-connection-for-uri) + (timeout 10)) + "Invoke the endpoint at URL using METHOD. Decode the resulting JSON body +using DECODE, a one-argument procedure that takes an input port; when DECODE +is false, return the input port. When FALSE-IF-404? is true, return #f upon +404 responses." + (let* ((url (if (string? url) (string->uri url) url)) + (port (or port (open-connection url #:timeout timeout))) + (response response-port + (if keep-alive + (method url #:streaming? #t + #:body body + #:port port + #:keep-alive? #t) + (method url #:streaming? #t + #:body body + #:port port + ;; IPFS daemon seems to responds with bad + ;; request if PUT requests are kept alive and + ;; do not have "Connection: close" header. + #:keep-alive? #f + #:headers `((connection close) + ,@headers))))) + (cond ((= 200 (response-code response)) + (if decode + (let ((result (decode response-port))) + (close-port response-port) + result) + response-port)) + ((and false-if-404? + (= 404 (response-code response))) + (close-port response-port) + #f) + (else + (close-port response-port) + (format #t "~a\n" response) + (throw 'ipfs-error url response))))) + +(define-syntax-rule (false-if-ipfs-error exp) + "Return $f if EXP triggers a network related or IPFS related exception." + (with-exception-handler + (lambda (exn) + (let ((kind (exception-kind exn)) + (errno (system-error-errno + (cons 'system-error (exception-args exn))))) + (cond + ((= errno ECONNREFUSED) #f) + (else (raise-exception exp))))) + (lambda () exp) + #:unwind? #t)) + +(define %multipart-boundary + ;; XXX: We might want to find a more reliable boundary. + (string-append (make-string 24 #\-) "2698127afd7425a6")) + +(define (bytevector->form-data bv port) + "Write to PORT a 'multipart/form-data' representation of BV." + (display (string-append "--" %multipart-boundary "\r\n" + "Content-Disposition: form-data\r\n" + "Content-Type: application/octet-stream\r\n\r\n") + port) + (put-bytevector port bv) + (display (string-append "\r\n--" %multipart-boundary "--\r\n") + port)) + +(define (ipfs-block-put bv) + "Store a block on IPFS and return the CID of the block" + (call (string-append (%ipfs-base-url) + "/api/v0/block/put" + "?format=raw&mhtype=blake2b-256") + (lambda (port) (assoc-ref (json->scm port) "Key")) + #:headers `((content-type + . (multipart/form-data + (boundary . ,%multipart-boundary)))) + #:body (call-with-bytevector-output-port + (lambda (port) (bytevector->form-data bv port))) + ;; IPFS daemon does not seem to accept connection re-use when putting + ;; blocks. + #:keep-alive #f)) + +(define* (ipfs-block-get cid #:key + (open-connection guix:open-connection-for-uri)) + "Get a block from IPFS via the HTTP API" + (false-if-ipfs-error + (call (string-append (%ipfs-base-url) + "/api/v0/block/get" + "?arg=" cid) + get-bytevector-all + #:timeout 5 + #:open-connection open-connection))) + +;; ERIS block reducer + +(define eris-ipfs-reducer + (case-lambda + ;; initialization. Nothing to do here. In an improved implementation we + ;; might create a single HTTP connection and reuse it for all blocks. + (() '()) + + ;; Completion. Again, nothing to do. + ((_) 'done) + + ;; store a block + ((_ ref-block) + ;; ref-block is a pair consisting of the reference to the block and the + ;; block itself. + (ipfs-block-put (cdr ref-block))))) + +(define* (eris-ipfs-ref ref #:key + (open-connection guix:open-connection-for-uri)) + "Dereference a block from IPFS" + (ipfs-block-get (blake2b-256->cid ref) + #:open-connection open-connection))