From patchwork Thu Dec 16 16:17:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: pukkamustard X-Patchwork-Id: 500 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E2F7827BBEA; Thu, 16 Dec 2021 16:18:15 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 3866327BBE9 for ; Thu, 16 Dec 2021 16:18:12 +0000 (GMT) Received: from localhost ([::1]:36622 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxtSN-0008Oo-7W for patchwork@mira.cbaines.net; Thu, 16 Dec 2021 11:18:11 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55614) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtSF-0008Kp-14 for guix-patches@gnu.org; Thu, 16 Dec 2021 11:18:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53180) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxtSE-0002zX-Nt for guix-patches@gnu.org; Thu, 16 Dec 2021 11:18:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mxtSE-0003s3-Jt for guix-patches@gnu.org; Thu, 16 Dec 2021 11:18:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Dec 2021 16:18:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.163967147614866 (code B ref -1); Thu, 16 Dec 2021 16:18:02 +0000 Received: (at submit) by debbugs.gnu.org; 16 Dec 2021 16:17:56 +0000 Received: from localhost ([127.0.0.1]:36493 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtS1-0003rY-Se for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:17:56 -0500 Received: from lists.gnu.org ([209.51.188.17]:37690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtRx-0003rN-Ab for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:17:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55568) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtRu-0007zZ-Gl for guix-patches@gnu.org; Thu, 16 Dec 2021 11:17:43 -0500 Received: from mout01.posteo.de ([185.67.36.65]:54721) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtRr-0002xU-RE for guix-patches@gnu.org; Thu, 16 Dec 2021 11:17:42 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id A7664240027 for ; Thu, 16 Dec 2021 17:17:36 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1639671456; bh=LCOS0Ruzr1yXnOxpCv/ZshQ+w7IBoj9ooeByvyiV5d8=; h=From:To:Cc:Subject:Date:From; b=owd1T1Cl32kJHXMZf3WHFiPOsjnFVOytH4ghqM0GkhJppwP2dm/+FhcWST8y0jQaY xaz4l+BbbwPuyq2VuvI6a5T3AR5gSTVIcDp78szjoVq5/1egww/hG5Xn+xt538NMSO bkLL6RW2bEKcnQ40FniNb4CeLpO5dJnjkHF7clurTmifj4HuQ1YIRf6FRtud1oivee 3WSf/fz6ccTB+TzqjCLdVnYfCtK4iHo/Nscb5VOx/cB4qL2/zZUT+th9JHJ7O1SDyo oMWAft+8/jnyeQvogoCndtkL+Te/69ZSuMIB90XFQdk/MKs2NoXv7VXTSYp8MDL4so UMHM/LeItkubw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JFHML4ndkz9rxQ; Thu, 16 Dec 2021 17:17:34 +0100 (CET) From: pukkamustard Date: Thu, 16 Dec 2021 16:17:24 +0000 Message-Id: <20211216161724.547-1-pukkamustard@posteo.net> MIME-Version: 1.0 Received-SPF: pass client-ip=185.67.36.65; envelope-from=pukkamustard@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hello Guix, This is an initial patch and proposal towards decentralizing substitute distribution with ERIS. ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into uniformly sized, encryped and content-addressed blocks. The original content can be reconstructed only with access to a read capability, which can be encoded as an URN. One key advantage of ERIS is that the encoding is protocol agnostic. Any protocol that can transfer small (32KiB) sized blocks referenced by the hash of their content will do. This can be done with things such as GNUNet, IPFS, OpenDHT, HTTP or a USB stick on a bicycle. The following patch allows substitutes to be published over IPFS using ERIS. This is inspired and very similar to previous work on distributing substitutes over IPFS [2]. The narinfos served by `guix publish` look like this: --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 Compression: gzip FileSize: 67363 ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 Compression: zstd FileSize: 64917 ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w NarSize: 220704 References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg== --8<---------------cut here---------------end--------------->8--- For every compressed nar the ERIS URN is computed and added. If the `--ipfs` is used for `guix publish` then the encoded blocks are also uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like this: --8<---------------cut here---------------start------------->8--- (use-modules (eris) (eris blocks ipfs)) (eris-decode->bytevector "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE" eris-blocks-ipfs-ref) --8<---------------cut here---------------end--------------->8--- These patches do not yet retrieve content from IPFS (TODO). But in principle, anybody connected to IPFS can get the nar with the ERIS URN. This could be used to reduce load on substitute server as they would only need to publish the ERIS URN directly - substitutes could be delivered much more peer-to-peer. Other transports that I have been looking in to and am pretty sure will work include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the advantage of ERIS over IPFS directly or GNUNet directly. The encoding and identifiers (URN) are abstracted away from specific transports (and also applications). ERIS is almost exactly the same encoding as used in GNUNet (ECRS). Blocks can be stored in any kind of databases (see for example the GDBM bindings [4]). A tricky things is figuring out how to multiplex all these different transports and storages... The ERIS specification is still considered "experimental". However we feel confident to stabilize it and intend to do so around February/March 2022 with a release 1.0.0 of the specification. This will ensure that the identifiers remain stable for the forseeable future (until the crypto breaks). Before that there is also a small external security audit of the specification planned (thanks to NGI0/NLnet!). This is just a little demo of the idea and some food for thought and discussion. Give it a try and let me know what you think! I've also pushed the patches to my personal Guix mirror if you want to check it out from there: https://inqlab.net/git/guix.git/log/?h=wip-eris Also CCing ~pukkamustard/eris@lists.sr.ht where there is some general ERIS related discussion. Thanks, -pukkamustard [1] http://purl.org/eris [2] https://issues.guix.gnu.org/33899 [3] https://www.ietf.org/rfc/rfc2169.txt [4] https://inqlab.net/git/guile-eris.git/tree/eris/blocks/gdbm.scm pukkamustard (3): publish: Add ERIS URN to narinfo WIP: gnu: guile-eris: Update to unreleased git version. publish: Add IPFS support. configure.ac | 5 ++ gnu/packages/guile-xyz.scm | 10 ++-- gnu/packages/package-management.scm | 1 + guix/narinfo.scm | 10 ++-- guix/scripts/publish.scm | 79 ++++++++++++++++++++++------- 5 files changed, 79 insertions(+), 26 deletions(-)