From patchwork Sun Feb 2 04:36:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Justin Veilleux X-Patchwork-Id: 38170 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8349D27BBE9; Sun, 2 Feb 2025 04:37:16 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B7F8B27BBE2 for ; Sun, 2 Feb 2025 04:37:13 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1teRjE-0008Sg-EP; Sat, 01 Feb 2025 23:37:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1teRjC-0008SQ-LV for guix-patches@gnu.org; Sat, 01 Feb 2025 23:37:02 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1teRjC-0004TB-Cw for guix-patches@gnu.org; Sat, 01 Feb 2025 23:37:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:From:To:Subject; bh=6UM/qvrRDNCxZQrmPvieXACSGuUJ3LBA2qNXfUzqdjM=; b=rdgI8b7RMdjPfUQAZPLanQr3d3SBGiFWqLUjDx7/lPXkWxoE4PoG5gdDul6RpZhANrwUg0ZHxwOxlTJevUrWy/Uq8T/KJ6RJuuDzVmwUv0sc7AL7blaS0KgH27+94/lM3lEBWeGZeNe+ntj37p0FB95LoPf32JfiF87VMso1Oh3UyvZPXTerZZGZuOLcuZTB+Tlrr7jjULpu0wxoy8XwBtB/AqjPZP995wFVWlbarqJp21/jA403bcpcCktQKblXznW6IRSjjLrw8rPirdp/iEguQkTgQ7H1YUIupw6YUrwBG90Ee/S81CBRncGmPZNN8Wyn0OdpM57kx7Wq85fbRA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1teRjC-0000QI-7a for guix-patches@gnu.org; Sat, 01 Feb 2025 23:37:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76003] [PATCH] distribute NARs through IPFS. Resent-From: Justin Veilleux Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 02 Feb 2025 04:37:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 76003 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76003@debbugs.gnu.org, ludo@gnu.org X-Debbugs-Original-To: guix-patches@gnu.org, ludo@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.17384710121607 (code B ref -1); Sun, 02 Feb 2025 04:37:02 +0000 Received: (at submit) by debbugs.gnu.org; 2 Feb 2025 04:36:52 +0000 Received: from localhost ([127.0.0.1]:60829 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1teRiy-0000Pj-Is for submit@debbugs.gnu.org; Sat, 01 Feb 2025 23:36:52 -0500 Received: from lists.gnu.org ([2001:470:142::17]:42778) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1teRiu-0000PG-UG for submit@debbugs.gnu.org; Sat, 01 Feb 2025 23:36:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1teRim-0008RG-AS for guix-patches@gnu.org; Sat, 01 Feb 2025 23:36:38 -0500 Received: from mail.cock.li ([37.120.193.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1teRij-0004Pa-9E; Sat, 01 Feb 2025 23:36:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cock.li; s=mail; t=1738470981; bh=5Q0RW10RU5uqiGJSRmQDM3Hh5kLzVVnDcv6iOhOcuNA=; h=From:To:Subject:Date:From; b=ZEv2ot7vENOAVLHr5FvGSjP9gYuZKqp79o4HebjaNmOA8e7hJ/1gVeYFtManEf3yl +u675+C/JvT3GWjjVIxjVUIimDr1HIfL3o2333Q503BjcMYszqg8vWmgXzZbIN0kdF cPhAlxae5SVBMWrAPUSKuFr4/w7theNS9E+Xle6Nixt5duHidRK0WkOmstI7jXoE9j 27X2yBoK3mE3UKC2vGWjCDI+gHMlRkP3knD45m/rb3Ill9nFST/+VWE+vV+qGlOj4f 7F6EOrOk0D9OQ8Ad8t54h+ZIT5IPsSLwqZTSGLIaSUv0AvDh615UkTL2Iy4eBFi0+V 1yZZJ6wz3zShw== Date: Sat, 01 Feb 2025 23:36:18 -0500 Message-ID: <875xltklod.fsf@cock.li> MIME-Version: 1.0 Received-SPF: pass client-ip=37.120.193.124; envelope-from=terramorpha@cock.li; helo=mail.cock.li X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_PBL=3.335, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Justin Veilleux X-ACL-Warn: , Justin Veilleux via Guix-patches X-Patchwork-Original-From: Justin Veilleux via Guix-patches via From: Justin Veilleux Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Hello everyone. I think Distributed substitute distribution has been on many people's radar for a time. However, from what I understood of the discussion at https://issues.guix.gnu.org/33899 (which has been stagnant for many years), actually achieving this is complicated because of the lack of a good way to encode into IPFS store items in a way that correctly deduplicates identical files (giving them the same hash) and works well with the rest of the IPFS ecosystem. Solving this problem is hard and until a spec for UnixFSv2 is created and implemented in the main IPFS implementations, making perfect use of all the IPFS features will be harder and require a lot of effort. However, I would really like to have distributed substitutes, even if it is done through a less-than optimal mechanism, and I feel like (I might be wrong) that is a common sentiment. The attached patch series adds to the `guix publish` daemon an option `--ipfs-api` which, when present, makes it publish through ipfs each compressed nar file it creates, and adds a corresponding entry in the narinfo through the URL: ipfs://Qm... property. on the `guix substitute` side, an `ipfs-fetch` function is also added to download nars through a local gateway. This means that the new URL: ipfs:Qm... fields can now be exploited as easily as the regular nar/lzip/... urls. These changes don't exploit every advantage of IPFS delivery (deduplication, "true decentralization", etc), but enable a non-trivial use-case: many machines on a local network can now distribute amongst themselves substitutes (thus reducing the total needed bandwidth) transparently. WDYT? From f71fb156b6b217cea0a181db13d88ccce69f0b5d Mon Sep 17 00:00:00 2001 Message-ID: In-Reply-To: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li> References: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li> From: terramorpha Date: Sat, 1 Feb 2025 22:41:28 -0500 Subject: [PATCH 4/4] services: guix-publish: Expose and document the --ipfs-api argument. Change-Id: Iecabcc79b11a263fd77b25e51ce9d02fcb3aeddc --- doc/guix.texi | 4 ++++ gnu/services/base.scm | 7 ++++++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/doc/guix.texi b/doc/guix.texi index fe1b3c08d9..ccd7b7d73f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -20352,6 +20352,10 @@ Base Services When true, advertise the service on the local network @i{via} the DNS-SD protocol, using Avahi. +@item @code{ipfs-api} (default: @code{#f}) +The url to an IPFS node api, when present (and @code{cache} has a +value), publish the compressed NARs on the ipfs network. + This allows neighboring Guix devices with discovery on (see @code{guix-configuration} above) to discover this @command{guix publish} instance and to automatically download substitutes from it. diff --git a/gnu/services/base.scm b/gnu/services/base.scm index 7331c030d7..f8c586d189 100644 --- a/gnu/services/base.scm +++ b/gnu/services/base.scm @@ -2252,6 +2252,8 @@ (define-record-type* (default "localhost")) (advertise? guix-publish-advertise? ;boolean (default #f)) + (ipfs-api guix-publish-ipfs-api ;string to a url + (default #f)) (compression guix-publish-configuration-compression (thunked) (default (default-compression this-record))) @@ -2291,7 +2293,7 @@ (define (guix-publish-shepherd-service config) (match-record config (guix port host nar-path cache workers ttl negative-ttl - cache-bypass-threshold advertise?) + cache-bypass-threshold advertise? ipfs-api) (let ((command #~(list #$(file-append guix "/bin/guix") "publish" "-u" "guix-publish" "-p" #$(number->string port) @@ -2301,6 +2303,9 @@ (define (guix-publish-shepherd-service config) #$@(if advertise? #~("--advertise") #~()) + #$@(if ipfs-api + #~((string-append "--ipfs-api=" ipfs-api)) + #~()) #$@(if workers #~((string-append "--workers=" #$(number->string -- 2.47.1