[bug#76003] distribute NARs through IPFS.

Message ID 875xltklod.fsf@cock.li
State New
Headers
Series [bug#76003] distribute NARs through IPFS. |

Commit Message

Justin Veilleux Feb. 2, 2025, 4:36 a.m. UTC
  Hello everyone.

I think Distributed substitute distribution has been on many people's
radar for a time. However, from what I understood of the discussion at
https://issues.guix.gnu.org/33899 (which has been stagnant for many
years), actually achieving this is complicated because of the lack of a
good way to encode into IPFS store items in a way that correctly
deduplicates identical files (giving them the same hash) and works well
with the rest of the IPFS ecosystem. Solving this problem is hard and
until a spec for UnixFSv2 is created and implemented in the main IPFS
implementations, making perfect use of all the IPFS features will be
harder and require a lot of effort.

However, I would really like to have distributed substitutes, even if it
is done through a less-than optimal mechanism, and I feel like (I might be
wrong) that is a common sentiment.

The attached patch series adds to the `guix publish` daemon an option
`--ipfs-api` which, when present, makes it publish through ipfs each
compressed nar file it creates, and adds a corresponding entry in the
narinfo through the URL: ipfs://Qm... property.

on the `guix substitute` side, an `ipfs-fetch` function is also added to
download nars through a local gateway. This means that the new URL:
ipfs:Qm... fields can now be exploited as easily as the regular
nar/lzip/... urls.

These changes don't exploit every advantage of IPFS delivery
(deduplication, "true decentralization", etc), but enable a non-trivial
use-case: many machines on a local network can now distribute amongst
themselves substitutes (thus reducing the total needed bandwidth)
transparently.

WDYT?
  

Comments

Maxim Cournoyer Feb. 5, 2025, 6:22 a.m. UTC | #1
Hi,

Justin Veilleux <terramorpha@cock.li> writes:

[...]

> These changes don't exploit every advantage of IPFS delivery
> (deduplication, "true decentralization", etc), but enable a non-trivial
> use-case: many machines on a local network can now distribute amongst
> themselves substitutes (thus reducing the total needed bandwidth)
> transparently.

Note that this is already possible via mdns discovery of substitute
servers advertising themselves via mDNS/DNS-SD (that's the
'--advertise?' option of guix-publish, coupled with the '--discover'
option of guix-daemon).
  

Patch

From f71fb156b6b217cea0a181db13d88ccce69f0b5d Mon Sep 17 00:00:00 2001
Message-ID: <f71fb156b6b217cea0a181db13d88ccce69f0b5d.1738470881.git.terramorpha@cock.li>
In-Reply-To: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
References: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
From: terramorpha <terramorpha@cock.li>
Date: Sat, 1 Feb 2025 22:41:28 -0500
Subject: [PATCH 4/4] services: guix-publish: Expose and document the
 --ipfs-api argument.

Change-Id: Iecabcc79b11a263fd77b25e51ce9d02fcb3aeddc
---
 doc/guix.texi         | 4 ++++
 gnu/services/base.scm | 7 ++++++-
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index fe1b3c08d9..ccd7b7d73f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20352,6 +20352,10 @@  Base Services
 When true, advertise the service on the local network @i{via} the DNS-SD
 protocol, using Avahi.
 
+@item @code{ipfs-api} (default: @code{#f})
+The url to an IPFS node api, when present (and @code{cache} has a
+value), publish the compressed NARs on the ipfs network.
+
 This allows neighboring Guix devices with discovery on (see
 @code{guix-configuration} above) to discover this @command{guix publish}
 instance and to automatically download substitutes from it.
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 7331c030d7..f8c586d189 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -2252,6 +2252,8 @@  (define-record-type* <guix-publish-configuration>
            (default "localhost"))
   (advertise? guix-publish-advertise?             ;boolean
               (default #f))
+  (ipfs-api guix-publish-ipfs-api                 ;string to a url
+            (default #f))
   (compression       guix-publish-configuration-compression
                      (thunked)
                      (default (default-compression this-record)))
@@ -2291,7 +2293,7 @@  (define (guix-publish-shepherd-service config)
 
   (match-record config <guix-publish-configuration>
     (guix port host nar-path cache workers ttl negative-ttl
-          cache-bypass-threshold advertise?)
+          cache-bypass-threshold advertise? ipfs-api)
     (let ((command #~(list #$(file-append guix "/bin/guix")
                            "publish" "-u" "guix-publish"
                            "-p" #$(number->string port)
@@ -2301,6 +2303,9 @@  (define (guix-publish-shepherd-service config)
                            #$@(if advertise?
                                   #~("--advertise")
                                   #~())
+                           #$@(if ipfs-api
+                                  #~((string-append "--ipfs-api=" ipfs-api))
+                                  #~())
                            #$@(if workers
                                   #~((string-append "--workers="
                                                     #$(number->string
-- 
2.47.1