[bug#76003] distribute NARs through IPFS.
Commit Message
Hello everyone.
I think Distributed substitute distribution has been on many people's
radar for a time. However, from what I understood of the discussion at
https://issues.guix.gnu.org/33899 (which has been stagnant for many
years), actually achieving this is complicated because of the lack of a
good way to encode into IPFS store items in a way that correctly
deduplicates identical files (giving them the same hash) and works well
with the rest of the IPFS ecosystem. Solving this problem is hard and
until a spec for UnixFSv2 is created and implemented in the main IPFS
implementations, making perfect use of all the IPFS features will be
harder and require a lot of effort.
However, I would really like to have distributed substitutes, even if it
is done through a less-than optimal mechanism, and I feel like (I might be
wrong) that is a common sentiment.
The attached patch series adds to the `guix publish` daemon an option
`--ipfs-api` which, when present, makes it publish through ipfs each
compressed nar file it creates, and adds a corresponding entry in the
narinfo through the URL: ipfs://Qm... property.
on the `guix substitute` side, an `ipfs-fetch` function is also added to
download nars through a local gateway. This means that the new URL:
ipfs:Qm... fields can now be exploited as easily as the regular
nar/lzip/... urls.
These changes don't exploit every advantage of IPFS delivery
(deduplication, "true decentralization", etc), but enable a non-trivial
use-case: many machines on a local network can now distribute amongst
themselves substitutes (thus reducing the total needed bandwidth)
transparently.
WDYT?
Comments
Hi,
Justin Veilleux <terramorpha@cock.li> writes:
[...]
> These changes don't exploit every advantage of IPFS delivery
> (deduplication, "true decentralization", etc), but enable a non-trivial
> use-case: many machines on a local network can now distribute amongst
> themselves substitutes (thus reducing the total needed bandwidth)
> transparently.
Note that this is already possible via mdns discovery of substitute
servers advertising themselves via mDNS/DNS-SD (that's the
'--advertise?' option of guix-publish, coupled with the '--discover'
option of guix-daemon).
From f71fb156b6b217cea0a181db13d88ccce69f0b5d Mon Sep 17 00:00:00 2001
Message-ID: <f71fb156b6b217cea0a181db13d88ccce69f0b5d.1738470881.git.terramorpha@cock.li>
In-Reply-To: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
References: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
From: terramorpha <terramorpha@cock.li>
Date: Sat, 1 Feb 2025 22:41:28 -0500
Subject: [PATCH 4/4] services: guix-publish: Expose and document the
--ipfs-api argument.
Change-Id: Iecabcc79b11a263fd77b25e51ce9d02fcb3aeddc
---
doc/guix.texi | 4 ++++
gnu/services/base.scm | 7 ++++++-
2 files changed, 10 insertions(+), 1 deletion(-)
@@ -20352,6 +20352,10 @@ Base Services
When true, advertise the service on the local network @i{via} the DNS-SD
protocol, using Avahi.
+@item @code{ipfs-api} (default: @code{#f})
+The url to an IPFS node api, when present (and @code{cache} has a
+value), publish the compressed NARs on the ipfs network.
+
This allows neighboring Guix devices with discovery on (see
@code{guix-configuration} above) to discover this @command{guix publish}
instance and to automatically download substitutes from it.
@@ -2252,6 +2252,8 @@ (define-record-type* <guix-publish-configuration>
(default "localhost"))
(advertise? guix-publish-advertise? ;boolean
(default #f))
+ (ipfs-api guix-publish-ipfs-api ;string to a url
+ (default #f))
(compression guix-publish-configuration-compression
(thunked)
(default (default-compression this-record)))
@@ -2291,7 +2293,7 @@ (define (guix-publish-shepherd-service config)
(match-record config <guix-publish-configuration>
(guix port host nar-path cache workers ttl negative-ttl
- cache-bypass-threshold advertise?)
+ cache-bypass-threshold advertise? ipfs-api)
(let ((command #~(list #$(file-append guix "/bin/guix")
"publish" "-u" "guix-publish"
"-p" #$(number->string port)
@@ -2301,6 +2303,9 @@ (define (guix-publish-shepherd-service config)
#$@(if advertise?
#~("--advertise")
#~())
+ #$@(if ipfs-api
+ #~((string-append "--ipfs-api=" ipfs-api))
+ #~())
#$@(if workers
#~((string-append "--workers="
#$(number->string
--
2.47.1