Message ID | af02fc3aaa36504cd63cdc5d48bb1f8f31a46d83.camel@telenet.be |
---|---|
State | Accepted |
Headers | show |
Series | [bug#45905] IPFS service definition | expand |
Context | Check | Description |
---|---|---|
cbaines/submitting builds | success | |
cbaines/comparison | success | View comparision |
cbaines/git branch | success | View Git branch |
cbaines/applying patch | fail | View Laminar job |
cbaines/issue | success | View issue |
Hi Maxime, Maxime Devos <maximedevos@telenet.be> skribis: > A patch defining simple ‘ipfs-service-type’ is attached. I've tested > this in a VM, and will test it on a ‘real’ system later. The gateway > is currently broken, it tries to redirect to non-existent subdomains > of localhost. Correcting this might require fiddling with the DNS > configuration. OK. That doesn’t prevent one from using it, right? > From c441bca727df67837652eb2f0b5ad23528fd11a3 Mon Sep 17 00:00:00 2001 > From: Maxime Devos <maximedevos@telenet.be> > Date: Fri, 15 Jan 2021 21:46:42 +0100 > Subject: [PATCH] services: Add ipfs-service-type > > * gnu/services/networking.scm (ipfs-service-type) > (%ipfs-home-mapping, %ipfs-environment) > (%ipfs-accounts, %ipfs-home): New variables. > (ipfs-configuration, ipfs-configuration?) > (ipfs-configuration-package, ipfs-configuration-gateway) > (ipfs-configuration-api, ipfs-shepherd-service) > (ipfs-binary, %ipfs-activation): New procedures. > * doc/guix.texi (Networking Services): Document it. [...] > +@lisp > +;; part of the operating-system declaration I think you can omit this line. > +(service ipfs-service-type > + (ipfs-configuration > + (gateway "/ip4/127.0.0.1/tcp/8880") > + (api "/ip4/127.0.0.1/tcp/8881"))) Indentation is left (should be aligned with ‘ipfs-service-type’.) > + (start #~(make-forkexec-constructor/container > + #$ipfs-daemon-command > + #:namespaces '#$(fold delq %namespaces '(user net)) > + #:mappings (list #$%ipfs-home-mapping) > + #:log-file "/var/log/ipfs.log" > + #:user "ipfs" > + #:group "ipfs" > + #:environment-variables #$%ipfs-environment)) Nice! > + ;; Run ipfs init and ipfs config from a container, > + ;; in case the IPFS daemon was compromised at some point > + ;; and ~/.ipfs is now a symlink to somewhere outside > + ;; %ipfs-home. > + (define container-gexp > + (with-extensions (list shepherd) > + (with-imported-modules (source-module-closure > + '((gnu build shepherd) > + (gnu system file-systems))) > + #~(begin > + (use-modules (gnu build shepherd) > + (gnu system file-systems)) > + (let* ((constructor > + (make-forkexec-constructor/container > + (list #$inner-script) > + #:namespaces '#$(fold delq %namespaces '(user)) > + #:mappings (list #$%ipfs-home-mapping) > + #:user "ipfs" > + #:group "ipfs" > + #:environment-variables #$%ipfs-environment)) > + (pid (constructor))) > + (waitpid pid)))))) > + ;; The activation may happen from the initrd, which uses > + ;; a statically-linked guile, while the guix container > + ;; procedures require a working dynamic-link. > + (define container-script > + (program-file "ipfs-activation-container" container-gexp)) > + #~(system* #$container-script)) That’s a bit involved, but it makes sense to me. The patch LGTM. However, we usually commit services along with a system test under (gnu tests …). The manual has info on how to run individual system tests: https://guix.gnu.org/manual/en/html_node/Running-the-Test-Suite.html Could you write a test that ensures that basic functionality works? It could be as simple as waiting for the service to be up, then invoking ‘ipfs add’ and ‘ipfs get’. WDYT? Thank you! Ludo’.
On Mon, 2021-03-22 at 18:17 +0100, Ludovic Courtès wrote: > Hi Maxime, Hi > Maxime Devos <maximedevos@telenet.be> skribis: > > > A patch defining simple ‘ipfs-service-type’ is attached. I've tested > > this in a VM, and will test it on a ‘real’ system later. The gateway > > is currently broken, it tries to redirect to non-existent subdomains > > of localhost. Correcting this might require fiddling with the DNS > > configuration. > > OK. That doesn’t prevent one from using it, right? Nah, the REST API presumably works just fine and there is plenty to see on the webui: http://localhost:5001/ipfs/bafybeif4zkmu7qdhkpf3pnhwxipylqleof7rl6ojbe7mq3fzogz6m4xk3i/#/ Not perfect, but it might suffice for your purposes. That reminds me the configuration can be modified from there. I didn't figure how to disable that. Not ideal from a security perspective, but at least its only loopback & ipfs is in a container. > > +@lisp > > +;; part of the operating-system declaration > I think you can omit this line. I think I found that line somewhere & copied it for consistency, but it has been some time ago. > > +(service ipfs-service-type > > + (ipfs-configuration > > + (gateway "/ip4/127.0.0.1/tcp/8880") > > + (api "/ip4/127.0.0.1/tcp/8881"))) > > Indentation is left (should be aligned with ‘ipfs-service-type’.) Ok, not sure how this happened. > > + (start #~(make-forkexec-constructor/container > > + [container stuff] > > + #:environment-variables #$%ipfs-environment)) > > Nice! Yep! Also, this reminds me I'm not sure what the distinction between #+ and #~ is in activation gexps, in shepherd services definitions, etc. > > + ;; Run ipfs init and ipfs config from a container, > > + ;; in case the IPFS daemon was compromised at some point > > + ;; and ~/.ipfs is now a symlink to somewhere outside > > + ;; %ipfs-home. > > + (define container-gexp [complicated container stuff]) > > > That’s a bit involved, but it makes sense to me. Unfortunately, there are (non-container related) some more issues. Last few weeks I've been seeing this error (/var/log/ipfs.log): (start snip) Error: fs-repo requires migration Initializing daemon... go-ipfs version: 0.8.0 Repo version: 11 System version: amd64/linux Golang version: go1.14.15 Found outdated fs-repo, migrations need to be run. Run migrations now? [y/N] Not running migrations of fs-repo now. Please get fs-repo-migrations from https://dist.ipfs.io Error: fs-repo requires migration (end snip) (Super hacky work-around: rm -r /var/lib/ipfs mkdir /var/lib/ipfs chmod a-rwx /var/lib/ipfs chmod u+rwx /var/lib/ipfs chown ipfs:ipfs /var/lib/ipfs sudo -u ipfs -g ipfs "`guix build go-ipfs`/bin/ipfs" init # ^ this can take some seconds to complete sudo -u ipfs -g ipfs "`guix build go-ipfs`/bin/ipfs" config API /ip4/127.0.0.1/tcp/5001 sudo -u ipfs -g ipfs "`guix build go-ipfs`/bin/ipfs" config Addresses.Gateway /ip4/127.0.0.1/tcp/8082 herd enable ipfs herd start ipfs) Unfortunately "fs-repo-migrations" does not seem to be packaged in Guix. Apparently there has been a change in repo format in the go-ipfs v0.7.0 --> v0.8.0 upgrade. I believe for most users simply automatically running the upgrades would be sufficient. Now, how could we do this safely from shepherd? Maybe before starting open a pipe, write "y\n" to it an pass it as file descriptor 0 (stdin) would be sufficient? But shepherd always closes /dev/stdin before exec IIRC .. Seems like shepherd needs support for file descriptor! I've a patch for that, but it needs to be verified (& corrected likely) on GNU/Hurd. Feel free to ask for the incomplete patch if you're impatient and want to finish it yourself! (Seems to work on GNU/Linux in any case.) > The patch LGTM. However, we usually commit services along with a system > test under (gnu tests …). The manual has info on how to run individual > system tests: > > https://guix.gnu.org/manual/en/html_node/Running-the-Test-Suite.html > > Could you write a test that ensures that basic functionality works? It > could be as simple as waiting for the service to be up, then invoking > ‘ipfs add’ and ‘ipfs get’. WDYT? Will look into it eventually, but I am currently occupied with other things that have deadlines )-:. (Not feeling very inspired for a writing/presentation assignment ...) (And I would rather hack on GNUnet frankly; IPFS is more of a stop-gap to me for having some distributed something for substitutes.) So feel free to beat me to it. Greetings, Maxime.
Hi Maxime! Maxime Devos <maximedevos@telenet.be> skribis: >> OK. That doesn’t prevent one from using it, right? > > Nah, the REST API presumably works just fine and there is plenty to see on > the webui: > > http://localhost:5001/ipfs/bafybeif4zkmu7qdhkpf3pnhwxipylqleof7rl6ojbe7mq3fzogz6m4xk3i/#/ > > Not perfect, but it might suffice for your purposes. > That reminds me the configuration can be modified from there. > I didn't figure how to disable that. Not ideal from a security > perspective, but at least its only loopback & ipfs is in a container. Good. [...] > Yep! Also, this reminds me I'm not sure what the distinction between > #+ and #~ is in activation gexps, in shepherd services definitions, > etc. #+ is ‘ungexp-native’. It makes sense if you consider a cross-compiled system. Code in an activation gexp is meant to run on the target system, so you want to use #$ (‘ungexp’) there. You might want to use #+ when building things that can just as well be built natively. For instance, the background image for GRUB must be built by running Inkscape natively on the host system, so we use #+inkscape (or similar) to do that. I hope that makes sense. > Unfortunately, there are (non-container related) some more issues. > Last few weeks I've been seeing this error (/var/log/ipfs.log): > > (start snip) > Error: fs-repo requires migration > Initializing daemon... > go-ipfs version: 0.8.0 > Repo version: 11 > System version: amd64/linux > Golang version: go1.14.15 > Found outdated fs-repo, migrations need to be run. > Run migrations now? [y/N] Not running migrations of fs-repo now. > Please get fs-repo-migrations from https://dist.ipfs.io > > Error: fs-repo requires migration > (end snip) Bah, I remember seeing that. > Unfortunately "fs-repo-migrations" does not seem to be packaged in Guix. > Apparently there has been a change in repo format in the go-ipfs v0.7.0 > --> v0.8.0 upgrade. I believe for most users simply automatically running > the upgrades would be sufficient. Yes, I think so. We “just” need to package ‘fs-repo-migrations’ first. Perhaps it’d be okay, as a first step, to provide an IPFS service that doesn’t handle migrations automatically. > Now, how could we do this safely from shepherd? Maybe before starting open > a pipe, write "y\n" to it an pass it as file descriptor 0 (stdin) would > be sufficient? But shepherd always closes /dev/stdin before exec IIRC .. You could have the ‘ipfs’ Shepherd service depend on, say, a one-shot ‘ipfs-migration’ service. The ‘ipfs-migration’ service would run ‘fs-repo-migrations’ if it’s necessary. >> The patch LGTM. However, we usually commit services along with a system >> test under (gnu tests …). The manual has info on how to run individual >> system tests: >> >> https://guix.gnu.org/manual/en/html_node/Running-the-Test-Suite.html >> >> Could you write a test that ensures that basic functionality works? It >> could be as simple as waiting for the service to be up, then invoking >> ‘ipfs add’ and ‘ipfs get’. WDYT? > > Will look into it eventually, but I am currently occupied with other things > that have deadlines )-:. (Not feeling very inspired for a > writing/presentation assignment ...) (And I would rather hack on GNUnet > frankly; IPFS is more of a stop-gap to me for having some distributed > something for substitutes.) So feel free to beat me to it. I’m not offering to work on it :-), but hopefully you or maybe some fellow contributor can finish it up in the coming weeks! Thanks, Ludo’.
From c441bca727df67837652eb2f0b5ad23528fd11a3 Mon Sep 17 00:00:00 2001 From: Maxime Devos <maximedevos@telenet.be> Date: Fri, 15 Jan 2021 21:46:42 +0100 Subject: [PATCH] services: Add ipfs-service-type * gnu/services/networking.scm (ipfs-service-type) (%ipfs-home-mapping, %ipfs-environment) (%ipfs-accounts, %ipfs-home): New variables. (ipfs-configuration, ipfs-configuration?) (ipfs-configuration-package, ipfs-configuration-gateway) (ipfs-configuration-api, ipfs-shepherd-service) (ipfs-binary, %ipfs-activation): New procedures. * doc/guix.texi (Networking Services): Document it. --- doc/guix.texi | 33 +++++++++ gnu/services/networking.scm | 138 ++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index cea7f8a8cf..49680b13e9 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -86,6 +86,7 @@ Copyright @copyright{} 2020 raingloom@* Copyright @copyright{} 2020 Daniel Brooks@* Copyright @copyright{} 2020 John Soo@* Copyright @copyright{} 2020 Jonathan Brielmaier@* +Copyright @copyright{} 2021 Maxime Devos@* Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -17221,6 +17222,38 @@ address, delete everything except these options: @end table @end deftp +@cindex IPFS +@defvr {Scheme Variable} ipfs-service-type +The service type for connecting to the @uref{https://ipfs.io,IPFS network}, +a global, versioned, peer-to-peer file system. Pass it a +@code{ipfs-configuration} to change the ports used for the gateway and API. + +Here's an example configuration, using some non-standard ports: + +@lisp +;; part of the operating-system declaration +(service ipfs-service-type + (ipfs-configuration + (gateway "/ip4/127.0.0.1/tcp/8880") + (api "/ip4/127.0.0.1/tcp/8881"))) +@end lisp +@end defvr + +@deftp {Data Type} ipfs-configuration +Data type representing the configuration of IPFS. + +@table @asis +@item @code{package} (default: @code{go-ipfs}) +Package object of IPFS. + +@item @code{gateway} (default: @code{"/ip4/127.0.0.1/tcp/8082"}) +Address of the gateway, in ‘multiaddress’ format. + +@item @code{api} (default: @code{"/ip4/127.0.0.1/tcp/5001"}) +Address of the API endpoint, in ‘multiaddress’ format. +@end table +@end deftp + @cindex keepalived @deffn {Scheme Variable} keepalived-service-type This is the type for the @uref{https://www.keepalived.org/, Keepalived} diff --git a/gnu/services/networking.scm b/gnu/services/networking.scm index dd4061341e..4a1d04dfbb 100644 --- a/gnu/services/networking.scm +++ b/gnu/services/networking.scm @@ -15,6 +15,7 @@ ;;; Copyright © 2019 Alex Griffin <a@ajgrf.com> ;;; Copyright © 2020 Brice Waegeneire <brice@waegenei.re> ;;; Copyright © 2021 Oleg Pykhalov <go.wigust@gmail.com> +;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be> ;;; ;;; This file is part of GNU Guix. ;;; @@ -54,6 +55,8 @@ #:use-module (gnu packages ntp) #:use-module (gnu packages wicd) #:use-module (gnu packages gnome) + #:use-module (gnu packages ipfs) + #:use-module (gnu build linux-container) #:use-module (guix gexp) #:use-module (guix records) #:use-module (guix modules) @@ -196,6 +199,13 @@ yggdrasil-configuration-json-config yggdrasil-configuration-package + ipfs-service-type + ipfs-configuration + ipfs-configuration? + ipfs-configuration-package + ipfs-configuration-gateway + ipfs-configuration-api + keepalived-configuration keepalived-configuration? keepalived-service-type)) @@ -1873,6 +1883,134 @@ See yggdrasil -genconf for config options.") (service-extension profile-service-type (compose list yggdrasil-configuration-package)))))) + +;;; +;;; IPFS +;;; + +(define-record-type* <ipfs-configuration> + ipfs-configuration + make-ipfs-configuration + ipfs-configuration? + (package ipfs-configuration-package + (default go-ipfs)) + (gateway ipfs-configuration-gateway + (default "/ip4/127.0.0.1/tcp/8082")) + (api ipfs-configuration-api + (default "/ip4/127.0.0.1/tcp/5001"))) + +(define %ipfs-home "/var/lib/ipfs") + +(define %ipfs-accounts + (list (user-account + (name "ipfs") + (group "ipfs") + (system? #t) + (comment "IPFS daemon user") + (home-directory "/var/lib/ipfs") + (shell (file-append shadow "/sbin/nologin"))) + (user-group + (name "ipfs") + (system? #t)))) + +(define (ipfs-binary config) + (file-append (ipfs-configuration-package config) "/bin/ipfs")) + +(define %ipfs-home-mapping + #~(file-system-mapping + (source #$%ipfs-home) + (target #$%ipfs-home) + (writable? #t))) + +(define %ipfs-environment + #~(list #$(string-append "HOME=" %ipfs-home))) + +(define (ipfs-shepherd-service config) + "Return a <shepherd-service> for IPFS with CONFIG." + (define ipfs-daemon-command + #~(list #$(ipfs-binary config) "daemon")) + (list + (with-imported-modules (source-module-closure + '((gnu build shepherd) + (gnu system file-systems))) + (shepherd-service + (provision '(ipfs)) + (requirement '(networking)) + (documentation "Connect to the IPFS network") + (modules '((gnu build shepherd) + (gnu system file-systems))) + (start #~(make-forkexec-constructor/container + #$ipfs-daemon-command + #:namespaces '#$(fold delq %namespaces '(user net)) + #:mappings (list #$%ipfs-home-mapping) + #:log-file "/var/log/ipfs.log" + #:user "ipfs" + #:group "ipfs" + #:environment-variables #$%ipfs-environment)) + (stop #~(make-kill-destructor)))))) + +(define (%ipfs-activation config) + "Return an activation gexp for IPFS with CONFIG" + (define (ipfs-config-command setting value) + #~(#$(ipfs-binary config) "config" #$setting #$value)) + (define (set-config!-gexp setting value) + #~(system* #$@(ipfs-config-command setting value))) + (define settings + `(("Addresses.API" ,(ipfs-configuration-api config)) + ("Addresses.Gateway" ,(ipfs-configuration-gateway config)))) + (define inner-gexp + #~(begin + (umask #o077) + ;; Create $HOME/.ipfs structure + (system* #$(ipfs-binary config) "init") + ;; Apply settings + #$@(map (cute apply set-config!-gexp <>) settings))) + (define inner-script + (program-file "ipfs-activation-inner" inner-gexp)) + ;; Run ipfs init and ipfs config from a container, + ;; in case the IPFS daemon was compromised at some point + ;; and ~/.ipfs is now a symlink to somewhere outside + ;; %ipfs-home. + (define container-gexp + (with-extensions (list shepherd) + (with-imported-modules (source-module-closure + '((gnu build shepherd) + (gnu system file-systems))) + #~(begin + (use-modules (gnu build shepherd) + (gnu system file-systems)) + (let* ((constructor + (make-forkexec-constructor/container + (list #$inner-script) + #:namespaces '#$(fold delq %namespaces '(user)) + #:mappings (list #$%ipfs-home-mapping) + #:user "ipfs" + #:group "ipfs" + #:environment-variables #$%ipfs-environment)) + (pid (constructor))) + (waitpid pid)))))) + ;; The activation may happen from the initrd, which uses + ;; a statically-linked guile, while the guix container + ;; procedures require a working dynamic-link. + (define container-script + (program-file "ipfs-activation-container" container-gexp)) + #~(system* #$container-script)) + +(define ipfs-service-type + (service-type + (name 'ipfs) + (extensions + (list (service-extension account-service-type + (const %ipfs-accounts)) + (service-extension activation-service-type + %ipfs-activation) + (service-extension shepherd-root-service-type + ipfs-shepherd-service))) + (default-value (ipfs-configuration)) + (description + "Run @command{ipfs daemon}, the reference implementation +of the IPFS p2p storage network."))) + ;;; ;;; Keepalived -- 2.30.0