| Message ID | 20211121234941.32724b68@tachikoma.lepiller.eu |
|---|---|
| State | New |
| Headers | show |
| Series | [bug#39728,v2] Allow parallel downloads and builds | expand |
| Context | Check | Description |
|---|---|---|
| cbaines/comparison | success | View comparision |
| cbaines/git branch | success | View Git branch |
| cbaines/applying patch | fail | View Laminar job |
| cbaines/issue | success | View issue |
Hello, Julien Lepiller <julien@lepiller.eu> skribis: > After so long, I managed to find the time to go over the comments and > improve my patches. I tested the new daemon for a bit, and it's working > as expected so far :D On a recent daemon, have you seen cases where having multiple downloads in parallel speeds things up? The analysis in <https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/> suggests that at the time you first submitted this patch, substitution speed (which is different from raw download speed) was often CPU-bound. This is no longer the case, meaning that downloads should now be network-bound or almost. Thanks, Ludo’.
Le Thu, 25 Nov 2021 13:53:25 +0100, Ludovic Courtès <ludo@gnu.org> a écrit : > Hello, > > Julien Lepiller <julien@lepiller.eu> skribis: > > > After so long, I managed to find the time to go over the comments > > and improve my patches. I tested the new daemon for a bit, and it's > > working as expected so far :D > > On a recent daemon, have you seen cases where having multiple > downloads in parallel speeds things up? > > The analysis in > <https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/> > suggests that at the time you first submitted this patch, substitution > speed (which is different from raw download speed) was often > CPU-bound. This is no longer the case, meaning that downloads should > now be network-bound or almost. > > Thanks, > Ludo’. I would still say yes, because the output from berlin is often much less than my throughput. With multiple downloads in parallel it at least feels quicker, probably because I can download at full speed. In any case, I see often a build start while downloads are in progress, so I think it's still a win if you can get a few derivations built while waiting for a big download to finish at the same time :) At some point we might want to prioritize builds/downloads that help unlock as much builds as possible early, so we don't have builds waiting for downloads.
Hi! Julien Lepiller <julien@lepiller.eu> skribis: > I would still say yes, because the output from berlin is often much > less than my throughput. With multiple downloads in parallel it at > least feels quicker, probably because I can download at full speed. It would be nice to measure that because like I wrote, I think we’re pretty much network-bound these days, at least with zstd and uncompressed downloads. > In any case, I see often a build start while downloads are in progress, > so I think it's still a win if you can get a few derivations built > while waiting for a big download to finish at the same time :) True! Overlapping downloads and builds sounds like a good idea. > At some point we might want to prioritize builds/downloads that help > unlock as much builds as possible early, so we don't have builds > waiting for downloads. Right now the daemon starts with substitutes and builds afterwards. BTW, we’re assuming downloads = substitutes in this whole discussion, but we could/should take fixed-output derivations into account too. I’ll take a closer look later on… Thanks, Ludo’.
From d8b39af21d97dc3f0d3057ceb7ba91a93ff8d3ec Mon Sep 17 00:00:00 2001 Message-Id: <d8b39af21d97dc3f0d3057ceb7ba91a93ff8d3ec.1637534792.git.julien@lepiller.eu> In-Reply-To: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> References: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> From: Julien Lepiller <julien@lepiller.eu> Date: Sun, 21 Nov 2021 23:26:11 +0100 Subject: [PATCH 2/2] guix: Support specifying max download jobs. * guix/store.scm (set-build-options): Optionally send max-download-jobs. * guix/scripts/build.scm (set-build-options-from-command-line) (%standard-build-options): Support `--max-downloads` * doc/guix.texi (Common Build Options): Document it. --- doc/guix.texi | 6 ++++++ guix/scripts/build.scm | 8 ++++++++ guix/store.scm | 5 +++++ 3 files changed, 19 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index b8de53c53b..0afefbd416 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -10846,6 +10846,12 @@ Common Build Options guix-daemon, @option{--max-jobs}}, for details about this option and the equivalent @command{guix-daemon} option. +@item --max-downloads=@var{n} +@itemx -D @var{n} +Allow at most @var{n} download jobs in parallel. @xref{Invoking +guix-daemon, @option{--max-downloads}}, for details about this option and the +equivalent @command{guix-daemon} option. + @item --debug=@var{level} Produce debugging output coming from the build daemon. @var{level} must be an integer between 0 and 5; higher means more verbose output. Setting a level of diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 97e2f5a167..0d0199eccd 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -207,6 +207,7 @@ (define (set-build-options-from-command-line store opts) #:rounds (assoc-ref opts 'rounds) #:build-cores (assoc-ref opts 'cores) #:max-build-jobs (assoc-ref opts 'max-jobs) + #:max-download-jobs (assoc-ref opts 'max-downloads) #:fallback? (assoc-ref opts 'fallback?) #:use-substitutes? (assoc-ref opts 'substitutes?) #:substitute-urls (assoc-ref opts 'substitute-urls) @@ -316,6 +317,13 @@ (define %standard-build-options (let ((c (false-if-exception (string->number arg)))) (if c (apply values (alist-cons 'max-jobs c result) rest) + (leave (G_ "not a number: '~a' option argument: ~a~%") + name arg))))) + (option '(#\D "max-downloads") #t #f + (lambda (opt name arg result . rest) + (let ((c (false-if-exception (string->number arg)))) + (if c + (apply values (alist-cons 'max-downloads c result) rest) (leave (G_ "not a number: '~a' option argument: ~a~%") name arg))))))) diff --git a/guix/store.scm b/guix/store.scm index 7388953d15..6b5b9262b1 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -803,6 +803,7 @@ (define* (set-build-options server (verbosity 0) rounds ;number of build rounds max-build-jobs + max-download-jobs timeout max-silent-time (offload? #t) @@ -896,6 +897,10 @@ (define* (set-build-options server `(("build-max-jobs" . ,(number->string max-build-jobs))) '()) + ,@(if max-download-jobs + `(("download-max-jobs" + . ,(number->string max-download-jobs))) + '()) ,@(if build-cores `(("build-cores" . ,(number->string build-cores))) '()) -- 2.33.1