diff mbox series

[bug#39728,v2] Allow parallel downloads and builds

Message ID 20211121234941.32724b68@tachikoma.lepiller.eu
State New
Headers show
Series [bug#39728,v2] Allow parallel downloads and builds | expand


Context Check Description
cbaines/comparison success View comparision
cbaines/git branch success View Git branch
cbaines/applying patch fail View Laminar job
cbaines/issue success View issue

Commit Message

Julien Lepiller Nov. 21, 2021, 10:49 p.m. UTC
Le Wed, 26 Feb 2020 11:36:32 +0100,
Pierre Neidhardt <mail@ambrevar.xyz> a écrit :

> Thank you so much for this, Julien!
> There was a thread on this topic on the mailing list:
> https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html
> (could not find the first email :p).
> There were a couple other issues that were mentioned there.
> This patch would be a first step towards more parallelization!


After so long, I managed to find the time to go over the comments and
improve my patches. I tested the new daemon for a bit, and it's working
as expected so far :D
diff mbox series


From d8b39af21d97dc3f0d3057ceb7ba91a93ff8d3ec Mon Sep 17 00:00:00 2001
Message-Id: <d8b39af21d97dc3f0d3057ceb7ba91a93ff8d3ec.1637534792.git.julien@lepiller.eu>
In-Reply-To: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu>
References: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu>
From: Julien Lepiller <julien@lepiller.eu>
Date: Sun, 21 Nov 2021 23:26:11 +0100
Subject: [PATCH 2/2] guix: Support specifying max download jobs.

* guix/store.scm (set-build-options): Optionally send max-download-jobs.
* guix/scripts/build.scm (set-build-options-from-command-line)
(%standard-build-options): Support `--max-downloads`
* doc/guix.texi (Common Build Options): Document it.
 doc/guix.texi          | 6 ++++++
 guix/scripts/build.scm | 8 ++++++++
 guix/store.scm         | 5 +++++
 3 files changed, 19 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index b8de53c53b..0afefbd416 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -10846,6 +10846,12 @@  Common Build Options
 guix-daemon, @option{--max-jobs}}, for details about this option and the
 equivalent @command{guix-daemon} option.
+@item --max-downloads=@var{n}
+@itemx -D @var{n}
+Allow at most @var{n} download jobs in parallel.  @xref{Invoking
+guix-daemon, @option{--max-downloads}}, for details about this option and the
+equivalent @command{guix-daemon} option.
 @item --debug=@var{level}
 Produce debugging output coming from the build daemon.  @var{level} must be an
 integer between 0 and 5; higher means more verbose output.  Setting a level of
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 97e2f5a167..0d0199eccd 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -207,6 +207,7 @@  (define (set-build-options-from-command-line store opts)
                      #:rounds (assoc-ref opts 'rounds)
                      #:build-cores (assoc-ref opts 'cores)
                      #:max-build-jobs (assoc-ref opts 'max-jobs)
+                     #:max-download-jobs (assoc-ref opts 'max-downloads)
                      #:fallback? (assoc-ref opts 'fallback?)
                      #:use-substitutes? (assoc-ref opts 'substitutes?)
                      #:substitute-urls (assoc-ref opts 'substitute-urls)
@@ -316,6 +317,13 @@  (define %standard-build-options
                   (let ((c (false-if-exception (string->number arg))))
                     (if c
                         (apply values (alist-cons 'max-jobs c result) rest)
+                        (leave (G_ "not a number: '~a' option argument: ~a~%")
+                               name arg)))))
+        (option '(#\D "max-downloads") #t #f
+                (lambda (opt name arg result . rest)
+                  (let ((c (false-if-exception (string->number arg))))
+                    (if c
+                        (apply values (alist-cons 'max-downloads c result) rest)
                         (leave (G_ "not a number: '~a' option argument: ~a~%")
                                name arg)))))))
diff --git a/guix/store.scm b/guix/store.scm
index 7388953d15..6b5b9262b1 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -803,6 +803,7 @@  (define* (set-build-options server
                             (verbosity 0)
                             rounds                ;number of build rounds
+                            max-download-jobs
                             (offload? #t)
@@ -896,6 +897,10 @@  (define* (set-build-options server
                               . ,(number->string max-build-jobs)))
+                     ,@(if max-download-jobs
+                           `(("download-max-jobs"
+                              . ,(number->string max-download-jobs)))
+                           '())
                      ,@(if build-cores
                            `(("build-cores" . ,(number->string build-cores)))