From patchwork Sun Nov 21 22:49:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Julien Lepiller X-Patchwork-Id: 34733 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id ECCB627BBE3; Sun, 21 Nov 2021 22:50:15 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,T_DKIM_INVALID, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id EEB9427BBE1 for ; Sun, 21 Nov 2021 22:50:14 +0000 (GMT) Received: from localhost ([::1]:46836 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1movf4-0005mW-2U for patchwork@mira.cbaines.net; Sun, 21 Nov 2021 17:50:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:37324) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1moves-0005m9-Uc for guix-patches@gnu.org; Sun, 21 Nov 2021 17:50:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34739) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1moves-0002TE-Lp for guix-patches@gnu.org; Sun, 21 Nov 2021 17:50:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1moves-0001Mf-FZ for guix-patches@gnu.org; Sun, 21 Nov 2021 17:50:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#39728] [PATCH v2] Allow parallel downloads and builds Resent-From: Julien Lepiller Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 21 Nov 2021 22:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39728 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , 39728@debbugs.gnu.org, zimoun Received: via spool by 39728-submit@debbugs.gnu.org id=B39728.16375350015234 (code B ref 39728); Sun, 21 Nov 2021 22:50:02 +0000 Received: (at 39728) by debbugs.gnu.org; 21 Nov 2021 22:50:01 +0000 Received: from localhost ([127.0.0.1]:46285 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1movel-0001MF-IR for submit@debbugs.gnu.org; Sun, 21 Nov 2021 17:50:01 -0500 Received: from lepiller.eu ([89.234.186.109]:39638) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1movei-0001M4-Sr for 39728@debbugs.gnu.org; Sun, 21 Nov 2021 17:49:54 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id f305089e; Sun, 21 Nov 2021 22:49:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type; s=dkim; bh=IsYhBHenT5UmJSqrlIR0fSyMLjS2/+gRmIoF6m MOaFU=; b=F9TAWIhF9oi5030Je8p5hoS0L9/TPs/tKwQZdRdNQsHfC0e7wUNNs2 AieE3bgorJzgU7PZv13XYwYDpEe/G8wHT2i1BXV1TfJZipQ3ezLioBq1oS8Gsi7m 7KXrsbj7AphftkKhZ9si9AWeVSkgyORYMFSpA3Z6//pPHFJ0deksZ08vn73gJTs8 XAxGsf+fQdRztC3TW+mG1i3M31rfz3gP8QRbChSwVoJfEf5CKrsdNinaAl8ux2jd igGeCzr6ePwEjMkcMaWyWFqb8lt9iP7LixEbz8AbJhbmOk0kFAbX2ZCRNLQc4wmt XkHR9GAkNJcO6rXcSTAbiwmXpu7bcqPw== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id cc2b7bcb (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Sun, 21 Nov 2021 22:49:48 +0000 (UTC) Date: Sun, 21 Nov 2021 23:49:41 +0100 From: Julien Lepiller Message-ID: <20211121234941.32724b68@tachikoma.lepiller.eu> In-Reply-To: <87zhd58wjz.fsf@ambrevar.xyz> References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> <87zhd58wjz.fsf@ambrevar.xyz> X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Le Wed, 26 Feb 2020 11:36:32 +0100, Pierre Neidhardt a écrit : > Thank you so much for this, Julien! > > There was a thread on this topic on the mailing list: > https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html > (could not find the first email :p). > > There were a couple other issues that were mentioned there. > This patch would be a first step towards more parallelization! > Hi! After so long, I managed to find the time to go over the comments and improve my patches. I tested the new daemon for a bit, and it's working as expected so far :D From d8b39af21d97dc3f0d3057ceb7ba91a93ff8d3ec Mon Sep 17 00:00:00 2001 Message-Id: In-Reply-To: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> References: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> From: Julien Lepiller Date: Sun, 21 Nov 2021 23:26:11 +0100 Subject: [PATCH 2/2] guix: Support specifying max download jobs. * guix/store.scm (set-build-options): Optionally send max-download-jobs. * guix/scripts/build.scm (set-build-options-from-command-line) (%standard-build-options): Support `--max-downloads` * doc/guix.texi (Common Build Options): Document it. --- doc/guix.texi | 6 ++++++ guix/scripts/build.scm | 8 ++++++++ guix/store.scm | 5 +++++ 3 files changed, 19 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index b8de53c53b..0afefbd416 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -10846,6 +10846,12 @@ Common Build Options guix-daemon, @option{--max-jobs}}, for details about this option and the equivalent @command{guix-daemon} option. +@item --max-downloads=@var{n} +@itemx -D @var{n} +Allow at most @var{n} download jobs in parallel. @xref{Invoking +guix-daemon, @option{--max-downloads}}, for details about this option and the +equivalent @command{guix-daemon} option. + @item --debug=@var{level} Produce debugging output coming from the build daemon. @var{level} must be an integer between 0 and 5; higher means more verbose output. Setting a level of diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 97e2f5a167..0d0199eccd 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -207,6 +207,7 @@ (define (set-build-options-from-command-line store opts) #:rounds (assoc-ref opts 'rounds) #:build-cores (assoc-ref opts 'cores) #:max-build-jobs (assoc-ref opts 'max-jobs) + #:max-download-jobs (assoc-ref opts 'max-downloads) #:fallback? (assoc-ref opts 'fallback?) #:use-substitutes? (assoc-ref opts 'substitutes?) #:substitute-urls (assoc-ref opts 'substitute-urls) @@ -316,6 +317,13 @@ (define %standard-build-options (let ((c (false-if-exception (string->number arg)))) (if c (apply values (alist-cons 'max-jobs c result) rest) + (leave (G_ "not a number: '~a' option argument: ~a~%") + name arg))))) + (option '(#\D "max-downloads") #t #f + (lambda (opt name arg result . rest) + (let ((c (false-if-exception (string->number arg)))) + (if c + (apply values (alist-cons 'max-downloads c result) rest) (leave (G_ "not a number: '~a' option argument: ~a~%") name arg))))))) diff --git a/guix/store.scm b/guix/store.scm index 7388953d15..6b5b9262b1 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -803,6 +803,7 @@ (define* (set-build-options server (verbosity 0) rounds ;number of build rounds max-build-jobs + max-download-jobs timeout max-silent-time (offload? #t) @@ -896,6 +897,10 @@ (define* (set-build-options server `(("build-max-jobs" . ,(number->string max-build-jobs))) '()) + ,@(if max-download-jobs + `(("download-max-jobs" + . ,(number->string max-download-jobs))) + '()) ,@(if build-cores `(("build-cores" . ,(number->string build-cores))) '()) -- 2.33.1