From patchwork Tue Feb 20 19:42:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60815 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 032F227BBEA; Tue, 20 Feb 2024 19:43:01 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7F33A27BBE2 for ; Tue, 20 Feb 2024 19:43:00 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcW0q-0004Cx-Oc; Tue, 20 Feb 2024 14:42:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcW0o-0004Bt-RE for guix-patches@gnu.org; Tue, 20 Feb 2024 14:42:42 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcW0o-000164-Id; Tue, 20 Feb 2024 14:42:42 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcW18-0002Z7-6X; Tue, 20 Feb 2024 14:43:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69291] [PATCH 2/5] scripts: substitute: Allow not using with-timeout in download-nar. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69291 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69291@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69291-submit@debbugs.gnu.org id=B69291.17084581769825 (code B ref 69291); Tue, 20 Feb 2024 19:43:02 +0000 Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:42:56 +0000 Received: from localhost ([127.0.0.1]:46673 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW0z-0002YA-7l for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:42:55 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43156) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW0x-0002Xp-As for 69291@debbugs.gnu.org; Tue, 20 Feb 2024 14:42:51 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id F204C27BBE9 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:28 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6addb520 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:28 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:42:24 +0000 Message-ID: <2bcc2278b5aefcd81fbc53b791d08622b6d93c7c.1708458147.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches I don't think the approach of using SIGALARM here for the timeout will work well in all cases (e.g. when using Guile Fibers), so make it possible to avoid this. * guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in as an option. Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6 --- guix/scripts/substitute.scm | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index f3eed0eb44..575fa2a0b3 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -452,7 +452,8 @@ (define-syntax-rule (catch-system-error exp) (define* (download-nar narinfo destination #:key status-port - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + (fetch-timeout %fetch-timeout)) "Download the nar prescribed in NARINFO, which is assumed to be authentic and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and if DESTINATION is in the store, deduplicate its files. Print a status line to @@ -473,20 +474,26 @@ (define* (download-nar narinfo destination (let ((port (open-file (uri-path uri) "r0b"))) (values port (stat:size (stat port))))) ((http https) - ;; Test this with: - ;; sudo tc qdisc add dev eth0 root netem delay 1500ms - ;; and then cancel with: - ;; sudo tc qdisc del dev eth0 root - (with-timeout %fetch-timeout - (begin - (warning (G_ "while fetching ~a: server is somewhat slow~%") - (uri->string uri)) - (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (with-cached-connection uri port - (http-fetch uri #:text? #f - #:port port - #:keep-alive? #t - #:buffered? #f)))) + (if fetch-timeout + ;; Test this with: + ;; sudo tc qdisc add dev eth0 root netem delay 1500ms + ;; and then cancel with: + ;; sudo tc qdisc del dev eth0 root + (with-timeout %fetch-timeout + (begin + (warning (G_ "while fetching ~a: server is somewhat slow~%") + (uri->string uri)) + (warning (G_ "try `--no-substitutes' if the problem persists~%"))) + (with-cached-connection uri port + (http-fetch uri #:text? #f + #:port port + #:keep-alive? #t + #:buffered? #f))) + (with-cached-connection uri port + (http-fetch uri #:text? #f + #:port port + #:keep-alive? #t + #:buffered? #f)))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) From patchwork Tue Feb 20 19:42:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60814 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2E53927BBEA; Tue, 20 Feb 2024 19:42:58 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B846127BBE2 for ; Tue, 20 Feb 2024 19:42:57 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcW0q-0004Cq-Gx; Tue, 20 Feb 2024 14:42:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcW0o-0004Br-J9 for guix-patches@gnu.org; Tue, 20 Feb 2024 14:42:42 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcW0o-00015x-AT; Tue, 20 Feb 2024 14:42:42 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcW19-0002ZV-45; Tue, 20 Feb 2024 14:43:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69291] [PATCH 3/5] scripts: substitute: Replace some leave calls with raise. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:43:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69291 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69291@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69291-submit@debbugs.gnu.org id=B69291.17084581809846 (code B ref 69291); Tue, 20 Feb 2024 19:43:03 +0000 Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:43:00 +0000 Received: from localhost ([127.0.0.1]:46677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW16-0002Yi-Ac for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:43:00 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43158) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW0y-0002Y0-BZ for 69291@debbugs.gnu.org; Tue, 20 Feb 2024 14:42:52 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id 0407A27BBE2 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:30 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 17dacf27 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:29 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:42:25 +0000 Message-ID: <3a88dc8a8369278f86f3a51a99aa62d991ba997d.1708458147.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches These calls happen inside of with-error-handling, so the effect should be the same, but this opens up the possibility of using this code in a program that doesn't want to exit when one of these error conditions is met. Change-Id: I15d963615d85d419559fa0f4333fa4dc1dfbfd3b * guix/scripts/substitute.scm (download-nar, process-substitution): Use raise formatted-message rather than leave. Change-Id: Idd0880206b69e3903e19e0536b87d65a52c200d5 --- guix/scripts/substitute.scm | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 575fa2a0b3..1875a4332d 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -495,8 +495,10 @@ (define* (download-nar narinfo destination #:keep-alive? #t #:buffered? #f)))) (else - (leave (G_ "unsupported substitute URI scheme: ~a~%") - (uri->string uri))))) + (raise + (formatted-message + (G_ "unsupported substitute URI scheme: ~a~%") + (uri->string uri)))))) (define (try-fetch choices) (match choices @@ -511,9 +513,11 @@ (define* (download-nar narinfo destination (G_ "Downloading ~a...~%") (uri->string uri))) (values port uri compression download-size)))) (() - (leave (G_ "no valid nar URLs for ~a at ~a~%") - (narinfo-path narinfo) - (narinfo-uri-base narinfo))))) + (raise + (formatted-message + (G_ "no valid nar URLs for ~a at ~a~%") + (narinfo-path narinfo) + (narinfo-uri-base narinfo)))))) ;; Delete DESTINATION first--necessary when starting over after a failed ;; download. @@ -678,8 +682,10 @@ (define* (process-substitution port store-item destination (cut valid-narinfo? <> acl)))) (unless narinfo - (leave (G_ "no valid substitute for '~a'~%") - store-item)) + (raise + (formatted-message + (G_ "no valid substitute for '~a'~%") + store-item))) (guard (c ((network-error? c) (when (http-get-error? c) From patchwork Tue Feb 20 19:42:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60816 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0502027BBEC; Tue, 20 Feb 2024 19:43:12 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id C6FE527BBE2 for ; Tue, 20 Feb 2024 19:43:10 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcW0s-0004EK-No; Tue, 20 Feb 2024 14:42:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcW0q-0004Ca-9O for guix-patches@gnu.org; Tue, 20 Feb 2024 14:42:44 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcW0q-00016S-0h; Tue, 20 Feb 2024 14:42:44 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcW19-0002Zf-Jc; Tue, 20 Feb 2024 14:43:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting fast vs small compressions. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:43:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69291 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69291@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69291-submit@debbugs.gnu.org id=B69291.17084581819859 (code B ref 69291); Tue, 20 Feb 2024 19:43:03 +0000 Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:43:01 +0000 Received: from localhost ([127.0.0.1]:46679 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW16-0002Yl-NQ for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:43:01 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43160) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW0y-0002Y2-CU for 69291@debbugs.gnu.org; Tue, 20 Feb 2024 14:42:53 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id 0AC2827BBE9 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:30 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 606a0cb8 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:29 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:42:26 +0000 Message-ID: <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Pulling the logic up to the script makes this code more portable and not reliant on setting a global variable. * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to… (%default-prefer-fast-decompression?): this. (display-narinfo-data): Update accordingly. (download-nar): Add prefer-fast-decompression? as a keyword argument, remove code to set! it and return the cpu-usage recorded. (process-substitution, process-substitution/fallback): Accept and pass through prefer-fast-decompression? to download-nar. (guix-substitute): Move the prefer fast decompression switching logic here. Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb --- guix/scripts/substitute.scm | 90 +++++++++++++++++++++---------------- 1 file changed, 52 insertions(+), 38 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 1875a4332d..61e16b22db 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -261,12 +261,7 @@ (define (show-help) ;;; Daemon/substituter protocol. ;;; -(define %prefer-fast-decompression? - ;; Whether to prefer fast decompression over good compression ratios. This - ;; serves in particular to choose between lzip (high compression ratio but - ;; low decompression throughput) and zstd (lower compression ratio but high - ;; decompression throughput). - #f) +(define %default-prefer-fast-decompression? #f) (define (call-with-cpu-usage-monitoring proc) (let ((before (times))) @@ -297,7 +292,7 @@ (define (display-narinfo-data port narinfo) (let ((uri compression file-size (narinfo-best-uri narinfo #:fast-decompression? - %prefer-fast-decompression?))) + %default-prefer-fast-decompression?))) (format port "~a\n~a\n" (or file-size 0) (or (narinfo-size narinfo) 0)))) @@ -453,7 +448,8 @@ (define-syntax-rule (catch-system-error exp) (define* (download-nar narinfo destination #:key status-port deduplicate? print-build-trace? - (fetch-timeout %fetch-timeout)) + (fetch-timeout %fetch-timeout) + prefer-fast-decompression?) "Download the nar prescribed in NARINFO, which is assumed to be authentic and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and if DESTINATION is in the store, deduplicate its files. Print a status line to @@ -525,7 +521,7 @@ (define* (download-nar narinfo destination (let ((choices (narinfo-preferred-uris narinfo #:fast-decompression? - %prefer-fast-decompression?))) + prefer-fast-decompression?))) ;; 'guix publish' without '--cache' doesn't specify a Content-Length, so ;; DOWNLOAD-SIZE is #f in this case. (let* ((raw uri compression download-size (try-fetch choices)) @@ -566,21 +562,6 @@ (define* (download-nar narinfo destination deduplicate?) dump-file/deduplicate* dump-file)))) - - ;; Create a hysteresis: depending on CPU usage, favor compression - ;; methods with faster decompression (like ztsd) or methods with better - ;; compression ratios (like lzip). This stems from the observation that - ;; substitution can be CPU-bound when high-speed networks are used: - ;; . - ;; To simulate "slow" networking or changing conditions, run: - ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency 50ms burst 1540 - ;; and then cancel with: - ;; sudo tc qdisc del dev eno1 root - (when (> cpu-usage .8) - (set! %prefer-fast-decompression? #t)) - (when (< cpu-usage .2) - (set! %prefer-fast-decompression? #f)) - (close-port hashed) (close-port input) @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination (format status-port "hash-mismatch ~a ~a ~a~%" (hash-algorithm-name algorithm) (bytevector->nix-base32-string expected) - (bytevector->nix-base32-string actual))))))) + (bytevector->nix-base32-string actual)))) + + cpu-usage))) (define (system-error? exception) "Return true if EXCEPTION is a Guile 'system-error exception." @@ -628,7 +611,8 @@ (define network-error? (define* (process-substitution/fallback port narinfo destination #:key cache-urls acl - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + prefer-fast-decompression?) "Attempt to substitute NARINFO, which is assumed to be authorized or equivalent, by trying to download its nar from each entry in CACHE-URLS. @@ -662,14 +646,17 @@ (define* (process-substitution/fallback port narinfo destination (download-nar alternate destination #:status-port port #:deduplicate? deduplicate? - #:print-build-trace? print-build-trace?)) + #:print-build-trace? print-build-trace? + #:prefer-fast-decompression? + prefer-fast-decompression?)) (loop rest))) (() (loop rest))))))) (define* (process-substitution port store-item destination #:key cache-urls acl - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + prefer-fast-decompression?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL, and verify its hash against what appears in the narinfo. When DEDUPLICATE? is true, and if @@ -701,11 +688,14 @@ (define* (process-substitution port store-item destination #:acl acl #:deduplicate? deduplicate? #:print-build-trace? - print-build-trace?))) + print-build-trace? + #:prefer-fast-decompression? + prefer-fast-decompression?))) (download-nar narinfo destination #:status-port port #:deduplicate? deduplicate? - #:print-build-trace? print-build-trace?))) + #:print-build-trace? print-build-trace? + #:prefer-fast-decompression? prefer-fast-decompression?))) ;;; @@ -895,18 +885,42 @@ (define-command (guix-substitute . args) ;; Specify the number of columns of the terminal so the progress ;; report displays nicely. (parameterize ((current-terminal-columns (client-terminal-columns))) - (let loop () + (let loop ((prefer-fast-decompression? + %default-prefer-fast-decompression?)) (match (read-line) ((? eof-object?) #t) ((= string-tokenize ("substitute" store-path destination)) - (process-substitution reply-port store-path destination - #:cache-urls (substitute-urls) - #:acl (current-acl) - #:deduplicate? deduplicate? - #:print-build-trace? - print-build-trace?) - (loop)))))) + (let ((cpu-usage + (process-substitution reply-port store-path destination + #:cache-urls (substitute-urls) + #:acl (current-acl) + #:deduplicate? deduplicate? + #:print-build-trace? + print-build-trace? + #:prefer-fast-decompression? + prefer-fast-decompression?))) + + ;; Create a hysteresis: depending on CPU usage, favor + ;; compression methods with faster decompression (like ztsd) + ;; or methods with better compression ratios (like lzip). + ;; This stems from the observation that substitution can be + ;; CPU-bound when high-speed networks are used: + ;; . + ;; To simulate "slow" networking or changing conditions, run: + ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency + ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del + ;; dev eno1 root + (loop (cond + ;; Whether to prefer fast decompression over good + ;; compression ratios. This serves in particular to + ;; choose between lzip (high compression ratio but low + ;; decompression throughput) and zstd (lower + ;; compression ratio but high decompression + ;; throughput). + ((> cpu-usage .8) #t) + ((< cpu-usage .2) #f) + (else prefer-fast-decompression?))))))))) (opts (leave (G_ "~a: unrecognized options~%") opts)))))) From patchwork Tue Feb 20 19:42:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60817 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 22F1A27BBE2; Tue, 20 Feb 2024 19:43:13 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id EEC7827BBEA for ; Tue, 20 Feb 2024 19:43:11 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcW0s-0004E2-DF; Tue, 20 Feb 2024 14:42:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcW0q-0004Cb-Be for guix-patches@gnu.org; Tue, 20 Feb 2024 14:42:44 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcW0q-00016T-2t; Tue, 20 Feb 2024 14:42:44 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcW1A-0002Zp-10; Tue, 20 Feb 2024 14:43:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69291] [PATCH 5/5] scripts: substitute: Extract script specific output from download-nar. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:43:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69291 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69291@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69291-submit@debbugs.gnu.org id=B69291.17084581829876 (code B ref 69291); Tue, 20 Feb 2024 19:43:03 +0000 Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:43:02 +0000 Received: from localhost ([127.0.0.1]:46681 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW17-0002Yy-Kj for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:43:02 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43162) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcW0y-0002Y3-Db for 69291@debbugs.gnu.org; Tue, 20 Feb 2024 14:42:53 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id 1138A27BBEA for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:30 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 23ba8778 for <69291@debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:29 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:42:27 +0000 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches As this moves download-nar in a direction where it could be used outside the substitute script. * guix/scripts/substitute.scm (download-nar): Return more information and move status-port output to… (guix-substitute): here. Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0 --- guix/scripts/substitute.scm | 49 +++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 61e16b22db..94eb6d2f71 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -568,26 +568,10 @@ (define* (download-nar narinfo destination ;; Wait for the reporter to finish. (every (compose zero? cdr waitpid) pids) - ;; Skip a line after what 'progress-reporter/file' printed, and another - ;; one to visually separate substitutions. When PRINT-BUILD-TRACE? is - ;; true, leave it up to (guix status) to prettify things. - (newline (current-error-port)) - (unless print-build-trace? - (newline (current-error-port))) - - ;; Check whether we got the data announced in NARINFO. - (let ((actual (get-hash))) - (if (bytevector=? actual expected) - ;; Tell the daemon that we're done. - (format status-port "success ~a ~a~%" - (narinfo-hash narinfo) (narinfo-size narinfo)) - ;; The actual data has a different hash than that in NARINFO. - (format status-port "hash-mismatch ~a ~a ~a~%" - (hash-algorithm-name algorithm) - (bytevector->nix-base32-string expected) - (bytevector->nix-base32-string actual)))) - - cpu-usage))) + (values narinfo + expected + (get-hash) + cpu-usage)))) (define (system-error? exception) "Return true if EXCEPTION is a Guile 'system-error exception." @@ -891,7 +875,10 @@ (define-command (guix-substitute . args) ((? eof-object?) #t) ((= string-tokenize ("substitute" store-path destination)) - (let ((cpu-usage + (let ((narinfo + expected-hash + actual-hash + cpu-usage (process-substitution reply-port store-path destination #:cache-urls (substitute-urls) #:acl (current-acl) @@ -901,6 +888,26 @@ (define-command (guix-substitute . args) #:prefer-fast-decompression? prefer-fast-decompression?))) + ;; Skip a line after what 'progress-reporter/file' printed, + ;; and another one to visually separate substitutions. When + ;; PRINT-BUILD-TRACE? is true, leave it up to (guix status) + ;; to prettify things. + (newline (current-error-port)) + (unless print-build-trace? + (newline (current-error-port))) + + ;; Check whether we got the data announced in NARINFO. + (if (bytevector=? actual-hash expected-hash) + ;; Tell the daemon that we're done. + (format reply-port "success ~a ~a~%" + (narinfo-hash narinfo) (narinfo-size narinfo)) + ;; The actual data has a different hash than that in NARINFO. + (format reply-port "hash-mismatch ~a ~a ~a~%" + (hash-algorithm-name + (narinfo-hash-algorithm+value narinfo)) + (bytevector->nix-base32-string expected-hash) + (bytevector->nix-base32-string actual-hash))) + ;; Create a hysteresis: depending on CPU usage, favor ;; compression methods with faster decompression (like ztsd) ;; or methods with better compression ratios (like lzip).