From patchwork Sat Feb 13 13:47:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27027 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0063E27BC2E; Sat, 13 Feb 2021 13:49:01 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id A9D0D27BC27 for ; Sat, 13 Feb 2021 13:49:01 +0000 (GMT) Received: from localhost ([::1]:46092 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIB-0003JN-9E for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55148) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHG-0002gO-E8 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50509) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHG-00076A-7G for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHG-0002wi-59 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 01/14] substitute: Remove buffer handling from fetch. References: <87y2hn9l8j.fsf@cbaines.net> In-Reply-To: <87y2hn9l8j.fsf@cbaines.net> Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404411150 (code B ref 45409); Sat, 13 Feb 2021 13:48:02 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:24 +0000 Received: from localhost ([127.0.0.1]:33786 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGc-0002tA-Mf for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:23 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48186) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGb-0002sk-9o for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:21 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 50D6E27BC27 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 821f6a77 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:06 +0000 Message-Id: <20210213134719.19625-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches http-fetch does this, so just set the right option. * guix/scripts/substitute.scm (fetch): Remove buffering code, and pass #:buffered? to http-fetch. --- guix/scripts/substitute.scm | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index f9bcead045..88610a0781 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -204,10 +204,9 @@ connection (typically PORT) is kept open once data has been fetched from URI." (when (or (not port) (port-closed? port)) (set! port (guix:open-connection-for-uri uri #:verify-certificate? #f))) - (unless (or buffered? (not (file-port? port))) - (setvbuf port 'none)) (http-fetch uri #:text? #f #:port port #:keep-alive? keep-alive? + #:buffered? buffered? #:verify-certificate? #f)))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") From patchwork Sat Feb 13 13:47:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27021 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8518927BC2E; Sat, 13 Feb 2021 13:48:32 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 3B12A27BC27 for ; Sat, 13 Feb 2021 13:48:32 +0000 (GMT) Received: from localhost ([::1]:44806 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvHj-0002i2-Bp for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55160) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHH-0002h4-0W for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50510) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHG-00076X-PZ for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHG-0002wp-JG for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 02/14] substitute: Remove connection handling from fetch. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404711186 (code B ref 45409); Sat, 13 Feb 2021 13:48:02 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:27 +0000 Received: from localhost ([127.0.0.1]:33793 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGh-0002uL-0L for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48192) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGb-0002sn-9q for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:22 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 69B1E27BC2E for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 48b691d5 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:07 +0000 Message-Id: <20210213134719.19625-2-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches http-fetch does this, so just use that code instead. * guix/scripts/substitute.scm (fetch): Remove connection handling when the port is closed. --- guix/scripts/substitute.scm | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 88610a0781..323957910a 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -200,14 +200,10 @@ connection (typically PORT) is kept open once data has been fetched from URI." (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (begin - (when (or (not port) (port-closed? port)) - (set! port (guix:open-connection-for-uri - uri #:verify-certificate? #f))) - (http-fetch uri #:text? #f #:port port - #:keep-alive? keep-alive? - #:buffered? buffered? - #:verify-certificate? #f)))))) + (http-fetch uri #:text? #f #:port port + #:keep-alive? keep-alive? + #:buffered? buffered? + #:verify-certificate? #f))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) From patchwork Sat Feb 13 13:47:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27029 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 9FD4B27BC2E; Sat, 13 Feb 2021 13:49:18 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 769B627BC27 for ; Sat, 13 Feb 2021 13:49:18 +0000 (GMT) Received: from localhost ([::1]:46548 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIT-0003Ue-JP for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55196) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHU-0002kS-TR for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:16 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50517) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHL-00078Z-Py for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:11 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHL-0002xh-6Z for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:07 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 03/14] substitute: Remove redundant let block from fetch. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:07 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404911237 (code B ref 45409); Sat, 13 Feb 2021 13:48:07 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:29 +0000 Received: from localhost ([127.0.0.1]:33807 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGj-0002v5-9u for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:29 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48190) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGb-0002sm-9o for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:23 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 7401B27BC45 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b385960b for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:08 +0000 Message-Id: <20210213134719.19625-3-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/scripts/substitute.scm (fetch): Remove redundant let block. --- guix/scripts/substitute.scm | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 323957910a..f01892776e 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -192,18 +192,17 @@ connection (typically PORT) is kept open once data has been fetched from URI." ;; sudo tc qdisc add dev eth0 root netem delay 1500ms ;; and then cancel with: ;; sudo tc qdisc del dev eth0 root - (let ((port port)) - (with-timeout (if timeout? - %fetch-timeout - 0) - (begin - (warning (G_ "while fetching ~a: server is somewhat slow~%") - (uri->string uri)) - (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f #:port port - #:keep-alive? keep-alive? - #:buffered? buffered? - #:verify-certificate? #f))))) + (with-timeout (if timeout? + %fetch-timeout + 0) + (begin + (warning (G_ "while fetching ~a: server is somewhat slow~%") + (uri->string uri)) + (warning (G_ "try `--no-substitutes' if the problem persists~%"))) + (http-fetch uri #:text? #f #:port port + #:keep-alive? keep-alive? + #:buffered? buffered? + #:verify-certificate? #f)))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) From patchwork Sat Feb 13 13:47:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27024 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 04F1F27BC27; Sat, 13 Feb 2021 13:48:41 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 409DE27BC45 for ; Sat, 13 Feb 2021 13:48:40 +0000 (GMT) Received: from localhost ([::1]:45542 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvHr-00035j-Ds for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHJ-0002iD-Dt for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50514) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHJ-00077T-3l for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHI-0002xK-Um for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 04/14] guix: Move http-multiple-get to (guix http-client). Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404811216 (code B ref 45409); Sat, 13 Feb 2021 13:48:04 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:28 +0000 Received: from localhost ([127.0.0.1]:33801 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGi-0002uj-7R for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:28 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48188) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGb-0002sl-Cq for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:23 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 8D1BE27BC46 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 14a92826 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:09 +0000 Message-Id: <20210213134719.19625-4-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From (guix scripts substitute). This will make it easier to reuse this code. * guix/scripts/substitute.scm (http-multiple-get): Remove, and move to… * guix/http-client.scm (http-multiple-get): …here. --- guix/http-client.scm | 76 +++++++++++++++++++++++++++++++++++++ guix/scripts/substitute.scm | 70 ---------------------------------- 2 files changed, 76 insertions(+), 70 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 553640fe9e..7ead493633 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -21,8 +21,11 @@ (define-module (guix http-client) #:use-module (web uri) + #:use-module (web http) #:use-module ((web client) #:hide (open-socket-for-uri)) + #:use-module (web request) #:use-module (web response) + #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) @@ -50,6 +53,7 @@ http-get-error-reason http-fetch + http-multiple-get %http-cache-ttl http-fetch/cached)) @@ -138,6 +142,78 @@ Raise an '&http-get-error' condition if downloading fails." (uri->string uri) code (response-reason-phrase resp)))))))))))) +(define* (http-multiple-get base-uri proc seed requests + #:key port (verify-certificate? #t) + (open-connection guix:open-connection-for-uri) + (keep-alive? #t) + (batch-size 1000)) + "Send all of REQUESTS to the server at BASE-URI. Call PROC for each +response, passing it the request object, the response, a port from which to +read the response body, and the previous result, starting with SEED, à la +'fold'. Return the final result. + +When PORT is specified, use it as the initial connection on which HTTP +requests are sent; otherwise call OPEN-CONNECTION to open a new connection for +a URI. When KEEP-ALIVE? is false, close the connection port before +returning." + (let connect ((port port) + (requests requests) + (result seed)) + (define batch + (if (>= batch-size (length requests)) + requests + (take requests batch-size))) + + ;; (format (current-error-port) "connecting (~a requests left)..." + ;; (length requests)) + (let ((p (or port (open-connection base-uri + #:verify-certificate? + verify-certificate?)))) + ;; For HTTPS, P is not a file port and does not support 'setvbuf'. + (when (file-port? p) + (setvbuf p 'block (expt 2 16))) + + ;; Send BATCH in a row. + ;; XXX: Do our own caching to work around inefficiencies when + ;; communicating over TLS: . + (let-values (((buffer get) (open-bytevector-output-port))) + ;; Inherit the HTTP proxying property from P. + (set-http-proxy-port?! buffer (http-proxy-port? p)) + + (for-each (cut write-request <> buffer) + batch) + (put-bytevector p (get)) + (force-output p)) + + ;; Now start processing responses. + (let loop ((sent batch) + (processed 0) + (result result)) + (match sent + (() + (match (drop requests processed) + (() + (unless keep-alive? + (close-port p)) + (reverse result)) + (remainder + (connect p remainder result)))) + ((head tail ...) + (let* ((resp (read-response p)) + (body (response-body-port resp)) + (result (proc head resp body result))) + ;; The server can choose to stop responding at any time, in which + ;; case we have to try again. Check whether that is the case. + ;; Note that even upon "Connection: close", we can read from BODY. + (match (assq 'connection (response-headers resp)) + (('connection 'close) + (close-port p) + (connect #f ;try again + (drop requests (+ 1 processed)) + result)) + (_ + (loop tail (+ 1 processed) result)))))))))) ;keep going + ;;; ;;; Caching. diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index f01892776e..fc6bb54301 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -299,76 +299,6 @@ return its MAX-LENGTH first elements and its tail." (values (reverse result) lst) (loop (+ 1 len) tail (cons head result))))))) -(define* (http-multiple-get base-uri proc seed requests - #:key port (verify-certificate? #t) - (open-connection guix:open-connection-for-uri) - (keep-alive? #t) - (batch-size 1000)) - "Send all of REQUESTS to the server at BASE-URI. Call PROC for each -response, passing it the request object, the response, a port from which to -read the response body, and the previous result, starting with SEED, à la -'fold'. Return the final result. - -When PORT is specified, use it as the initial connection on which HTTP -requests are sent; otherwise call OPEN-CONNECTION to open a new connection for -a URI. When KEEP-ALIVE? is false, close the connection port before -returning." - (let connect ((port port) - (requests requests) - (result seed)) - (define batch - (at-most batch-size requests)) - - ;; (format (current-error-port) "connecting (~a requests left)..." - ;; (length requests)) - (let ((p (or port (open-connection base-uri - #:verify-certificate? - verify-certificate?)))) - ;; For HTTPS, P is not a file port and does not support 'setvbuf'. - (when (file-port? p) - (setvbuf p 'block (expt 2 16))) - - ;; Send BATCH in a row. - ;; XXX: Do our own caching to work around inefficiencies when - ;; communicating over TLS: . - (let-values (((buffer get) (open-bytevector-output-port))) - ;; Inherit the HTTP proxying property from P. - (set-http-proxy-port?! buffer (http-proxy-port? p)) - - (for-each (cut write-request <> buffer) - batch) - (put-bytevector p (get)) - (force-output p)) - - ;; Now start processing responses. - (let loop ((sent batch) - (processed 0) - (result result)) - (match sent - (() - (match (drop requests processed) - (() - (unless keep-alive? - (close-port p)) - (reverse result)) - (remainder - (connect p remainder result)))) - ((head tail ...) - (let* ((resp (read-response p)) - (body (response-body-port resp)) - (result (proc head resp body result))) - ;; The server can choose to stop responding at any time, in which - ;; case we have to try again. Check whether that is the case. - ;; Note that even upon "Connection: close", we can read from BODY. - (match (assq 'connection (response-headers resp)) - (('connection 'close) - (close-port p) - (connect #f ;try again - (drop requests (+ 1 processed)) - result)) - (_ - (loop tail (+ 1 processed) result)))))))))) ;keep going - (define (read-to-eof port) "Read from PORT until EOF is reached. The data are discarded." (dump-port port (%make-void-port "w"))) From patchwork Sat Feb 13 13:47:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27030 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B677627BC2E; Sat, 13 Feb 2021 13:49:23 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 649D027BC27 for ; Sat, 13 Feb 2021 13:49:23 +0000 (GMT) Received: from localhost ([::1]:46910 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIY-0003eI-I6 for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55166) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHH-0002hb-GI for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50511) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHH-00076d-8D for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHH-0002wx-4B for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 05/14] http-client: Add error handling to http-multiple-get. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404711194 (code B ref 45409); Sat, 13 Feb 2021 13:48:03 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:27 +0000 Received: from localhost ([127.0.0.1]:33795 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGh-0002uN-8f for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48194) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGb-0002so-AC for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:22 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id A34D927BC47 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 81cf0d6a for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:10 +0000 Message-Id: <20210213134719.19625-5-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Making sure to close the port if it looks to be unusable. This closing of the port will allow for caching connections, without caching broken connections, as the cache can avoid handing out closed ports. * guix/http-client.scm (http-multiple-get): Try to catch exceptions that happen if the port is unusable, this is a adaptation of code within the (guix scripts substitute) module. --- guix/http-client.scm | 74 +++++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 18 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 7ead493633..3aba3b28c1 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -38,6 +38,7 @@ #:use-module (guix utils) #:use-module (guix base64) #:autoload (gcrypt hash) (sha256) + #:autoload (gnutls) (error/invalid-session) #:use-module ((guix build utils) #:select (mkdir-p dump-port)) #:use-module ((guix build download) @@ -180,10 +181,25 @@ returning." ;; Inherit the HTTP proxying property from P. (set-http-proxy-port?! buffer (http-proxy-port? p)) - (for-each (cut write-request <> buffer) - batch) - (put-bytevector p (get)) - (force-output p)) + (catch #t + (lambda () + (for-each (cut write-request <> buffer) + batch) + (put-bytevector p (get)) + (force-output p)) + (lambda (key . args) + ;; If PORT becomes unusable, open a fresh connection and + ;; retry. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session))) + (begin + (close-port p) ; close the broken port + (connect #f + requests + result)) + (apply throw key args))))) ;; Now start processing responses. (let loop ((sent batch) @@ -199,20 +215,42 @@ returning." (remainder (connect p remainder result)))) ((head tail ...) - (let* ((resp (read-response p)) - (body (response-body-port resp)) - (result (proc head resp body result))) - ;; The server can choose to stop responding at any time, in which - ;; case we have to try again. Check whether that is the case. - ;; Note that even upon "Connection: close", we can read from BODY. - (match (assq 'connection (response-headers resp)) - (('connection 'close) - (close-port p) - (connect #f ;try again - (drop requests (+ 1 processed)) - result)) - (_ - (loop tail (+ 1 processed) result)))))))))) ;keep going + (catch #t + (lambda () + (let* ((resp (read-response p)) + (body (response-body-port resp)) + (result (proc head resp body result))) + ;; The server can choose to stop responding at any time, + ;; in which case we have to try again. Check whether + ;; that is the case. Note that even upon "Connection: + ;; close", we can read from BODY. + (match (assq 'connection (response-headers resp)) + (('connection 'close) + (close-port p) + (connect #f ;try again + (drop requests (+ 1 processed)) + result)) + (_ + (loop tail (+ 1 processed) result))))) ;keep going + (lambda (key . args) + ;; If PORT was cached and the server closed the connection + ;; in the meantime, we get EPIPE. In that case, open a + ;; fresh connection and retry. We might also get + ;; 'bad-response or a similar exception from (web response) + ;; later on, once we've sent the request, or a + ;; ERROR/INVALID-SESSION from GnuTLS. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session)) + (memq key + '(bad-response bad-header bad-header-component))) + (begin + (close-port p) + (connect #f ; try again + (drop requests (+ 1 processed)) + result)) + (apply throw key args)))))))))) ;;; From patchwork Sat Feb 13 13:47:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27032 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id D715C27BC2E; Sat, 13 Feb 2021 13:49:30 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id ACE5C27BC27 for ; Sat, 13 Feb 2021 13:49:30 +0000 (GMT) Received: from localhost ([::1]:47762 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIf-00041L-SZ for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55172) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHI-0002i1-4q for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50512) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHH-00076z-O9 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHH-0002x5-L0 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 06/14] substitute: open-connection-for-uri/maybe add #:verify-certificate?. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404811201 (code B ref 45409); Sat, 13 Feb 2021 13:48:03 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:28 +0000 Received: from localhost ([127.0.0.1]:33797 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGh-0002uV-LF for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48196) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGc-0002sw-78 for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:22 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id B6D9127BC48 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 426c47ef for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:11 +0000 Message-Id: <20210213134719.19625-6-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches As this is used by http-fetch and http-multiple-get when they call the specified open connection procedure. * guix/scripts/substitute.scm (open-connection-for-uri/maybe): Support #:verify-certificate?. --- guix/scripts/substitute.scm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index fc6bb54301..f01c11b020 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -322,7 +322,8 @@ if file doesn't exist, and the narinfo otherwise." (define* (open-connection-for-uri/maybe uri #:key fresh? - (time %fetch-timeout)) + (time %fetch-timeout) + verify-certificate?) "Open a connection to URI via 'open-connection-for-uri/cached' and return a port to it, or, if connection failed, print a warning and return #f. Pass #:fresh? to 'open-connection-for-uri/cached'." @@ -332,7 +333,8 @@ port to it, or, if connection failed, print a warning and return #f. Pass (catch #t (lambda () (open-connection-for-uri/cached uri #:timeout time - #:fresh? fresh?)) + #:fresh? fresh? + #:verify-certificate? verify-certificate?)) (match-lambda* (('getaddrinfo-error error) (unless (hash-ref %unreachable-hosts host) From patchwork Sat Feb 13 13:47:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27022 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A261F27BC2E; Sat, 13 Feb 2021 13:48:33 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 462ED27BC27 for ; Sat, 13 Feb 2021 13:48:33 +0000 (GMT) Received: from localhost ([::1]:44862 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvHk-0002kG-FP for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55174) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHJ-0002iC-0H for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50513) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHI-00077N-MG for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHI-0002xC-4B for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 07/14] substitute: Stop using call-with-cached-connection in fetch-narinfos. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404811208 (code B ref 45409); Sat, 13 Feb 2021 13:48:04 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:28 +0000 Received: from localhost ([127.0.0.1]:33799 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGh-0002uc-Tw for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:28 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48198) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGc-0002sx-7C for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:23 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id D04BB27BC49 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 08f393b6 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:12 +0000 Message-Id: <20210213134719.19625-7-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Instead, just pass open-connection-for-uri/maybe to http-multiple-get. This code should be functionaly similar to the previous code. The eventual aim of this is to make the connection caching not mandatory in fetch-narinfos. * guix/scripts/substitute.scm (fetch-narinfos): Remove use of call-with-cached-connection. --- guix/scripts/substitute.scm | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index f01c11b020..cd52ad747e 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -412,20 +412,14 @@ port to it, or, if connection failed, print a warning and return #f. Pass ;; on the X.509 PKI. We can do it because we authenticate ;; narinfos, which provides a much stronger guarantee. (let* ((requests (map (cut narinfo-request url <>) paths)) - (result (call-with-cached-connection uri - (lambda (port) - (if port - (begin - (update-progress!) - (http-multiple-get uri - handle-narinfo-response '() - requests - #:open-connection - open-connection-for-uri/cached - #:verify-certificate? #f - #:port port)) - '())) - open-connection-for-uri/maybe))) + (result (begin + (update-progress!) + (http-multiple-get uri + handle-narinfo-response '() + requests + #:open-connection + open-connection-for-uri/maybe + #:verify-certificate? #f)))) (newline (current-error-port)) result)) ((file #f) From patchwork Sat Feb 13 13:47:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27033 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 357F927BC2E; Sat, 13 Feb 2021 13:49:36 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 0878A27BC27 for ; Sat, 13 Feb 2021 13:49:36 +0000 (GMT) Received: from localhost ([::1]:48256 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIl-0004DL-8G for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55178) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHL-0002jF-KX for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50515) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHK-000785-1y for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHJ-0002xS-EB for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 08/14] http-client: Accept #:open-connection in http-fetch. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:05 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404911223 (code B ref 45409); Sat, 13 Feb 2021 13:48:05 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:29 +0000 Received: from localhost ([127.0.0.1]:33803 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGi-0002ur-Nu for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:28 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48200) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGc-0002t5-HP for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:23 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id DC17627BC4A for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id fda78696 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:13 +0000 Message-Id: <20210213134719.19625-8-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches So that an alternative procedure can be passed in, perhaps to perform connection caching. * guix/http-client.scm (http-fetch): Add an #:open-connection keyword argument. --- guix/http-client.scm | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 3aba3b28c1..2d7458a56e 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -75,6 +75,7 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) + (open-connection guix:open-connection-for-uri) (keep-alive? #f) (verify-certificate? #t) (headers '((user-agent . "GNU Guile"))) @@ -97,10 +98,10 @@ Raise an '&http-get-error' condition if downloading fails." (let loop ((uri (if (string? uri) (string->uri uri) uri))) - (let ((port (or port (guix:open-connection-for-uri uri - #:verify-certificate? - verify-certificate? - #:timeout timeout))) + (let ((port (or port (open-connection uri + #:verify-certificate? + verify-certificate? + #:timeout timeout))) (headers (match (uri-userinfo uri) ((? string? str) (cons (cons 'Authorization From patchwork Sat Feb 13 13:47:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27025 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B266E27BC45; Sat, 13 Feb 2021 13:48:46 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6A73827BC27 for ; Sat, 13 Feb 2021 13:48:46 +0000 (GMT) Received: from localhost ([::1]:45660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvHx-00038i-0Z for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55182) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHL-0002jJ-Kw for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50516) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHK-000786-Rn for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHK-0002xZ-A8 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:06 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 09/14] substitute: Change connection cache handling in process-substitution. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404911230 (code B ref 45409); Sat, 13 Feb 2021 13:48:06 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:29 +0000 Received: from localhost ([127.0.0.1]:33805 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGj-0002uy-05 for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:29 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48202) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGc-0002t7-HQ for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:23 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 008AA27BC4B for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b4b82949 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:14 +0000 Message-Id: <20210213134719.19625-9-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Just pass open-connection-for-uri/maybe to http-fetch, this removes the need for with-cached-connection and passing the port in. * guix/scripts/substitute.scm (fetch): Don't take a port as an argument, and pass open-connection-for-uri/maybe to http-fetch. (process-substitution): Don't call fetch with with-cached-connection. --- guix/scripts/substitute.scm | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index cd52ad747e..5d4884a7db 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -170,7 +170,7 @@ again." (apply values result))))) (define* (fetch uri #:key (buffered? #t) (timeout? #t) - (keep-alive? #f) (port #f)) + (keep-alive? #f)) "Return a binary input port to URI and the number of bytes it's expected to provide. @@ -199,7 +199,8 @@ connection (typically PORT) is kept open once data has been fetched from URI." (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f #:port port + (http-fetch uri #:text? #f + #:open-connection open-connection-for-uri/maybe #:keep-alive? keep-alive? #:buffered? buffered? #:verify-certificate? #f)))) @@ -752,10 +753,8 @@ the current output port." (let*-values (((raw download-size) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. - (with-cached-connection uri port - (fetch uri #:buffered? #f #:timeout? #f - #:port port - #:keep-alive? #t))) + (fetch uri #:buffered? #f #:timeout? #f + #:keep-alive? #t)) ((progress) (let* ((dl-size (or download-size (and (equal? compression "none") From patchwork Sat Feb 13 13:47:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27026 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6061D27BC2E; Sat, 13 Feb 2021 13:48:49 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id F337127BC27 for ; Sat, 13 Feb 2021 13:48:48 +0000 (GMT) Received: from localhost ([::1]:45878 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvI0-0003E3-0T for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55200) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHV-0002kn-5o for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:17 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50519) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHN-00079S-Sw for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:11 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHN-0002xy-3y for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:09 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 10/14] substitute: Remove now redundant connection caching helpers. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:09 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322405011252 (code B ref 45409); Sat, 13 Feb 2021 13:48:09 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:30 +0000 Received: from localhost ([127.0.0.1]:33811 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGj-0002vK-Vi for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:30 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48204) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGd-0002tJ-2g for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:24 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0F47727BC4C for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:21 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 94cae6e3 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:15 +0000 Message-Id: <20210213134719.19625-10-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Failures now should be handled where they occur, and if there's a problem that's symptomatic of an issue with the connection, the port should be closed. * guix/scripts/substitute.scm (call-with-cached-connection): Remove procedure. (with-cached-connection): Remove syntax rule. --- guix/scripts/substitute.scm | 28 ---------------------------- 1 file changed, 28 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 5d4884a7db..a2b1526cc6 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -690,32 +690,6 @@ server certificates." (drain-input socket) socket)))))))) -(define* (call-with-cached-connection uri proc - #:optional - (open-connection - open-connection-for-uri/cached)) - (let ((port (open-connection uri))) - (catch #t - (lambda () - (proc port)) - (lambda (key . args) - ;; If PORT was cached and the server closed the connection in the - ;; meantime, we get EPIPE. In that case, open a fresh connection and - ;; retry. We might also get 'bad-response or a similar exception from - ;; (web response) later on, once we've sent the request, or a - ;; ERROR/INVALID-SESSION from GnuTLS. - (if (or (and (eq? key 'system-error) - (= EPIPE (system-error-errno `(,key ,@args)))) - (and (eq? key 'gnutls-error) - (eq? (first args) error/invalid-session)) - (memq key '(bad-response bad-header bad-header-component))) - (proc (open-connection uri #:fresh? #t)) - (apply throw key args)))))) - -(define-syntax-rule (with-cached-connection uri port exp ...) - "Bind PORT with EXP... to a socket connected to URI." - (call-with-cached-connection uri (lambda (port) exp ...))) - (define* (process-substitution store-item destination #:key cache-urls acl deduplicate? print-build-trace?) @@ -1011,8 +985,6 @@ default value." ;;; Local Variables: ;;; eval: (put 'with-timeout 'scheme-indent-function 1) -;;; eval: (put 'with-cached-connection 'scheme-indent-function 2) -;;; eval: (put 'call-with-cached-connection 'scheme-indent-function 1) ;;; End: ;;; substitute.scm ends here From patchwork Sat Feb 13 13:47:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27023 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A706E27BC2E; Sat, 13 Feb 2021 13:48:35 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 747E427BC27 for ; Sat, 13 Feb 2021 13:48:35 +0000 (GMT) Received: from localhost ([::1]:45044 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvHm-0002qk-J6 for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:48:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55206) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHV-0002kt-7O for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:17 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50521) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHQ-0007AB-Cr for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:16 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHP-0002yE-OK for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:11 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 11/14] substitute: Remove redundant fetch arguments. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:11 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322405111267 (code B ref 45409); Sat, 13 Feb 2021 13:48:11 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:31 +0000 Received: from localhost ([127.0.0.1]:33815 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGk-0002vZ-TF for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:31 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48206) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGd-0002tK-3P for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:24 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 1C3C627BC4D for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:21 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 7b36c04b for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:16 +0000 Message-Id: <20210213134719.19625-11-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches It's just called in one place, with hardcoded argument values, so just inline them. * guix/scripts/substitute.scm (fetch): Remove arguments that don't vary, copy the values from the call site in process-substitution. (process-substitution): Remove unnecessary argument values from fetch call. --- guix/scripts/substitute.scm | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index a2b1526cc6..26fd05429f 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -169,18 +169,12 @@ again." (sigaction SIGALRM SIG_DFL) (apply values result))))) -(define* (fetch uri #:key (buffered? #t) (timeout? #t) - (keep-alive? #f)) +(define (fetch uri) "Return a binary input port to URI and the number of bytes it's expected to -provide. - -When PORT is true, use it as the underlying I/O port for HTTP transfers; when -PORT is false, open a new connection for URI. When KEEP-ALIVE? is true, the -connection (typically PORT) is kept open once data has been fetched from URI." +provide." (case (uri-scheme uri) ((file) - (let ((port (open-file (uri-path uri) - (if buffered? "rb" "r0b")))) + (let ((port (open-file (uri-path uri) "r0b"))) (values port (stat:size (stat port))))) ((http https) (guard (c ((http-get-error? c) @@ -192,17 +186,15 @@ connection (typically PORT) is kept open once data has been fetched from URI." ;; sudo tc qdisc add dev eth0 root netem delay 1500ms ;; and then cancel with: ;; sudo tc qdisc del dev eth0 root - (with-timeout (if timeout? - %fetch-timeout - 0) + (with-timeout %fetch-timeout (begin (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) (http-fetch uri #:text? #f #:open-connection open-connection-for-uri/maybe - #:keep-alive? keep-alive? - #:buffered? buffered? + #:keep-alive? #t + #:buffered? #f #:verify-certificate? #f)))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") @@ -727,8 +719,7 @@ the current output port." (let*-values (((raw download-size) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. - (fetch uri #:buffered? #f #:timeout? #f - #:keep-alive? #t)) + (fetch uri)) ((progress) (let* ((dl-size (or download-size (and (equal? compression "none") From patchwork Sat Feb 13 13:47:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27028 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 731BB27BC2E; Sat, 13 Feb 2021 13:49:17 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 37E9627BC27 for ; Sat, 13 Feb 2021 13:49:17 +0000 (GMT) Received: from localhost ([::1]:46414 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIS-0003RM-Bb for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55198) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHU-0002kT-Tk for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:16 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50518) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHM-00079Q-Oi for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:11 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHM-0002xq-4f for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:08 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 12/14] substitute: Inline fetch in to process-substitutes. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:08 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322405011245 (code B ref 45409); Sat, 13 Feb 2021 13:48:08 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:30 +0000 Received: from localhost ([127.0.0.1]:33809 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGj-0002vC-If for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:29 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48196) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGc-0002sw-RC for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:24 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 3331827BC4E for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:21 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 99b9c809 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:17 +0000 Message-Id: <20210213134719.19625-12-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches As it's only called in one place, and this should make the code easier to read. * guix/scripts/substitute.scm (fetch): Move procedure inside… (process-substitution): …here. --- guix/scripts/substitute.scm | 60 ++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 26fd05429f..717c232633 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -169,37 +169,6 @@ again." (sigaction SIGALRM SIG_DFL) (apply values result))))) -(define (fetch uri) - "Return a binary input port to URI and the number of bytes it's expected to -provide." - (case (uri-scheme uri) - ((file) - (let ((port (open-file (uri-path uri) "r0b"))) - (values port (stat:size (stat port))))) - ((http https) - (guard (c ((http-get-error? c) - (leave (G_ "download from '~a' failed: ~a, ~s~%") - (uri->string (http-get-error-uri c)) - (http-get-error-code c) - (http-get-error-reason c)))) - ;; Test this with: - ;; sudo tc qdisc add dev eth0 root netem delay 1500ms - ;; and then cancel with: - ;; sudo tc qdisc del dev eth0 root - (with-timeout %fetch-timeout - (begin - (warning (G_ "while fetching ~a: server is somewhat slow~%") - (uri->string uri)) - (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f - #:open-connection open-connection-for-uri/maybe - #:keep-alive? #t - #:buffered? #f - #:verify-certificate? #f)))) - (else - (leave (G_ "unsupported substitute URI scheme: ~a~%") - (uri->string uri))))) - (define (narinfo-cache-file cache-url path) "Return the name of the local file that contains an entry for PATH. The entry is stored in a sub-directory specific to CACHE-URL." @@ -706,6 +675,35 @@ the current output port." (apply dump-file/deduplicate (append args (list #:store (%store-prefix))))) + (define (fetch uri) + (case (uri-scheme uri) + ((file) + (let ((port (open-file (uri-path uri) "r0b"))) + (values port (stat:size (stat port))))) + ((http https) + (guard (c ((http-get-error? c) + (leave (G_ "download from '~a' failed: ~a, ~s~%") + (uri->string (http-get-error-uri c)) + (http-get-error-code c) + (http-get-error-reason c)))) + ;; Test this with: + ;; sudo tc qdisc add dev eth0 root netem delay 1500ms + ;; and then cancel with: + ;; sudo tc qdisc del dev eth0 root + (with-timeout %fetch-timeout + (begin + (warning (G_ "while fetching ~a: server is somewhat slow~%") + (uri->string uri)) + (warning (G_ "try `--no-substitutes' if the problem persists~%"))) + (http-fetch uri #:text? #f + #:open-connection open-connection-for-uri/maybe + #:keep-alive? #t + #:buffered? #f + #:verify-certificate? #f)))) + (else + (leave (G_ "unsupported substitute URI scheme: ~a~%") + (uri->string uri))))) + (unless narinfo (leave (G_ "no valid substitute for '~a'~%") store-item)) From patchwork Sat Feb 13 13:47:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27034 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8B0B627BC2E; Sat, 13 Feb 2021 13:49:43 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 2A08D27BC27 for ; Sat, 13 Feb 2021 13:49:43 +0000 (GMT) Received: from localhost ([::1]:48474 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIs-0004If-Ci for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55204) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHV-0002kp-6v for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:17 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50520) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHP-00079Y-Bi for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:13 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHO-0002y6-8w for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:10 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 13/14] substitute: Remove fetch-narinfos use open-connection-for-uri/maybe. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:10 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322405111260 (code B ref 45409); Sat, 13 Feb 2021 13:48:10 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:31 +0000 Received: from localhost ([127.0.0.1]:33813 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGk-0002vR-H7 for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:30 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48208) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGd-0002tL-3O for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:24 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 49CC527BC4F for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:21 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 050eadf2 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:18 +0000 Message-Id: <20210213134719.19625-13-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches At least by default. Instead, make the open-connection procedure a parameter, and make the default guix:open-connection-for-uri. Do so similarly for lookup-narinfos and lookup-narinfos/diverse which work towards calling fetch-narinfos. This means this code can be moved to a different module, without having use/move the connection caching code. * guix/scripts/substitute.scm (fetch-narinfos): Add #:open-connection argument, and call http-multiple-get with it. (lookup-narinfos) Add #:open-connection argument, and call fetch-narinfos with it. (lookup-narinfos/diverse): Add #:open-connection argument, and call lookup-narinfos with it. (process-query): Call lookup-narinfos/diverse with #:open-connection open-connection-for-uri/maybe. --- guix/scripts/substitute.scm | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 717c232633..fea2cecef0 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -314,7 +314,8 @@ port to it, or, if connection failed, print a warning and return #f. Pass (args (apply throw args))))) -(define (fetch-narinfos url paths) +(define* (fetch-narinfos url paths + #:key (open-connection guix:open-connection-for-uri)) "Retrieve all the narinfos for PATHS from the cache at URL and return them." (define update-progress! (let ((done 0) @@ -379,8 +380,7 @@ port to it, or, if connection failed, print a warning and return #f. Pass (http-multiple-get uri handle-narinfo-response '() requests - #:open-connection - open-connection-for-uri/maybe + #:open-connection open-connection #:verify-certificate? #f)))) (newline (current-error-port)) result)) @@ -396,7 +396,8 @@ port to it, or, if connection failed, print a warning and return #f. Pass (do-fetch (string->uri url))) -(define (lookup-narinfos cache paths) +(define* (lookup-narinfos cache paths + #:key (open-connection guix:open-connection-for-uri)) "Return the narinfos for PATHS, invoking the server at CACHE when no information is available locally." (let-values (((cached missing) @@ -413,10 +414,13 @@ information is available locally." paths))) (if (null? missing) cached - (let ((missing (fetch-narinfos cache missing))) + (let ((missing (fetch-narinfos cache missing + #:open-connection open-connection))) (append cached (or missing '())))))) -(define (lookup-narinfos/diverse caches paths authorized?) +(define* (lookup-narinfos/diverse caches paths authorized? + #:key (open-connection + guix:open-connection-for-uri)) "Look up narinfos for PATHS on all of CACHES, a list of URLS, in that order. That is, when a cache lacks an AUTHORIZED? narinfo, look it up in the next cache, and so on. @@ -448,7 +452,8 @@ AUTHORIZED? narinfo." (_ (match caches ((cache rest ...) - (let* ((narinfos (lookup-narinfos cache paths)) + (let* ((narinfos (lookup-narinfos cache paths + #:open-connection open-connection)) (definite (map narinfo-path (filter authorized? narinfos))) (missing (lset-difference string=? paths definite))) ;XXX: perf (loop rest missing @@ -588,14 +593,18 @@ authorized substitutes." (match (string-tokenize command) (("have" paths ..1) ;; Return the subset of PATHS available in CACHE-URLS. - (let ((substitutable (lookup-narinfos/diverse cache-urls paths valid?))) + (let ((substitutable (lookup-narinfos/diverse + cache-urls paths valid? + #:open-connection open-connection-for-uri/maybe))) (for-each (lambda (narinfo) (format #t "~a~%" (narinfo-path narinfo))) substitutable) (newline))) (("info" paths ..1) ;; Reply info about PATHS if it's in CACHE-URLS. - (let ((substitutable (lookup-narinfos/diverse cache-urls paths valid?))) + (let ((substitutable (lookup-narinfos/diverse + cache-urls paths valid? + #:open-connection open-connection-for-uri/maybe))) (for-each display-narinfo-data substitutable) (newline))) (wtf From patchwork Sat Feb 13 13:47:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 27031 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3CE8627BC45; Sat, 13 Feb 2021 13:49:27 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id AFE2C27BC27 for ; Sat, 13 Feb 2021 13:49:26 +0000 (GMT) Received: from localhost ([::1]:47258 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIb-0003nS-TT for patchwork@mira.cbaines.net; Sat, 13 Feb 2021 08:49:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55208) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHV-0002l0-6g for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:17 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50522) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHS-0007Ao-L5 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:16 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHQ-0002yM-R1 for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:13 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 14/14] substitute: Rework connection error handling. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:12 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322405111273 (code B ref 45409); Sat, 13 Feb 2021 13:48:12 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:31 +0000 Received: from localhost ([127.0.0.1]:33817 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGl-0002vg-7t for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:31 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48198) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGd-0002sx-Ab for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:25 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 6526227BC50 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:21 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 8265f8c0 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:19 +0000 Message-Id: <20210213134719.19625-14-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches This is part of trying to reduce the interdependency of code within the substitute module. This commit addresses some of the error handling that was performed through open-connection-for-uri/maybe. The new approach is to use call-with-connection-error-handling, and wrap calls to http-multiple-get and http-fetch with that procedure, which takes care of handling connection errors. I think this is even slightly more rigerous than the previous setup, because this approach handles connection errors that occur when http-multiple-get reconnects to a host. * guix/scripts/substitute.scm (open-connection-for-uri/maybe): Transform in to call-with-connection-error-handling. (fetch-narinfos): Use call-with-connection-error-handling. (process-query): Replace open-connection-for-uri/maybe with open-connection-for-uri/cached. (open-connection-for-uri/cached): Set a default timeout, matching the behaviour in open-connection-for-uri/maybe. (process-substitution): Use call-with-connection-error-handling. --- guix/scripts/substitute.scm | 47 +++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 25 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index fea2cecef0..a3a0349530 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -281,22 +281,13 @@ if file doesn't exist, and the narinfo otherwise." ;; Set of names of unreachable hosts. (make-hash-table)) -(define* (open-connection-for-uri/maybe uri - #:key - fresh? - (time %fetch-timeout) - verify-certificate?) - "Open a connection to URI via 'open-connection-for-uri/cached' and return a -port to it, or, if connection failed, print a warning and return #f. Pass -#:fresh? to 'open-connection-for-uri/cached'." +(define* (call-with-connection-error-handling uri proc) + "Call PROC, and catch if a connection fails, print a warning and return #f." (define host (uri-host uri)) (catch #t - (lambda () - (open-connection-for-uri/cached uri #:timeout time - #:fresh? fresh? - #:verify-certificate? verify-certificate?)) + proc (match-lambda* (('getaddrinfo-error error) (unless (hash-ref %unreachable-hosts host) @@ -377,11 +368,14 @@ port to it, or, if connection failed, print a warning and return #f. Pass (let* ((requests (map (cut narinfo-request url <>) paths)) (result (begin (update-progress!) - (http-multiple-get uri - handle-narinfo-response '() - requests - #:open-connection open-connection - #:verify-certificate? #f)))) + (call-with-connection-error-handling + uri + (lambda () + (http-multiple-get uri + handle-narinfo-response '() + requests + #:open-connection open-connection + #:verify-certificate? #f)))))) (newline (current-error-port)) result)) ((file #f) @@ -595,7 +589,7 @@ authorized substitutes." ;; Return the subset of PATHS available in CACHE-URLS. (let ((substitutable (lookup-narinfos/diverse cache-urls paths valid? - #:open-connection open-connection-for-uri/maybe))) + #:open-connection open-connection-for-uri/cached))) (for-each (lambda (narinfo) (format #t "~a~%" (narinfo-path narinfo))) substitutable) @@ -604,7 +598,7 @@ authorized substitutes." ;; Reply info about PATHS if it's in CACHE-URLS. (let ((substitutable (lookup-narinfos/diverse cache-urls paths valid? - #:open-connection open-connection-for-uri/maybe))) + #:open-connection open-connection-for-uri/cached))) (for-each display-narinfo-data substitutable) (newline))) (wtf @@ -617,7 +611,7 @@ authorized substitutes." (define open-connection-for-uri/cached (let ((cache '())) - (lambda* (uri #:key fresh? timeout verify-certificate?) + (lambda* (uri #:key fresh? (timeout %fetch-timeout) verify-certificate?) "Return a connection for URI, possibly reusing a cached connection. When FRESH? is true, delete any cached connections for URI and open a new one. Return #f if URI's scheme is 'file' or #f. @@ -704,11 +698,14 @@ the current output port." (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f - #:open-connection open-connection-for-uri/maybe - #:keep-alive? #t - #:buffered? #f - #:verify-certificate? #f)))) + (call-with-connection-error-handling + uri + (lambda () + (http-fetch uri #:text? #f + #:open-connection open-connection-for-uri/cached + #:keep-alive? #t + #:buffered? #f + #:verify-certificate? #f)))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri)))))