From patchwork Sat Jan 16 13:57:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26405 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 862E127BC13; Sat, 16 Jan 2021 13:59:49 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 488B627BC13 for ; Sat, 16 Jan 2021 13:59:49 +0000 (GMT) Received: from localhost ([::1]:33030 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m7I-0007b9-Eu for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 08:59:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60154) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6Z-0007CI-7D for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59298) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0004vp-VT for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6Y-0005wW-Ux for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 01/13] substitute: Remove buffer handling from fetch. References: <87y2hn9l8j.fsf@cbaines.net> In-Reply-To: <87y2hn9l8j.fsf@cbaines.net> Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080549622698 (code B ref 45409); Sat, 16 Jan 2021 13:59:02 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:16 +0000 Received: from localhost ([127.0.0.1]:42589 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5o-0005u1-45 for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:16 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49410) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5g-0005sK-3B for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:09 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id A3E3C27BC13 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:06 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 42d2a158 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:51 +0000 Message-Id: <20210116135803.21955-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches http-fetch does this, so just set the right option. * guix/scripts/substitute.scm (fetch): Remove buffering code, and pass #:buffered? to http-fetch. --- guix/scripts/substitute.scm | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 2eefdb79d8..aaafb5d605 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -204,10 +204,9 @@ connection (typically PORT) is kept open once data has been fetched from URI." (when (or (not port) (port-closed? port)) (set! port (guix:open-connection-for-uri uri #:verify-certificate? #f))) - (unless (or buffered? (not (file-port? port))) - (setvbuf port 'none)) (http-fetch uri #:text? #f #:port port #:keep-alive? keep-alive? + #:buffered? buffered? #:verify-certificate? #f)))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") From patchwork Sat Jan 16 13:57:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26403 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 66C6827BC14; Sat, 16 Jan 2021 13:59:35 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 3D8F027BC13 for ; Sat, 16 Jan 2021 13:59:35 +0000 (GMT) Received: from localhost ([::1]:60470 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m74-0007EB-9k for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 08:59:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60152) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0007Ba-RD for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59297) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0004vE-JP for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6Y-0005wO-Hm for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 02/13] substitute: Remove connection handling from fetch. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080549222682 (code B ref 45409); Sat, 16 Jan 2021 13:59:02 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:12 +0000 Received: from localhost ([127.0.0.1]:42587 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5j-0005tV-Hc for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:12 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49412) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5g-0005sL-1G for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:09 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id D0A9327BC14 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:06 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6a747df8 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:52 +0000 Message-Id: <20210116135803.21955-2-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches http-fetch does this, so just use that code instead. * guix/scripts/substitute.scm (fetch): Remove connection handling when the port is closed. --- guix/scripts/substitute.scm | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index aaafb5d605..74fce15117 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -200,14 +200,10 @@ connection (typically PORT) is kept open once data has been fetched from URI." (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (begin - (when (or (not port) (port-closed? port)) - (set! port (guix:open-connection-for-uri - uri #:verify-certificate? #f))) - (http-fetch uri #:text? #f #:port port - #:keep-alive? keep-alive? - #:buffered? buffered? - #:verify-certificate? #f)))))) + (http-fetch uri #:text? #f #:port port + #:keep-alive? keep-alive? + #:buffered? buffered? + #:verify-certificate? #f))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) From patchwork Sat Jan 16 13:57:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26409 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E583027BC14; Sat, 16 Jan 2021 14:00:47 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id A8D5427BC13 for ; Sat, 16 Jan 2021 14:00:47 +0000 (GMT) Received: from localhost ([::1]:34598 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m8E-0008GA-Qe for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:00:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60160) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007Cz-80 for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59299) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6Z-0004vu-BQ for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6Z-0005wd-Ad for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 03/13] substitute: Remove redundant let block from fetch. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080549622705 (code B ref 45409); Sat, 16 Jan 2021 13:59:03 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:16 +0000 Received: from localhost ([127.0.0.1]:42591 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5o-0005u3-Cg for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:16 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49414) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5g-0005sM-3G for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:09 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id ED72427BC15 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:06 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 8f52af8d for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:53 +0000 Message-Id: <20210116135803.21955-3-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/scripts/substitute.scm (fetch): Remove redundant let block. --- guix/scripts/substitute.scm | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 74fce15117..ecc2bd9035 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -192,18 +192,17 @@ connection (typically PORT) is kept open once data has been fetched from URI." ;; sudo tc qdisc add dev eth0 root netem delay 1500ms ;; and then cancel with: ;; sudo tc qdisc del dev eth0 root - (let ((port port)) - (with-timeout (if timeout? - %fetch-timeout - 0) - (begin - (warning (G_ "while fetching ~a: server is somewhat slow~%") - (uri->string uri)) - (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f #:port port - #:keep-alive? keep-alive? - #:buffered? buffered? - #:verify-certificate? #f))))) + (with-timeout (if timeout? + %fetch-timeout + 0) + (begin + (warning (G_ "while fetching ~a: server is somewhat slow~%") + (uri->string uri)) + (warning (G_ "try `--no-substitutes' if the problem persists~%"))) + (http-fetch uri #:text? #f #:port port + #:keep-alive? keep-alive? + #:buffered? buffered? + #:verify-certificate? #f)))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) From patchwork Sat Jan 16 13:57:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26408 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 17B7227BC13; Sat, 16 Jan 2021 13:59:58 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5F63027BC14 for ; Sat, 16 Jan 2021 13:59:57 +0000 (GMT) Received: from localhost ([::1]:33794 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m7Q-0007un-GV for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 08:59:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60172) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D5-9b for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59300) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6Z-0004vv-N1 for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6Z-0005wl-Mz for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 04/13] guix: Move http-multiple-get to (guix http-client). Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080549722714 (code B ref 45409); Sat, 16 Jan 2021 13:59:03 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:17 +0000 Received: from localhost ([127.0.0.1]:42593 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5o-0005uA-Lb for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:17 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49418) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5g-0005sN-0y for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:10 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 19DA827BC16 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 7a92d3e8 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:54 +0000 Message-Id: <20210116135803.21955-4-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches From (guix scripts substitute). This will make it easier to reuse this code. * guix/scripts/substitute.scm (http-multiple-get): Remove, and move to… * guix/http-client.scm (http-multiple-get): …here. --- guix/http-client.scm | 76 +++++++++++++++++++++++++++++++++++++ guix/scripts/substitute.scm | 70 ---------------------------------- 2 files changed, 76 insertions(+), 70 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 553640fe9e..7ead493633 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -21,8 +21,11 @@ (define-module (guix http-client) #:use-module (web uri) + #:use-module (web http) #:use-module ((web client) #:hide (open-socket-for-uri)) + #:use-module (web request) #:use-module (web response) + #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) @@ -50,6 +53,7 @@ http-get-error-reason http-fetch + http-multiple-get %http-cache-ttl http-fetch/cached)) @@ -138,6 +142,78 @@ Raise an '&http-get-error' condition if downloading fails." (uri->string uri) code (response-reason-phrase resp)))))))))))) +(define* (http-multiple-get base-uri proc seed requests + #:key port (verify-certificate? #t) + (open-connection guix:open-connection-for-uri) + (keep-alive? #t) + (batch-size 1000)) + "Send all of REQUESTS to the server at BASE-URI. Call PROC for each +response, passing it the request object, the response, a port from which to +read the response body, and the previous result, starting with SEED, à la +'fold'. Return the final result. + +When PORT is specified, use it as the initial connection on which HTTP +requests are sent; otherwise call OPEN-CONNECTION to open a new connection for +a URI. When KEEP-ALIVE? is false, close the connection port before +returning." + (let connect ((port port) + (requests requests) + (result seed)) + (define batch + (if (>= batch-size (length requests)) + requests + (take requests batch-size))) + + ;; (format (current-error-port) "connecting (~a requests left)..." + ;; (length requests)) + (let ((p (or port (open-connection base-uri + #:verify-certificate? + verify-certificate?)))) + ;; For HTTPS, P is not a file port and does not support 'setvbuf'. + (when (file-port? p) + (setvbuf p 'block (expt 2 16))) + + ;; Send BATCH in a row. + ;; XXX: Do our own caching to work around inefficiencies when + ;; communicating over TLS: . + (let-values (((buffer get) (open-bytevector-output-port))) + ;; Inherit the HTTP proxying property from P. + (set-http-proxy-port?! buffer (http-proxy-port? p)) + + (for-each (cut write-request <> buffer) + batch) + (put-bytevector p (get)) + (force-output p)) + + ;; Now start processing responses. + (let loop ((sent batch) + (processed 0) + (result result)) + (match sent + (() + (match (drop requests processed) + (() + (unless keep-alive? + (close-port p)) + (reverse result)) + (remainder + (connect p remainder result)))) + ((head tail ...) + (let* ((resp (read-response p)) + (body (response-body-port resp)) + (result (proc head resp body result))) + ;; The server can choose to stop responding at any time, in which + ;; case we have to try again. Check whether that is the case. + ;; Note that even upon "Connection: close", we can read from BODY. + (match (assq 'connection (response-headers resp)) + (('connection 'close) + (close-port p) + (connect #f ;try again + (drop requests (+ 1 processed)) + result)) + (_ + (loop tail (+ 1 processed) result)))))))))) ;keep going + ;;; ;;; Caching. diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index ecc2bd9035..64b8ae2a15 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -299,76 +299,6 @@ return its MAX-LENGTH first elements and its tail." (values (reverse result) lst) (loop (+ 1 len) tail (cons head result))))))) -(define* (http-multiple-get base-uri proc seed requests - #:key port (verify-certificate? #t) - (open-connection guix:open-connection-for-uri) - (keep-alive? #t) - (batch-size 1000)) - "Send all of REQUESTS to the server at BASE-URI. Call PROC for each -response, passing it the request object, the response, a port from which to -read the response body, and the previous result, starting with SEED, à la -'fold'. Return the final result. - -When PORT is specified, use it as the initial connection on which HTTP -requests are sent; otherwise call OPEN-CONNECTION to open a new connection for -a URI. When KEEP-ALIVE? is false, close the connection port before -returning." - (let connect ((port port) - (requests requests) - (result seed)) - (define batch - (at-most batch-size requests)) - - ;; (format (current-error-port) "connecting (~a requests left)..." - ;; (length requests)) - (let ((p (or port (open-connection base-uri - #:verify-certificate? - verify-certificate?)))) - ;; For HTTPS, P is not a file port and does not support 'setvbuf'. - (when (file-port? p) - (setvbuf p 'block (expt 2 16))) - - ;; Send BATCH in a row. - ;; XXX: Do our own caching to work around inefficiencies when - ;; communicating over TLS: . - (let-values (((buffer get) (open-bytevector-output-port))) - ;; Inherit the HTTP proxying property from P. - (set-http-proxy-port?! buffer (http-proxy-port? p)) - - (for-each (cut write-request <> buffer) - batch) - (put-bytevector p (get)) - (force-output p)) - - ;; Now start processing responses. - (let loop ((sent batch) - (processed 0) - (result result)) - (match sent - (() - (match (drop requests processed) - (() - (unless keep-alive? - (close-port p)) - (reverse result)) - (remainder - (connect p remainder result)))) - ((head tail ...) - (let* ((resp (read-response p)) - (body (response-body-port resp)) - (result (proc head resp body result))) - ;; The server can choose to stop responding at any time, in which - ;; case we have to try again. Check whether that is the case. - ;; Note that even upon "Connection: close", we can read from BODY. - (match (assq 'connection (response-headers resp)) - (('connection 'close) - (close-port p) - (connect #f ;try again - (drop requests (+ 1 processed)) - result)) - (_ - (loop tail (+ 1 processed) result)))))))))) ;keep going - (define (read-to-eof port) "Read from PORT until EOF is reached. The data are discarded." (dump-port port (%make-void-port "w"))) From patchwork Sat Jan 16 13:57:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26412 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4072A27BC14; Sat, 16 Jan 2021 14:02:44 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id AA57527BC13 for ; Sat, 16 Jan 2021 14:02:42 +0000 (GMT) Received: from localhost ([::1]:37334 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0mA5-00016F-TU for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:02:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0007BG-LQ for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59296) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0004ur-6W for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6Y-0005wH-4s for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 05/13] http-client: Add error handling to http-multiple-get. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080549122665 (code B ref 45409); Sat, 16 Jan 2021 13:59:02 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:11 +0000 Received: from localhost ([127.0.0.1]:42581 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5i-0005sz-6Y for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:11 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49420) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5g-0005sO-1F for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:09 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 3CD5927BC17 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 78ff9e06 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:55 +0000 Message-Id: <20210116135803.21955-5-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Making sure to close the port if it looks to be unusable. This closing of the port will allow for caching connections, without caching broken connections, as the cache can avoid handing out closed ports. * guix/http-client.scm (http-multiple-get): Try to catch exceptions that happen if the port is unusable, this is a adaptation of code within the (guix scripts substitute) module. --- guix/http-client.scm | 74 +++++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 18 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 7ead493633..3aba3b28c1 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -38,6 +38,7 @@ #:use-module (guix utils) #:use-module (guix base64) #:autoload (gcrypt hash) (sha256) + #:autoload (gnutls) (error/invalid-session) #:use-module ((guix build utils) #:select (mkdir-p dump-port)) #:use-module ((guix build download) @@ -180,10 +181,25 @@ returning." ;; Inherit the HTTP proxying property from P. (set-http-proxy-port?! buffer (http-proxy-port? p)) - (for-each (cut write-request <> buffer) - batch) - (put-bytevector p (get)) - (force-output p)) + (catch #t + (lambda () + (for-each (cut write-request <> buffer) + batch) + (put-bytevector p (get)) + (force-output p)) + (lambda (key . args) + ;; If PORT becomes unusable, open a fresh connection and + ;; retry. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session))) + (begin + (close-port p) ; close the broken port + (connect #f + requests + result)) + (apply throw key args))))) ;; Now start processing responses. (let loop ((sent batch) @@ -199,20 +215,42 @@ returning." (remainder (connect p remainder result)))) ((head tail ...) - (let* ((resp (read-response p)) - (body (response-body-port resp)) - (result (proc head resp body result))) - ;; The server can choose to stop responding at any time, in which - ;; case we have to try again. Check whether that is the case. - ;; Note that even upon "Connection: close", we can read from BODY. - (match (assq 'connection (response-headers resp)) - (('connection 'close) - (close-port p) - (connect #f ;try again - (drop requests (+ 1 processed)) - result)) - (_ - (loop tail (+ 1 processed) result)))))))))) ;keep going + (catch #t + (lambda () + (let* ((resp (read-response p)) + (body (response-body-port resp)) + (result (proc head resp body result))) + ;; The server can choose to stop responding at any time, + ;; in which case we have to try again. Check whether + ;; that is the case. Note that even upon "Connection: + ;; close", we can read from BODY. + (match (assq 'connection (response-headers resp)) + (('connection 'close) + (close-port p) + (connect #f ;try again + (drop requests (+ 1 processed)) + result)) + (_ + (loop tail (+ 1 processed) result))))) ;keep going + (lambda (key . args) + ;; If PORT was cached and the server closed the connection + ;; in the meantime, we get EPIPE. In that case, open a + ;; fresh connection and retry. We might also get + ;; 'bad-response or a similar exception from (web response) + ;; later on, once we've sent the request, or a + ;; ERROR/INVALID-SESSION from GnuTLS. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session)) + (memq key + '(bad-response bad-header bad-header-component))) + (begin + (close-port p) + (connect #f ; try again + (drop requests (+ 1 processed)) + result)) + (apply throw key args)))))))))) ;;; From patchwork Sat Jan 16 13:57:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26404 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B82F727BC14; Sat, 16 Jan 2021 13:59:35 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 68CC327BC13 for ; Sat, 16 Jan 2021 13:59:35 +0000 (GMT) Received: from localhost ([::1]:60490 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m74-0007Ev-I3 for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 08:59:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60158) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007Cy-7d for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59301) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6a-0004vw-3N for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6a-0005ws-2w for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 06/13] substitute: open-connection-for-uri/maybe add #:verify-certificate?. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550222730 (code B ref 45409); Sat, 16 Jan 2021 13:59:04 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:22 +0000 Received: from localhost ([127.0.0.1]:42595 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5u-0005uW-Cl for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:22 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49426) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5h-0005sZ-Hp for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:10 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 5D6C527BC18 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 19dd725f for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:56 +0000 Message-Id: <20210116135803.21955-6-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches As this is used by http-fetch and http-multiple-get when they call the specified open connection procedure. * guix/scripts/substitute.scm (open-connection-for-uri/maybe): Support #:verify-certificate?. --- guix/scripts/substitute.scm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 64b8ae2a15..259b109cc6 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -322,7 +322,8 @@ if file doesn't exist, and the narinfo otherwise." (define* (open-connection-for-uri/maybe uri #:key fresh? - (time %fetch-timeout)) + (time %fetch-timeout) + verify-certificate?) "Open a connection to URI via 'open-connection-for-uri/cached' and return a port to it, or, if connection failed, print a warning and return #f. Pass #:fresh? to 'open-connection-for-uri/cached'." @@ -332,7 +333,8 @@ port to it, or, if connection failed, print a warning and return #f. Pass (catch #t (lambda () (open-connection-for-uri/cached uri #:timeout time - #:fresh? fresh?)) + #:fresh? fresh? + #:verify-certificate? verify-certificate?)) (match-lambda* (('getaddrinfo-error error) (unless (hash-ref %unreachable-hosts host) From patchwork Sat Jan 16 13:57:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26406 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id C843027BC13; Sat, 16 Jan 2021 13:59:49 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7073C27BC14 for ; Sat, 16 Jan 2021 13:59:49 +0000 (GMT) Received: from localhost ([::1]:33032 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m7I-0007bB-GN for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 08:59:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60162) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D0-85 for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59303) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6a-0004vy-VC for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6a-0005x7-Te for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 07/13] substitute: Stop using call-with-cached-connection in fetch-narinfos. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550322745 (code B ref 45409); Sat, 16 Jan 2021 13:59:04 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:23 +0000 Received: from localhost ([127.0.0.1]:42599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5v-0005uh-0Y for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:23 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49428) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5h-0005sY-Hx for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:10 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 7AC6C27BC19 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 171b4850 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:57 +0000 Message-Id: <20210116135803.21955-7-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Instead, just pass open-connection-for-uri/maybe to http-multiple-get. This code should be functionaly similar to the previous code. The eventual aim of this is to make the connection caching not mandatory in fetch-narinfos. * guix/scripts/substitute.scm (fetch-narinfos): Remove use of call-with-cached-connection. --- guix/scripts/substitute.scm | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 259b109cc6..88219ea7f6 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -412,20 +412,14 @@ port to it, or, if connection failed, print a warning and return #f. Pass ;; on the X.509 PKI. We can do it because we authenticate ;; narinfos, which provides a much stronger guarantee. (let* ((requests (map (cut narinfo-request url <>) paths)) - (result (call-with-cached-connection uri - (lambda (port) - (if port - (begin - (update-progress!) - (http-multiple-get uri - handle-narinfo-response '() - requests - #:open-connection - open-connection-for-uri/cached - #:verify-certificate? #f - #:port port)) - '())) - open-connection-for-uri/maybe))) + (result (begin + (update-progress!) + (http-multiple-get uri + handle-narinfo-response '() + requests + #:open-connection + open-connection-for-uri/maybe + #:verify-certificate? #f)))) result)) ((file #f) (let* ((base (string-append (uri-path uri) "/")) From patchwork Sat Jan 16 13:57:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26407 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8CE8A27BC15; Sat, 16 Jan 2021 13:59:57 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 42BC727BC13 for ; Sat, 16 Jan 2021 13:59:57 +0000 (GMT) Received: from localhost ([::1]:33796 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m7Q-0007up-E4 for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 08:59:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60156) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007Cx-7c for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59302) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6a-0004vx-HK for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6a-0005wz-Gh for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 08/13] http-client: Accept #:open-connection in http-fetch. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550322738 (code B ref 45409); Sat, 16 Jan 2021 13:59:04 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:23 +0000 Received: from localhost ([127.0.0.1]:42597 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5u-0005uZ-MX for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:22 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49430) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5h-0005si-SQ for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:10 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 9134927BC1A for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 27b3d66f for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:58 +0000 Message-Id: <20210116135803.21955-8-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches So that an alternative procedure can be passed in, perhaps to perform connection caching. * guix/http-client.scm (http-fetch): Add an #:open-connection keyword argument. --- guix/http-client.scm | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 3aba3b28c1..2d7458a56e 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -75,6 +75,7 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) + (open-connection guix:open-connection-for-uri) (keep-alive? #f) (verify-certificate? #t) (headers '((user-agent . "GNU Guile"))) @@ -97,10 +98,10 @@ Raise an '&http-get-error' condition if downloading fails." (let loop ((uri (if (string? uri) (string->uri uri) uri))) - (let ((port (or port (guix:open-connection-for-uri uri - #:verify-certificate? - verify-certificate? - #:timeout timeout))) + (let ((port (or port (open-connection uri + #:verify-certificate? + verify-certificate? + #:timeout timeout))) (headers (match (uri-userinfo uri) ((? string? str) (cons (cons 'Authorization From patchwork Sat Jan 16 13:57:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26414 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 7F02227BC14; Sat, 16 Jan 2021 14:03:34 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E57CC27BC13 for ; Sat, 16 Jan 2021 14:03:31 +0000 (GMT) Received: from localhost ([::1]:38552 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0mAt-0001cI-5P for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:03:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60170) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D4-9X for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59304) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6b-0004vz-D3 for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6b-0005xE-CC for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 09/13] substitute: Change connection cache handling in process-substitution. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:05 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550322752 (code B ref 45409); Sat, 16 Jan 2021 13:59:05 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:23 +0000 Received: from localhost ([127.0.0.1]:42601 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5v-0005uo-8d for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:23 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49432) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5h-0005sj-TR for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:10 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id B3DDA27BC1B for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 41331557 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:59 +0000 Message-Id: <20210116135803.21955-9-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Just pass open-connection-for-uri/maybe to http-fetch, this removes the need for with-cached-connection and passing the port in. * guix/scripts/substitute.scm (fetch): Don't take a port as an argument, and pass open-connection-for-uri/maybe to http-fetch. (process-substitution): Don't call fetch with with-cached-connection. --- guix/scripts/substitute.scm | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 88219ea7f6..fc5a19124e 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -170,7 +170,7 @@ again." (apply values result))))) (define* (fetch uri #:key (buffered? #t) (timeout? #t) - (keep-alive? #f) (port #f)) + (keep-alive? #f)) "Return a binary input port to URI and the number of bytes it's expected to provide. @@ -199,7 +199,8 @@ connection (typically PORT) is kept open once data has been fetched from URI." (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f #:port port + (http-fetch uri #:text? #f + #:open-connection open-connection-for-uri/maybe #:keep-alive? keep-alive? #:buffered? buffered? #:verify-certificate? #f)))) @@ -751,10 +752,8 @@ the current output port." (let*-values (((raw download-size) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. - (with-cached-connection uri port - (fetch uri #:buffered? #f #:timeout? #f - #:port port - #:keep-alive? #t))) + (fetch uri #:buffered? #f #:timeout? #f + #:keep-alive? #t)) ((progress) (let* ((dl-size (or download-size (and (equal? compression "none") From patchwork Sat Jan 16 13:58:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26410 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id EB2F427BC14; Sat, 16 Jan 2021 14:01:05 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 8D8D427BC13 for ; Sat, 16 Jan 2021 14:01:05 +0000 (GMT) Received: from localhost ([::1]:35274 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m8W-0000BI-Lu for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:01:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D7-9X for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59305) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6b-0004w0-SJ for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6b-0005xL-Rp for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 10/13] substitute: Remove now redundant connection caching helpers. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:05 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550422759 (code B ref 45409); Sat, 16 Jan 2021 13:59:05 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:24 +0000 Received: from localhost ([127.0.0.1]:42603 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5v-0005uv-HF for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:23 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49434) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5i-0005sq-4D for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:11 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id CA43F27BC1C for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 113d4c3d for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:58:00 +0000 Message-Id: <20210116135803.21955-10-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Failures now should be handled where they occur, and if there's a problem that's symptomatic of an issue with the connection, the port should be closed. * guix/scripts/substitute.scm (call-with-cached-connection): Remove procedure. (with-cached-connection): Remove syntax rule. --- guix/scripts/substitute.scm | 28 ---------------------------- 1 file changed, 28 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index fc5a19124e..d316bdef15 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -689,32 +689,6 @@ server certificates." (drain-input socket) socket)))))))) -(define* (call-with-cached-connection uri proc - #:optional - (open-connection - open-connection-for-uri/cached)) - (let ((port (open-connection uri))) - (catch #t - (lambda () - (proc port)) - (lambda (key . args) - ;; If PORT was cached and the server closed the connection in the - ;; meantime, we get EPIPE. In that case, open a fresh connection and - ;; retry. We might also get 'bad-response or a similar exception from - ;; (web response) later on, once we've sent the request, or a - ;; ERROR/INVALID-SESSION from GnuTLS. - (if (or (and (eq? key 'system-error) - (= EPIPE (system-error-errno `(,key ,@args)))) - (and (eq? key 'gnutls-error) - (eq? (first args) error/invalid-session)) - (memq key '(bad-response bad-header bad-header-component))) - (proc (open-connection uri #:fresh? #t)) - (apply throw key args)))))) - -(define-syntax-rule (with-cached-connection uri port exp ...) - "Bind PORT with EXP... to a socket connected to URI." - (call-with-cached-connection uri (lambda (port) exp ...))) - (define* (process-substitution store-item destination #:key cache-urls acl deduplicate? print-build-trace?) @@ -1010,8 +984,6 @@ default value." ;;; Local Variables: ;;; eval: (put 'with-timeout 'scheme-indent-function 1) -;;; eval: (put 'with-cached-connection 'scheme-indent-function 2) -;;; eval: (put 'call-with-cached-connection 'scheme-indent-function 1) ;;; End: ;;; substitute.scm ends here From patchwork Sat Jan 16 13:58:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26415 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id CD05C27BC14; Sat, 16 Jan 2021 14:03:38 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 9862F27BC13 for ; Sat, 16 Jan 2021 14:03:38 +0000 (GMT) Received: from localhost ([::1]:39038 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0mAz-0001od-Pd for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:03:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60174) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D6-9a for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59306) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6c-0004w1-AX for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6c-0005xT-A3 for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:06 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 11/13] substitute: Remove redundant fetch arguments. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550422766 (code B ref 45409); Sat, 16 Jan 2021 13:59:06 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:24 +0000 Received: from localhost ([127.0.0.1]:42605 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5v-0005v2-Qv for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:24 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49436) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5i-0005sr-4F for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:11 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id E0C4D27BC1D for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id afce44bc for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:58:01 +0000 Message-Id: <20210116135803.21955-11-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches It's just called in one place, with hardcoded argument values, so just inline them. * guix/scripts/substitute.scm (fetch): Remove arguments that don't vary, copy the values from the call site in process-substitution. (process-substitution): Remove unnecessary argument values from fetch call. --- guix/scripts/substitute.scm | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index d316bdef15..b5a4c08325 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -169,18 +169,12 @@ again." (sigaction SIGALRM SIG_DFL) (apply values result))))) -(define* (fetch uri #:key (buffered? #t) (timeout? #t) - (keep-alive? #f)) +(define (fetch uri) "Return a binary input port to URI and the number of bytes it's expected to -provide. - -When PORT is true, use it as the underlying I/O port for HTTP transfers; when -PORT is false, open a new connection for URI. When KEEP-ALIVE? is true, the -connection (typically PORT) is kept open once data has been fetched from URI." +provide." (case (uri-scheme uri) ((file) - (let ((port (open-file (uri-path uri) - (if buffered? "rb" "r0b")))) + (let ((port (open-file (uri-path uri) "r0b"))) (values port (stat:size (stat port))))) ((http https) (guard (c ((http-get-error? c) @@ -192,17 +186,15 @@ connection (typically PORT) is kept open once data has been fetched from URI." ;; sudo tc qdisc add dev eth0 root netem delay 1500ms ;; and then cancel with: ;; sudo tc qdisc del dev eth0 root - (with-timeout (if timeout? - %fetch-timeout - 0) + (with-timeout %fetch-timeout (begin (warning (G_ "while fetching ~a: server is somewhat slow~%") (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) (http-fetch uri #:text? #f #:open-connection open-connection-for-uri/maybe - #:keep-alive? keep-alive? - #:buffered? buffered? + #:keep-alive? #t + #:buffered? #f #:verify-certificate? #f)))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") @@ -726,8 +718,7 @@ the current output port." (let*-values (((raw download-size) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. - (fetch uri #:buffered? #f #:timeout? #f - #:keep-alive? #t)) + (fetch uri)) ((progress) (let* ((dl-size (or download-size (and (equal? compression "none") From patchwork Sat Jan 16 13:58:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26413 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 16D4927BC14; Sat, 16 Jan 2021 14:03:12 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 0127427BC13 for ; Sat, 16 Jan 2021 14:03:08 +0000 (GMT) Received: from localhost ([::1]:37934 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0mAV-0001LQ-2d for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:03:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60178) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D8-9u for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59307) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6c-0004w4-OY for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6c-0005xa-NV for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:06 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 12/13] substitute: Inline fetch in to process-substitutes. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550422773 (code B ref 45409); Sat, 16 Jan 2021 13:59:06 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:24 +0000 Received: from localhost ([127.0.0.1]:42607 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5w-0005v9-5d for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:24 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49438) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5i-0005sw-8L for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:11 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 0306B27BC1E for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 533ad5a2 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:58:02 +0000 Message-Id: <20210116135803.21955-12-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches As it's only called in one place, and this should make the code easier to read. * guix/scripts/substitute.scm (fetch): Move procedure inside… (process-substitution): …here. --- guix/scripts/substitute.scm | 60 ++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index b5a4c08325..858ce1dcc4 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -169,37 +169,6 @@ again." (sigaction SIGALRM SIG_DFL) (apply values result))))) -(define (fetch uri) - "Return a binary input port to URI and the number of bytes it's expected to -provide." - (case (uri-scheme uri) - ((file) - (let ((port (open-file (uri-path uri) "r0b"))) - (values port (stat:size (stat port))))) - ((http https) - (guard (c ((http-get-error? c) - (leave (G_ "download from '~a' failed: ~a, ~s~%") - (uri->string (http-get-error-uri c)) - (http-get-error-code c) - (http-get-error-reason c)))) - ;; Test this with: - ;; sudo tc qdisc add dev eth0 root netem delay 1500ms - ;; and then cancel with: - ;; sudo tc qdisc del dev eth0 root - (with-timeout %fetch-timeout - (begin - (warning (G_ "while fetching ~a: server is somewhat slow~%") - (uri->string uri)) - (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (http-fetch uri #:text? #f - #:open-connection open-connection-for-uri/maybe - #:keep-alive? #t - #:buffered? #f - #:verify-certificate? #f)))) - (else - (leave (G_ "unsupported substitute URI scheme: ~a~%") - (uri->string uri))))) - (define (narinfo-cache-file cache-url path) "Return the name of the local file that contains an entry for PATH. The entry is stored in a sub-directory specific to CACHE-URL." @@ -705,6 +674,35 @@ the current output port." (apply dump-file/deduplicate (append args (list #:store (%store-prefix))))) + (define (fetch uri) + (case (uri-scheme uri) + ((file) + (let ((port (open-file (uri-path uri) "r0b"))) + (values port (stat:size (stat port))))) + ((http https) + (guard (c ((http-get-error? c) + (leave (G_ "download from '~a' failed: ~a, ~s~%") + (uri->string (http-get-error-uri c)) + (http-get-error-code c) + (http-get-error-reason c)))) + ;; Test this with: + ;; sudo tc qdisc add dev eth0 root netem delay 1500ms + ;; and then cancel with: + ;; sudo tc qdisc del dev eth0 root + (with-timeout %fetch-timeout + (begin + (warning (G_ "while fetching ~a: server is somewhat slow~%") + (uri->string uri)) + (warning (G_ "try `--no-substitutes' if the problem persists~%"))) + (http-fetch uri #:text? #f + #:open-connection open-connection-for-uri/maybe + #:keep-alive? #t + #:buffered? #f + #:verify-certificate? #f)))) + (else + (leave (G_ "unsupported substitute URI scheme: ~a~%") + (uri->string uri))))) + (unless narinfo (leave (G_ "no valid substitute for '~a'~%") store-item)) From patchwork Sat Jan 16 13:58:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26411 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B9D0E27BC14; Sat, 16 Jan 2021 14:01:55 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5D79427BC13 for ; Sat, 16 Jan 2021 14:01:55 +0000 (GMT) Received: from localhost ([::1]:35866 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0m9K-0000TA-Hi for patchwork@mira.cbaines.net; Sat, 16 Jan 2021 09:01:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60166) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6e-0007D2-8I for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:08 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59308) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6d-0004w6-5u for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6d-0005xi-5T for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:07 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 13/13] substitute: Remove fetch-narinfos use open-connection-for-uri/maybe. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:07 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080550422779 (code B ref 45409); Sat, 16 Jan 2021 13:59:07 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:24 +0000 Received: from localhost ([127.0.0.1]:42609 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5w-0005vG-Gj for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:24 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49440) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5i-0005sx-8O for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:12 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 20F5C27BC1F for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:08 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id c1fc769d for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:58:03 +0000 Message-Id: <20210116135803.21955-13-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches At least by default. Instead, make the open-connection procedure a parameter, and make the default guix:open-connection-for-uri. Do so similarly for lookup-narinfos and lookup-narinfos/diverse which work towards calling fetch-narinfos. This means this code can be moved to a different module, without having use/move the connection caching code. * guix/scripts/substitute.scm (fetch-narinfos): Add #:open-connection argument, and call http-multiple-get with it. (lookup-narinfos) Add #:open-connection argument, and call fetch-narinfos with it. (lookup-narinfos/diverse): Add #:open-connection argument, and call lookup-narinfos with it. (process-query): Call lookup-narinfos/diverse with #:open-connection open-connection-for-uri/maybe. --- guix/scripts/substitute.scm | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 858ce1dcc4..c2a8dd419f 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -314,7 +314,8 @@ port to it, or, if connection failed, print a warning and return #f. Pass (args (apply throw args))))) -(define (fetch-narinfos url paths) +(define* (fetch-narinfos url paths + #:key (open-connection guix:open-connection-for-uri)) "Retrieve all the narinfos for PATHS from the cache at URL and return them." (define update-progress! (let ((done 0) @@ -379,8 +380,7 @@ port to it, or, if connection failed, print a warning and return #f. Pass (http-multiple-get uri handle-narinfo-response '() requests - #:open-connection - open-connection-for-uri/maybe + #:open-connection open-connection #:verify-certificate? #f)))) result)) ((file #f) @@ -395,7 +395,8 @@ port to it, or, if connection failed, print a warning and return #f. Pass (do-fetch (string->uri url))) -(define (lookup-narinfos cache paths) +(define* (lookup-narinfos cache paths + #:key (open-connection guix:open-connection-for-uri)) "Return the narinfos for PATHS, invoking the server at CACHE when no information is available locally." (let-values (((cached missing) @@ -412,10 +413,13 @@ information is available locally." paths))) (if (null? missing) cached - (let ((missing (fetch-narinfos cache missing))) + (let ((missing (fetch-narinfos cache missing + #:open-connection open-connection))) (append cached (or missing '())))))) -(define (lookup-narinfos/diverse caches paths authorized?) +(define* (lookup-narinfos/diverse caches paths authorized? + #:key (open-connection + guix:open-connection-for-uri)) "Look up narinfos for PATHS on all of CACHES, a list of URLS, in that order. That is, when a cache lacks an AUTHORIZED? narinfo, look it up in the next cache, and so on. @@ -447,7 +451,8 @@ AUTHORIZED? narinfo." (_ (match caches ((cache rest ...) - (let* ((narinfos (lookup-narinfos cache paths)) + (let* ((narinfos (lookup-narinfos cache paths + #:open-connection open-connection)) (definite (map narinfo-path (filter authorized? narinfos))) (missing (lset-difference string=? paths definite))) ;XXX: perf (loop rest missing @@ -587,14 +592,18 @@ authorized substitutes." (match (string-tokenize command) (("have" paths ..1) ;; Return the subset of PATHS available in CACHE-URLS. - (let ((substitutable (lookup-narinfos/diverse cache-urls paths valid?))) + (let ((substitutable (lookup-narinfos/diverse + cache-urls paths valid? + #:open-connection open-connection-for-uri/maybe))) (for-each (lambda (narinfo) (format #t "~a~%" (narinfo-path narinfo))) substitutable) (newline))) (("info" paths ..1) ;; Reply info about PATHS if it's in CACHE-URLS. - (let ((substitutable (lookup-narinfos/diverse cache-urls paths valid?))) + (let ((substitutable (lookup-narinfos/diverse + cache-urls paths valid? + #:open-connection open-connection-for-uri/maybe))) (for-each display-narinfo-data substitutable) (newline))) (wtf