From patchwork Thu Mar 3 21:14:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 37599 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4A74A27BBEB; Thu, 3 Mar 2022 21:29:27 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 02F0A27BBEA for ; Thu, 3 Mar 2022 21:29:27 +0000 (GMT) Received: from localhost ([::1]:58940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nPt0o-0008VP-5Q for patchwork@mira.cbaines.net; Thu, 03 Mar 2022 16:29:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35380) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnv-0005ey-5l for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50053) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsnq-0007Ii-Mk for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nPsnq-0002HI-HB for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 1/4] http-client: Add response headers to '&http-get-error'. References: <20220303211326.19884-1-ludo@gnu.org> In-Reply-To: <20220303211326.19884-1-ludo@gnu.org> Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421258666 (code B ref 54241); Thu, 03 Mar 2022 21:16:02 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:25 +0000 Received: from localhost ([127.0.0.1]:43939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnE-0002Fd-TF for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:25 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57758) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnC-0002FM-RT for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:24 -0500 Received: from [2001:470:142:3::e] (port=58918 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsn7-00078m-LD; Thu, 03 Mar 2022 16:15:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=KbRkngTko0T7lcr8FWw0zRyzmJMdgMzzPqVooLXmPlE=; b=YDFscPTLrHTpBE 9GNIhs2cpaE0s2coo40j0IlBRQtLyFSPtFl0kuihbGYOYp55Ra03VuGpNSKN2zqgv195D+7qBTdCr Jsb5PffUBKHPthZ8fj2eyBIGCUgh/D/Ag9LloCqGk2WtSfmaHzuZPb603sZ9Fey8dq7rN3pZ0B+D9 JXiva/E4vFJX+J5/+6YF54+WFgwG2Cj99xh2nZu2k94zabYwDhNYHAoHVJPdgARGgbDIz/pFWuzBs XD5nnP9VTREpUiHbWGwS8/6fYrTD6EKqYfkcSp8nYPAUAwCFWR1lQCtAW3bOk2YzLZeN5/QeKsXFQ lv3bI2DWFq0g7o/LZhzg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsmu-0006KA-Tf; Thu, 03 Mar 2022 16:15:16 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:41 +0100 Message-Id: <20220303211444.19928-1-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/http-client.scm (&http-get-error)[headers]: New field. (http-fetch): Initialize the 'headers' field. --- guix/http-client.scm | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 10bc278023..4b01e31165 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020, 2021 Ludovic Courtès +;;; Copyright © 2012-2018, 2020-2022 Ludovic Courtès ;;; Copyright © 2015 Mark H Weaver ;;; Copyright © 2012, 2015 Free Software Foundation, Inc. ;;; Copyright © 2017 Tobias Geerinckx-Rice @@ -52,6 +52,7 @@ (define-module (guix http-client) http-get-error-uri http-get-error-code http-get-error-reason + http-get-error-headers http-fetch http-multiple-get @@ -69,9 +70,10 @@ (define-module (guix http-client) ;; HTTP GET error. (define-condition-type &http-get-error &error http-get-error? - (uri http-get-error-uri) ; URI - (code http-get-error-code) ; integer - (reason http-get-error-reason)) ; string + (uri http-get-error-uri) ;URI + (code http-get-error-code) ;integer + (reason http-get-error-reason) ;string + (headers http-get-error-headers)) ;alist (define* (http-fetch uri #:key port (text? #f) (buffered? #t) @@ -138,7 +140,8 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) (raise (condition (&http-get-error (uri uri) (code code) - (reason (response-reason-phrase resp))) + (reason (response-reason-phrase resp)) + (headers (response-headers resp))) (&message (message (format From patchwork Thu Mar 3 21:14:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 37601 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id CBD8727BBEA; Thu, 3 Mar 2022 21:30:16 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 2CE4027BBE9 for ; Thu, 3 Mar 2022 21:30:16 +0000 (GMT) Received: from localhost ([::1]:60966 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nPt1b-0001RB-Cl for patchwork@mira.cbaines.net; Thu, 03 Mar 2022 16:30:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35382) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnv-0005fq-L8 for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:09 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50054) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsnr-0007In-5u for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nPsnq-0002HS-W4 for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421298690 (code B ref 54241); Thu, 03 Mar 2022 21:16:02 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:29 +0000 Received: from localhost ([127.0.0.1]:43945 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnI-0002Fz-8D for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57766) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnE-0002FO-A5 for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:24 -0500 Received: from [2001:470:142:3::e] (port=58924 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsn9-000794-4S; Thu, 03 Mar 2022 16:15:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=YqTGlDNK8xosVwE96k92CcKvx/luyf2dhcbIRo7hYDU=; b=kTlPDjCbdWhaHgcd0bVk 4X+I2zmI1xyDhCPXdXO9/Y1jbGAcHiowkZFl8IdePdKhv+tV4w3KNMw6gnAc7uRxudcS+nZQeR0WY yBJ3eqWHhJyKOgtakw5XZHfjbXFl3jIDOipvishe7y4/z8OGDe4MbFPGB8AnENZ3i4K8msxapir5d cQsgWE3sPKKe0/0F4hOBUREf0WfJfJokF13yUOf12qNZiF+B+mIfLxTCtdp7BbSGBhVGqG0uXZuD8 AhsePOkOpEN/9wHZvBNKdADCvEfGEGDo2a4OSpygVHcn8YnMrPdOaGrEzxoovFNajfmX2PRcqLHn4 Z3+pSyiCw5lNFQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsn7-0006KA-Q5; Thu, 03 Mar 2022 16:15:18 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:42 +0100 Message-Id: <20220303211444.19928-2-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220303211444.19928-1-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Previously, 'guix refresh' would literally crash when the rate limit was reached due to the call to 'error'. With this change, the updater notices when the rate limit is reached and it turns itself into a no-op until the rate limit has been reset. When running "guix refresh" (with no arguments), the 'github' updater gets used until the rate limit has been reached, after which "guix refresh" automatically picks up the next valid updater, typically 'generic-git'. * guix/import/github.scm (fetch-releases-or-tags): Use 'http-fetch' directly instead of 'json-fetch' to let 'http-get-error?' exceptions through. Handle 403 errors with an 'X-RateLimit-Remaining' header. (%rate-limit-reset-time): New variable. (update-rate-limit-reset-time!, request-rate-limit-reached?): New procedures. (latest-released-version): Remove calls to 'error'. --- guix/import/github.scm | 113 +++++++++++++++++++++++++++++------------ 1 file changed, 80 insertions(+), 33 deletions(-) diff --git a/guix/import/github.scm b/guix/import/github.scm index 8c1898c0c5..5062bad80d 100644 --- a/guix/import/github.scm +++ b/guix/import/github.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 Ben Woodcroft -;;; Copyright © 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2017-2020, 2022 Ludovic Courtès ;;; Copyright © 2018 Eric Bavier ;;; Copyright © 2019 Arun Isaac ;;; Copyright © 2019 Efraim Flashner @@ -30,15 +30,16 @@ (define-module (guix import github) #:use-module (guix utils) #:use-module (guix i18n) #:use-module (guix diagnostics) + #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix download) #:prefix download:) #:use-module ((guix git-download) #:prefix download:) #:use-module (guix import utils) - #:use-module (guix import json) #:use-module (json) #:use-module (guix packages) #:use-module (guix upstream) #:use-module (guix http-client) #:use-module (web uri) + #:use-module (web response) #:export (%github-api %github-updater)) ;; For tests. @@ -140,6 +141,30 @@ (define %github-token ;; limit, or #f. (make-parameter (getenv "GUIX_GITHUB_TOKEN"))) +(define %rate-limit-reset-time + ;; Time (seconds since the Epoch, UTC) when the rate limit for GitHub + ;; requests will be reset, or #f if the rate limit hasn't been reached. + #f) + +(define (update-rate-limit-reset-time! headers) + "Update the rate limit reset time based on HEADERS, the HTTP response +headers." + (match (assq-ref headers 'x-ratelimit-reset) + ((= string->number (? number? reset)) + (set! %rate-limit-reset-time reset) + reset) + (_ + 0))) + +(define (request-rate-limit-reached?) + "Return true if the rate limit has been reached." + (and %rate-limit-reset-time + (match (< (car (gettimeofday)) %rate-limit-reset-time) + (#t #t) + (#f + (set! %rate-limit-reset-time #f) + #f)))) + (define (fetch-releases-or-tags url) "Fetch the list of \"releases\" or, if it's empty, the list of tags for the repository at URL. Return the corresponding JSON dictionaries (alists), @@ -170,20 +195,49 @@ (define headers `((Authorization . ,(string-append "token " (%github-token)))) '()))) - (guard (c ((and (http-get-error? c) - (= 404 (http-get-error-code c))) - (warning (G_ "~a is unreachable (~a)~%") - release-url (http-get-error-code c)) - '#())) ;return an empty release set - (let* ((port (http-fetch release-url #:headers headers)) - (result (json->scm port))) - (close-port port) - (match result - (#() - ;; We got the empty list, presumably because the user didn't use GitHub's - ;; "release" mechanism, but hopefully they did use Git tags. - (json-fetch tag-url #:headers headers)) - (x x))))) + (and (not (request-rate-limit-reached?)) + (guard (c ((and (http-get-error? c) + (= 404 (http-get-error-code c))) + (warning (G_ "~a is unreachable (~a)~%") + (uri->string (http-get-error-uri c)) + (http-get-error-code c)) + '#()) ;return an empty release set + ((and (http-get-error? c) + (= 403 (http-get-error-code c))) + ;; See + ;; . + (match (assq-ref (http-get-error-headers c) + 'x-ratelimit-remaining) + (#f + (raise c)) + ((? (compose zero? string->number)) + (let ((reset (update-rate-limit-reset-time! + (http-get-error-headers c)))) + (warning (G_ "GitHub rate limit exceeded; \ +disallowing requests for ~a seconds~%") + (- reset (car (gettimeofday)))) + (display-hint (G_ "You can waive the rate limit by +setting the @env{GUIX_GITHUB_TOKEN} environment variable to a token obtained +from @url{https://github.com/settings/tokens} with your GitHub account. + +Alternatively, you can wait until your rate limit is reset, or use the +@code{generic-git} updater instead.")) + #f)) ;bail out + (_ + (raise c))))) + + (let* ((port (http-fetch release-url #:headers headers)) + (result (json->scm port))) + (close-port port) + (match result + (#() + ;; We got the empty list, presumably because the user didn't use GitHub's + ;; "release" mechanism, but hopefully they did use Git tags. + (let* ((port (http-fetch tag-url #:headers headers)) + (json (json->scm port))) + (close-port port) + json)) + (x x)))))) (define (latest-released-version url package-name) "Return the newest released version and its tag given a string URL like @@ -223,23 +277,16 @@ (define (release->version release) (cons tag tag)) (else #f)))) - (let* ((json (and=> (fetch-releases-or-tags url) - vector->list))) - (if (eq? json #f) - (if (%github-token) - (error "Error downloading release information through the GitHub -API when using a GitHub token") - (error "Error downloading release information through the GitHub -API. This may be fixed by using an access token and setting the environment -variable GUIX_GITHUB_TOKEN, for instance one procured from -https://github.com/settings/tokens")) - (match (sort (filter-map release->version - (match (remove pre-release? json) - (() json) ; keep everything - (releases releases))) - (lambda (x y) (version>? (car x) (car y)))) - (((latest-version . tag) . _) (values latest-version tag)) - (() (values #f #f)))))) + (match (and=> (fetch-releases-or-tags url) vector->list) + (#f (values #f #f)) + (json + (match (sort (filter-map release->version + (match (remove pre-release? json) + (() json) ; keep everything + (releases releases))) + (lambda (x y) (version>? (car x) (car y)))) + (((latest-version . tag) . _) (values latest-version tag)) + (() (values #f #f)))))) (define (latest-release pkg) "Return an for the latest release of PKG." From patchwork Thu Mar 3 21:14:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 37602 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3A39127BBEA; Thu, 3 Mar 2022 21:30:39 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 02FD827BBE9 for ; Thu, 3 Mar 2022 21:30:39 +0000 (GMT) Received: from localhost ([::1]:32956 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nPt1x-0001af-QX for patchwork@mira.cbaines.net; Thu, 03 Mar 2022 16:30:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35386) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnv-0005ft-Lr for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:09 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50055) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsnr-0007Io-Fn for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nPsnr-0002Hb-C2 for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 3/4] http-client: Correctly handle redirects when #:keep-alive? #t. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421298696 (code B ref 54241); Thu, 03 Mar 2022 21:16:03 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:29 +0000 Received: from localhost ([127.0.0.1]:43947 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnJ-0002G7-4z for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57770) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnF-0002FQ-Us for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:26 -0500 Received: from [2001:470:142:3::e] (port=58926 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnA-00079I-Oz; Thu, 03 Mar 2022 16:15:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=3FOgxt6RAntKXgIo3ctekWMUYLNuoI7nrkNNHJj7ILo=; b=UETos/z1a/NobplX+7K7 k/BWR4KDemZttuqwBGsmlqIyX2+2VXhsHdU/Swn5I2TJvlO+4RKkXSgoj0Krrmdd2QmVlyvU2IgUf slTxcRUqyPR935THz3526G7drgxlIoxpfPa4nDPpLxzseC9gAIzIhXCRJIuKbAeYbheXOPbmUud+o buPuCtyZU84uPtMWC1uWhgFVguB3JAITR21K/IYi4o0OnD3DeoYR7ktDYNAKlHorQqrajHM/7kHOE G2QNTWDuxZzVNongb47zeW6hnDJ8IqPFqOjF4kNA5asxogbbWu0LFj2HaNg4ew1K7AVVpuqrEtKyU +CXXViqfP/aDcQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsn9-0006KA-9e; Thu, 03 Mar 2022 16:15:20 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:43 +0100 Message-Id: <20220303211444.19928-3-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220303211444.19928-1-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Previously PORT would be closed unconditionally, which broke redirects when #:keep-alive? #t is given. * guix/http-client.scm (http-fetch): Make 'port' a parameter of 'loop'. Upon 3xx responses, do not close PORT is KEEP-ALIVE? is true, but consume RESP's body. Add second argument to 'loop'. --- guix/http-client.scm | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 4b01e31165..4784497e5f 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -102,12 +102,12 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) Raise an '&http-get-error' condition if downloading fails." (let loop ((uri (if (string? uri) (string->uri uri) - uri))) - (let ((port (or port (open-connection uri - #:verify-certificate? - verify-certificate? - #:timeout timeout))) - (headers (match (uri-userinfo uri) + uri)) + (port (or port (open-connection uri + #:verify-certificate? + verify-certificate? + #:timeout timeout)))) + (let ((headers (match (uri-userinfo uri) ((? string? str) (cons (cons 'Authorization (string-append "Basic " @@ -131,11 +131,19 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) 303 ; see other 307 ; temporary redirection 308) ; permanent redirection - (let ((uri (resolve-uri-reference (response-location resp) uri))) - (close-port port) + (let ((host (uri-host uri)) + (uri (resolve-uri-reference (response-location resp) uri))) + (if keep-alive? + (dump-port data (%make-void-port "w0") + (response-content-length resp)) + (close-port port)) (format log-port (G_ "following redirection to `~a'...~%") (uri->string uri)) - (loop uri))) + (loop uri + (and keep-alive? + (or (not (uri-host uri)) + (string=? host (uri-host uri))) + port)))) (else (raise (condition (&http-get-error (uri uri) From patchwork Thu Mar 3 21:14:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 37600 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3F09A27BBEA; Thu, 3 Mar 2022 21:30:04 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E877E27BBE9 for ; Thu, 3 Mar 2022 21:30:03 +0000 (GMT) Received: from localhost ([::1]:60682 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nPt1P-0001Eh-3e for patchwork@mira.cbaines.net; Thu, 03 Mar 2022 16:30:03 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35388) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnv-0005g4-Qg for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:09 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50056) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsnr-0007Iu-Rb for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:06 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nPsnr-0002Hk-OG for guix-patches@gnu.org; Thu, 03 Mar 2022 16:16:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 4/4] import: github: Reuse HTTP connection for the /tags URL fallback. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421308704 (code B ref 54241); Thu, 03 Mar 2022 21:16:03 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:30 +0000 Received: from localhost ([127.0.0.1]:43949 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnK-0002GK-Eh for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:30 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57782) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnI-0002FV-63 for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:28 -0500 Received: from [2001:470:142:3::e] (port=58928 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnC-00079S-6p; Thu, 03 Mar 2022 16:15:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=yWJeafeePwydXKt0CTt4A2Wg0EUsJ40BZd0C7uxqd5U=; b=T8qUP8WETPgn+gygoMB8 5P6dtkg/38+jCHrH7s7bW/gSfWXrgJaB9KRkrJuJP3u8EFTbe7RDXZ95giYUiGqnxDm0ck8UfcOt3 48FuXDDBnm25U0GMkZi7QBIINVmpimcLwtbXGeDsxsee9cO5THEi5k5DT617cFjthFddH6JRVwYtY dzXcqeBcINglirLl97iunJzi9/AT2Kq0V0eFa1Qzu4JLELsaSyb8NazbjpuqZYl01Ke7j1unbHfRY uWK153XYXk0g8tEpfB//A83gxMSc/jpomRMYz67oKHGtmAE25TvBOZbPUF1QapTEJ79smqK9Aj64d rR+iDm/QQ+0ghw==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsnA-0006KA-Un; Thu, 03 Mar 2022 16:15:21 -0500 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:44 +0100 Message-Id: <20220303211444.19928-4-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220303211444.19928-1-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/import/github.scm (fetch-releases-or-tags): Call 'open-connection-for-uri' and reuse the same connection for the two 'http-fetch' calls. * .dir-locals.el (scheme-mode): Add 'call-with-port'. --- .dir-locals.el | 1 + guix/import/github.scm | 30 ++++++++++++++++++------------ 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/.dir-locals.el b/.dir-locals.el index 0edf2a8d23..bee745cc27 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -52,6 +52,7 @@ (eval . (put 'test-equal 'scheme-indent-function 1)) (eval . (put 'test-eq 'scheme-indent-function 1)) (eval . (put 'call-with-input-string 'scheme-indent-function 1)) + (eval . (put 'call-with-port 'scheme-indent-function 1)) (eval . (put 'guard 'scheme-indent-function 1)) (eval . (put 'lambda* 'scheme-indent-function 1)) (eval . (put 'substitute* 'scheme-indent-function 1)) diff --git a/guix/import/github.scm b/guix/import/github.scm index 5062bad80d..84f70eed0f 100644 --- a/guix/import/github.scm +++ b/guix/import/github.scm @@ -33,6 +33,7 @@ (define-module (guix import github) #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix download) #:prefix download:) #:use-module ((guix git-download) #:prefix download:) + #:autoload (guix build download) (open-connection-for-uri) #:use-module (guix import utils) #:use-module (json) #:use-module (guix packages) @@ -226,18 +227,23 @@ (define headers (_ (raise c))))) - (let* ((port (http-fetch release-url #:headers headers)) - (result (json->scm port))) - (close-port port) - (match result - (#() - ;; We got the empty list, presumably because the user didn't use GitHub's - ;; "release" mechanism, but hopefully they did use Git tags. - (let* ((port (http-fetch tag-url #:headers headers)) - (json (json->scm port))) - (close-port port) - json)) - (x x)))))) + (let ((release-uri (string->uri release-url))) + (call-with-port (open-connection-for-uri release-uri) + (lambda (connection) + (let* ((result (json->scm + (http-fetch release-uri + #:port connection + #:keep-alive? #t + #:headers headers)))) + (match result + (#() + ;; We got the empty list, presumably because the user didn't use GitHub's + ;; "release" mechanism, but hopefully they did use Git tags. + (json->scm (http-fetch tag-url + #:port connection + #:keep-alive? #t + #:headers headers))) + (x x))))))))) (define (latest-released-version url package-name) "Return the newest released version and its tag given a string URL like