From patchwork Mon May 22 15:30:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 50234 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E981027BBE9; Mon, 22 May 2023 16:32:28 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 2FAB027BBE2 for ; Mon, 22 May 2023 16:32:28 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q17WH-0007Qk-On; Mon, 22 May 2023 11:32:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17W0-0007ND-Lc for guix-patches@gnu.org; Mon, 22 May 2023 11:32:08 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q17W0-0001LL-1v; Mon, 22 May 2023 11:32:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q17Vx-0004tl-UH; Mon, 22 May 2023 11:32:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63646] [PATCH] substitute: If a server's nar URL is 404, try the next one(s). Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: mail@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Mon, 22 May 2023 15:32:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 63646 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 63646@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.168476946918760 (code B ref -1); Mon, 22 May 2023 15:32:01 +0000 Received: (at submit) by debbugs.gnu.org; 22 May 2023 15:31:09 +0000 Received: from localhost ([127.0.0.1]:35934 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q17V6-0004sV-Kw for submit@debbugs.gnu.org; Mon, 22 May 2023 11:31:09 -0400 Received: from lists.gnu.org ([209.51.188.17]:43202) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q17V4-0004sN-F8 for submit@debbugs.gnu.org; Mon, 22 May 2023 11:31:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17V3-0007JC-CE for guix-patches@gnu.org; Mon, 22 May 2023 11:31:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17V1-0001E5-PC; Mon, 22 May 2023 11:31:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=W8FgW+p0K19l9XTbOwJpeMnNIILn3UGIhOSd+7VQ6f4=; b=WMKZnyqJO8dQpq ehNwetFFkQANvEyS2EOB5ibdDFU8hFVRlzuJECOyndywjI6tKiPFz38iMvXaYEM10bPSH9qk5MCAl NHx81MVHbamkLOuZeS8H31hZ0Kxal5VI78bt/mm+Utn8ixgCaGqDbYt/eXHPHiNsy0ylIXR+DwpAA cYV+ft79LqFlcjg8erkwN7psQG2tEDjE4vC/IOX2gre10KDf02wViXvIAPGIkw6VnZtjzI+7RNvgm I+Cbe/ec6RzNB7gEO4CFck3EbH5biPeBLF7EzjYdd3CS3i2i/h8v8iXg876GK45InULqX4aDHxE07 QcSMdPHJsAWhkcceLWOw==; Received: from [193.50.110.247] (helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17V0-0003C9-65; Mon, 22 May 2023 11:31:02 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Mon, 22 May 2023 17:30:50 +0200 Message-Id: X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches If a substitute server advertises in its narinfo, for example, both a /zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip URL. Fixes . * guix/narinfo.scm (narinfo-preferred-uris): New procedure. (narinfo-best-uri): Rebase on top of it. * guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure. Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of NARINFO. * tests/substitute.scm (request-substitution): Remove 'parameterize'. Delete DESTINATION. ("substitute, preferred nar URL is 404, other is 200"): New test. --- guix/narinfo.scm | 21 +++++++++++++++++---- guix/scripts/substitute.scm | 33 ++++++++++++++++++++++----------- tests/substitute.scm | 36 +++++++++++++++++++++++++++++++----- 3 files changed, 70 insertions(+), 20 deletions(-) base-commit: dff1689bb37e5303868584d3f1d7a33cbcb7f51e diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 741c7ad406..a149d9a901 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -54,6 +54,7 @@ (define-module (guix narinfo) narinfo-hash-algorithm+value narinfo-hash->sha256 + narinfo-preferred-uris narinfo-best-uri valid-narinfo? @@ -309,9 +310,11 @@ (define (decompresses-faster? compression1 compression2) ("gzip" (string=? compression2 "lzip")) (_ #f))) -(define* (narinfo-best-uri narinfo #:key fast-decompression?) - "Select the \"best\" URI to download NARINFO's nar, and return three values: -the URI, its compression method (a string), and the compressed file size. +(define* (narinfo-preferred-uris narinfo #:key fast-decompression?) + "Return the sorted list of \"preferred\" nar URIs from NARINFO (preferred +comes first) where each entry is a tuple containing: the URI, its compression +method (a string), and the compressed file size. + When FAST-DECOMPRESSION? is true, prefer substitutes with faster decompression (typically zstd) rather than substitutes with a higher compression ratio (typically lzip)." @@ -343,6 +346,16 @@ (define* (narinfo-best-uri narinfo #:key fast-decompression?) ((uri2 compression2 . _) (decompresses-faster? compression2 compression1)))))) - (match (sort choices (if fast-decompression? (negate speedstring uri))))) - (let ((uri compression file-size - (narinfo-best-uri narinfo - #:fast-decompression? - %prefer-fast-decompression?))) - (unless print-build-trace? - (format (current-error-port) - (G_ "Downloading ~a...~%") (uri->string uri))) + (define (try-fetch choices) + (match choices + (((uri compression file-size) rest ...) + (guard (c ((and (pair? rest) (network-error? c)) + (warning (G_ "download from '~a' failed, trying next URL~%") + (uri->string uri)) + (try-fetch rest))) + (let ((port download-size (fetch uri))) + (unless print-build-trace? + (format (current-error-port) + (G_ "Downloading ~a...~%") (uri->string uri))) + (values port uri compression download-size)))) + (() + (leave (G_ "no valid nar URLs for ~a at ~a~%") + (narinfo-path narinfo) + (narinfo-uri-base narinfo))))) - (let* ((raw download-size - ;; 'guix publish' without '--cache' doesn't specify a - ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. - (fetch uri)) + (let ((choices (narinfo-preferred-uris narinfo + #:fast-decompression? + %prefer-fast-decompression?))) + ;; 'guix publish' without '--cache' doesn't specify a Content-Length, so + ;; DOWNLOAD-SIZE is #f in this case. + (let* ((raw uri compression download-size (try-fetch choices)) (progress (let* ((dl-size (or download-size (and (equal? compression "none") diff --git a/tests/substitute.scm b/tests/substitute.scm index 9032a50268..8df3938b59 100644 --- a/tests/substitute.scm +++ b/tests/substitute.scm @@ -64,11 +64,11 @@ (define-syntax-rule (test-quit name error-rx exp) (define (request-substitution item destination) "Run 'guix substitute --substitute' to fetch ITEM to DESTINATION." - (parameterize ((guix-warning-port (current-error-port))) - (with-input-from-string (string-append "substitute " item " " - destination "\n") - (lambda () - (guix-substitute "--substitute"))))) + (false-if-exception (delete-file destination)) + (with-input-from-string (string-append "substitute " item " " + destination "\n") + (lambda () + (guix-substitute "--substitute")))) (define %public-key ;; This key is known to be in the ACL by default. @@ -613,6 +613,32 @@ (define-syntax-rule (with-narinfo* narinfo directory body ...) (lambda () (false-if-exception (delete-file "substitute-retrieved"))))))))))) +(test-equal "substitute, preferred nar URL is 404, other is 200" + "Substitutable data." + (with-narinfo* (string-append %narinfo "Signature: " (signature-field %narinfo)) + %main-substitute-directory + + (with-http-server `((200 ,(string-append %narinfo "Signature: " + (signature-field %narinfo) + "\n" + "URL: example.nar.lz\n" + "Compression: lzip\n")) + (404 "Sorry, nar.lz is missing!") + (200 ,(call-with-input-file + (string-append %main-substitute-directory + "/example.nar") + get-bytevector-all))) + (dynamic-wind + (const #t) + (lambda () + (parameterize ((substitute-urls (list (%local-url)))) + (request-substitution (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + "substitute-retrieved")) + (call-with-input-file "substitute-retrieved" get-string-all)) + (lambda () + (false-if-exception (delete-file "substitute-retrieved"))))))) + (test-quit "substitute, narinfo is available but nar is missing" "failed to find alternative substitute" (with-narinfo*