From patchwork Mon Mar 20 14:09:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 48214 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E477716F1D; Mon, 20 Mar 2023 14:10:28 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7D70C16EC7 for ; Mon, 20 Mar 2023 14:10:27 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1peGD8-0008K1-Mp; Mon, 20 Mar 2023 10:10:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1peGD6-0008Ik-C3 for guix-patches@gnu.org; Mon, 20 Mar 2023 10:10:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1peGD4-0004Cj-Hp for guix-patches@gnu.org; Mon, 20 Mar 2023 10:10:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1peGD4-0004DZ-Ar for guix-patches@gnu.org; Mon, 20 Mar 2023 10:10:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43442] [PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020 Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 20 Mar 2023 14:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43442 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: zimoun Cc: 43442@debbugs.gnu.org Received: via spool by 43442-submit@debbugs.gnu.org id=B43442.167932136116148 (code B ref 43442); Mon, 20 Mar 2023 14:10:02 +0000 Received: (at 43442) by debbugs.gnu.org; 20 Mar 2023 14:09:21 +0000 Received: from localhost ([127.0.0.1]:55443 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1peGCO-0004CN-Vw for submit@debbugs.gnu.org; Mon, 20 Mar 2023 10:09:21 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:51064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1peGCM-0004C2-DC for 43442@debbugs.gnu.org; Mon, 20 Mar 2023 10:09:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=ZXgWW4DhDQrUfTaQOnOoMqLhzH2AVg6/Tt35qHUcGk0=; b=PXyR6HVLP05J1oetvEKzKq8zZoISabxVafitfcnSvHIXQJe9iXpqb0uy oninQB2KYeepWSAYlmNFGSHR3HpazwUDh2lkieI2ANMJqLtpLfYD7JiVL e1eiNTjz4BxFZrTQ1qAmEdUOXgjmW/P9RlwcfZ3Wuy42v6d1gKwg3DM2m A=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludovic.courtes@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="5.98,274,1673910000"; d="scan'208";a="50699762" Received: from unknown (HELO ribbon) ([193.50.110.213]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2023 15:09:12 +0100 From: Ludovic =?utf-8?q?Court=C3=A8s?= References: <20200916081411.5801-1-zimon.toutoune@gmail.com> <20200916081642.6716-1-zimon.toutoune@gmail.com> <87o8lyj090.fsf@gnu.org> <86zh5iol1n.fsf@gmail.com> <87h7rocvkt.fsf@gnu.org> <877dsi44lu.fsf@gnu.org> <87y2knhei3.fsf@gnu.org> Date: Mon, 20 Mar 2023 15:09:11 +0100 In-Reply-To: <87y2knhei3.fsf@gnu.org> ("Ludovic =?utf-8?q?Court=C3=A8s?= "'s message of "Sat, 03 Oct 2020 10:59:00 +0200") Message-ID: <87jzzbms54.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Hi! Ludovic Courtès skribis: > Ah yes, under “extra_headers” there’s the SVN revision number. (guix > swh) doesn’t expose “extra_headers” yet, but once it does, we can walk > snapshots until we find the SVN revision we’re looking for. > > scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/") > $3 = #< visits-url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visits/" type: #f url: "https://scm.gforge.inria.fr/anonscm/svn/mpfi"> > scheme@(guile-user)> (origin-visits $3) > $4 = (#< date: # origin: "https://scm.gforge.inria.fr/anonscm/svn/mpfi" url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/1/" snapshot-url: "https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f710dbc55c1b308804fa1b5f51f0/" status: full number: 1>) > scheme@(guile-user)> (visit-snapshot (car $4)) > $5 = #< branches: (#< name: "HEAD" target-type: revision target-url: "https://archive.softwareheritage.org/api/1/revision/f7b445a6bdc38bf075f29265120ca49824f698ea/">)> > > So the next step is to augment (guix swh) with a > ‘lookup-subversion-revision’ procedure. The attached patch does that: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.fr/anonscm/svn/mpfi" 680) $12 = #< id: "72102de7605a2459ebcb016338ebbf1a997e8c8d" date: # directory: "5c89c025a4cd9d16befdfec12dfc23f7318d0d5b" directory-url: "https://archive.softwareheritage.org/api/1/directory/5c89c025a4cd9d16befdfec12dfc23f7318d0d5b/" parents-ids: ("16da41f1848d77a93aec565320b72b460c429b61") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-4709-9413-9660a31d4696") ("svn_revision" . "680"))> scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.fr/anonscm/svn/mpfi" 666) $13 = #< id: "148eb1e7206b111af4075c73c656e54c9efed6cb" date: # directory: "ed7b0bd7019fb85cd86d948a97c23b9d43aa8728" directory-url: "https://archive.softwareheritage.org/api/1/directory/ed7b0bd7019fb85cd86d948a97c23b9d43aa8728/" parents-ids: ("0ba2aa7e0d3fc0a1eb3ba72b32094515415ae47a") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-4709-9413-9660a31d4696") ("svn_revision" . "666"))> --8<---------------cut here---------------end--------------->8--- The implementation is pretty bad though, because it walks the revision history until it finds the right revision number—so you’re likely to reach the bandwidth rate limit before you’ve found the revision you’re looking for. More importantly, most svn origins cannot be found, or at least not by passing the URL as-is: https://sympa.inria.fr/sympa/arc/swh-devel/2023-03/msg00009.html This whole hack looks like a dead end. It would be ideal if SWH would compute nar hashes, as you proposed: https://gitlab.softwareheritage.org/swh/meta/-/issues/4538 As a stopgap, I wonder if we could use “double hashing” on our side, but only for svn: we’d store both the nar sha256 as we currently do, plus the swhid. It still seems to me that it’d be hard to scale and to maintain that over time, even if it’s limited to svn. Plus, there’d still be the problem of ‘svn-multi-fetch’, which is what most TeX Live packages use. Thoughts? Ludo’. diff --git a/guix/swh.scm b/guix/swh.scm index c7c1c873a2..a65635b1db 100644 --- a/guix/swh.scm +++ b/guix/swh.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2018, 2019, 2020, 2021, 2023 Ludovic Courtès ;;; Copyright © 2020 Jakub Kądziołka ;;; Copyright © 2021 Xinglu Chen ;;; Copyright © 2021 Simon Tournier @@ -75,8 +75,10 @@ (define-module (guix swh) revision-id revision-date revision-directory + revision-parents lookup-revision lookup-origin-revision + lookup-subversion-revision content? content-checksums @@ -207,6 +209,14 @@ (define string* ((? null?) #f) ;Guile-JSON 3.x ('null #f))) ;Guile-JSON 4.x +(define pair-vector->alist + (match-lambda + ('null '()) + ((= vector->list lst) + (map (match-lambda + (#(key value) (cons key value))) + lst)))) + (define %allow-request? ;; Takes a URL and method (e.g., the 'http-get' procedure) and returns true ;; to keep going. This can be used to disallow requests when @@ -346,7 +356,14 @@ (define-json-mapping make-revision revision? (id revision-id) (date revision-date "date" (maybe-null string->date*)) (directory revision-directory) - (directory-url revision-directory-url "directory_url")) + (directory-url revision-directory-url "directory_url") + (parents-ids revision-parent-ids "parents" + (lambda (vector) + (map (lambda (alist) + (assoc-ref alist "id")) + (vector->list vector)))) + (extra-headers revision-extra-headers ;alist--e.g., with "svn_revision" + "extra_headers" pair-vector->alist)) ;; (define-json-mapping make-content content? @@ -524,6 +541,50 @@ (define (lookup-origin-revision url tag) (() #f))))) +(define (revision-parents revision) + "Return the parent revision(s) of REVISION." + (filter-map lookup-revision (revision-parent-ids revision))) + +(define (lookup-subversion-revision-in-history revision revision-number) + "Look for Subversion REVISION-NUMBER starting from REVISION and going back +in history." + (let loop ((revision revision)) + (let ((number (and=> (assoc-ref (revision-extra-headers revision) + "svn_revision") + string->number))) + (and number + (cond ((= number revision-number) + ;; Found it! + revision) + ((< number revision-number) + ;; REVISION is ancestor of REVISION-NUMBER, so stop here. + #f) + (else + ;; Check the parent(s) of REVISION. + (any loop (revision-parents revision)))))))) + +(define (lookup-subversion-revision url revision-number) + "Return either #f or the revision of the Subversion repository once +available at URL with the given REVISION-NUMBER." + (match (lookup-origin url) + (#f #f) + (origin + (match (filter (lambda (visit) + ;; Return #f if (visit-snapshot VISIT) would return #f. + (and (visit-snapshot-url visit) + (eq? 'full (visit-status visit)))) + (origin-visits origin)) + (() + #f) + ((visit . _) + (any (lambda (branch) + (match (branch-target branch) + ((? revision? revision) + (lookup-subversion-revision-in-history revision + revision-number)) + (_ #f))) + (snapshot-branches (visit-snapshot visit)))))))) + (define (release-target release) "Return the revision that is the target of RELEASE." (match (release-target-type release)