From patchwork Mon Sep 18 18:03:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Tournier X-Patchwork-Id: 54036 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id C79C627BBE9; Mon, 18 Sep 2023 19:08:12 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B90B627BBE2 for ; Mon, 18 Sep 2023 19:08:09 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qiIcx-0004AH-DX; Mon, 18 Sep 2023 14:05:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qiIcF-0003DB-FW for guix-patches@gnu.org; Mon, 18 Sep 2023 14:04:59 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qiIcF-0003ZN-7Z for guix-patches@gnu.org; Mon, 18 Sep 2023 14:04:59 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qiIcN-0002Oi-HY for guix-patches@gnu.org; Mon, 18 Sep 2023 14:05:07 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#62202] [PATCH v2 12/23] DRAFT import: juliahub: Add functions to parse the git repo for a git tag. Resent-From: Simon Tournier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 18 Sep 2023 18:05:07 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62202 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch moreinfo To: 62202@debbugs.gnu.org Cc: zimon.toutoune@gmail.com X-Debbugs-Original-Cc: Nicolas Graves via Guix-patches via , Simon Tournier Received: via spool by 62202-submit@debbugs.gnu.org id=B62202.16950602568893 (code B ref 62202); Mon, 18 Sep 2023 18:05:07 +0000 Received: (at 62202) by debbugs.gnu.org; 18 Sep 2023 18:04:16 +0000 Received: from localhost ([127.0.0.1]:54393 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qiIbY-0002JG-7r for submit@debbugs.gnu.org; Mon, 18 Sep 2023 14:04:16 -0400 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]:40127) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qiIbH-0002Eo-KZ for 62202@debbugs.gnu.org; Mon, 18 Sep 2023 14:04:04 -0400 Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-40471c054f9so14350515e9.0 for <62202@debbugs.gnu.org>; Mon, 18 Sep 2023 11:03:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1695060225; x=1695665025; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=naePonU76gTGTfpRR8tIbQvC/x6wxVESjDneXTICeZ0=; b=I9rp2BDaxnl+0HJPhzTE26DzMfHHf8FVuLl0xZPaa83Gki1f/6OkXzvLp+He9VMmip yWLm+AtSu1E/BglBzU1GqJuz4MqE3uQ6wSow6Pqk2h10zAr5jha6STUfynZSxgUF9r6p +E+Y4KTm8CZ+IuAgkR8g6cLRRMcibzoXHYiekMQIHZ8keoqk23lll6guwopRpZ7oz2YJ 0/Vzsbdhua7pUy9Q3v1TysjqxTBEAce9JXVXJJQ/QWz3WZBgkLLl+/STJonm/mKAUIFz FH7g0GrWmK+TRaXz/vfMOM0uWWpPIAbYI7VSEpboJ14juE4Lw4HZltCfW+gKGP2lXLES J44A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695060225; x=1695665025; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=naePonU76gTGTfpRR8tIbQvC/x6wxVESjDneXTICeZ0=; b=UKhYf3Ys1jQ5eZF99EdV5VcI0EQSRhtUTBp2/MKmcftmQG4ezGBP2YJ9AXpTbJl4xt IfhIs+9c/DmB1edVSyJBOvSMyFm/Us+adMPxPXCg3z5/0lgr6LikH0HPbb/iiIhFCdio V5Zlu2Kr4seFkoiK24h+XCQZ3vgn8kTBTFTkylEKw1XZrRAI+M20eLv/rq0R8vFe8mm2 gRYVtImcRDCvaejtPINOqrgsAVNaXQAhnzdgBu3UvrUnzl7VtfJIXQJUBoNr5ev+dALK YFG7wAS6ZhJr2xiB+slWeVulVl4OVI0oheeH/kf2ZACSps9w6u8NcWrRAz7zLLHPoN1J QwMw== X-Gm-Message-State: AOJu0Yxii844+3THq0A1EsGChhnimi5sJtuXT65jzCgd24hFBsAVUts9 VBKXsFo9XDeeGOTCAOsUMqTptGMJNN4= X-Google-Smtp-Source: AGHT+IELLI9+EqhFGs/m2GoRd8GSftNhN7iYJqP9e6nqsk98K9forq+ChwIxCQuTWCWoxBiUiZXvKQ== X-Received: by 2002:adf:ecc7:0:b0:31f:899b:a47 with SMTP id s7-20020adfecc7000000b0031f899b0a47mr7943960wro.4.1695060225415; Mon, 18 Sep 2023 11:03:45 -0700 (PDT) Received: from localhost.localdomain ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id z2-20020adfd0c2000000b0030ada01ca78sm2210465wrh.10.2023.09.18.11.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 11:03:45 -0700 (PDT) From: Simon Tournier Date: Mon, 18 Sep 2023 20:03:19 +0200 Message-Id: X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Nicolas Graves via Guix-patches via Signed-off-by: Simon Tournier --- guix/import/juliahub.scm | 62 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 6 deletions(-) diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm index fc25ba1d4220..5327e923251d 100644 --- a/guix/import/juliahub.scm +++ b/guix/import/juliahub.scm @@ -26,6 +26,7 @@ (define-module (guix import juliahub) #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (guix http-client) + #:use-module (guix git) #:use-module (guix import utils) #:use-module (guix import json) #:use-module (guix base32) @@ -38,6 +39,48 @@ (define-module (guix import juliahub) %juliahub-updater juliahub-recursive-import)) +;; Juliahub may be more up-to-date than the General registry or the actual git +;; tag (it seems around 6 hours pass between the time a commit is supplied to +;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes +;; the git tag). We have no simple way to get the commit of the latest-version. +;; Thus the best simple thing we can do is get the latest-git-version, and +;; import this version instead. We do this by parsing Package.toml in the General +;; registry, and then getting the refs of the git repo supplied by this +;; file. Parsing this file is also necessary if the package is in a subdir of a +;; git repository, because the information isn't present in Juliahub. +;; There's a last case where some Julia packages are not based on a particular +;; git tag, and where looking for the commit is tedious and artificial. In these +;; cases, we introduce the tree-commit which is available in the Versions.toml +;; file in the General repository. This is equivalent to a commit, since we have +;; a unique hash of the listing of files and directories, thus it can be used to +;; identify the state of a repository. + +(define %general-base-url + "https://raw.githubusercontent.com/JuliaRegistries/General/master/") + +(define (general-url package-name file) + (let ((folder (string-capitalize (string-take package-name 1)))) + (string-append + %general-base-url folder "/" package-name "/" file))) + +(define (ini-list->alist lst) + (match lst + ((attribute '= value ooo ...) + `((,attribute . ,value) ,@(ini-list->alist ooo))) + ('() + '()))) + +(define (ini-fetch url) + (let* ((port (http-fetch url #:text? #t)) + (ini-list (stream->list (port->stream port read)))) + (close-port port) + (ini-list->alist ini-list))) + +(define (latest-git-tag repo) + (let* ((last-ref (last (remote-refs repo #:tags? #t))) + (last-git-tag (last (string-split last-ref #\/)))) + (string-drop last-git-tag 1))) + ;; To update, see file sysimg.jl (define %julia-stdlibs (list "julia" @@ -194,14 +237,21 @@ (define (make-julia-sexp name source home-page synopsis description ((license) (license->symbol license)) (_ `(list ,@(map license->symbol licenses))))))) +;; TODO handle subdir case properly. + (define* (juliahub->guix-package package-name #:key version #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the `package' s-expression corresponding to that package, or #f on failure. Optionally include a VERSION string to fetch a specific version juliahub." - (let ((package (if version - (juliahub-fetch package-name version) - (juliahub-fetch package-name)))) + (let* ((package-toml (ini-fetch (general-url name "Package.toml"))) + (subdir (assoc-ref package-toml 'subdir)) + (tag (latest-git-tag (assoc-ref package-toml 'repo))) + (package (if version + (juliahub-fetch package-name version) + (if tag + (juliahub-fetch package-name tag) + (juliahub-fetch package-name))))) (if package (let-values (((source directory) (git->origin+dir @@ -233,13 +283,13 @@ (define* (juliahub->guix-package package-name (define (guix-package->juliahub-name package) (let* ((url (juliahub-package-url package)) - (git-name (car (last-pair (string-split url #\/)))) + (git-name (last (string-split url #\/))) (ungitted-name (if (string-suffix? ".git" git-name) (string-drop-right git-name 4) git-name)) (package-name (if (string-suffix? ".jl" ungitted-name) - (string-drop-right ungitted-name 4) - ungitted-name))) + (string-drop-right ungitted-name 4) + ungitted-name))) package-name)) (define* (import-release package #:key (version #f))