From patchwork Thu Dec 21 14:01:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nicolas Graves X-Patchwork-Id: 57956 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6C38527BBE9; Thu, 21 Dec 2023 14:02:22 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B5FE527BBE2 for ; Thu, 21 Dec 2023 14:02:19 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGJcg-0005Sh-GZ; Thu, 21 Dec 2023 09:02:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGJcc-0005S2-Vz for guix-patches@gnu.org; Thu, 21 Dec 2023 09:01:59 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGJcc-0005c0-MZ for guix-patches@gnu.org; Thu, 21 Dec 2023 09:01:58 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGJcg-0000ya-6H for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#62202] [PATCH v3 1/4] import: utils: Add function git->origin. References: <871qlq89kz.fsf@ngraves.fr> In-Reply-To: <871qlq89kz.fsf@ngraves.fr> Resent-From: Nicolas Graves Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 14:02:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62202 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch moreinfo To: 62202@debbugs.gnu.org Cc: ngraves@ngraves.fr Received: via spool by 62202-submit@debbugs.gnu.org id=B62202.17031673203738 (code B ref 62202); Thu, 21 Dec 2023 14:02:02 +0000 Received: (at 62202) by debbugs.gnu.org; 21 Dec 2023 14:02:00 +0000 Received: from localhost ([127.0.0.1]:42693 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcc-0000yD-Nd for submit@debbugs.gnu.org; Thu, 21 Dec 2023 09:02:00 -0500 Received: from 4.mo576.mail-out.ovh.net ([46.105.42.102]:40603) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcY-0000xn-6k for 62202@debbugs.gnu.org; Thu, 21 Dec 2023 09:01:57 -0500 Received: from director6.ghost.mail-out.ovh.net (unknown [10.108.17.219]) by mo576.mail-out.ovh.net (Postfix) with ESMTP id 28AC92A0FD for <62202@debbugs.gnu.org>; Thu, 21 Dec 2023 14:01:48 +0000 (UTC) Received: from ghost-submission-6684bf9d7b-v96f7 (unknown [10.110.96.188]) by director6.ghost.mail-out.ovh.net (Postfix) with ESMTPS id 49A241FEC6; Thu, 21 Dec 2023 14:01:47 +0000 (UTC) Received: from ngraves.fr ([37.59.142.105]) by ghost-submission-6684bf9d7b-v96f7 with ESMTPSA id b/ceDEtFhGUOPgAAgEA/0g (envelope-from ); Thu, 21 Dec 2023 14:01:47 +0000 Authentication-Results: garm.ovh; auth=pass (GARM-105G0067c578b33-e0e4-42e2-a5ba-1545eed2567e, 9D5CD1A767DEA5A8C66A052B36C551B71BBC44F2) smtp.auth=ngraves@ngraves.fr X-OVh-ClientIp: 89.85.249.141 Date: Thu, 21 Dec 2023 15:01:00 +0100 Message-ID: <20231221140142.16523-1-ngraves@ngraves.fr> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Ovh-Tracer-Id: 4993366087554491106 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvkedrvdduhedgfeegucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpefhvfevufffkffogggtgfesthekredtredtjeenucfhrhhomheppfhitgholhgrshcuifhrrghvvghsuceonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqeenucggtffrrghtthgvrhhnpefgfeejveevtdfgieekieejueffhefhtdetudffgedvkedvudejgeejieffvdetffenucffohhmrghinhepghhithhlrggsrdgtohhmpdhsphgugidrohhrghdpghhithhhuhgsrdgtohhmnecukfhppeduvdejrddtrddtrddupdekledrkeehrddvgeelrddugedupdefjedrheelrddugedvrddutdehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpeeonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqedpnhgspghrtghpthhtohepuddprhgtphhtthhopeeivddvtddvseguvggssghughhsrdhgnhhurdhorhhgpdfovfetjfhoshhtpehmohehjeeipdhmohguvgepshhmthhpohhuth X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Nicolas Graves X-ACL-Warn: , Nicolas Graves via Guix-patches X-Patchwork-Original-From: Nicolas Graves via Guix-patches via From: Nicolas Graves Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/import/utils.scm: (git->origin): Add function. * guix/import/elpa.scm (download-git-repository): Remove function download-git-repository. (git-repository->origin): Remove function git-repository->origin. (ref): Add function ref. (melpa-recipe->origin): Use functions git->origin and ref. * guix/import/go.scm (git-checkout-hash): Remove function git-checkout-hash. (transform-version): Add function transform-version. (vcs->origin): Use functions git->origin and transform-version. Add optional argument transform-version. * tests/import/go.scm (go-module->guix-package): Adapt test case to changes in guix/import/go.scm. * guix/import/minetest.scm (download-git-repository): Remove function download-git-repository. (make-minetest-sexp): Use function git->origin. * tests/minetest.scm (make-package-sexp): Use function git->origin. (example-package): Adapt test-case to git->origin. * guix/import/composer.scm (make-php-sexp): Use function git->origin. Change-Id: Ied05a63bdd60fbafe26fbbb4e115ff6f0bb9db3c --- guix/import/composer.scm | 86 ++++++++++++++-------------------------- guix/import/elpa.scm | 43 ++++++-------------- guix/import/go.scm | 57 ++++++++------------------ guix/import/minetest.scm | 28 ++----------- guix/import/utils.scm | 39 ++++++++++++++++++ tests/go.scm | 29 ++++++++++---- tests/minetest.scm | 15 ++----- 7 files changed, 127 insertions(+), 170 deletions(-) diff --git a/guix/import/composer.scm b/guix/import/composer.scm index 1ad608964b..dabc5423ed 100644 --- a/guix/import/composer.scm +++ b/guix/import/composer.scm @@ -19,22 +19,17 @@ (define-module (guix import composer) #:use-module (ice-9 match) #:use-module (json) - #:use-module (guix hash) - #:use-module (guix base32) - #:use-module (guix build git) - #:use-module (guix build utils) - #:use-module (guix build-system) #:use-module (guix build-system composer) + #:use-module ((guix download) #:select (download-to-store)) #:use-module (guix import json) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix memoization) #:use-module (guix packages) - #:use-module (guix serialization) + #:use-module (guix store) #:use-module (guix upstream) #:use-module (guix utils) #:use-module (srfi srfi-1) - #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:export (composer->guix-package %composer-updater @@ -141,55 +136,34 @@ (define (make-php-sexp composer-package) (dependencies (map php-package-name (composer-package-require composer-package))) (dev-dependencies (map php-package-name - (composer-package-dev-require composer-package))) - (git? (equal? (composer-source-type source) "git"))) - ((if git? call-with-temporary-directory call-with-temporary-output-file) - (lambda* (temp #:optional port) - (and (if git? - (begin - (mkdir-p temp) - (git-fetch (composer-source-url source) - (composer-source-reference source) - temp)) - (url-fetch (composer-source-url source) temp)) - `(package - (name ,(composer-package-name composer-package)) - (version ,(composer-package-version composer-package)) - (source - (origin - ,@(if git? - `((method git-fetch) - (uri (git-reference - (url ,(if (string-suffix? - ".git" - (composer-source-url source)) - (string-drop-right - (composer-source-url source) - (string-length ".git")) - (composer-source-url source))) - (commit ,(composer-source-reference source)))) - (file-name (git-file-name name version)) - (sha256 - (base32 - ,(bytevector->nix-base32-string - (file-hash* temp))))) - `((method url-fetch) - (uri ,(composer-source-url source)) - (sha256 (base32 ,(guix-hash-url temp))))))) - (build-system composer-build-system) - ,@(if (null? dependencies) - '() - `((inputs - (list ,@(map string->symbol dependencies))))) - ,@(if (null? dev-dependencies) - '() - `((native-inputs - (list ,@(map string->symbol dev-dependencies))))) - (synopsis "") - (description ,(composer-package-description composer-package)) - (home-page ,(composer-package-homepage composer-package)) - (license ,(or (composer-package-license composer-package) - 'unknown-license!)))))))) + (composer-package-dev-require composer-package)))) + `(package + (name ,(composer-package-name composer-package)) + (version ,(composer-package-version composer-package)) + (source + ,(if (string= (composer-source-type source) "git") + (git->origin (composer-source-url source) + `(tag-or-commit . ,(composer-source-reference source))) + (let* ((source (composer-source-url source)) + (tarball (with-store store (download-to-store store source)))) + `(origin + (method url-fetch) + (uri ,source) + (sha256 (base32 ,(guix-hash-url tarball))))))) + (build-system composer-build-system) + ,@(if (null? dependencies) + '() + `((inputs + (list ,@(map string->symbol dependencies))))) + ,@(if (null? dev-dependencies) + '() + `((native-inputs + (list ,@(map string->symbol dev-dependencies))))) + (synopsis "") + (description ,(composer-package-description composer-package)) + (home-page ,(composer-package-homepage composer-package)) + (license ,(or (composer-package-license composer-package) + 'unknown-license!))))) (define composer->guix-package (memoize diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm index d1855b3698..a755387242 100644 --- a/guix/import/elpa.scm +++ b/guix/import/elpa.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2021 Sarah Morgensen ;;; Copyright © 2021 Simon Tournier ;;; Copyright © 2022 Hartmut Goebel +;;; Copyright © 2023 Nicolas Graves ;;; ;;; This file is part of GNU Guix. ;;; @@ -208,11 +209,6 @@ (define* (fetch-elpa-package name #:optional (repo 'gnu)) url))) (_ #f)))) -(define* (download-git-repository url ref) - "Fetch the given REF from the Git repository at URL." - (with-store store - (latest-repository-commit store url #:ref ref))) - (define (package-name->melpa-recipe package-name) "Fetch the MELPA recipe for PACKAGE-NAME, represented as an alist from keywords to values." @@ -232,28 +228,15 @@ (define (data->recipe data) (close-port port) (data->recipe (cons ':name data)))) -(define (git-repository->origin recipe url) - "Fetch origin details from the Git repository at URL for the provided MELPA -RECIPE." - (define ref - (cond - ((assoc-ref recipe #:branch) - => (lambda (branch) (cons 'branch branch))) - ((assoc-ref recipe #:commit) - => (lambda (commit) (cons 'commit commit))) - (else - '()))) - - (let-values (((directory commit) (download-git-repository url ref))) - `(origin - (method git-fetch) - (uri (git-reference - (url ,url) - (commit ,commit))) - (sha256 - (base32 - ,(bytevector->nix-base32-string - (file-hash* directory #:recursive? #true))))))) +(define (ref recipe) + "Create REF from MELPA RECIPE." + (cond + ((assoc-ref recipe #:branch) + => (lambda (branch) (cons 'branch branch))) + ((assoc-ref recipe #:commit) + => (lambda (commit) (cons 'commit commit))) + (else + '()))) (define* (melpa-recipe->origin recipe) "Fetch origin details from the MELPA recipe and associated repository for @@ -264,9 +247,9 @@ (define (gitlab-repo->url repo) (string-append "https://gitlab.com/" repo ".git")) (match (assq-ref recipe ':fetcher) - ('github (git-repository->origin recipe (github-repo->url (assq-ref recipe ':repo)))) - ('gitlab (git-repository->origin recipe (gitlab-repo->url (assq-ref recipe ':repo)))) - ('git (git-repository->origin recipe (assq-ref recipe ':url))) + ('github (git->origin (github-repo->url (assq-ref recipe ':repo)) (ref recipe))) + ('gitlab (git->origin (gitlab-repo->url (assq-ref recipe ':repo)) (ref recipe))) + ('git (git->origin (assq-ref recipe ':url) (ref recipe))) (#f #f) ; if we're not using melpa then this stops us printing a warning (_ (warning (G_ "unsupported MELPA fetcher: ~a, falling back to unstable MELPA source~%") (assq-ref recipe ':fetcher)) diff --git a/guix/import/go.scm b/guix/import/go.scm index dd9298808d..6e2ce2ed00 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2021 Sarah Morgensen ;;; Copyright © 2021 Simon Tournier ;;; Copyright © 2023 Efraim Flashner +;;; Copyright © 2023 Nicolas Graves ;;; ;;; This file is part of GNU Guix. ;;; @@ -514,49 +515,24 @@ (define (module-meta-data-repo-url meta-data goproxy) goproxy (module-meta-repo-root meta-data))) -(define* (git-checkout-hash url reference algorithm) - "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or -tag." - (define cache - (string-append (or (getenv "TMPDIR") "/tmp") - "/guix-import-go-" - (passwd:name (getpwuid (getuid))))) +;; This is done because the version field of the package, which the generated +;; quoted expression refers to, has been stripped of any 'v' prefixed. +(define (transform-version version) + (let ((plain-version? (string=? version (go-version->git-ref version))) + (v-prefixed? (string-prefix? "v" version))) + (if (and plain-version? v-prefixed?) + '(string-append "v" version) + '(go-version->git-ref version)))) - ;; Use a custom cache to avoid cluttering the default one under - ;; ~/.cache/guix, but choose one under /tmp so that it's persistent across - ;; subsequent "guix import" invocations. - (mkdir-p cache) - (chmod cache #o700) - (let-values (((checkout commit _) - (parameterize ((%repository-cache-directory cache)) - (update-cached-checkout url - #:ref - `(tag-or-commit . ,reference))))) - (file-hash* checkout #:algorithm algorithm #:recursive? #true))) - -(define (vcs->origin vcs-type vcs-repo-url version) +(define* (vcs->origin vcs-type vcs-repo-url version + #:key (transform-version #f)) "Generate the `origin' block of a package depending on what type of source -control system is being used." +control system is being used. Optionally use the function TRANSFORM-VERSION +which takes version as an input." (case vcs-type ((git) - (let ((plain-version? (string=? version (go-version->git-ref version))) - (v-prefixed? (string-prefix? "v" version))) - `(origin - (method git-fetch) - (uri (git-reference - (url ,vcs-repo-url) - ;; This is done because the version field of the package, - ;; which the generated quoted expression refers to, has been - ;; stripped of any 'v' prefixed. - (commit ,(if (and plain-version? v-prefixed?) - '(string-append "v" version) - '(go-version->git-ref version))))) - (file-name (git-file-name name version)) - (sha256 - (base32 - ,(bytevector->nix-base32-string - (git-checkout-hash vcs-repo-url (go-version->git-ref version) - (hash-algorithm sha256)))))))) + (git->origin vcs-repo-url `(tag-or-commit . ,version) + #:ref->commit transform-version)) ((hg) `(origin (method hg-fetch) @@ -649,7 +625,8 @@ (define* (go-module->guix-package module-path #:key (name ,guix-name) (version ,(strip-v-prefix version*)) (source - ,(vcs->origin vcs-type vcs-repo-url version*)) + ,(vcs->origin vcs-type vcs-repo-url version* + #:transform-version transform-version)) (build-system go-build-system) (arguments (list ,@(if (version>? min-go-version (package-version (go-package))) diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm index 5ea6e023ce..65ef242431 100644 --- a/guix/import/minetest.scm +++ b/guix/import/minetest.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2021, 2022 Maxime Devos ;;; Copyright © 2022 Hartmut Goebel +;;; Copyright © 2023 Nicolas Graves ;;; ;;; This file is part of GNU Guix. ;;; @@ -32,7 +33,6 @@ (define-module (guix import minetest) #:use-module (guix import utils) #:use-module (guix import json) #:use-module (json) - #:use-module (guix base32) #:use-module (guix git) #:use-module ((guix git-download) #:prefix download:) #:use-module (guix hash) @@ -277,12 +277,6 @@ (define url (string-append (%contentdb-api) "packages/?type=" type -;; XXX copied from (guix import elpa) -(define* (download-git-repository url ref) - "Fetch the given REF from the Git repository at URL." - (with-store store - (latest-repository-commit store url #:ref ref))) - (define (make-minetest-sexp author/name version repository commit inputs home-page synopsis description media-license license) @@ -293,24 +287,8 @@ (define (make-minetest-sexp author/name version repository commit (name ,(contentdb->package-name author/name)) (version ,version) (source - (origin - (method git-fetch) - (uri (git-reference - (url ,repository) - (commit ,commit))) - (sha256 - (base32 - ;; The git commit is not always available. - ,(and commit - (bytevector->nix-base32-string - (file-hash* - (download-git-repository repository - `(commit . ,commit)) - ;; 'download-git-repository' already filtered out the '.git' - ;; directory. - #:select? (const #true) - #:recursive? #true))))) - (file-name (git-file-name name version)))) + ,(git->origin + repository `(tag-or-commit . ,commit) #:ref->commit #t)) (build-system minetest-mod-build-system) ,@(maybe-propagated-inputs (map contentdb->package-name inputs)) (home-page ,home-page) diff --git a/guix/import/utils.scm b/guix/import/utils.scm index 0cf52cdbde..47254539a1 100644 --- a/guix/import/utils.scm +++ b/guix/import/utils.scm @@ -13,6 +13,7 @@ ;;; Copyright © 2022 Alice Brenon ;;; Copyright © 2022 Kyle Meyer ;;; Copyright © 2022 Philip McGrath +;;; Copyright © 2023 Nicolas Graves ;;; ;;; This file is part of GNU Guix. ;;; @@ -39,6 +40,8 @@ (define-module (guix import utils) #:use-module (guix packages) #:use-module (guix discovery) #:use-module (guix build-system) + #:use-module (guix git) + #:use-module (guix hash) #:use-module ((guix i18n) #:select (G_)) #:use-module (guix store) #:use-module (guix download) @@ -63,6 +66,7 @@ (define-module (guix import utils) url-fetch guix-hash-url + git->origin package-names->package-inputs maybe-inputs @@ -161,6 +165,41 @@ (define (guix-hash-url filename) "Return the hash of FILENAME in nix-base32 format." (bytevector->nix-base32-string (file-sha256 filename))) +(define* (git->origin repo-url ref #:key (ref->commit #f)) + "Returns a generated `origin' block of a package depending on the git source +control system, and the directory in the store where the package has been +downloaded, in case further processing is necessary. REPO-URL or REF can be +null. REF->COMMIT can be a function or #t, in which case the commit matching +ref is used. If REF->COMMIT is not used, the value inside REF is used." + (let* ((version (and (pair? ref) (cdr ref))) + (directory commit + (if version + (with-store store + (latest-repository-commit store repo-url + #:ref (if version ref '()))) + (values #f #f))) + (vcommit (match ref->commit + (#t commit) + (#f version) + ((? procedure?) (ref->commit version)) + (_ #f)))) + (values + `(origin + (method git-fetch) + (uri (git-reference + (url ,(and (not (eq? repo-url 'null)) repo-url)) + (commit ,vcommit))) + (file-name (git-file-name name version)) + (sha256 + (base32 + ,(and version ; Version or commit is not always available. + (bytevector->nix-base32-string + (file-hash* directory + ;; 'git-fetch' already filtered out '.git'. + #:select? (const #true) + #:recursive? #true)))))) + directory))) + (define %spdx-license-identifiers ;; https://spdx.org/licenses/ ;; The gfl1.0, nmap, repoze diff --git a/tests/go.scm b/tests/go.scm index d2e8846b30..4644e1bd1c 100644 --- a/tests/go.scm +++ b/tests/go.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2021 François Joulaud ;;; Copyright © 2021 Sarah Morgensen +;;; Copyright © 2023 Nicolas Graves ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,7 +25,6 @@ (define-module (tests-import-go) #:use-module (guix base32) #:use-module (guix build-system go) #:use-module (guix import go) - #:use-module (guix base32) #:use-module ((guix utils) #:select (call-with-temporary-directory)) #:use-module (guix tests) #:use-module (ice-9 match) @@ -403,13 +403,26 @@ (define (mock-http-get testcase) (mock-http-get fixtures-go-check-test)) (mock ((guix http-client) http-fetch (mock-http-fetch fixtures-go-check-test)) - (mock ((guix git) update-cached-checkout - (lambda* (url #:key ref) - ;; Return an empty directory and its hash. - (values checkout - (nix-base32-string->bytevector - "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5") - #f))) + (mock ((guix import utils) git->origin + ;; Mock an empty directory by replacing hash. + (lambda* (repo-url ref #:key (ref->commit #f)) + (let* ((version (if (pair? ref) + (cdr ref) + #f)) + (vcommit (match ref->commit + (#t commit) + (#f version) + ((? procedure?) (ref->commit version)) + (_ #f)))) + `(origin + (method git-fetch) + (uri (git-reference + (url ,(and (not (eq? repo-url 'null)) repo-url)) + (commit ,vcommit))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))))) (go-module->guix-package* "github.com/go-check/check"))))))) (test-end "go") diff --git a/tests/minetest.scm b/tests/minetest.scm index 78469bf95b..7ff72dbfdc 100644 --- a/tests/minetest.scm +++ b/tests/minetest.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2021 Maxime Devos +;;; Copyright © 2023 Nicolas Graves ;;; ;;; This file is part of GNU Guix. ;;; @@ -57,15 +58,7 @@ (define* (make-package-sexp #:key `(package (name ,guix-name) (version ,version) - (source - (origin - (method git-fetch) - (uri (git-reference - (url ,(and (not (eq? repo 'null)) repo)) - (commit #f))) - (sha256 - (base32 #f)) - (file-name (git-file-name name version)))) + (source ,(git->origin repo #f)) (build-system minetest-mod-build-system) ,@(maybe-propagated-inputs inputs) (home-page ,home-page) @@ -419,8 +412,8 @@ (define* (example-package #:key (uri (git-reference (url repo) (commit commit #;"808f9ffbd3106da4c92d2367b118b98196c9e81e"))) - (sha256 #f) ; not important for the following tests - (file-name (git-file-name name version))) + (file-name (git-file-name name version)) + (sha256 #f)) ; not important for the following tests source)) (build-system minetest-mod-build-system) (license #f) From patchwork Thu Dec 21 14:01:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nicolas Graves X-Patchwork-Id: 57959 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0205227BBEA; Thu, 21 Dec 2023 14:03:29 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6E7F327BBE9 for ; Thu, 21 Dec 2023 14:03:26 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGJdc-0005iL-92; Thu, 21 Dec 2023 09:03:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGJda-0005hm-Tz for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:58 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGJda-0005v6-Kp for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:58 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGJde-00010a-GW for guix-patches@gnu.org; Thu, 21 Dec 2023 09:03:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#62202] [PATCH v3 2/4] import: Add juliahub importer. Resent-From: Nicolas Graves Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 14:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62202 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch moreinfo To: 62202@debbugs.gnu.org Cc: ngraves@ngraves.fr Received: via spool by 62202-submit@debbugs.gnu.org id=B62202.17031673243786 (code B ref 62202); Thu, 21 Dec 2023 14:03:02 +0000 Received: (at 62202) by debbugs.gnu.org; 21 Dec 2023 14:02:04 +0000 Received: from localhost ([127.0.0.1]:42698 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcg-0000yt-Sx for submit@debbugs.gnu.org; Thu, 21 Dec 2023 09:02:04 -0500 Received: from 9.mo550.mail-out.ovh.net ([178.32.108.172]:37231) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcZ-0000xq-Vl for 62202@debbugs.gnu.org; Thu, 21 Dec 2023 09:01:58 -0500 Received: from director7.ghost.mail-out.ovh.net (unknown [10.109.140.215]) by mo550.mail-out.ovh.net (Postfix) with ESMTP id 9C00427F13 for <62202@debbugs.gnu.org>; Thu, 21 Dec 2023 14:01:49 +0000 (UTC) Received: from ghost-submission-6684bf9d7b-7vdqn (unknown [10.111.174.161]) by director7.ghost.mail-out.ovh.net (Postfix) with ESMTPS id 024311FE7A; Thu, 21 Dec 2023 14:01:48 +0000 (UTC) Received: from ngraves.fr ([37.59.142.105]) by ghost-submission-6684bf9d7b-7vdqn with ESMTPSA id SgJAMExFhGWjLgAAFFRZrQ (envelope-from ); Thu, 21 Dec 2023 14:01:48 +0000 Authentication-Results: garm.ovh; auth=pass (GARM-105G006f7c80d17-d1d2-4227-8716-02146d4f6d9a, 9D5CD1A767DEA5A8C66A052B36C551B71BBC44F2) smtp.auth=ngraves@ngraves.fr X-OVh-ClientIp: 89.85.249.141 Date: Thu, 21 Dec 2023 15:01:01 +0100 Message-ID: <20231221140142.16523-2-ngraves@ngraves.fr> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231221140142.16523-1-ngraves@ngraves.fr> References: <20231221140142.16523-1-ngraves@ngraves.fr> MIME-Version: 1.0 X-Ovh-Tracer-Id: 4993647564968288994 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvkedrvdduhedgfeegucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpefhvfevufffkffojghfgggtgfesthekredtredtjeenucfhrhhomheppfhitgholhgrshcuifhrrghvvghsuceonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqeenucggtffrrghtthgvrhhnpeevkeetkefffffgheekgeekhfeugedvheejteelheejudeffeffgfelleeghffhieenucffohhmrghinhepghhithhhuhgsrdgtohhmpdhjuhhlihgrhhhusgdrtghomhdpghhnuhdrohhrghdpghhithhhuhgsuhhsvghrtghonhhtvghnthdrtghomhdpjhhulhhirghhuhgsrdhorhhgnecukfhppeduvdejrddtrddtrddupdekledrkeehrddvgeelrddugedupdefjedrheelrddugedvrddutdehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpeeonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqedpnhgspghrtghpthhtohepuddprhgtphhtthhopeeivddvtddvseguvggssghughhsrdhgnhhurdhorhhgpdfovfetjfhoshhtpehmohehhedtpdhmohguvgepshhmthhpohhuth X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Nicolas Graves X-ACL-Warn: , Nicolas Graves via Guix-patches X-Patchwork-Original-From: Nicolas Graves via Guix-patches via From: Nicolas Graves Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches --- doc/guix.texi | 27 +++ guix/import/juliahub.scm | 309 +++++++++++++++++++++++++++++++ guix/scripts/import.scm | 2 +- guix/scripts/import/juliahub.scm | 107 +++++++++++ 4 files changed, 444 insertions(+), 1 deletion(-) create mode 100644 guix/import/juliahub.scm create mode 100644 guix/scripts/import/juliahub.scm diff --git a/doc/guix.texi b/doc/guix.texi index b742a3d5b2..f50bb3f328 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14678,6 +14678,33 @@ guix import hexpm cf@@0.3.0 Additional options include: +@table @code +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. +@end table + +@item juliahub +@cindex juliahub +Import metadata from both the General +@uref{https://github.com/JuliaRegistries/General} and Juliahub +@uref{https://juliahub.com} Julia package repositories, as in this +example: + +@example +guix import juliahub Cthulhu@@2.8.9 +@end example + +The supplied package name must have the same case as in the +aforementioned package repositories, and the version used must be an +exact version (e.g. @code{2.8.9} instead of @code{2.8}). The command +will also fail in the case of a Julia package that doesn't use a git +tag. + +Additional options include: + @table @code @item --recursive @itemx -r diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm new file mode 100644 index 0000000000..ab838b6035 --- /dev/null +++ b/guix/import/juliahub.scm @@ -0,0 +1,309 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2023 Nicolas Graves +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import juliahub) + #:use-module (ice-9 textual-ports) + #:use-module (ice-9 regex) + #:use-module (ice-9 match) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) + #:use-module (srfi srfi-71) + #:use-module (guix http-client) + #:use-module (guix git) + #:use-module (guix import utils) + #:use-module (guix import json) + #:use-module (guix base32) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (json) + #:use-module ((guix licenses) #:prefix license:) + + #:export (juliahub->guix-package + %juliahub-updater + juliahub-recursive-import)) + + +;; JuliaHub API. +(define (juliahub-redirect-uri name) + (let* ((url (string-append "https://docs.juliahub.com/" name "/")) + (port (http-fetch url #:text? #t)) + (_ (get-line port)) + (meta (get-line port)) + (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*") + (redirect (match:substring (string-match regex meta)))) + (close-port port) + (string-drop redirect 4))) + +(define (juliahub-url name) + (let* ((url (string-append "https://docs.juliahub.com/" name "/")) + (uri (juliahub-redirect-uri name))) + (string-append url uri "/"))) + +;; General package repository. +(define %general-base-url + "https://raw.githubusercontent.com/JuliaRegistries/General/master/") + +(define (general-url package-name file) + (let ((folder (string-capitalize (string-take package-name 1)))) + (string-append + %general-base-url folder "/" package-name "/" file))) + +(define (ini-line->alist line) + (let* ((l (string-split line #\=)) + (attribute (string->symbol (string-drop-right (car l) 1))) + (value (string-drop (string-drop-right (cadr l) 1) 2))) + `(,attribute . ,value))) + +(define (ini-fetch url) + (let* ((port (http-fetch url #:text? #t)) + (raw (get-string-all port)) + (lines (drop-right (string-split raw #\newline) 1))) + (close-port port) + (map ini-line->alist lines))) + +;; Filtering out julia-stdlibs. +;; To update them, see file sysimg.jl. +(define %julia-stdlibs + (list "julia" + "ArgTools" + "Artifacts" + "Base64" + "CRC32c" + "FileWatching" + "Libdl" + "Logging" + "Mmap" + "NetworkOptions" + "SHA" + "Serialization" + "Sockets" + "Unicode" + "DelimitedFiles" + "LinearAlgebra" + "Markdown" + "Printf" + "Random" + "Tar" + "Dates" + "Distributed" + "Future" + "InteractiveUtils" + "LibGit2" + "Profile" + "SparseArrays" + "UUIDs" + "REPL" + "SharedArrays" + "Statistics" + "SuiteSparse" + "TOML" + "Test" + "LibCURL" + "Downloads" + "Pkg" + "LazyArtifacts")) + +;; Julia package. +(define-json-mapping make-juliahub-package juliahub-package? + json->juliahub-package + (homepage juliahub-package-homepage) ;string + (readme juliahub-package-readme) ;string + (version juliahub-package-version) ;string + (description juliahub-package-description) ;string + (dependencies + juliahub-package-dependencies "deps" + json->juliahub-dependencies) ;list of + (url juliahub-package-url) ;string + (uuid juliahub-package-uuid) ;string + (license juliahub-package-license)) ;string + +(define-json-mapping + make-juliahub-dependency juliahub-dependency? + json->juliahub-dependency + (direct? juliahub-dependency-direct? "direct") ;boolean + (name juliahub-dependency-name) ;string + (uuid juliahub-dependency-uuid) ;string + (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings + +(define (julia-name->guix-name name) + (string-append "julia-" (snake-case name))) + +(define* (juliahub-fetch name #:key (version #f)) + "Return a record for package NAME, or #f on failure." + (let* ((uri (juliahub-redirect-uri name)) + (slug (string-take uri 5)) + (url (if version + (string-append "https://docs.juliahub.com/" name "/" + slug "/" version "/pkg.json") + (string-append (juliahub-url name) "pkg.json")))) + (and=> (json-fetch url) json->juliahub-package))) + +(define (make-julia-sexp name version source home-page synopsis description + direct-dependencies test-dependencies-names license) + "Return the `package' s-expression for a Julia package with the given NAME, +VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES, +TEST-DEPENDENCIES-NAMES and LICENSE." + `(package + (name ,(julia-name->guix-name name)) + (version ,version) + (source ,source) + (build-system julia-build-system) + ,@(if (null? direct-dependencies) + '() + `((propagated-inputs + (list ,@(map (compose string->symbol + julia-name->guix-name + juliahub-dependency-name) + direct-dependencies))))) + ,@(if (null? test-dependencies-names) + '() + `((native-inputs + (list ,@(map (compose string->symbol julia-name->guix-name) + test-dependencies-names))))) + (synopsis ,synopsis) + (description ,description) + (home-page ,home-page) + (license ,(if license (spdx-string->license license) #f)))) + +;; Dependencies helpers. +(define (json->juliahub-dependencies vector) + (if (vector? vector) + (filter-map + (lambda (el) + (let ((dep (json->juliahub-dependency el))) + (if (and (juliahub-dependency-direct? dep) + (not (member (juliahub-dependency-name dep) + %julia-stdlibs))) + dep + #f))) + (vector->list vector)))) + +(define (parse-test-dependencies directory) + (let* ((port (open-input-file (string-append directory "/Project.toml"))) + (project.toml (get-string-all port)) + (regex "\ntest = \\[.*\\]") + (deps (match:substring (string-match regex project.toml))) + (pure (string-delete (list->char-set (list #\" #\ )) deps))) + (close-port port) + (filter (lambda (x) (not (member x %julia-stdlibs))) + (string-split (string-drop (string-drop-right pure 1) 7) #\,)))) + +;; Juliahub may be more up-to-date than the General registry or the actual git +;; tag (it seems around 6 hours pass between the time a commit is supplied to +;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes +;; the git tag). We have no simple way to get the commit of the latest-version. +;; Thus the best simple thing we can do is get the latest-git-tag, and import +;; this version instead. We do this by parsing Package.toml in the General +;; registry, and then getting the refs of the git repo supplied by this +;; file. Parsing this file is also necessary if the package is in a subdir of a +;; git repository, because the information isn't present in Juliahub. + +;; There's a last case where some Julia packages are not based on a particular +;; git tag. In this case, the script fails, but it seems quite rare. We could +;; introduce the tree-commit which is available in the Versions.toml file in the +;; General repository. This can be used to identify the state of a repository, +;; since we have a unique hash of the listing of files and directories. + +(define (latest-git-tag repo) + (let* ((last-ref (last (remote-refs repo #:tags? #t))) + (last-git-tag (last (string-split last-ref #\/)))) + (string-drop last-git-tag 1))) + +(define* (juliahub->guix-package package-name + #:key version #:allow-other-keys) + "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the +`package' s-expression corresponding to that package, or #f on failure. +Optionally include a VERSION string to fetch a specific version juliahub." + (let* ((package-toml (ini-fetch (general-url package-name "Package.toml"))) + (subdir (assoc-ref package-toml 'subdir)) + (tag (latest-git-tag (assoc-ref package-toml 'repo))) + (package (if version + (juliahub-fetch package-name #:version version) + (if tag + (juliahub-fetch package-name #:version tag) + (juliahub-fetch package-name))))) + (if package + (let* ((source directory + (git->origin + (juliahub-package-url package) + `(tag-or-commit + . ,(string-append + "v" (juliahub-package-version package))))) + (direct-dependencies + (filter juliahub-dependency-direct? + (juliahub-package-dependencies package))) + (dependencies-names (map juliahub-dependency-name + direct-dependencies)) + (test-dependencies-names + (if subdir + (parse-test-dependencies + (string-append subdir "/" directory)) + (parse-test-dependencies directory))) + (homepage (juliahub-package-homepage package))) + (values (make-julia-sexp + package-name + (juliahub-package-version package) + source + (match homepage + ("" (juliahub-package-url package)) + ((? string?) homepage) + (_ (juliahub-package-url package))) + (juliahub-package-description package) + (beautify-description + (juliahub-package-readme package)) + direct-dependencies + test-dependencies-names + (juliahub-package-license package)) + (append dependencies-names test-dependencies-names))) + (values #f '())))) + +;; We must use the url to get a name with the true case of juliahub/general. +(define (guix-package->juliahub-name package) + (let* ((url (juliahub-package-url package)) + (git-name (last (string-split url #\/))) + (ungitted-name (if (string-suffix? ".git" git-name) + (string-drop-right git-name 4) + git-name)) + (package-name (if (string-suffix? ".jl" ungitted-name) + (string-drop-right ungitted-name 4) + ungitted-name))) + package-name)) + +(define* (import-release package #:key (version #f)) + "Return an for the latest release of PACKAGE." + (let* ((package-name (guix-package->juliahub-name package)) + (package (juliahub-fetch package-name)) + (version (or version (juliahub-package-version package)))) + (upstream-source + (package (package-name package)) + (version version) + (urls (list (juliahub-package-url package)))))) + +(define %juliahub-updater + (upstream-updater + (name 'juliahub) + (description "Updater for Juliahub packages") + (pred juliahub-package?) + (import import-release))) + +(define* (juliahub-recursive-import package-name #:optional version) + (recursive-import package-name + #:repo '() + #:repo->guix-package juliahub->guix-package + #:guix-name julia-name->guix-name + #:version version)) diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index d2a1cee56e..8926c9610f 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -47,7 +47,7 @@ (define %standard-import-options '()) (define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa" "gem" "go" "cran" "crate" "texlive" "json" "opam" - "minetest" "elm" "hexpm" "composer")) + "minetest" "elm" "hexpm" "composer" "juliahub")) (define (resolve-importer name) (let ((module (resolve-interface diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm new file mode 100644 index 0000000000..1317c67aa3 --- /dev/null +++ b/guix/scripts/import/juliahub.scm @@ -0,0 +1,107 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2023 Nicolas Graves +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix scripts import juliahub) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (guix scripts) + #:use-module (guix import juliahub) + #:use-module (guix scripts import) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-37) + #:use-module (ice-9 match) + #:use-module (ice-9 format) + #:use-module (ice-9 receive) + #:export (guix-import-juliahub)) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '()) + +(define (show-help) + (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and +convert the Julia package for PACKAGE-NAME. Optionally, a version can be +specified after the at-sign (@) character.\n")) + (display (G_ " + -h, --help display this help and exit")) + (display (G_ " + -V, --version display version information and exit")) + (display (G_ " + -r, --recursive generate package expressions for all Gem packages\ + that are not yet in Guix")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (cons* (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix import gem"))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive #t result))) + %standard-import-options)) + + +;;; +;;; Entry point. +;;; + +(define (guix-import-juliahub . args) + (define (parse-options) + ;; Return the alist of option values. + (parse-command-line args %options (list %default-options) + #:build-options? #f)) + + (let* ((opts (parse-options)) + (args (filter-map (match-lambda + (('argument . value) + value) + (_ #f)) + (reverse opts)))) + (match args + ((spec) + (receive (package-name package-version) + (package-name->name+version spec) + (let ((code (if (assoc-ref opts 'recursive) + (map (match-lambda + ((and ('package ('name name) . rest) pkg) + `(define-public ,(string->symbol name) + ,pkg)) + (_ #f)) + (juliahub-recursive-import package-name package-version)) + (let ((sexp (juliahub->guix-package package-name #:version package-version))) + (if sexp sexp #f))))) + (match code + ((or #f '(#f)) + (leave (G_ "failed to download meta-data for package '~a'~%") + package-name)) + (_ code))))) + (() + (leave (G_ "too few arguments~%"))) + ((many ...) + (leave (G_ "too many arguments~%")))))) From patchwork Thu Dec 21 14:01:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Graves X-Patchwork-Id: 57958 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2866A27BBEA; Thu, 21 Dec 2023 14:03:25 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 3DA5127BBE2 for ; Thu, 21 Dec 2023 14:03:24 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGJdc-0005iK-0O; Thu, 21 Dec 2023 09:03:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGJda-0005hk-Pd for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:58 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGJda-0005uv-5r for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:58 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGJde-00010S-1u for guix-patches@gnu.org; Thu, 21 Dec 2023 09:03:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#62202] [PATCH v3 3/4] import: juliahub: Beautify description. Resent-From: Nicolas Graves Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 14:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62202 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch moreinfo To: 62202@debbugs.gnu.org Cc: ngraves@ngraves.fr Received: via spool by 62202-submit@debbugs.gnu.org id=B62202.17031673233778 (code B ref 62202); Thu, 21 Dec 2023 14:03:02 +0000 Received: (at 62202) by debbugs.gnu.org; 21 Dec 2023 14:02:03 +0000 Received: from localhost ([127.0.0.1]:42695 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcg-0000yd-7m for submit@debbugs.gnu.org; Thu, 21 Dec 2023 09:02:02 -0500 Received: from 5.mo576.mail-out.ovh.net ([46.105.43.105]:48553) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJca-0000xs-Bg for 62202@debbugs.gnu.org; Thu, 21 Dec 2023 09:01:57 -0500 Received: from director5.ghost.mail-out.ovh.net (unknown [10.109.139.217]) by mo576.mail-out.ovh.net (Postfix) with ESMTP id 2F74E28DF1 for <62202@debbugs.gnu.org>; Thu, 21 Dec 2023 14:01:51 +0000 (UTC) Received: from ghost-submission-6684bf9d7b-qnh7v (unknown [10.110.168.40]) by director5.ghost.mail-out.ovh.net (Postfix) with ESMTPS id 894721FEB9; Thu, 21 Dec 2023 14:01:50 +0000 (UTC) Received: from ngraves.fr ([37.59.142.107]) by ghost-submission-6684bf9d7b-qnh7v with ESMTPSA id TfEXGU5FhGUvHwAAfKi07w (envelope-from ); Thu, 21 Dec 2023 14:01:50 +0000 Authentication-Results: garm.ovh; auth=pass (GARM-107S001b38d7a09-3c56-45ee-98c2-64c9a3714c9d, 9D5CD1A767DEA5A8C66A052B36C551B71BBC44F2) smtp.auth=ngraves@ngraves.fr X-OVh-ClientIp: 89.85.249.141 Date: Thu, 21 Dec 2023 15:01:02 +0100 Message-ID: <20231221140142.16523-3-ngraves@ngraves.fr> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231221140142.16523-1-ngraves@ngraves.fr> References: <20231221140142.16523-1-ngraves@ngraves.fr> MIME-Version: 1.0 X-Ovh-Tracer-Id: 4994210511922848482 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvkedrvdduhedgfeegucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefpihgtohhlrghsucfirhgrvhgvshcuoehnghhrrghvvghssehnghhrrghvvghsrdhfrheqnecuggftrfgrthhtvghrnhepleffjeetueethfefkeffffefvddukeejkefgleduiedthfekvefhiedvhfffgeegnecukfhppeduvdejrddtrddtrddupdekledrkeehrddvgeelrddugedupdefjedrheelrddugedvrddutdejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpeeonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqedpnhgspghrtghpthhtohepuddprhgtphhtthhopeeivddvtddvseguvggssghughhsrdhgnhhurdhorhhgpdfovfetjfhoshhtpehmohehjeeipdhmohguvgepshhmthhpohhuth X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Nicolas Graves X-ACL-Warn: , Nicolas Graves via Guix-patches X-Patchwork-Original-From: Nicolas Graves via Guix-patches via From: Nicolas Graves Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches --- guix/import/juliahub.scm | 53 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 51 insertions(+), 2 deletions(-) diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm index ab838b6035..e4540de06d 100644 --- a/guix/import/juliahub.scm +++ b/guix/import/juliahub.scm @@ -30,8 +30,10 @@ (define-module (guix import juliahub) #:use-module (guix base32) #:use-module (guix packages) #:use-module (guix upstream) - #:use-module (json) #:use-module ((guix licenses) #:prefix license:) + #:use-module (json) + #:use-module (htmlprag) + #:use-module (sxml transform) #:export (juliahub->guix-package %juliahub-updater @@ -76,6 +78,53 @@ (define (ini-fetch url) (close-port port) (map ini-line->alist lines))) +;; Beautify description. +(define %juliahub-beautify-description-rules + `((h1 *preorder* . ,(lambda args #f)) + (h2 *preorder* . ,(lambda args #f)) + (h3 *preorder* . ,(lambda args #f)) + (h4 *preorder* . ,(lambda args #f)) + (h5 *preorder* . ,(lambda args #f)) + (h6 *preorder* . ,(lambda args #f)) + (hr *preorder* . ,(lambda args #f)) + (span *preorder* . ,(lambda args #f)) + (img *preorder* . ,(lambda args #f)) + (pre *preorder* . ,(lambda args #f)) + (div *preorder* . ,(lambda args #f)) + (table *preorder* . ,(lambda args #f)) + (imgalt *preorder* . ,(lambda args #f)) + (@ *preorder* . ,(lambda args #f)) + (*TOP* . ,(lambda args (cdr args))) + (p . ,(lambda args (cdr args))) + (em . ,(lambda args (cdr args))) + (strong . ,(lambda args (cdr args))) + (a . ,(lambda args + (match args + ((tag link ref) + (if ref ref #f)) + (_ #f)))) + (ul . ,(lambda args + `("@itemize" ,@(cdr args) "\n@end itemize"))) + (ol . ,(lambda args + `("@enumerate" ,@(cdr args) "@end enumerate"))) + (blockquote . ,(lambda args + `("@quotation" ,@(cdr args) "@end quotation"))) + (li . ,(lambda args + `("\n@item" ,@(cdr args)))) + (code . ,(lambda args + `("@code{" ,@(cdr args) "}"))) + (*text* . ,(lambda (tag x) x)) + (*default* . ,(lambda (tag . body) + (cons tag body))))) + +(define (juliahub-beautify-description description) + (string-join + (filter (lambda (x) (if (equal? x " ") #f x)) + (flatten + (pre-post-order (html->sxml description) + %juliahub-beautify-description-rules))) + " ")) + ;; Filtering out julia-stdlibs. ;; To update them, see file sysimg.jl. (define %julia-stdlibs @@ -264,7 +313,7 @@ (define* (juliahub->guix-package package-name ((? string?) homepage) (_ (juliahub-package-url package))) (juliahub-package-description package) - (beautify-description + ((compose beautify-description juliahub-beautify-description) (juliahub-package-readme package)) direct-dependencies test-dependencies-names From patchwork Thu Dec 21 14:01:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Graves X-Patchwork-Id: 57957 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 7754827BBEA; Thu, 21 Dec 2023 14:03:13 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id AD99527BBE9 for ; Thu, 21 Dec 2023 14:03:12 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGJde-0005kh-P2; Thu, 21 Dec 2023 09:03:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGJdb-0005iC-Et for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:59 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGJdb-0005vR-2S for guix-patches@gnu.org; Thu, 21 Dec 2023 09:02:59 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGJde-00010i-Ui for guix-patches@gnu.org; Thu, 21 Dec 2023 09:03:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#62202] [PATCH v3 4/4] import: utils: Rule out texinfo common syntax from @ escape. Resent-From: Nicolas Graves Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 14:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62202 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch moreinfo To: 62202@debbugs.gnu.org Cc: ngraves@ngraves.fr Received: via spool by 62202-submit@debbugs.gnu.org id=B62202.17031673283796 (code B ref 62202); Thu, 21 Dec 2023 14:03:02 +0000 Received: (at 62202) by debbugs.gnu.org; 21 Dec 2023 14:02:08 +0000 Received: from localhost ([127.0.0.1]:42700 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcm-0000zA-AD for submit@debbugs.gnu.org; Thu, 21 Dec 2023 09:02:08 -0500 Received: from 2.mo560.mail-out.ovh.net ([188.165.53.149]:38653) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGJcc-0000xu-7E for 62202@debbugs.gnu.org; Thu, 21 Dec 2023 09:02:01 -0500 Received: from director10.ghost.mail-out.ovh.net (unknown [10.109.140.215]) by mo560.mail-out.ovh.net (Postfix) with ESMTP id 6F18B2C92B for <62202@debbugs.gnu.org>; Thu, 21 Dec 2023 14:01:52 +0000 (UTC) Received: from ghost-submission-6684bf9d7b-kwgz9 (unknown [10.110.96.132]) by director10.ghost.mail-out.ovh.net (Postfix) with ESMTPS id 114A41FEA3; Thu, 21 Dec 2023 14:01:51 +0000 (UTC) Received: from ngraves.fr ([37.59.142.102]) by ghost-submission-6684bf9d7b-kwgz9 with ESMTPSA id 0wIFAFBFhGWnJgAA8T+B1w (envelope-from ); Thu, 21 Dec 2023 14:01:51 +0000 Authentication-Results: garm.ovh; auth=pass (GARM-102R004bb7d06d5-3e2e-454c-a5f4-c17aef2200e3, 9D5CD1A767DEA5A8C66A052B36C551B71BBC44F2) smtp.auth=ngraves@ngraves.fr X-OVh-ClientIp: 89.85.249.141 Date: Thu, 21 Dec 2023 15:01:03 +0100 Message-ID: <20231221140142.16523-4-ngraves@ngraves.fr> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231221140142.16523-1-ngraves@ngraves.fr> References: <20231221140142.16523-1-ngraves@ngraves.fr> MIME-Version: 1.0 X-Ovh-Tracer-Id: 4994491990046139106 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedvkedrvdduhedgfeegucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefpihgtohhlrghsucfirhgrvhgvshcuoehnghhrrghvvghssehnghhrrghvvghsrdhfrheqnecuggftrfgrthhtvghrnhepleffjeetueethfefkeffffefvddukeejkefgleduiedthfekvefhiedvhfffgeegnecukfhppeduvdejrddtrddtrddupdekledrkeehrddvgeelrddugedupdefjedrheelrddugedvrddutddvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpeeonhhgrhgrvhgvshesnhhgrhgrvhgvshdrfhhrqedpnhgspghrtghpthhtohepuddprhgtphhtthhopeeivddvtddvseguvggssghughhsrdhgnhhurdhorhhgpdfovfetjfhoshhtpehmohehiedtpdhmohguvgepshhmthhpohhuth X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Nicolas Graves X-ACL-Warn: , Nicolas Graves via Guix-patches X-Patchwork-Original-From: Nicolas Graves via Guix-patches via From: Nicolas Graves Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches --- guix/import/utils.scm | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/guix/import/utils.scm b/guix/import/utils.scm index 47254539a1..57e4ec0ce7 100644 --- a/guix/import/utils.scm +++ b/guix/import/utils.scm @@ -375,7 +375,13 @@ (define* (beautify-description description #:optional (length 80)) (cut string-trim-both <> #\') ;; Escape single @ to prevent it from being understood as ;; invalid Texinfo syntax. - (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post) + (lambda (word) + (if ; Rule out some valid Texinfo syntax. + (member word '("@itemize" "@item" "@end" "@quotation" + "@enumerate" "@code" "@code{")) + word + ((cut regexp-substitute/global + #f "@" <> 'pre "@@" 'post) word))) ;; Wrap camelCase or PascalCase words in @code{...}. (lambda (word) (let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))