From patchwork Thu Dec 21 22:01:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Elsing X-Patchwork-Id: 57968 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 30BBD27BBE9; Thu, 21 Dec 2023 22:03:18 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 54A6F27BBE2 for ; Thu, 21 Dec 2023 22:03:16 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGR88-0004gb-8F; Thu, 21 Dec 2023 17:03:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGR86-0004fR-6A for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:58 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGR85-0005Iw-Tj for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:57 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGR8A-0004i0-0s for guix-patches@gnu.org; Thu, 21 Dec 2023 17:03:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#67960] [PATCH 1/4] gnu: import: Fix memoization in crate-recursive-import. Resent-From: David Elsing Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 22:03:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 67960@debbugs.gnu.org Cc: David Elsing Received: via spool by 67960-submit@debbugs.gnu.org id=B67960.170319616218039 (code B ref 67960); Thu, 21 Dec 2023 22:03:01 +0000 Received: (at 67960) by debbugs.gnu.org; 21 Dec 2023 22:02:42 +0000 Received: from localhost ([127.0.0.1]:45614 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7q-0004gt-1P for submit@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:42 -0500 Received: from mout02.posteo.de ([185.67.36.66]:43197) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7n-0004gW-G2 for 67960@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:40 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 40CB2240105 for <67960@debbugs.gnu.org>; Thu, 21 Dec 2023 23:02:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703196148; bh=UV4KlIQQh1jOE4HaoulPcZQ3wHUJ5zlK7m8zckqJTC8=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=aqlM0qCHLEogJ37ektND48kxW2J0TKYaLF390pcQrWBJdiTaYUSvQGK4nvJGMb+0L qw3C75tm0M1Ko4g5+C1MtJ3oQ/thREsKf3ijTO35Ghi9saUHeTaHw6iiSuOSnh/gpN zKtFS1rpmePVNZwv0kAim9qX7eorfYWnKHewhcoVJyQUMpGltkFZ5kfyIz2WBhMQow t5ZLDPhnIDXeaiKHEMzzS1eII+mqw+EfaOTKNKdA57O9pP+D2pWyMx0/PLX5tMLIJJ RAFfpgzUFpMB/YON1Ifv3Qlx6BYDFYtG3csCaXfMkG1125pKJur85rhRwf6NUNFrqS aYYEuoYB6V6cQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Sx4D35mMtz6tw4; Thu, 21 Dec 2023 23:02:27 +0100 (CET) From: David Elsing Date: Thu, 21 Dec 2023 22:01:49 +0000 Message-ID: <7b04c6fca16b404565f8d4d019a76252eb66c50c.1703195451.git.david.elsing@posteo.net> In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * guix/import/crate.scm (crate-recursive-import): Apply memoize outside the lambda passed to recursive-import in order to actually use the memoization. --- guix/import/crate.scm | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/guix/import/crate.scm b/guix/import/crate.scm index 43823d006e..07874bdb26 100644 --- a/guix/import/crate.scm +++ b/guix/import/crate.scm @@ -6,6 +6,7 @@ ;;; Copyright © 2022 Hartmut Goebel ;;; Copyright © 2023 Simon Tournier ;;; Copyright © 2023 Efraim Flashner +;;; Copyright © 2023 David Elsing ;;; ;;; This file is part of GNU Guix. ;;; @@ -328,15 +329,17 @@ (define (sort-map-dependencies deps) (values #f '()))) (define* (crate-recursive-import crate-name #:key version) - (recursive-import crate-name - #:repo->guix-package (lambda* params - ;; download development dependencies only for the top level package - (let ((include-dev-deps? (equal? (car params) crate-name)) - (crate->guix-package* (memoize crate->guix-package))) - (apply crate->guix-package* - (append params `(#:include-dev-deps? ,include-dev-deps?))))) - #:version version - #:guix-name crate-name->package-name)) + (recursive-import + crate-name + #:repo->guix-package + (let ((crate->guix-package* (memoize crate->guix-package))) + (lambda* params + ;; download development dependencies only for the top level package + (let ((include-dev-deps? (equal? (car params) crate-name))) + (apply crate->guix-package* + (append params `(#:include-dev-deps? ,include-dev-deps?)))))) + #:version version + #:guix-name crate-name->package-name)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE." From patchwork Thu Dec 21 22:01:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Elsing X-Patchwork-Id: 57970 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 966A327BBE2; Thu, 21 Dec 2023 22:03:36 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 3E01C27BBEA for ; Thu, 21 Dec 2023 22:03:32 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGR88-0004gc-9x; Thu, 21 Dec 2023 17:03:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGR87-0004fp-44 for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:59 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGR86-0005JB-Rx for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:58 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGR8A-0004iF-VC for guix-patches@gnu.org; Thu, 21 Dec 2023 17:03:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#67960] [PATCH 2/4] import: crate: Optionally import dev-dependencies recursively. Resent-From: David Elsing Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 22:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 67960@debbugs.gnu.org Cc: David Elsing Received: via spool by 67960-submit@debbugs.gnu.org id=B67960.170319617118076 (code B ref 67960); Thu, 21 Dec 2023 22:03:02 +0000 Received: (at 67960) by debbugs.gnu.org; 21 Dec 2023 22:02:51 +0000 Received: from localhost ([127.0.0.1]:45620 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7x-0004hT-Oo for submit@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:50 -0500 Received: from mout02.posteo.de ([185.67.36.66]:59121) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7p-0004gc-8w for 67960@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:44 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 7761B240103 for <67960@debbugs.gnu.org>; Thu, 21 Dec 2023 23:02:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703196151; bh=ypS3YGMTYlH2BzJ9VbUqH7laHNlWak9KlOqJOPwdIWs=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=CeoODpYeDxc0jgbYBP5Kz5L0DcXcQtaIPC2CTeMz0JuYYuWXLjR57ukZZJAxW1jwt K83p5dYTA9N2Z0v4JaFN1FipKx30MwRc9Ym09RyIoUy1kfOU83f3AFmO7PYYAkcLmA q2dsYTjSpgXNqVDPcK4LWQAmI6//lMIv/m4guKR4whoGV6lVV7vat1YijMxTAI5Izr Eini62hHYKEmkAwsN0uh6WM8r5AqJ5tz6XqRIdqAPvFGrtrmvoMbX9GFXLhsuh62Lc IkQlfrCoTI56H9uglFw4bpVRjtl6s8LoHGU3HMru2ysaruj038m2GzL9szQk7QTT0n bbeDabPuHBC6w== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Sx4D70WZvz6tvh; Thu, 21 Dec 2023 23:02:31 +0100 (CET) From: David Elsing Date: Thu, 21 Dec 2023 22:01:50 +0000 Message-ID: <068ee00edf887f21bea0267064f729ecaff05190.1703195451.git.david.elsing@posteo.net> In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches If --recursive-dev-dependencies is specified, development dependencies are also included for all recursivly imported packages. * doc/guix.texi (Invoking guix import): Mention --recursive-dev-dependencies. * guix/import/crate.scm (crate-recursive-import): Add recursive-dev-dependencies? argument. * guix/scripts/import/crate.scm (show-help, guix-import-crate): Add "--recursive-dev-dependencies". * tests/crate.scm: Test both #f and #t for #:recursive-dev-dependencies? in the 'cargo-recursive-import' test. (test-root-dependencies): Add intermediate-c as dev-dependency. (test-intermediate-c-crate,test-intermediate-c-dependencies): New variables. --- doc/guix.texi | 4 + guix/import/crate.scm | 7 +- guix/scripts/import/crate.scm | 12 +- tests/crate.scm | 228 +++++++++++++++++++++++++++++++++- 4 files changed, 244 insertions(+), 7 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index b742a3d5b2..a19671643b 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14512,6 +14512,10 @@ Additional options include: Traverse the dependency graph of the given upstream package recursively and generate package expressions for all those packages that are not yet in Guix. +@item --recursive-dev-dependencies +If @option{--recursive} is specified, also the recursively imported +packages contain their development dependencies, which are recursively +imported as well. @end table @item elm diff --git a/guix/import/crate.scm b/guix/import/crate.scm index 07874bdb26..db5461312f 100644 --- a/guix/import/crate.scm +++ b/guix/import/crate.scm @@ -328,14 +328,17 @@ (define (sort-map-dependencies deps) (append cargo-inputs cargo-development-inputs))) (values #f '()))) -(define* (crate-recursive-import crate-name #:key version) +(define* (crate-recursive-import + crate-name #:key version recursive-dev-dependencies?) (recursive-import crate-name #:repo->guix-package (let ((crate->guix-package* (memoize crate->guix-package))) (lambda* params ;; download development dependencies only for the top level package - (let ((include-dev-deps? (equal? (car params) crate-name))) + (let ((include-dev-deps? + (or (equal? (car params) crate-name) + recursive-dev-dependencies?))) (apply crate->guix-package* (append params `(#:include-dev-deps? ,include-dev-deps?)))))) #:version version diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scm index 038faa87db..b13b6636a6 100644 --- a/guix/scripts/import/crate.scm +++ b/guix/scripts/import/crate.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2019, 2020 Martin Becze ;;; Copyright © 2021 Sarah Morgensen ;;; Copyright © 2023 Simon Tournier +;;; Copyright © 2023 David Elsing ;;; ;;; This file is part of GNU Guix. ;;; @@ -47,6 +48,9 @@ (define (show-help) Import and convert the crates.io package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) + (display (G_ " + --recursive-dev-dependencies + include dev-dependencies recursively")) (newline) (display (G_ " -h, --help display this help and exit")) @@ -67,6 +71,9 @@ (define %options (option '(#\r "recursive") #f #f (lambda (opt name arg result) (alist-cons 'recursive #t result))) + (option '("recursive-dev-dependencies") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive-dev-dependencies #t result))) %standard-import-options)) @@ -92,7 +99,10 @@ (define-values (name version) (package-name->name+version spec)) (match (if (assoc-ref opts 'recursive) - (crate-recursive-import name #:version version) + (crate-recursive-import + name #:version version + #:recursive-dev-dependencies? + (assoc-ref opts 'recursive-dev-dependencies)) (crate->guix-package name #:version version #:include-dev-deps? #t)) ((or #f '()) (leave (G_ "failed to download meta-data for package '~a'~%") diff --git a/tests/crate.scm b/tests/crate.scm index 5aea5efaf3..1b9ad88358 100644 --- a/tests/crate.scm +++ b/tests/crate.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2019, 2020, 2022 Ludovic Courtès ;;; Copyright © 2020 Martin Becze ;;; Copyright © 2023 Efraim Flashner +;;; Copyright © 2023 David Elsing ;;; ;;; This file is part of GNU Guix. ;;; @@ -40,10 +41,11 @@ (define-module (test-crate) ;; ;; root-1.0.0 ;; root-1.0.4 -;; intermediate-a 1.0.42 -;; intermeidate-b ^1.0.0 +;; intermediate-a 1.0.42 +;; intermediate-b ^1.0.0 ;; leaf-alice ^0.7 -;; leaf-bob ^3 +;; leaf-bob ^3 +;; intermediate-c 1 (dev-dependency) ;; ;; intermediate-a-1.0.40 ;; intermediate-a-1.0.42 @@ -55,6 +57,9 @@ (define-module (test-crate) ;; intermediate-b-1.2.3 ;; leaf-bob 3.0.1 ;; +;; intermediate-c-1.0.1 +;; leaf-alice 0.7.5 (dev-dependency) +;; ;; leaf-alice-0.7.3 ;; leaf-alice-0.7.5 ;; @@ -164,6 +169,11 @@ (define test-root-dependencies \"crate_id\": \"leaf-bob\", \"kind\": \"normal\", \"req\": \"^3\" + }, + { + \"crate_id\": \"intermediate-c\", + \"kind\": \"dev\", + \"req\": \"1\" } ] }") @@ -262,6 +272,40 @@ (define test-intermediate-b-dependencies ] }") +(define test-intermediate-c-crate + "{ + \"crate\": { + \"max_version\": \"1.0.1\", + \"name\": \"intermediate-c\", + \"description\": \"summary\", + \"homepage\": \"http://example.com\", + \"repository\": \"http://example.com\", + \"keywords\": [\"dummy\", \"test\"], + \"categories\": [\"test\"], + \"actual_versions\": [ + { \"id\": 234290, + \"num\": \"1.0.1\", + \"license\": \"MIT OR Apache-2.0\", + \"links\": { + \"dependencies\": \"/api/v1/crates/intermediate-c/1.0.1/dependencies\" + }, + \"yanked\": false + } + ] + } +}") + +(define test-intermediate-c-dependencies + "{ + \"dependencies\": [ + { + \"crate_id\": \"leaf-alice\", + \"kind\": \"dev\", + \"req\": \"0.7.5\" + } + ] +}") + (define test-leaf-alice-crate "{ \"crate\": { @@ -430,6 +474,15 @@ (define have-guile-semver? (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/intermediate-b/1.2.3/dependencies" (open-input-string test-intermediate-b-dependencies)) + ("https://crates.io/api/v1/crates/intermediate-c" + (open-input-string test-intermediate-c-crate)) + ("https://crates.io/api/v1/crates/intermediate-c/1.0.1/download" + (set! test-source-hash + (bytevector->nix-base32-string + (sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/intermediate-c/1.0.1/dependencies" + (open-input-string test-intermediate-c-dependencies)) ("https://crates.io/api/v1/crates/leaf-alice" (open-input-string test-leaf-alice-crate)) ("https://crates.io/api/v1/crates/leaf-alice/0.7.5/download" @@ -452,7 +505,27 @@ (define have-guile-semver? (match (crate-recursive-import "root") ;; rust-intermediate-b has no dependency on the rust-leaf-alice ;; package, so this is a valid ordering - (((define-public 'rust-leaf-alice-0.7 + (((define-public 'rust-intermediate-c-1 + (package + (name "rust-intermediate-c") + (version "1.0.1") + (source + (origin + (method url-fetch) + (uri (crate-uri "intermediate-c" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:skip-build? #t))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-leaf-alice-0.7 (package (name "rust-leaf-alice") (version "0.7.5") @@ -562,11 +635,158 @@ (define-public 'rust-root-1 ('unquote rust-intermediate-b-1)) ("rust-leaf-alice" ('unquote 'rust-leaf-alice-0.7)) + ("rust-leaf-bob" + ('unquote rust-leaf-bob-3))) + #:cargo-development-inputs + (("rust-intermediate-c" + ('unquote rust-intermediate-c-1)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0))))) + #t) + (x + (pk 'fail x #f))) + (match (crate-recursive-import "root" + #:recursive-dev-dependencies? #t) + ;; rust-intermediate-b has no dependency on the rust-leaf-alice + ;; package, so this is a valid ordering + (((define-public 'rust-intermediate-c-1 + (package + (name "rust-intermediate-c") + (version "1.0.1") + (source + (origin + (method url-fetch) + (uri (crate-uri "intermediate-c" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:cargo-development-inputs + (("rust-leaf-alice" + ('unquote rust-leaf-alice-0.7)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-leaf-alice-0.7 + (package + (name "rust-leaf-alice") + (version "0.7.5") + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-alice" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-leaf-bob-3 + (package + (name "rust-leaf-bob") + (version "3.0.1") + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-bob" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-intermediate-b-1 + (package + (name "rust-intermediate-b") + (version "1.2.3") + (source + (origin + (method url-fetch) + (uri (crate-uri "intermediate-b" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:cargo-inputs + (("rust-leaf-bob" + ('unquote rust-leaf-bob-3)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-intermediate-a-1 + (package + (name "rust-intermediate-a") + (version "1.0.42") + (source + (origin + (method url-fetch) + (uri (crate-uri "intermediate-a" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:cargo-inputs + (("rust-intermediate-b" + ('unquote rust-intermediate-b-1)) + ("rust-leaf-alice" + ('unquote 'rust-leaf-alice-0.7)) ("rust-leaf-bob" ('unquote rust-leaf-bob-3)))))) (home-page "http://example.com") (synopsis "summary") (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-root-1 + (package + (name "rust-root") + (version "1.0.4") + (source + (origin + (method url-fetch) + (uri (crate-uri "root" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:cargo-inputs + (("rust-intermediate-a" + ('unquote rust-intermediate-a-1)) + ("rust-intermediate-b" + ('unquote rust-intermediate-b-1)) + ("rust-leaf-alice" + ('unquote 'rust-leaf-alice-0.7)) + ("rust-leaf-bob" + ('unquote rust-leaf-bob-3))) + #:cargo-development-inputs + (("rust-intermediate-c" + ('unquote rust-intermediate-c-1)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") (license (list license:expat license:asl2.0))))) #t) (x From patchwork Thu Dec 21 22:01:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Elsing X-Patchwork-Id: 57971 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2CA4A27BBE9; Thu, 21 Dec 2023 22:03:40 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id AFB3827BBE2 for ; Thu, 21 Dec 2023 22:03:37 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGR88-0004gh-Gh; Thu, 21 Dec 2023 17:03:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGR86-0004fa-L3 for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:58 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGR86-0005J2-CY for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:58 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGR8A-0004i8-E1 for guix-patches@gnu.org; Thu, 21 Dec 2023 17:03:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#67960] [PATCH 3/4] tests: Mock find-packages-by-name in crate importer test. Resent-From: David Elsing Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 22:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 67960@debbugs.gnu.org Cc: David Elsing Received: via spool by 67960-submit@debbugs.gnu.org id=B67960.170319616618061 (code B ref 67960); Thu, 21 Dec 2023 22:03:02 +0000 Received: (at 67960) by debbugs.gnu.org; 21 Dec 2023 22:02:46 +0000 Received: from localhost ([127.0.0.1]:45618 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7t-0004hE-HL for submit@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:46 -0500 Received: from mout02.posteo.de ([185.67.36.66]:33167) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7r-0004gh-0v for 67960@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:43 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 43A2B240103 for <67960@debbugs.gnu.org>; Thu, 21 Dec 2023 23:02:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703196153; bh=vIWSDFMaf2qGMHu+uoIQcd8H7KbxW2mof6vXpDXrOFk=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=OTCKnPnX4RVGSqzUGfWorV7wml1AMN1iu90C3h4bPuCzNmZNHlzaBimhMMxYwxJYc DD2rIz6u3bhwk+vJr1SkXPSiNg8MRqrPBvAaYmoVuEFzB05jiYrl3E1Fi8iGJ91F/h oOko8G6WvylHDS5lu/3bvFyg97Wc4X+t9NNzRO6GB4abm40RyJEuKGiqzO1ENf4Hd6 mosN6ZPXOH+HlLZfNqIB5hTCAvtq0k4ELCxVjr3SUHJTb+Dm2re+tHiHp01/6rCWD2 n5L01w+WSG4ac0DYPkQ8r4H8aFWdH+bK6ZBjiH0h5xo2uUTVOIS510R/81XrtsTTw3 M5VQBtC8hUnjw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Sx4D869LMz6tvr; Thu, 21 Dec 2023 23:02:32 +0100 (CET) From: David Elsing Date: Thu, 21 Dec 2023 22:01:51 +0000 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * tests/crate.scm: Import only sha256 from (gcrypt hash) as gcrypt-sha256 to prevent a name collision. Rename test 'cargo-recursive-import' to 'crate-recursive-import' and 'cargo-recursive-import-hoors-existing-packages' to 'crate-recursive-import-honors-existing-packages'. Mock find-packages-by-name from (gnu packages). Adjust test to import fake 'bar' crate instead of doctool. (test-bar-crate): New variable. (test-bar-dependencies): New variable. (test-root-crate): Adjust sha256 -> gcrypt-sha256. (test-doctool-crate,test-doctool-dependencies): Remove variables. (rust-leaf-bob-3): New variable. --- tests/crate.scm | 203 +++++++++++++++++++++++++++--------------------- 1 file changed, 115 insertions(+), 88 deletions(-) diff --git a/tests/crate.scm b/tests/crate.scm index 1b9ad88358..e779f738b3 100644 --- a/tests/crate.scm +++ b/tests/crate.scm @@ -25,7 +25,9 @@ (define-module (test-crate) #:use-module (guix import crate) #:use-module (guix base32) #:use-module (guix build-system cargo) - #:use-module (gcrypt hash) + #:use-module ((gcrypt hash) + #:select ((sha256 . gcrypt-sha256))) + #:use-module (guix packages) #:use-module (guix tests) #:use-module (gnu packages) #:use-module (ice-9 iconv) @@ -38,6 +40,8 @@ (define-module (test-crate) ;; foo-1.0.0 ;; foo-1.0.3 ;; leaf-alice 0.7.5 +;; bar-1.0.0 +;; leaf-bob 3.0.1 ;; ;; root-1.0.0 ;; root-1.0.4 @@ -116,6 +120,40 @@ (define test-foo-dependencies ] }") +(define test-bar-crate + "{ + \"crate\": { + \"max_version\": \"1.0.0\", + \"name\": \"bar\", + \"description\": \"summary\", + \"homepage\": \"http://example.com\", + \"repository\": \"http://example.com\", + \"keywords\": [\"dummy\", \"test\"], + \"categories\": [\"test\"], + \"actual_versions\": [ + { \"id\": 234100, + \"num\": \"1.0.0\", + \"license\": \"MIT OR Apache-2.0\", + \"links\": { + \"dependencies\": \"/api/v1/crates/bar/1.0.0/dependencies\" + }, + \"yanked\": false + } + ] + } +}") + +(define test-bar-dependencies + "{ + \"dependencies\": [ + { + \"crate_id\": \"leaf-bob\", + \"kind\": \"normal\", + \"req\": \"3.0.1\" + } + ] +}") + (define test-root-crate "{ \"crate\": { @@ -399,7 +437,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/foo/1.0.3/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.3/dependencies" (open-input-string test-foo-dependencies)) @@ -408,7 +446,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/leaf-alice/0.7.5/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/leaf-alice/0.7.5/dependencies" (open-input-string test-leaf-alice-dependencies)) @@ -442,7 +480,7 @@ (define have-guile-semver? (pk 'fail x #f))))) (unless have-guile-semver? (test-skip 1)) -(test-assert "cargo-recursive-import" +(test-assert "crate-recursive-import" ;; Replace network resources with sample data. (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -452,7 +490,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/root/1.0.4/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/root/1.0.4/dependencies" (open-input-string test-root-dependencies)) @@ -461,7 +499,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/intermediate-a/1.0.42/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/intermediate-a/1.0.42/dependencies" (open-input-string test-intermediate-a-dependencies)) @@ -470,7 +508,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/intermediate-b/1.2.3/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/intermediate-b/1.2.3/dependencies" (open-input-string test-intermediate-b-dependencies)) @@ -479,7 +517,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/intermediate-c/1.0.1/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/intermediate-c/1.0.1/dependencies" (open-input-string test-intermediate-c-dependencies)) @@ -488,7 +526,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/leaf-alice/0.7.5/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/leaf-alice/0.7.5/dependencies" (open-input-string test-leaf-alice-dependencies)) @@ -497,7 +535,7 @@ (define have-guile-semver? ("https://crates.io/api/v1/crates/leaf-bob/3.0.1/download" (set! test-source-hash (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/leaf-bob/3.0.1/dependencies" (open-input-string test-leaf-bob-dependencies)) @@ -814,85 +852,74 @@ (define-public 'rust-root-1 -(define test-doctool-crate - "{ - \"crate\": { - \"max_version\": \"2.2.2\", - \"name\": \"leaf-bob\", - \"description\": \"summary\", - \"homepage\": \"http://example.com\", - \"repository\": \"http://example.com\", - \"keywords\": [\"dummy\", \"test\"], - \"categories\": [\"test\"] - \"actual_versions\": [ - { \"id\": 234280, - \"num\": \"2.2.2\", - \"license\": \"MIT OR Apache-2.0\", - \"links\": { - \"dependencies\": \"/api/v1/crates/doctool/2.2.2/dependencies\" - }, - \"yanked\": false - } - ] - } -}") - -;; FIXME: This test depends on some existing packages -(define test-doctool-dependencies - "{ - \"dependencies\": [ - { - \"crate_id\": \"docopt\", - \"kind\": \"normal\", - \"req\": \"^0.8.1\" - } - ] -}") - - -(test-assert "self-test: rust-docopt 0.8.x is gone, please adjust the test case" - (not (null? (find-packages-by-name "rust-docopt" "0.8")))) +(define rust-leaf-bob-3 + (package + (name "rust-leaf-bob") + (version "3.0.1") + (source #f) + (build-system #f) + (home-page #f) + (synopsis #f) + (description #f) + (license #f))) (unless have-guile-semver? (test-skip 1)) -(test-assert "cargo-recursive-import-hoors-existing-packages" - (mock ((guix http-client) http-fetch - (lambda (url . rest) - (match url - ("https://crates.io/api/v1/crates/doctool" - (open-input-string test-doctool-crate)) - ("https://crates.io/api/v1/crates/doctool/2.2.2/download" - (set! test-source-hash - (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) - (open-input-string "empty file\n")) - ("https://crates.io/api/v1/crates/doctool/2.2.2/dependencies" - (open-input-string test-doctool-dependencies)) - (_ (error "Unexpected URL: " url))))) - (match (crate-recursive-import "doctool") - (((define-public 'rust-doctool-2 - (package - (name "rust-doctool") - (version "2.2.2") - (source - (origin - (method url-fetch) - (uri (crate-uri "doctool" version)) - (file-name - (string-append name "-" version ".tar.gz")) - (sha256 - (base32 - (? string? hash))))) - (build-system cargo-build-system) - (arguments - ('quasiquote (#:cargo-inputs - (("rust-docopt" - ('unquote 'rust-docopt-0.8)))))) - (home-page "http://example.com") - (synopsis "summary") - (description "summary") - (license (list license:expat license:asl2.0))))) - #t) - (x - (pk 'fail x #f))))) +(test-assert "crate-recursive-import-honors-existing-packages" + (mock + ((gnu packages) find-packages-by-name + (lambda* (name #:optional version) + (match name + ("rust-leaf-bob" + (list rust-leaf-bob-3)) + (_ '())))) + (mock + ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://crates.io/api/v1/crates/bar" + (open-input-string test-bar-crate)) + ("https://crates.io/api/v1/crates/bar/1.0.0/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/bar/1.0.0/dependencies" + (open-input-string test-bar-dependencies)) + ("https://crates.io/api/v1/crates/leaf-bob" + (open-input-string test-leaf-bob-crate)) + ("https://crates.io/api/v1/crates/leaf-bob/3.0.2/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/leaf-bob/3.0.2/dependencies" + (open-input-string test-leaf-bob-dependencies)) + (_ (error "Unexpected URL: " url))))) + (match (crate-recursive-import "bar") + (((define-public 'rust-bar-1 + (package + (name "rust-bar") + (version "1.0.0") + (source + (origin + (method url-fetch) + (uri (crate-uri "bar" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:cargo-inputs + (("rust-leaf-bob" + ('unquote 'rust-leaf-bob-3)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0))))) + #t) + (x + (pk 'fail x #f)))))) (test-end "crate") From patchwork Thu Dec 21 22:01:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Elsing X-Patchwork-Id: 57969 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id ED68427BBEC; Thu, 21 Dec 2023 22:03:33 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 06A0C27BBE2 for ; Thu, 21 Dec 2023 22:03:30 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGR88-0004gu-J4; Thu, 21 Dec 2023 17:03:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGR87-0004fw-I4 for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:59 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGR87-0005JI-AP for guix-patches@gnu.org; Thu, 21 Dec 2023 17:02:59 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rGR8B-0004iN-E9 for guix-patches@gnu.org; Thu, 21 Dec 2023 17:03:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#67960] [PATCH 4/4] guix: import: Optionally import necessary yanked crates. Resent-From: David Elsing Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Dec 2023 22:03:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 67960@debbugs.gnu.org Cc: David Elsing Received: via spool by 67960-submit@debbugs.gnu.org id=B67960.170319617218085 (code B ref 67960); Thu, 21 Dec 2023 22:03:03 +0000 Received: (at 67960) by debbugs.gnu.org; 21 Dec 2023 22:02:52 +0000 Received: from localhost ([127.0.0.1]:45622 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7y-0004ha-Uo for submit@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:52 -0500 Received: from mout01.posteo.de ([185.67.36.65]:57003) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGR7s-0004gj-W6 for 67960@debbugs.gnu.org; Thu, 21 Dec 2023 17:02:46 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id DA432240027 for <67960@debbugs.gnu.org>; Thu, 21 Dec 2023 23:02:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703196154; bh=kf/QQ3w2o3n41m+q1lqJnC3/E83qa71KG/PfIeEZkLY=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=khquDJgR+/YoB7X7bvvv0wxph3szue740Crdr0RAUwmZNgFeWH0Ih9NPkRjjPgIHj acuyKCplauLnS/CxPbm+LiR++2C0MbvGNNIGXSwZ9UcynsPltvB14fNTfCagKrXIpn 1A+/9TuPshbwU18F8iu5XLkz7lrzMOzx5sRPtzvPoUNhi2G4w4s/ieoJBX9iVX5Wrb QZpYOvVDaRdrm3iif//G1z/JXVU7sOfy57E0OCCeCbFlLMP6hlTg9zcYoZM4n04n2s M7WFy8dV1s4yQZle/goeALUIW5gtTxk7ePRGmHqCQr97faSQ0uqPk6LRmewuCkh3aW CrFisQ92XOR6A== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Sx4DB23rzz6twM; Thu, 21 Dec 2023 23:02:34 +0100 (CET) From: David Elsing Date: Thu, 21 Dec 2023 22:01:52 +0000 Message-ID: <7b351acd4d85a1b934ac898c217fe7b9b40bedf5.1703195451.git.david.elsing@posteo.net> In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * doc/guix.texi (Invoking guix import): Mention '--allow-yanked'. * guix/import/crate.scm (make-crate-sexp): Add yanked? argument. For yanked packages, use the full version suffixed by "-yanked" for generated variable names and add a comment and package property. (crate->guix-package): Add allow-yanked? argument and if it is set to #t, allow importing yanked crates if no other version matching the requirements exists. [find-package-version]: Packages previously marked as yanked are only included if allow-yanked? is #t and then take the lowest priority. [find-crate-version]: If allow-yanked? is #t, also consider yanked versions with the lowest priority. [dependency-name+version]: Rename to ... [dependency-name+version+yanked] ...this. Honor allow-yanked? and choose between an existing package and an upstream package. Exit with an error message if no version fulfilling the requirement is found. [version*]: Exit with an error message if the crate version is not found. (cargo-recursive-import): Add allow-yanked? argument. * guix/read-print.scm: Export . * guix/scripts/import/crate.scm: Add "--allow-yanked". * tests/crate.scm: Add test 'crate-recursive-import-only-yanked-available'. [sort-map-dependencies]: Adjust accordingly. [remove-yanked-info]: New variable. Adjust test 'crate-recursive-import-honors-existing-packages'. (test-bar-dependencies): Add yanked dev-dependencies. (test-leaf-bob-crate): Add yanked versions. (rust-leaf-bob-3.0.2-yanked): New variable. --- doc/guix.texi | 3 + guix/import/crate.scm | 139 ++++++++++++++++++------ guix/read-print.scm | 1 + guix/scripts/import/crate.scm | 14 ++- tests/crate.scm | 193 +++++++++++++++++++++++++++++++++- 5 files changed, 310 insertions(+), 40 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index a19671643b..da36f90e9b 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14516,6 +14516,9 @@ in Guix. If @option{--recursive} is specified, also the recursively imported packages contain their development dependencies, which are recursively imported as well. +@item --allow-yanked +If no non-yanked version of a crate is available, use the latest yanked +version instead instead of aborting. @end table @item elm diff --git a/guix/import/crate.scm b/guix/import/crate.scm index db5461312f..e3b8286350 100644 --- a/guix/import/crate.scm +++ b/guix/import/crate.scm @@ -26,12 +26,15 @@ (define-module (guix import crate) #:use-module (guix base32) #:use-module (guix build-system cargo) + #:use-module (guix diagnostics) #:use-module (gcrypt hash) #:use-module (guix http-client) + #:use-module (guix i18n) #:use-module (guix import json) #:use-module (guix import utils) #:use-module (guix memoization) #:use-module (guix packages) + #:use-module (guix read-print) #:use-module (guix upstream) #:use-module (guix utils) #:use-module (gnu packages) @@ -41,6 +44,7 @@ (define-module (guix import crate) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-69) #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name @@ -100,7 +104,7 @@ (define-json-mapping make-crate-dependency ;; Autoload Guile-Semver so we only have a soft dependency. (module-autoload! (current-module) - '(semver) '(string->semver semver->string semversemver semver->string semversemver-range semver-range-contains?)) @@ -165,16 +169,18 @@ (define (version->semver-prefix version) (list-matches "^(0+\\.){,2}[0-9]+" version)))) (define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs - home-page synopsis description license build?) + home-page synopsis description license build? yanked?) "Return the `package' s-expression for a rust package with the given NAME, VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." (define (format-inputs inputs) (map (match-lambda - ((name version) + ((name version yanked) (list (crate-name->package-name name) - (version->semver-prefix version)))) + (if yanked + (string-append version "-yanked") + (version->semver-prefix version))))) inputs)) (let* ((port (http-fetch (crate-uri name version))) @@ -184,6 +190,9 @@ (define (format-inputs inputs) (pkg `(package (name ,guix-name) (version ,version) + ,@(if yanked? + `(,(comment "; This version was yanked!\n" #t)) + '()) (source (origin (method url-fetch) (uri (crate-uri ,name version)) @@ -191,6 +200,9 @@ (define (format-inputs inputs) (sha256 (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) + ,@(if yanked? + `((properties '((crate-version-yanked? . #t)))) + '()) (build-system cargo-build-system) ,@(maybe-arguments (append (if build? '() @@ -207,7 +219,10 @@ (define (format-inputs inputs) ((license) license) (_ `(list ,@license))))))) (close-port port) - (package->definition pkg (version->semver-prefix version)))) + (package->definition pkg + (if yanked? + (string-append version "-yanked") + (version->semver-prefix version))))) (define (string->license string) (filter-map (lambda (license) @@ -218,8 +233,9 @@ (define (string->license string) 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:key version include-dev-deps? - #:allow-other-keys) +(define* (crate->guix-package + crate-name + #:key version include-dev-deps? allow-yanked? #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, convert it into a semver range and attempt to fetch @@ -243,63 +259,112 @@ (define version-number (or version (crate-latest-version crate)))) - ;; find the highest existing package that fulfills the semver + ;; Find the highest existing package that fulfills the semver + ;; . Packages previously marked as yanked take lower priority. (define (find-package-version name range) (let* ((semver-range (string->semver-range range)) - (versions + (package-versions (sort - (filter (lambda (version) - (semver-range-contains? semver-range version)) + (filter (match-lambda ((semver yanked) + (and + (or allow-yanked? (not yanked)) + (semver-range-contains? semver-range semver)))) (map (lambda (pkg) - (string->semver (package-version pkg))) + (let ((version (package-version pkg))) + (list + (string->semver version) + (assoc-ref (package-properties pkg) 'crate-version-yanked?)))) (find-packages-by-name (crate-name->package-name name)))) - semverstring (last versions))))) - - ;; Find the highest version of a crate that fulfills the semver - ;; and hasn't been yanked. + (match-lambda* (((semver1 yanked1) (semver2 yanked2)) + (or + (and yanked1 (not yanked2)) + (and + (eq? yanked1 yanked2) + (semverstring semver) yanked))))) + + ;; Find the highest version of a crate that fulfills the semver . If + ;; no matching non-yanked version has been found and allow-yanked? is #t, + ;; also consider yanked packages. (define (find-crate-version crate range) (let* ((semver-range (string->semver-range range)) (versions (sort (filter (lambda (entry) (and - (not (crate-version-yanked? (second entry))) - (semver-range-contains? semver-range (first entry)))) + (or allow-yanked? (not (crate-version-yanked? (second entry)))) + (semver-range-contains? semver-range (first entry)))) (map (lambda (ver) (list (string->semver (crate-version-number ver)) ver)) (crate-versions crate))) - (match-lambda* (((semver _) ...) - (apply semversemver (first existing-version)) + (string->semver (crate-version-number ver))) + (begin + (warning (G_ "~A: version ~a is no longer yanked~%") name (first existing-version)) + (cons name existing-version)) + (list name + (crate-version-number ver) + (crate-version-yanked? ver))) + (begin + (warning (G_ "~A: using existing version ~a, which was yanked~%") name (first existing-version)) + (cons name existing-version))) + (begin + (unless ver + (leave (G_ "~A: no version found for requirement ~a~%") name req)) + (if (crate-version-yanked? ver) + (warning (G_ "~A: imported version ~a was yanked~%") name (crate-version-number ver))) + (list name + (crate-version-number ver) + (crate-version-yanked? ver)))))))) (define version* (and crate - (find-crate-version crate version-number))) + (or + (find-crate-version crate version-number) + (leave (G_ "~A: version ~a not found~%") crate-name version-number)))) ;; sort and map the dependencies to a list containing ;; pairs of (name version) (define (sort-map-dependencies deps) - (sort (map dependency-name+version + (sort (map dependency-name+version+yanked deps) - (match-lambda* (((name _) ...) + (match-lambda* (((name _ _) ...) (apply string-ci (crate-version-license version*) string->license)) - (append cargo-inputs cargo-development-inputs))) + (append + (remove-yanked-info cargo-inputs) + (remove-yanked-info cargo-development-inputs)))) (values #f '()))) (define* (crate-recursive-import - crate-name #:key version recursive-dev-dependencies?) + crate-name #:key version recursive-dev-dependencies? allow-yanked?) (recursive-import crate-name #:repo->guix-package @@ -340,7 +408,8 @@ (define* (crate-recursive-import (or (equal? (car params) crate-name) recursive-dev-dependencies?))) (apply crate->guix-package* - (append params `(#:include-dev-deps? ,include-dev-deps?)))))) + (append params `(#:include-dev-deps? ,include-dev-deps? + #:allow-yanked? ,allow-yanked?)))))) #:version version #:guix-name crate-name->package-name)) diff --git a/guix/read-print.scm b/guix/read-print.scm index 690f5dacdd..6421b79737 100644 --- a/guix/read-print.scm +++ b/guix/read-print.scm @@ -46,6 +46,7 @@ (define-module (guix read-print) page-break page-break? + comment comment? comment->string diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scm index b13b6636a6..082a973aee 100644 --- a/guix/scripts/import/crate.scm +++ b/guix/scripts/import/crate.scm @@ -51,6 +51,10 @@ (define (show-help) (display (G_ " --recursive-dev-dependencies include dev-dependencies recursively")) + (display (G_ " + --allow-yanked + allow importing yanked crates if no alternative + satisfying the version requirement exists")) (newline) (display (G_ " -h, --help display this help and exit")) @@ -74,6 +78,9 @@ (define %options (option '("recursive-dev-dependencies") #f #f (lambda (opt name arg result) (alist-cons 'recursive-dev-dependencies #t result))) + (option '("allow-yanked") #f #f + (lambda (opt name arg result) + (alist-cons 'allow-yanked #t result))) %standard-import-options)) @@ -102,8 +109,11 @@ (define-values (name version) (crate-recursive-import name #:version version #:recursive-dev-dependencies? - (assoc-ref opts 'recursive-dev-dependencies)) - (crate->guix-package name #:version version #:include-dev-deps? #t)) + (assoc-ref opts 'recursive-dev-dependencies) + #:allow-yanked? (assoc-ref opts 'allow-yanked)) + (crate->guix-package + name #:version version #:include-dev-deps? #t + #:allow-yanked? (assoc-ref opts 'allow-yanked))) ((or #f '()) (leave (G_ "failed to download meta-data for package '~a'~%") (if version diff --git a/tests/crate.scm b/tests/crate.scm index e779f738b3..ce2f08aade 100644 --- a/tests/crate.scm +++ b/tests/crate.scm @@ -28,6 +28,7 @@ (define-module (test-crate) #:use-module ((gcrypt hash) #:select ((sha256 . gcrypt-sha256))) #:use-module (guix packages) + #:use-module (guix read-print) #:use-module (guix tests) #:use-module (gnu packages) #:use-module (ice-9 iconv) @@ -42,6 +43,8 @@ (define-module (test-crate) ;; leaf-alice 0.7.5 ;; bar-1.0.0 ;; leaf-bob 3.0.1 +;; leaf-bob 3.0.2 (dev-dependency) +;; leaf-bob 4.0.0 (dev-dependency) ;; ;; root-1.0.0 ;; root-1.0.4 @@ -68,6 +71,8 @@ (define-module (test-crate) ;; leaf-alice-0.7.5 ;; ;; leaf-bob-3.0.1 +;; leaf-bob-3.0.2 (yanked) +;; leaf-bob-4.0.0 (yanked) (define test-foo-crate @@ -150,6 +155,16 @@ (define test-bar-dependencies \"crate_id\": \"leaf-bob\", \"kind\": \"normal\", \"req\": \"3.0.1\" + }, + { + \"crate_id\": \"leaf-bob\", + \"kind\": \"dev\", + \"req\": \"^3.0.2\" + }, + { + \"crate_id\": \"leaf-bob\", + \"kind\": \"dev\", + \"req\": \"^4.0.0\" } ] }") @@ -398,6 +413,22 @@ (define test-leaf-bob-crate \"dependencies\": \"/api/v1/crates/leaf-bob/3.0.1/dependencies\" }, \"yanked\": false + }, + { \"id\": 234281, + \"num\": \"3.0.2\", + \"license\": \"MIT OR Apache-2.0\", + \"links\": { + \"dependencies\": \"/api/v1/crates/leaf-bob/3.0.2/dependencies\" + }, + \"yanked\": true + }, + { \"id\": 234282, + \"num\": \"4.0.0\", + \"license\": \"MIT OR Apache-2.0\", + \"links\": { + \"dependencies\": \"/api/v1/crates/leaf-bob/4.0.0/dependencies\" + }, + \"yanked\": true } ] } @@ -863,6 +894,18 @@ (define rust-leaf-bob-3 (description #f) (license #f))) +(define rust-leaf-bob-3.0.2-yanked + (package + (name "rust-leaf-bob") + (version "3.0.2") + (source #f) + (properties '((crate-version-yanked? . #t))) + (build-system #f) + (home-page #f) + (synopsis #f) + (description #f) + (license #f))) + (unless have-guile-semver? (test-skip 1)) (test-assert "crate-recursive-import-honors-existing-packages" (mock @@ -870,7 +913,7 @@ (define rust-leaf-bob-3 (lambda* (name #:optional version) (match name ("rust-leaf-bob" - (list rust-leaf-bob-3)) + (list rust-leaf-bob-3 rust-leaf-bob-3.0.2-yanked)) (_ '())))) (mock ((guix http-client) http-fetch @@ -894,8 +937,16 @@ (define rust-leaf-bob-3 (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/leaf-bob/3.0.2/dependencies" (open-input-string test-leaf-bob-dependencies)) + ("https://crates.io/api/v1/crates/leaf-bob/4.0.0/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/leaf-bob/4.0.0/dependencies" + (open-input-string test-leaf-bob-dependencies)) (_ (error "Unexpected URL: " url))))) - (match (crate-recursive-import "bar") + (match (crate-recursive-import "bar" + #:allow-yanked? #t) (((define-public 'rust-bar-1 (package (name "rust-bar") @@ -913,7 +964,12 @@ (define rust-leaf-bob-3 (arguments ('quasiquote (#:cargo-inputs (("rust-leaf-bob" - ('unquote 'rust-leaf-bob-3)))))) + ('unquote 'rust-leaf-bob-3))) + #:cargo-development-inputs + (("rust-leaf-bob" + ('unquote 'rust-leaf-bob-3.0.2-yanked)) + ("rust-leaf-bob" + ('unquote 'rust-leaf-bob-4.0.0-yanked)))))) (home-page "http://example.com") (synopsis "summary") (description "summary") @@ -922,4 +978,135 @@ (define rust-leaf-bob-3 (x (pk 'fail x #f)))))) +(unless have-guile-semver? (test-skip 1)) +(test-assert "crate-import-only-yanked-available" + (mock + ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://crates.io/api/v1/crates/bar" + (open-input-string test-bar-crate)) + ("https://crates.io/api/v1/crates/bar/1.0.0/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/bar/1.0.0/dependencies" + (open-input-string test-bar-dependencies)) + ("https://crates.io/api/v1/crates/leaf-bob" + (open-input-string test-leaf-bob-crate)) + ("https://crates.io/api/v1/crates/leaf-bob/3.0.1/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/leaf-bob/3.0.1/dependencies" + (open-input-string test-leaf-bob-dependencies)) + ("https://crates.io/api/v1/crates/leaf-bob/3.0.2/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/leaf-bob/3.0.2/dependencies" + (open-input-string test-leaf-bob-dependencies)) + ("https://crates.io/api/v1/crates/leaf-bob/4.0.0/download" + (set! test-source-hash + (bytevector->nix-base32-string + (gcrypt-sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/leaf-bob/4.0.0/dependencies" + (open-input-string test-leaf-bob-dependencies)) + (_ (error "Unexpected URL: " url))))) + (match (crate-recursive-import "bar" + #:recursive-dev-dependencies? #t + #:allow-yanked? #t) + (((define-public 'rust-leaf-bob-4.0.0-yanked + (package + (name "rust-leaf-bob") + (version "4.0.0") + ($ "; This version was yanked!\n" #t) + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-bob" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (properties ('quote (('crate-version-yanked? . #t)))) + (build-system cargo-build-system) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-leaf-bob-3.0.2-yanked + (package + (name "rust-leaf-bob") + (version "3.0.2") + ($ "; This version was yanked!\n" #t) + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-bob" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (properties ('quote (('crate-version-yanked? . #t)))) + (build-system cargo-build-system) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-leaf-bob-3 + (package + (name "rust-leaf-bob") + (version "3.0.1") + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-bob" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public 'rust-bar-1 + (package + (name "rust-bar") + (version "1.0.0") + (source + (origin + (method url-fetch) + (uri (crate-uri "bar" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:cargo-inputs + (("rust-leaf-bob" + ('unquote 'rust-leaf-bob-3))) + #:cargo-development-inputs + (("rust-leaf-bob" + ('unquote 'rust-leaf-bob-3.0.2-yanked)) + ("rust-leaf-bob" + ('unquote 'rust-leaf-bob-4.0.0-yanked)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0))))) + #t) + (x + (pk 'fail (pretty-print-with-comments (current-output-port) x) #f))))) + (test-end "crate")