From patchwork Mon Jan 24 22:07:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Vivien Kraus X-Patchwork-Id: 36794 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E709B27BBE9; Mon, 24 Jan 2022 22:24:10 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 10DD027BBE9 for ; Mon, 24 Jan 2022 22:24:10 +0000 (GMT) Received: from localhost ([::1]:33352 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nC7kv-00071q-6T for patchwork@mira.cbaines.net; Mon, 24 Jan 2022 17:24:09 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38108) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nC7ko-00071f-FD for guix-patches@gnu.org; Mon, 24 Jan 2022 17:24:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53224) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nC7ko-0005gK-5p for guix-patches@gnu.org; Mon, 24 Jan 2022 17:24:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nC7ko-0000cq-2v for guix-patches@gnu.org; Mon, 24 Jan 2022 17:24:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53395] Fix pypi import for flake8-array-spacing Resent-From: Vivien Kraus Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Jan 2022 22:24:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53395 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Ludovic =?utf-8?q?Court=C3=A8s?= Cc: 53395@debbugs.gnu.org Received: via spool by 53395-submit@debbugs.gnu.org id=B53395.16430629922338 (code B ref 53395); Mon, 24 Jan 2022 22:24:02 +0000 Received: (at 53395) by debbugs.gnu.org; 24 Jan 2022 22:23:12 +0000 Received: from localhost ([127.0.0.1]:46127 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nC7jz-0000be-8R for submit@debbugs.gnu.org; Mon, 24 Jan 2022 17:23:11 -0500 Received: from planete-kraus.eu ([89.234.140.182]:39032) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nC7jv-0000bP-U5 for 53395@debbugs.gnu.org; Mon, 24 Jan 2022 17:23:09 -0500 Received: from planete-kraus.eu (localhost.lan [127.0.0.1]) by planete-kraus.eu (OpenSMTPD) with ESMTP id c151ced7; Mon, 24 Jan 2022 22:22:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=planete-kraus.eu; h= references:from:to:cc:subject:date:in-reply-to:message-id :mime-version:content-type; s=dkim; bh=GIZo3yaCh6J3SWbjr9kmp3AT8 80=; b=iU1fbLVtusIXgSpBuLdXBAsOZzTO6EBJU707r/vTmIYRya53lybjENIPO kA8sqQl1Qxwsgz0GQM3WEJzA7BIM+YKlYOYMOqATQ52lU99v6k994UARnh4lL6OS AFDQ/7EF4G3yo+7nreDjmPU22QORSbdK0p4dsdxv4rk67pAebg= Received: by planete-kraus.eu (OpenSMTPD) with ESMTPSA id 10614eca (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Mon, 24 Jan 2022 22:21:51 +0000 (UTC) References: <87czkmi434.fsf@planete-kraus.eu> <87ee4xut38.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 Date: Mon, 24 Jan 2022 23:07:46 +0100 In-reply-to: <87ee4xut38.fsf@gnu.org> Message-ID: <87mtjk22rn.fsf@planete-kraus.eu> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" Reply-to: Vivien Kraus X-ACL-Warn: , Vivien Kraus via Guix-patches X-Patchwork-Original-From: Vivien Kraus via Guix-patches via From: Vivien Kraus X-getmail-retrieved-from-mailbox: Patches Hi :) Ludovic Courtès writes: > As a rule of thumb, warnings are one-line messages (not sentences) > describing the problem. Here you could emit such a warning, followed by > a call to ‘display-hint’, which accepts Texinfo markup. display-hint seems to always add \n\n at the end of the message, is it intended that way? Should I do one big display-hint instead of 3? > Also, what about adding a unit test? I added two by copying the "foo" example and pretending it’s named "Foo" and "goo" respectively (fixing the package name in the json data), while keeping release URLs as /foo-… so the importer should in the first case recognize "Foo" in the URL and in the second case it should not recognize "goo" in the URL so it should emit a warning. I didn’t test the dash-to-underscore transformation, because it would require test data with a package named "foo-something" but with release URIs containing "foo_something". That data point does not exist currently in the test module. Vivien From 3e7cfd8b2c3a76ab8aac82a7d2f2a9f3e1a5139d Mon Sep 17 00:00:00 2001 From: Vivien Kraus Date: Thu, 20 Jan 2022 20:11:56 +0100 Subject: [PATCH] pypi importer: Convert - to _ in pypi urls if needed. * guix/import/pypi.scm (find-project-url): New function. (make-pypi-sexp): Use find-project-url. * tests/pypi.scm ("Package Foo has a correct pypi-uri of foo"): New test. ("If the package goo is released as foo, the importer warns"): New test. --- guix/import/pypi.scm | 30 ++++++--- tests/pypi.scm | 150 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index b4284f5c33..3839c37a95 100644 --- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -41,6 +41,7 @@ (define-module (guix import pypi) #:use-module (guix memoization) #:use-module (guix diagnostics) #:use-module (guix i18n) + #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) @@ -418,6 +419,25 @@ (define process-requirements (values (map process-requirements dependencies) (concatenate dependencies)))) +(define (find-project-url name pypi-url) + "Try different project name substitution until the result is found in +pypi-uri. Downcase is required for \"uWSGI\", and +underscores are required for flake8-array-spacing." + (or (find (cut string-contains pypi-url <>) + (list name + (string-downcase name) + (string-replace-substring name "-" "_"))) + (begin + (warning + (G_ "The project name does not appear verbatim in the pypi URI~%")) + (display-hint + (format #f (G_ "The project name is: @strong{~a}") name)) + (display-hint + (format #f (G_ "The pypi URI is: @strong{@url{~a}}") pypi-url)) + (display-hint + (G_ "The pypi-uri declaration in the generated package might need to be changed.")) + name))) + (define (make-pypi-sexp name version source-url wheel-url home-page synopsis description license) "Return the `package' s-expression for a python package with the given NAME, @@ -446,15 +466,7 @@ (define (maybe-upstream-name name) (origin (method url-fetch) (uri (pypi-uri - ;; PyPI URL are case sensitive, but sometimes - ;; a project named using mixed case has a URL - ;; using lower case, so we must work around this - ;; inconsistency. For actual examples, compare - ;; the URLs of the "Deprecated" and "uWSGI" PyPI - ;; packages. - ,(if (string-contains source-url name) - name - (string-downcase name)) + ,(find-project-url name source-url) version ;; Some packages have been released as `.zip` ;; instead of the more common `.tar.gz`. For diff --git a/tests/pypi.scm b/tests/pypi.scm index 1ea5f02643..b0aff2f0a1 100644 --- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,10 +23,12 @@ (define-module (test-pypi) #:use-module (guix import pypi) #:use-module (guix base32) #:use-module (guix memoization) + #:use-module (guix utils) #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which mkdir-p)) + #:use-module ((guix diagnostics) #:select (guix-warning-port)) #:use-module (srfi srfi-64) #:use-module (ice-9 match)) @@ -428,4 +430,152 @@ (define test-metadata-with-extras-jedi "\ (x (pk 'fail x #f)))))) +(test-equal "Package Foo has a correct pypi-uri of foo" + "" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" + (lambda () + (display test-requires.txt))) + (parameterize ((current-output-port (%make-void-port "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0"))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ;; Note the Foo capitalization in the project URL + ("https://pypi.org/pypi/Foo/json" + ;; The document advertises the package name as "Foo" + ;; too + (let ((name-fixed + (string-replace-substring + test-json-1 + "\"name\": \"foo\"," + "\"name\": \"Foo\","))) + (values (open-input-string name-fixed) + (string-length name-fixed)))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) + (_ (error "Unexpected URL: " url))))) + (match (pypi->guix-package "Foo") + (('package + ('name "python-foo") ;; Guix wants downcase package + ;; names + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-baz)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + +(test-equal "If the package goo is released as foo, the importer warns" + "warning: The project name does not appear verbatim in the pypi URI +hint: The project name is: *goo* + +hint: The pypi URI is: *`https://example.com/foo-1.0.0.tar.gz'* + +hint: The pypi-uri declaration in the generated package might need to be changed. + +" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port) + (current-error-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" + (lambda () + (display test-requires.txt))) + (parameterize ((current-output-port (%make-void-port "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0"))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/goo/json" + (let ((name-fixed ;; The true name of the package is + ;; "goo", only the releases are + ;; named foo- + (string-replace-substring + test-json-1 + "\"name\": \"foo\"," + "\"name\": \"goo\","))) + (values (open-input-string name-fixed) + (string-length name-fixed)))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) + (_ (error "Unexpected URL: " url))))) + (match (pypi->guix-package "goo") + (('package + ('name "python-goo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "goo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-baz)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + (test-end "pypi") -- 2.34.0