From patchwork Mon Mar 22 04:20:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 28011 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3FCE427BC57; Mon, 22 Mar 2021 04:27:18 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,SPF_HELO_PASS, T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 75EFF27BC58 for ; Mon, 22 Mar 2021 04:27:17 +0000 (GMT) Received: from localhost ([::1]:44484 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOC9s-0003t1-Jb for patchwork@mira.cbaines.net; Mon, 22 Mar 2021 00:27:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48248) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOC3q-0007GH-AK for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44284) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOC3q-00011q-0U for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOC3p-0005tb-T6 for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47310] [PATCH 1/4] import: utils: Refactor maybe-inputs and add maybe-propagated-inputs. References: <87eeg7vn3d.fsf@gmail.com> In-Reply-To: <87eeg7vn3d.fsf@gmail.com> Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 22 Mar 2021 04:21:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47310 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47310@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 47310-submit@debbugs.gnu.org id=B47310.161638684022577 (code B ref 47310); Mon, 22 Mar 2021 04:21:01 +0000 Received: (at 47310) by debbugs.gnu.org; 22 Mar 2021 04:20:40 +0000 Received: from localhost ([127.0.0.1]:55822 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3U-0005s5-54 for submit@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:40 -0400 Received: from mail-qv1-f53.google.com ([209.85.219.53]:35822) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3S-0005rq-S5 for 47310@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:39 -0400 Received: by mail-qv1-f53.google.com with SMTP id x27so8066318qvd.2 for <47310@debbugs.gnu.org>; Sun, 21 Mar 2021 21:20:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=/72utIhoWxFa/Ss4qr/M4ZbG5RnS99CklZw/lswtn24=; b=bewJRSipJEr1KhqibE0SLotkbmZPCz6xA8pEUxVnOuokfSYeX6U5HJ5BODDLbRYuyr 3cXDxc0KdpqzMCp+hsJ2rlQqemY+STvagnnfCL2IUwoNSbt5x/XH0HnWlmgQfytGvAy+ JGSPjWmiU3XJnMNOhVU2s8igG/M+/DAZgcEhi4Tf+6xU0ptSpocnnLJHwuz/AHOx8Ovl uZNhLhHZMbfqaAzSR7miLl1yh7oHDKdYw77LsKVLCYihGtBUWkSSuFmDoEvpYGX0ExSK UlSbVrjSagmJ2EwsJsxgrPmy1Io/Dx2wNKwSgOziU1bCqBY4eEncPFzRwisgrb6PM4sl OJkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=/72utIhoWxFa/Ss4qr/M4ZbG5RnS99CklZw/lswtn24=; b=Kthtoy67O8GRjVP5JVdvXHcxC/Ru4uNaCJZKmWao6j6A1vw7D2dbmD5Xc9IP9Zg7L1 WhTTsGCW8UMnowDk5rwlWLw8stRjGBWqdPCYWR6CPVCF3qVx0IFt7EzBoOvcUiXAarlP IaTVsiDTvk7K6b6YmaOCgmqJuSxIFFF23dc6DcllUB1W2iViUn9ZhX+PTPWdmVzwb7n9 Ni7R8i+gfDFwUbxbGrXOAx+D073atDPh2/+z7iY4PN3nVgbTzqFOkBgLgKK/tnal4R56 E7pXu4fCHDNecurdBhFMUZtaLWtQJ5Fir+IGmugRyArXOkhNLa0o9yI9tCbZ4AcR4BxC Xaew== X-Gm-Message-State: AOAM5318s1iv9XxTJQ3PPs9MY8F/fDyiiF3PX5qxCehFkGinmmTAmGPi Ass9tMmz42PmNeKDp8zLtN+e7aaD3rA= X-Google-Smtp-Source: ABdhPJzrCdlCuDS4q1TbjuUy+v+pZcxP0rUKyQSNEMEPS4rPnbTC6wKNEmfcyT0A/RwioWJ2VCgD5Q== X-Received: by 2002:a0c:f092:: with SMTP id g18mr19944409qvk.11.1616386833044; Sun, 21 Mar 2021 21:20:33 -0700 (PDT) Received: from localhost.localdomain (dsl-10-134-34.b2b2c.ca. [72.10.134.34]) by smtp.gmail.com with ESMTPSA id z8sm8340478qtn.12.2021.03.21.21.20.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Mar 2021 21:20:32 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 22 Mar 2021 00:20:19 -0400 Message-Id: <20210322042022.17385-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/import/utils.scm (maybe-inputs)[type]: New argument. Update docstring. The argument is used to derive the input field name to use. (maybe-native-inputs): Adjust to use the above. (maybe-propagated-inputs): New procedure. --- guix/import/utils.scm | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/guix/import/utils.scm b/guix/import/utils.scm index 64d1385164..e55258e306 100644 --- a/guix/import/utils.scm +++ b/guix/import/utils.scm @@ -7,6 +7,7 @@ ;;; Copyright © 2019 Robert Vollmert ;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> ;;; Copyright © 2020 Martin Becze +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -57,6 +58,7 @@ package-names->package-inputs maybe-inputs maybe-native-inputs + maybe-propagated-inputs package->definition spdx-string->license @@ -246,23 +248,29 @@ use in an 'inputs' field of a package definition." (input (make-input input #f))) names)) -(define* (maybe-inputs package-names #:optional (output #f)) +(define* (maybe-inputs package-names #:optional (output #f) + #:key (type #f)) "Given a list of PACKAGE-NAMES, tries to generate the 'inputs' field of a -package definition." - (match (package-names->package-inputs package-names output) - (() - '()) - ((package-inputs ...) - `((inputs (,'quasiquote ,package-inputs)))))) +package definition. TYPE can be used to specify the type of the inputs; +either the 'native or 'propagated symbols are accepted. Left unspecified, the +snippet generated is for regular inputs." + (let ((field-name (match type + ('native 'native-inputs) + ('propagated 'propagated-inputs) + (_ 'inputs)))) + (match (package-names->package-inputs package-names output) + (() + '()) + ((package-inputs ...) + `((,field-name (,'quasiquote ,package-inputs))))))) (define* (maybe-native-inputs package-names #:optional (output #f)) - "Given a list of PACKAGE-NAMES, tries to generate the 'inputs' field of a -package definition." - (match (package-names->package-inputs package-names output) - (() - '()) - ((package-inputs ...) - `((native-inputs (,'quasiquote ,package-inputs)))))) + "Same as MAYBE-INPUTS, but for native inputs." + (maybe-inputs package-names output #:type 'native)) + +(define* (maybe-propagated-inputs package-names #:optional (output #f)) + "Same as MAYBE-INPUTS, but for propagated inputs." + (maybe-inputs package-names output #:type 'propagated)) (define* (package->definition guix-package #:optional append-version?/string) "If APPEND-VERSION?/STRING is #t, append the package's major+minor From patchwork Mon Mar 22 04:20:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 28013 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 87B4827BC58; Mon, 22 Mar 2021 04:31:21 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,SPF_HELO_PASS, T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 9DF8527BC57 for ; Mon, 22 Mar 2021 04:31:19 +0000 (GMT) Received: from localhost ([::1]:51304 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOCDm-0006vy-Qa for patchwork@mira.cbaines.net; Mon, 22 Mar 2021 00:31:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48252) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOC3s-0007HM-2h for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44285) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOC3q-00011x-Dd for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOC3q-0005ti-9S for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47310] [PATCH 2/4] import: go: Improve synopsis and description parsing. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 22 Mar 2021 04:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47310 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47310@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 47310-submit@debbugs.gnu.org id=B47310.161638684522608 (code B ref 47310); Mon, 22 Mar 2021 04:21:02 +0000 Received: (at 47310) by debbugs.gnu.org; 22 Mar 2021 04:20:45 +0000 Received: from localhost ([127.0.0.1]:55826 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3Y-0005sX-Pl for submit@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:45 -0400 Received: from mail-qv1-f47.google.com ([209.85.219.47]:42851) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3T-0005rs-Pv for 47310@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:41 -0400 Received: by mail-qv1-f47.google.com with SMTP id 30so8039871qva.9 for <47310@debbugs.gnu.org>; Sun, 21 Mar 2021 21:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=obJLkDQgYRFgIjtmq0TR44mKYsgvPVnZ/e3hq8EHR7M=; b=L/jrlR3u5RXJPOWDWiU5rXL/2ES3Pm/KwQYQBgjq9twl0tbSneJfXeJggujJii3rXa V5d5d5UjudVlsms8U1F4WQ1TkafpF1ZQDG4PShGqtSzdSNI7Qvnuikim441lSKJjt/L/ OZAZBGZifaLIe9khuq6VPFiIeIvcPNcfIQMgtiG7JT03ghQQ3HiCtpnLBENPGyLEliL6 jAECstFn8BBnsO08qx73kxwdYr4vyzXGjcEVpKCYqfqC6IAyYD8ubfGk9jnxYsuUaN2M LwazM1q8cd0tytW0RoYbzAn3lFQDikDK6wIsaGZ+y5fdzmCsffTBcIxvVphU5ADZuWfF /+GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=obJLkDQgYRFgIjtmq0TR44mKYsgvPVnZ/e3hq8EHR7M=; b=ql+P4lqiKFH9VTKUu92wAQEm9XH2q1PTvcvQcJBWdaKyClX7/CRUM9ucWSZ86TVqUQ 38be5bGJ2i18oboc7I3nfflv1PMrWOMWGLsw8wiPovsqwpq+ggABRDfK55KR3C90ZJuV /6Bgn+P5DsApr5bmZBkc7XWWCZDRp0eTjEhXS5JuyxTHLsPy+ewzxue54wjmjFE82nmi 49EIsUCKZ/XSGYCOCGwM7NjypakeWC9XqWyd9mZosTXcCWo+CNIR3V0/ZwBw6EmnwJe5 cx8oe9vkfAvI6ZqpCcFQeQ0wmo+9VNr2PtjPAdT6AzJRLiNrQSwpW6g4HJL+pf1xOF3E 55ow== X-Gm-Message-State: AOAM531bKDWiU4cn4CFmqbYC1hGOV2PDXv8+ax7do3fcznkmiN8TUarY Df7Wm0DMxf477u6cdvS52ngZemYBoIQ= X-Google-Smtp-Source: ABdhPJxM8NfVQ6T0RUR/AXSpz1Bg0heVLFG352KPAqC1GCnpXChkzwOQoc4qqYrOh6bEmDNaQIXFQw== X-Received: by 2002:a0c:bdaf:: with SMTP id n47mr19443732qvg.48.1616386833721; Sun, 21 Mar 2021 21:20:33 -0700 (PDT) Received: from localhost.localdomain (dsl-10-134-34.b2b2c.ca. [72.10.134.34]) by smtp.gmail.com with ESMTPSA id z8sm8340478qtn.12.2021.03.21.21.20.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Mar 2021 21:20:33 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 22 Mar 2021 00:20:20 -0400 Message-Id: <20210322042022.17385-2-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210322042022.17385-1-maxim.cournoyer@gmail.com> References: <20210322042022.17385-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/import/go.scm (%strict-tokenizer?): Set parameter to #t. (go-path-escape): Redefine to prevent inlining. (http-get*): Replace by ... (http-fetch*): this ... (json-fetch*): New procedure. (go.pkg.dev-info): Use http-fetch*. (go-package-licenses): Rewrite in terms of go.pkg.dev-info. (go-package-description): Likewise. (go-package-synopsis): Likewise. (fetch-go.mod): Use the memoized http-fetch*. (parse-go.mod): Adjust to receive content as a string. (fetch-module-meta-data): Adjust to use http-fetch*. (go-module->guix-package): Adjust to the modified fetch-go.mod return value. [inputs]: Use propagated inputs, which is the most common situations for Go libraries. [description]: Beautify description. [licenses]: Do no check for #f. The result of the license parsing is always a list. * tests/go.scm: Adjust following above changes. --- guix/import/go.scm | 213 ++++++++++++++++++++++++++---------------- guix/import/utils.scm | 4 +- tests/go.scm | 75 +++++++-------- 3 files changed, 170 insertions(+), 122 deletions(-) diff --git a/guix/import/go.scm b/guix/import/go.scm index 7452b4c903..2ef2d872e8 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -32,7 +32,7 @@ #:use-module (guix http-client) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix memoization) - #:autoload (htmlprag) (html->sxml) ;from Guile-Lib + #:use-module (htmlprag) ;from Guile-Lib #:autoload (guix git) (update-cached-checkout) #:autoload (gcrypt hash) (open-hash-port hash-algorithm sha256) #:autoload (guix serialization) (write-file) @@ -42,20 +42,28 @@ #:use-module (ice-9 rdelim) #:use-module (ice-9 receive) #:use-module (ice-9 regex) + #:use-module (ice-9 textual-ports) #:use-module ((rnrs io ports) #:select (call-with-port)) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-2) #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) - #:use-module (sxml xpath) + #:use-module (sxml match) + #:use-module ((sxml xpath) #:renamer (lambda (s) + (if (eq? 'filter s) + 'xfilter + s))) #:use-module (web client) #:use-module (web response) #:use-module (web uri) - #:export (go-path-escape - go-module->guix-package + #:export (go-module->guix-package go-module-recursive-import)) +;;; Parameterize htmlprag to parse valid HTML more reliably. +(%strict-tokenizer? #t) + ;;; Commentary: ;;; ;;; (guix import go) attempts to make it easier to create Guix package @@ -89,6 +97,14 @@ ;;; Code: +(define http-fetch* + ;; Like http-fetch, but memoized and returning the body as a string. + (memoize (lambda args + (call-with-port (apply http-fetch args) get-string-all)))) + +(define json-fetch* + (memoize json-fetch)) + (define (go-path-escape path) "Escape a module path by replacing every uppercase letter with an exclamation mark followed with its lowercase equivalent, as per the module @@ -98,54 +114,73 @@ https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)." (string-append "!" (string-downcase (match:substring occurrence)))) (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) +;; Prevent inlining of this procedure, which is accessed by unit tests. +(set! go-path-escape go-path-escape) + +(define (go.pkg.dev-info name) + (http-fetch* (string-append "https://pkg.go.dev/" name))) + (define (go-module-latest-version goproxy-url module-path) "Fetch the version number of the latest version for MODULE-PATH from the given GOPROXY-URL server." - (assoc-ref (json-fetch (format #f "~a/~a/@latest" goproxy-url - (go-path-escape module-path))) + (assoc-ref (json-fetch* (format #f "~a/~a/@latest" goproxy-url + (go-path-escape module-path))) "Version")) - (define (go-package-licenses name) "Retrieve the list of licenses that apply to NAME, a Go package or module -name (e.g. \"github.com/golang/protobuf/proto\"). The data is scraped from -the https://pkg.go.dev/ web site." - (let*-values (((url) (string-append "https://pkg.go.dev/" name - "?tab=licenses")) - ((response body) (http-get url)) - ;; Extract the text contained in a h2 child node of any - ;; element marked with a "License" class attribute. - ((select) (sxpath `(// (* (@ (equal? (class "License")))) - h2 // *text*)))) - (and (eq? (response-code response) 200) - (match (select (html->sxml body)) - (() #f) ;nothing selected - (licenses licenses))))) - -(define (go.pkg.dev-info name) - (http-get (string-append "https://pkg.go.dev/" name))) -(define go.pkg.dev-info* - (memoize go.pkg.dev-info)) +name (e.g. \"github.com/golang/protobuf/proto\")." + (let* ((body (go.pkg.dev-info (string-append name "?tab=licenses"))) + ;; Extract the text contained in a h2 child node of any + ;; element marked with a "License" class attribute. + (select (sxpath `(// (* (@ (equal? (class "License")))) + h2 // *text*)))) + (select (html->sxml body)))) + +(define (sxml->texi sxml-node) + "A very basic SXML to Texinfo converter which attempts to preserve HTML +formatting and links as text." + (sxml-match sxml-node + ((strong ,text) + (format #f "@strong{~a}" text)) + ((a (@ (href ,url)) ,text) + (format #f "@url{~a,~a}" url text)) + ((code ,text) + (format #f "@code{~a}" text)) + (,something-else something-else))) (define (go-package-description name) "Retrieve a short description for NAME, a Go package name, -e.g. \"google.golang.org/protobuf/proto\". The data is scraped from the -https://pkg.go.dev/ web site." - (let*-values (((response body) (go.pkg.dev-info* name)) - ;; Extract the text contained in a h2 child node of any - ;; element marked with a "License" class attribute. - ((select) (sxpath - `(// (section - (@ (equal? (class "Documentation-overview")))) - (p 1))))) - (and (eq? (response-code response) 200) - (match (select (html->sxml body)) - (() #f) ;nothing selected - (((p . strings)) - ;; The paragraph text is returned as a list of strings embedding - ;; newline characters. Join them and strip the newline - ;; characters. - (string-delete #\newline (string-join strings))))))) +e.g. \"google.golang.org/protobuf/proto\"." + (let* ((body (go.pkg.dev-info name)) + (sxml (html->sxml body)) + (overview ((sxpath + `(// + (* (@ (equal? (class "Documentation-overview")))) + (p 1))) sxml)) + ;; Sometimes, the first paragraph just contains images/links that + ;; has only "\n" for text. The following filter is designed to + ;; omit it. + (contains-text? (lambda (node) + (remove string-null? + (map string-trim-both + (filter (node-typeof? '*text*) + (cdr node)))))) + (select-content (sxpath + `(// + (* (@ (equal? (class "UnitReadme-content")))) + div // p ,(xfilter contains-text?)))) + ;; Fall-back to use content; this is less desirable as it is more + ;; verbose, but not every page has an overview. + (description (if (not (null? overview)) + overview + (select-content sxml))) + (description* (and (not (null? description)) + (first description)))) + (match description* + (() #f) ;nothing selected + ((p elements ...) + (apply string-append (filter string? (map sxml->texi elements))))))) (define (go-package-synopsis module-name) "Retrieve a short synopsis for a Go module named MODULE-NAME, @@ -153,17 +188,17 @@ e.g. \"google.golang.org/protobuf\". The data is scraped from the https://pkg.go.dev/ web site." ;; Note: Only the *module* (rather than package) page has the README title ;; used as a synopsis on the https://pkg.go.dev web site. - (let*-values (((response body) (go.pkg.dev-info* module-name)) - ;; Extract the text contained in a h2 child node of any - ;; element marked with a "License" class attribute. - ((select) (sxpath - `(// (div (@ (equal? (class "UnitReadme-content")))) - // h3 *text*)))) - (and (eq? (response-code response) 200) - (match (select (html->sxml body)) - (() #f) ;nothing selected - ((title more ...) ;title is the first string of the list - (string-trim-both title)))))) + (let* ((url (string-append "https://pkg.go.dev/" module-name)) + (body (http-fetch* url)) + ;; Extract the text contained in a h2 child node of any + ;; element marked with a "License" class attribute. + (select-title (sxpath + `(// (div (@ (equal? (class "UnitReadme-content")))) + // h3 *text*)))) + (match (select-title (html->sxml body)) + (() #f) ;nothing selected + ((title more ...) ;title is the first string of the list + (string-trim-both title))))) (define (list->licenses licenses) "Given a list of LICENSES mostly following the SPDX conventions, return the @@ -188,13 +223,13 @@ corresponding Guix license or 'unknown-license!" 'unknown-license!))) licenses)) -(define (fetch-go.mod goproxy-url module-path version) - "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH -and VERSION." - (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url +(define (fetch-go.mod goproxy module-path version) + "Fetch go.mod from the given GOPROXY server for the given MODULE-PATH +and VERSION and return an input port." + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy (go-path-escape module-path) (go-path-escape version)))) - (http-fetch url))) + (http-fetch* url))) (define %go.mod-require-directive-rx ;; A line in a require directive is composed of a module path and @@ -215,9 +250,8 @@ and VERSION." "[[:blank:]]+" "=>" "[[:blank:]]+" "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) -(define (parse-go.mod port) - "Parse the go.mod file accessible via the input PORT, returning a list of -requirements." +(define (parse-go.mod content) + "Parse the go.mod file CONTENT, returning a list of requirements." (define-record-type (make-results requirements replacements) results? @@ -228,7 +262,7 @@ requirements." (define (toplevel results) "Main parser, RESULTS is a pair of alist serving as accumulator for all encountered requirements and replacements." - (let ((line (read-line port))) + (let ((line (read-line))) (cond ((eof-object? line) ;; parsing ended, give back the result @@ -254,7 +288,7 @@ requirements." (toplevel results))))) (define (in-require results) - (let ((line (read-line port))) + (let ((line (read-line))) (cond ((eof-object? line) ;; this should never happen here but we ignore silently @@ -266,7 +300,7 @@ requirements." (in-require (require-directive results line)))))) (define (in-replace results) - (let ((line (read-line port))) + (let ((line (read-line))) (cond ((eof-object? line) ;; this should never happen here but we ignore silently @@ -305,7 +339,9 @@ requirements." (($ requirements replaced) (make-results (alist-cons module-path version requirements) replaced))))) - (let ((results (toplevel (make-results '() '())))) + (let ((results (with-input-from-string content + (lambda _ + (toplevel (make-results '() '())))))) (match results (($ requirements replaced) ;; At last we remove replaced modules from the requirements list @@ -324,8 +360,10 @@ requirements." (url-prefix vcs-url-prefix) (root-regex vcs-root-regex) (type vcs-type)) + (define (make-vcs prefix regexp type) - (%make-vcs prefix (make-regexp regexp) type)) + (%make-vcs prefix (make-regexp regexp) type)) + (define known-vcs ;; See the following URL for the official Go equivalent: ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 @@ -384,6 +422,14 @@ hence the need to derive this information." "." "-") "/" "-")))) +(define (strip-.git-suffix/maybe repo-url) + "Strip a repository URL '.git' suffix from REPO-URL if hosted at GitHub." + (match repo-url + ((and (? (cut string-prefix? "https://github.com" <>)) + (? (cut string-suffix? ".git" <>))) + (string-drop-right repo-url 4)) + (_ repo-url))) + (define-record-type (make-module-meta import-prefix vcs repo-root) module-meta? @@ -396,21 +442,22 @@ hence the need to derive this information." because goproxy servers don't currently provide all the information needed to build a package." ;; - (let* ((port (http-fetch (format #f "https://~a?go-get=1" module-path))) + (let* ((meta-data (http-fetch* (format #f "https://~a?go-get=1" module-path))) (select (sxpath `(// head (meta (@ (equal? (name "go-import")))) // content)))) - (match (select (call-with-port port html->sxml)) - (() #f) ;nothing selected + (match (select (html->sxml meta-data)) + (() #f) ;nothing selected (((content content-text)) (match (string-split content-text #\space) ((root-path vcs repo-url) - (make-module-meta root-path (string->symbol vcs) repo-url))))))) + (make-module-meta root-path (string->symbol vcs) + (strip-.git-suffix/maybe repo-url)))))))) -(define (module-meta-data-repo-url meta-data goproxy-url) +(define (module-meta-data-repo-url meta-data goproxy) "Return the URL where the fetcher which will be used can download the source." (if (member (module-meta-vcs meta-data) '(fossil mod)) - goproxy-url + goproxy (module-meta-repo-root meta-data))) ;; XXX: Copied from (guix scripts hash). @@ -463,6 +510,9 @@ control system is being used." (method git-fetch) (uri (git-reference (url ,vcs-repo-url) + ;; This is done because the version field of the package, + ;; which the generated quoted expression refers to, has been + ;; stripped of any 'v' prefixed. (commit ,(if (and plain-version? v-prefixed?) '(string-append "v" version) '(go-version->git-ref version))))) @@ -502,8 +552,8 @@ control system is being used." (define* (go-module->guix-package module-path #:key (goproxy-url "https://proxy.golang.org")) (let* ((latest-version (go-module-latest-version goproxy-url module-path)) - (port (fetch-go.mod goproxy-url module-path latest-version)) - (dependencies (map car (call-with-port port parse-go.mod))) + (content (fetch-go.mod goproxy-url module-path latest-version)) + (dependencies (map car (parse-go.mod content))) (guix-name (go-module->guix-package-name module-path)) (root-module-path (module-path->repository-root module-path)) ;; The VCS type and URL are not included in goproxy information. For @@ -524,14 +574,17 @@ control system is being used." (build-system go-build-system) (arguments '(#:import-path ,root-module-path)) - ,@(maybe-inputs (map go-module->guix-package-name dependencies)) + ,@(maybe-propagated-inputs + (map go-module->guix-package-name dependencies)) (home-page ,(format #f "https://~a" root-module-path)) (synopsis ,synopsis) - (description ,description) - (license ,(match (and=> licenses list->licenses) - ((license) license) - ((licenses ...) `(list ,@licenses)) - (x x)))) + (description ,(and=> description beautify-description)) + (license ,(match (list->licenses licenses) + (() #f) ;unknown license + ((license) ;a single license + license) + ((license ...) ;a list of licenses + `(list ,@license))))) dependencies))) (define go-module->guix-package* (memoize go-module->guix-package)) diff --git a/guix/import/utils.scm b/guix/import/utils.scm index e55258e306..21fafad388 100644 --- a/guix/import/utils.scm +++ b/guix/import/utils.scm @@ -445,8 +445,8 @@ obtain a node's uniquely identifying \"key\"." "Return a list of package expressions for PACKAGE-NAME and all its dependencies, sorted in topological order. For each package, call (REPO->GUIX-PACKAGE NAME :KEYS version repo), which should return a -package expression and a list of dependencies; call (GUIX-NAME NAME) to -obtain the Guix package name corresponding to the upstream name." +package expression and a list of dependencies; call (GUIX-NAME PACKAGE-NAME) +to obtain the Guix package name corresponding to the upstream name." (define-record-type (make-node name version package dependencies) node? diff --git a/tests/go.scm b/tests/go.scm index 6ab99f508a..fa8fa7a2a6 100644 --- a/tests/go.scm +++ b/tests/go.scm @@ -180,13 +180,9 @@ require github.com/kr/pretty v0.2.1 (define (testing-parse-mod name expected input) (define (inf? p1 p2) (stringguix-package" '(package - (name "go-github-com-go-check-check") - (version "0.0.0-20201130134442-10cb98267c6c") - (source - (origin - (method git-fetch) - (uri (git-reference - (url "https://github.com/go-check/check.git") - (commit (go-version->git-ref version)))) - (file-name (git-file-name name version)) - (sha256 - (base32 - "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))) - (build-system go-build-system) - (arguments - (quote (#:import-path "github.com/go-check/check"))) - (inputs - (quasiquote (("go-github-com-kr-pretty" - (unquote go-github-com-kr-pretty))))) - (home-page "https://github.com/go-check/check") - (synopsis "Instructions") - (description #f) - (license license:bsd-2)) + (name "go-github-com-go-check-check") + (version "0.0.0-20201130134442-10cb98267c6c") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/go-check/check") + (commit (go-version->git-ref version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))) + (build-system go-build-system) + (arguments + '(#:import-path "github.com/go-check/check")) + (propagated-inputs + `(("go-github-com-kr-pretty" ,go-github-com-kr-pretty))) + (home-page "https://github.com/go-check/check") + (synopsis "Instructions") + (description "Package check is a rich testing extension for Go's testing \ +package.") + (license license:bsd-2)) ;; Replace network resources with sample data. (call-with-temporary-directory (lambda (checkout) (mock ((web client) http-get (mock-http-get fixtures-go-check-test)) - (mock ((guix http-client) http-fetch - (mock-http-fetch fixtures-go-check-test)) - (mock ((guix git) update-cached-checkout - (lambda* (url #:key ref) - ;; Return an empty directory and its hash. - (values checkout - (nix-base32-string->bytevector - "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5") - #f))) - (go-module->guix-package "github.com/go-check/check"))))))) + (mock ((guix http-client) http-fetch + (mock-http-fetch fixtures-go-check-test)) + (mock ((guix git) update-cached-checkout + (lambda* (url #:key ref) + ;; Return an empty directory and its hash. + (values checkout + (nix-base32-string->bytevector + "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5") + #f))) + (go-module->guix-package "github.com/go-check/check"))))))) (test-end "go") - From patchwork Mon Mar 22 04:20:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 28010 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 02B9727BC57; Mon, 22 Mar 2021 04:21:25 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,SPF_HELO_PASS, T_DKIM_INVALID,URIBL_BLOCKED,URIBL_SBL,URIBL_SBL_A autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 4B2C627BC58 for ; Mon, 22 Mar 2021 04:21:21 +0000 (GMT) Received: from localhost ([::1]:33204 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOC48-0007Lw-71 for patchwork@mira.cbaines.net; Mon, 22 Mar 2021 00:21:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48258) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOC3s-0007HR-31 for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44287) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOC3r-00012H-76 for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOC3r-0005tx-3f for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47310] [PATCH 3/4] import: go: Add an option to use pinned versions. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 22 Mar 2021 04:21:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47310 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47310@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 47310-submit@debbugs.gnu.org id=B47310.161638685222628 (code B ref 47310); Mon, 22 Mar 2021 04:21:03 +0000 Received: (at 47310) by debbugs.gnu.org; 22 Mar 2021 04:20:52 +0000 Received: from localhost ([127.0.0.1]:55830 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3a-0005si-Du for submit@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:52 -0400 Received: from mail-qv1-f52.google.com ([209.85.219.52]:37393) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3U-0005rt-II for 47310@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:42 -0400 Received: by mail-qv1-f52.google.com with SMTP id dc12so7164496qvb.4 for <47310@debbugs.gnu.org>; Sun, 21 Mar 2021 21:20:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t/6m7OBCu6RJyx7JDmnb2jgDXOf3Q+TlguWZhAuaSHY=; b=q6M6/OvK43C2kP5BW4e4f8STZOwHYYXZgYkMAGm0Gp/tfzJaVvnG7Bn27R4JjKLPoE vFAGfCb04xXUjAFn2GGYUqwx9LX1fXxA1SHgJGfwsc9tq2YZTWIsCw0eiLXcztnRUDBK fU35xQX/IOISEp/ILZOkeG9iOqwM7IVP2KswEzsmHNrL0EJYpRfUo7zQd2Z1SZ8rdW8C IFKyWu6c5yyYBuFYTao5RzruzNqlB5sEvouIQZOOQjNOyxUaATJmnB7GW6NcrvGsHTmt KYRE9n5HZdgH3Dlr8E1ia87OOnJ4ATVyPaOZe6IsNGM26Xgc9VyvgDhNjqo2iHL05fZj mpeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t/6m7OBCu6RJyx7JDmnb2jgDXOf3Q+TlguWZhAuaSHY=; b=K69ovmtYV8sUv2xFUaJrkVe6+vbrqBPhCUiHbY6kU7eK2wex2/W+B1kXUOTtEb/1eU HPs8LTGCl6DkNuqyWieWXmZxLctx77M0wiSIfXUpmrMGurym01icxEGRKWTqJvepRMNZ +/LUBC3vPI0xvcdjDUSUUCAdCiEccd07oZzx/unsqSVM2RXp2cpxTsy/WJdTsL2Q6c7j rDMn8YTwdxfxhMxNEFXuinIqqwvA8Y9qbiTn0iT/2uu46LIesO2AT2dNwMKj4N3NRBBF CuHItlSFYsYG7VM42iBm6XOT2tEMo2CFfto0hGwtQmU7YQngM89f+KkE0d6Q5ErlJo9s cUqQ== X-Gm-Message-State: AOAM531LOsR3IMZGtqrcRmb7H/+Yo++Te+bedvwXKmaun3GLQPR3Chs+ F4fLbeqk8hln+yGfQUftmM+wOaVEVA8= X-Google-Smtp-Source: ABdhPJwhBmFqj4ejna/aseQO54bBzkLncvOc/SKAH/tfvqbQrnG3OGftCkIZD4vlY3f4f2W6abtXDA== X-Received: by 2002:a05:6214:9c9:: with SMTP id dp9mr19655765qvb.34.1616386834386; Sun, 21 Mar 2021 21:20:34 -0700 (PDT) Received: from localhost.localdomain (dsl-10-134-34.b2b2c.ca. [72.10.134.34]) by smtp.gmail.com with ESMTPSA id z8sm8340478qtn.12.2021.03.21.21.20.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Mar 2021 21:20:34 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 22 Mar 2021 00:20:21 -0400 Message-Id: <20210322042022.17385-3-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210322042022.17385-1-maxim.cournoyer@gmail.com> References: <20210322042022.17385-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches The ability to pin versions is handy when having to deal to packages that bootstrap themselves through a chain of former versions. Not using pinned versions in these case could introduce dependency cycles. * guix/build-system/go.scm (guix) (%go-version-rx): Rename to... (%go-pseudo-version-rx): ... this. Simplify the regular expression, which in turns makes it more robust. * guix/build-system/go.scm (go-version->git-ref): Adjust following the above rename. (go-pseudo-version?): New predicate. (go-module-latest-version): Rename to ... (go-module-version-string): ... this. Rename goproxy-url argument to just goproxy. Add a VERSION keyword argument, update docstring and adjust to have it used. (go-module-available-versions): New procedure. (%go.mod-require-directive-rx): Document regexp. (parse-go.mod): Harmonize the way dependencies are recorded to a list of lists rather than a list of pairs, as done for other importers. Rewrite to directly pass multiple values rather than a record object. Filter the replaced modules in a functional style. (go-module->guix-package): Add docstring. [version, pin-versions?]: New arguments. Rename the GOPROXY-URL argument to GOPROXY. Adjust to the new returned value of fetch-go.mod, which is a string. Fail when the provided version doesn't exist. Return a list dependencies and their versions when in pinned versions mode, else just the dependencies. (go-module-recursive-import)[version, pin-versions?]: New arguments. Honor the new arguments and guard against network errors. * guix/scripts/import/go.scm (%default-options): Register a default value for the goproxy argument. (show-help): Document that a version can be specified. Remove the --version argument and add a --pin-versions argument. (%options)[version]: Remove option. [pin-versions]: Add option. (guix-import-go): Adjust so the version provided from the module name is honored, along the new pin-versions? argument. * tests/go.scm: Adjust and add new tests. --- guix/build-system/go.scm | 24 ++-- guix/import/go.scm | 239 ++++++++++++++++++++++--------------- guix/scripts/import/go.scm | 70 ++++++----- tests/go.scm | 64 +++++----- 4 files changed, 234 insertions(+), 163 deletions(-) diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm index 0e2c1cd2ee..8f55796e86 100644 --- a/guix/build-system/go.scm +++ b/guix/build-system/go.scm @@ -31,6 +31,7 @@ go-build go-build-system + go-pseudo-version? go-version->git-ref)) ;; Commentary: @@ -40,17 +41,19 @@ ;; ;; Code: -(define %go-version-rx +(define %go-pseudo-version-rx + ;; Match only the end of the version string; this is so that matching the + ;; more complex leading semantic version pattern is not required. (make-regexp (string-append - "(v?[0-9]\\.[0-9]\\.[0-9])" ;"v" prefix can be omitted in version prefix - "(-|-pre\\.0\\.|-0\\.)" ;separator - "([0-9]{14})-" ;timestamp - "([0-9A-Fa-f]{12})"))) ;commit hash + "([0-9]{14}-)" ;timestamp + "([0-9A-Fa-f]{12})" ;commit hash + "(\\+incompatible)?$"))) ;optional +incompatible tag (define (go-version->git-ref version) "Parse VERSION, a \"pseudo-version\" as defined at , and extract the commit hash from -it, defaulting to full VERSION if a pseudo-version pattern is not recognized." +it, defaulting to full VERSION (stripped from the \"+incompatible\" suffix if +present) if a pseudo-version pattern is not recognized." ;; A module version like v1.2.3 is introduced by tagging a revision in the ;; underlying source repository. Untagged revisions can be referred to ;; using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef, where @@ -65,11 +68,16 @@ it, defaulting to full VERSION if a pseudo-version pattern is not recognized." (if (string-suffix? "+incompatible" version) (string-drop-right version 13) version)) - (match (regexp-exec %go-version-rx version))) + (match (regexp-exec %go-pseudo-version-rx version))) (if match - (match:substring match 4) + (match:substring match 2) version))) +(define (go-pseudo-version? version) + "True if VERSION is a Go pseudo-version, i.e., a version string made of a +commit hash and its date rather than a proper release tag." + (regexp-exec %go-pseudo-version-rx version)) + (define %go-build-system-modules ;; Build-side modules imported and used by default. `((guix build go-build-system) diff --git a/guix/import/go.scm b/guix/import/go.scm index 2ef2d872e8..2376336265 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -49,6 +49,7 @@ #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) #:use-module (sxml match) #:use-module ((sxml xpath) #:renamer (lambda (s) (if (eq? 'filter s) @@ -91,9 +92,7 @@ ;;; assumption that there will be no collision. ;;; TODO list -;;; - get correct hash in vcs->origin -;;; - print partial result during recursive imports (need to catch -;;; exceptions) +;;; - get correct hash in vcs->origin for Mercurial and Subversion ;;; Code: @@ -120,12 +119,26 @@ https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)." (define (go.pkg.dev-info name) (http-fetch* (string-append "https://pkg.go.dev/" name))) -(define (go-module-latest-version goproxy-url module-path) - "Fetch the version number of the latest version for MODULE-PATH from the -given GOPROXY-URL server." - (assoc-ref (json-fetch* (format #f "~a/~a/@latest" goproxy-url - (go-path-escape module-path))) - "Version")) +(define* (go-module-version-string goproxy name #:key version) + "Fetch the version string of the latest version for NAME from the given +GOPROXY server, or for VERSION when specified." + (let ((file (if version + (string-append "@v/" version ".info") + "@latest"))) + (assoc-ref (json-fetch* (format #f "~a/~a/~a" + goproxy (go-path-escape name) file)) + "Version"))) + +(define* (go-module-available-versions goproxy name) + "Retrieve the available versions for a given module from the module proxy. +Versions are being returned **unordered** and may contain different versioning +styles for the same package." + (let* ((url (string-append goproxy "/" (go-path-escape name) "/@v/list")) + (body (http-fetch* url)) + (versions (remove string-null? (string-split body #\newline)))) + (if (null? versions) + (list (go-module-version-string goproxy name)) ;latest version + versions))) (define (go-package-licenses name) "Retrieve the list of licenses that apply to NAME, a Go package or module @@ -237,119 +250,119 @@ and VERSION and return an input port." ;; the end. (make-regexp (string-append - "^[[:blank:]]*" - "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" - "([[:blank:]]+//.*)?"))) + "^[[:blank:]]*([^[:blank:]]+)[[:blank:]]+" ;the module path + "([^[:blank:]]+)" ;the version + "([[:blank:]]+//.*)?"))) ;an optional comment (define %go.mod-replace-directive-rx ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline ;; | ModulePath [ Version ] "=>" ModulePath Version newline . (make-regexp (string-append - "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" - "[[:blank:]]+" "=>" "[[:blank:]]+" - "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + "([^[:blank:]]+)" ;the module path + "([[:blank:]]+([^[:blank:]]+))?" ;optional version + "[[:blank:]]+=>[[:blank:]]+" + "([^[:blank:]]+)" ;the file or module path + "([[:blank:]]+([^[:blank:]]+))?"))) ;the version (if a module path) (define (parse-go.mod content) "Parse the go.mod file CONTENT, returning a list of requirements." - (define-record-type - (make-results requirements replacements) - results? - (requirements results-requirements) - (replacements results-replacements)) ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar ;; which we think necessary for our use case. - (define (toplevel results) - "Main parser, RESULTS is a pair of alist serving as accumulator for - all encountered requirements and replacements." + (define (toplevel requirements replaced) + "This is the main parser. The results are accumulated in THE REQUIREMENTS +and REPLACED lists." (let ((line (read-line))) (cond ((eof-object? line) ;; parsing ended, give back the result - results) + (values requirements replaced)) ((string=? line "require (") ;; a require block begins, delegate parsing to IN-REQUIRE - (in-require results)) + (in-require requirements replaced)) ((string=? line "replace (") ;; a replace block begins, delegate parsing to IN-REPLACE - (in-replace results)) + (in-replace requirements replaced)) ((string-prefix? "require " line) - ;; a standalone require directive - (let* ((stripped-line (string-drop line 8)) - (new-results (require-directive results stripped-line))) - (toplevel new-results))) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8))) + (call-with-values + (lambda () + (require-directive requirements replaced stripped-line)) + toplevel))) ((string-prefix? "replace " line) - ;; a standalone replace directive - (let* ((stripped-line (string-drop line 8)) - (new-results (replace-directive results stripped-line))) - (toplevel new-results))) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8))) + (call-with-values + (lambda () + (replace-directive requirements replaced stripped-line)) + toplevel))) (#t ;; unrecognised line, ignore silently - (toplevel results))))) + (toplevel requirements replaced))))) - (define (in-require results) + (define (in-require requirements replaced) (let ((line (read-line))) (cond ((eof-object? line) ;; this should never happen here but we ignore silently - results) + (values requirements replaced)) ((string=? line ")") ;; end of block, coming back to toplevel - (toplevel results)) + (toplevel requirements replaced)) (#t - (in-require (require-directive results line)))))) + (call-with-values (lambda () + (require-directive requirements replaced line)) + in-require))))) - (define (in-replace results) + (define (in-replace requirements replaced) (let ((line (read-line))) (cond ((eof-object? line) ;; this should never happen here but we ignore silently - results) + (values requirements replaced)) ((string=? line ")") ;; end of block, coming back to toplevel - (toplevel results)) + (toplevel requirements replaced)) (#t - (in-replace (replace-directive results line)))))) - - (define (replace-directive results line) - "Extract replaced modules and new requirements from replace directive - in LINE and add to RESULTS." - (match results - (($ requirements replaced) - (let* ((rx-match (regexp-exec %go.mod-replace-directive-rx line)) - (module-path (match:substring rx-match 1)) - (version (match:substring rx-match 3)) - (new-module-path (match:substring rx-match 4)) - (new-version (match:substring rx-match 6)) - (new-replaced (alist-cons module-path version replaced)) - (new-requirements - (if (string-match "^\\.?\\./" new-module-path) - requirements - (alist-cons new-module-path new-version requirements)))) - (make-results new-requirements new-replaced))))) - (define (require-directive results line) - "Extract requirement from LINE and add it to RESULTS." + (call-with-values (lambda () + (replace-directive requirements replaced line)) + in-replace))))) + + (define (replace-directive requirements replaced line) + "Extract replaced modules and new requirements from the replace directive +in LINE and add them to the REQUIREMENTS and REPLACED lists." + (let* ((rx-match (regexp-exec %go.mod-replace-directive-rx line)) + (module-path (match:substring rx-match 1)) + (version (match:substring rx-match 3)) + (new-module-path (match:substring rx-match 4)) + (new-version (match:substring rx-match 6)) + (new-replaced (cons (list module-path version) replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (cons (list new-module-path new-version) requirements)))) + (values new-requirements new-replaced))) + + (define (require-directive requirements replaced line) + "Extract requirement from LINE and augment the REQUIREMENTS and REPLACED +lists." (let* ((rx-match (regexp-exec %go.mod-require-directive-rx line)) (module-path (match:substring rx-match 1)) - ;; we saw double-quoted string in the wild without escape - ;; sequences so we just trim the quotes + ;; Double-quoted strings were seen in the wild without escape + ;; sequences; trim the quotes to be on the safe side. (module-path (string-trim-both module-path #\")) (version (match:substring rx-match 2))) - (match results - (($ requirements replaced) - (make-results (alist-cons module-path version requirements) replaced))))) - - (let ((results (with-input-from-string content - (lambda _ - (toplevel (make-results '() '())))))) - (match results - (($ requirements replaced) - ;; At last we remove replaced modules from the requirements list - (fold - (lambda (replacedelem requirements) - (alist-delete! (car replacedelem) requirements)) - requirements - replaced))))) + (values (cons (list module-path version) requirements) replaced))) + + (with-input-from-string content + (lambda () + (receive (requirements replaced) + (toplevel '() '()) + ;; At last remove the replaced modules from the requirements list. + (remove (lambda (r) + (assoc (car r) replaced)) + requirements))))) ;; Prevent inlining of this procedure, which is accessed by unit tests. (set! parse-go.mod parse-go.mod) @@ -550,17 +563,32 @@ control system is being used." vcs-type vcs-repo-url))))) (define* (go-module->guix-package module-path #:key - (goproxy-url "https://proxy.golang.org")) - (let* ((latest-version (go-module-latest-version goproxy-url module-path)) - (content (fetch-go.mod goproxy-url module-path latest-version)) - (dependencies (map car (parse-go.mod content))) + (goproxy "https://proxy.golang.org") + version + pin-versions?) + "Return the package S-expression corresponding to MODULE-PATH at VERSION, a Go package. +The meta-data is fetched from the GOPROXY server and https://pkg.go.dev/. +When VERSION is unspecified, the latest version available is used." + (let* ((available-versions (go-module-available-versions goproxy module-path)) + (version* (or version + (go-module-version-string goproxy module-path))) ;latest + ;; Pseudo-versions do not appear in the versions list; skip the + ;; following check. + (_ (unless (or (go-pseudo-version? version*) + (member version* available-versions)) + (error (format #f "error: version ~s is not available +hint: use one of the following available versions ~a\n" + version* available-versions)))) + (content (fetch-go.mod goproxy module-path version*)) + (dependencies+versions (parse-go.mod content)) + (dependencies (map car dependencies+versions)) (guix-name (go-module->guix-package-name module-path)) (root-module-path (module-path->repository-root module-path)) ;; The VCS type and URL are not included in goproxy information. For ;; this we need to fetch it from the official module page. (meta-data (fetch-module-meta-data root-module-path)) (vcs-type (module-meta-vcs meta-data)) - (vcs-repo-url (module-meta-data-repo-url meta-data goproxy-url)) + (vcs-repo-url (module-meta-data-repo-url meta-data goproxy)) (synopsis (go-package-synopsis root-module-path)) (description (go-package-description module-path)) (licenses (go-package-licenses module-path))) @@ -568,14 +596,14 @@ control system is being used." `(package (name ,guix-name) ;; Elide the "v" prefix Go uses - (version ,(string-trim latest-version #\v)) + (version ,(string-trim version* #\v)) (source - ,(vcs->origin vcs-type vcs-repo-url latest-version)) + ,(vcs->origin vcs-type vcs-repo-url version*)) (build-system go-build-system) (arguments '(#:import-path ,root-module-path)) - ,@(maybe-propagated-inputs - (map go-module->guix-package-name dependencies)) + ,@(maybe-propagated-inputs (map go-module->guix-package-name + dependencies)) (home-page ,(format #f "https://~a" root-module-path)) (synopsis ,synopsis) (description ,(and=> description beautify-description)) @@ -585,16 +613,37 @@ control system is being used." license) ((license ...) ;a list of licenses `(list ,@license))))) - dependencies))) + (if pin-versions? + dependencies+versions + dependencies)))) (define go-module->guix-package* (memoize go-module->guix-package)) (define* (go-module-recursive-import package-name - #:key (goproxy-url "https://proxy.golang.org")) + #:key (goproxy "https://proxy.golang.org") + version + pin-versions?) + (recursive-import package-name - #:repo->guix-package (lambda* (name . _) - (go-module->guix-package* - name - #:goproxy-url goproxy-url)) - #:guix-name go-module->guix-package-name)) + #:repo->guix-package + (lambda* (name #:key version repo) + ;; Disable output buffering so that the following warning gets printed + ;; consistently. + (setvbuf (current-error-port) 'none) + (guard (c ((http-get-error? c) + (warning (G_ "Failed to import package ~s. +reason: ~s could not be fetched: HTTP error ~a (~s). +This package and its dependencies won't be imported.~%") + name + (uri->string (http-get-error-uri c)) + (http-get-error-code c) + (http-get-error-reason c)) + (values '() '()))) + (receive (package-sexp dependencies) + (go-module->guix-package* name #:goproxy goproxy + #:version version + #:pin-versions? pin-versions?) + (values package-sexp dependencies)))) + #:guix-name go-module->guix-package-name + #:version version)) diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm index afdba4e8f1..33d2470ce1 100644 --- a/guix/scripts/import/go.scm +++ b/guix/scripts/import/go.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2020 Katherine Cox-Buday +;;; Copyright © 2020 Katherine Cox-Buday +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -27,28 +28,30 @@ #:use-module (srfi srfi-37) #:use-module (ice-9 match) #:use-module (ice-9 format) + #:use-module (ice-9 receive) #:export (guix-import-go)) - + ;;; ;;; Command-line options. ;;; (define %default-options - '()) + '((goproxy . "https://proxy.golang.org"))) (define (show-help) - (display (G_ "Usage: guix import go PACKAGE-PATH -Import and convert the Go module for PACKAGE-PATH.\n")) + (display (G_ "Usage: guix import go PACKAGE-PATH[@VERSION] +Import and convert the Go module for PACKAGE-PATH. Optionally, a version +can be specified after the arobas (@) character.\n")) (display (G_ " -h, --help display this help and exit")) (display (G_ " - -V, --version display version information and exit")) - (display (G_ " - -r, --recursive generate package expressions for all Go modules\ - that are not yet in Guix")) + -r, --recursive generate package expressions for all Go modules +that are not yet in Guix")) (display (G_ " -p, --goproxy=GOPROXY specify which goproxy server to use")) + (display (G_ " + --pin-versions use the exact versions of a module's dependencies")) (newline) (show-bug-report-information)) @@ -58,9 +61,6 @@ Import and convert the Go module for PACKAGE-PATH.\n")) (lambda args (show-help) (exit 0))) - (option '(#\V "version") #f #f - (lambda args - (show-version-and-exit "guix import go"))) (option '(#\r "recursive") #f #f (lambda (opt name arg result) (alist-cons 'recursive #t result))) @@ -69,9 +69,12 @@ Import and convert the Go module for PACKAGE-PATH.\n")) (alist-cons 'goproxy (string->symbol arg) (alist-delete 'goproxy result)))) + (option '("pin-versions") #f #f + (lambda (opt name arg result) + (alist-cons 'pin-versions? #t result))) %standard-import-options)) - + ;;; ;;; Entry point. ;;; @@ -93,25 +96,28 @@ Import and convert the Go module for PACKAGE-PATH.\n")) (_ #f)) (reverse opts)))) (match args - ((module-name) - (if (assoc-ref opts 'recursive) - (map (match-lambda - ((and ('package ('name name) . rest) pkg) - `(define-public ,(string->symbol name) - ,pkg)) - (_ #f)) - (go-module-recursive-import module-name - #:goproxy-url - (or (assoc-ref opts 'goproxy) - "https://proxy.golang.org"))) - (let ((sexp (go-module->guix-package module-name - #:goproxy-url - (or (assoc-ref opts 'goproxy) - "https://proxy.golang.org")))) - (unless sexp - (leave (G_ "failed to download meta-data for module '~a'~%") - module-name)) - sexp))) + ((spec) ;e.g., github.com/golang/protobuf@v1.3.1 + (receive (name version) + (package-name->name+version spec) + (let ((arguments (list name + #:goproxy (assoc-ref opts 'goproxy) + #:version version + #:pin-versions? + (assoc-ref opts 'pin-versions?)))) + (if (assoc-ref opts 'recursive) + ;; Recursive import. + (map (match-lambda + ((and ('package ('name name) . rest) pkg) + `(define-public ,(string->symbol name) + ,pkg)) + (_ #f)) + (apply go-module-recursive-import arguments)) + ;; Single import. + (let ((sexp (apply go-module->guix-package arguments))) + (unless sexp + (leave (G_ "failed to download meta-data for module '~a'~%") + module-name)) + sexp))))) (() (leave (G_ "too few arguments~%"))) ((many ...) diff --git a/tests/go.scm b/tests/go.scm index fa8fa7a2a6..e5780e68b0 100644 --- a/tests/go.scm +++ b/tests/go.scm @@ -19,7 +19,7 @@ ;;; Summary ;; Tests for guix/import/go.scm -(define-module (test-import-go) +(define-module (tests-import-go) #:use-module (guix base32) #:use-module (guix build-system go) #:use-module (guix import go) @@ -147,7 +147,8 @@ require github.com/kr/pretty v0.2.1 ("https://pkg.go.dev/github.com/go-check/check" . ,pkg.go.dev) ("https://pkg.go.dev/github.com/go-check/check?tab=licenses" - . ,pkg.go.dev-licence)))) + . ,pkg.go.dev-licence) + ("https://proxy.golang.org/github.com/go-check/check/@v/list" . "")))) (test-begin "go") @@ -169,6 +170,12 @@ require github.com/kr/pretty v0.2.1 "daa7c04131f5" (go-version->git-ref "v1.2.4-0.20191109021931-daa7c04131f5")) +(test-assert "go-pseudo-version? multi-digit version number" + (go-pseudo-version? "v1.23.1-0.20200526195155-81db48ad09cc")) + +(test-assert "go-pseudo-version? semantic version with rc" + (go-pseudo-version? "v1.4.0-rc.4.0.20200313231945-b860323f09d0")) + ;;; Unit tests for (guix import go) (test-equal "go-path-escape" @@ -185,37 +192,38 @@ require github.com/kr/pretty v0.2.1 (sort ((@@ (guix import go) parse-go.mod) input) inf?))) (testing-parse-mod "parse-go.mod-simple" - '(("good/thing" . "v1.4.5") - ("new/thing/v2" . "v2.3.4") - ("other/thing" . "v1.0.2")) + '(("good/thing" "v1.4.5") + ("new/thing/v2" "v2.3.4") + ("other/thing" "v1.0.2")) fixture-go-mod-simple) (testing-parse-mod "parse-go.mod-with-block" - '(("A" . "v1") - ("B" . "v1.0.0") - ("C" . "v1.0.0") - ("D" . "v1.2.3") - ("E" . "dev")) + '(("A" "v1") + ("B" "v1.0.0") + ("C" "v1.0.0") + ("D" "v1.2.3") + ("E" "dev")) fixture-go-mod-with-block) -(testing-parse-mod "parse-go.mod-complete" - '(("github.com/corp/arbitrary-repo" . "v0.0.2") - ("quoted.example.com/abitrary/repo" . "v0.0.2") - ("one.example.com/abitrary/repo" . "v1.1.111") - ("hub.jazz.net/git/user/project/sub/directory" . "v1.1.19") - ("hub.jazz.net/git/user/project" . "v1.1.18") - ("launchpad.net/~user/project/branch/sub/directory" . "v1.1.17") - ("launchpad.net/~user/project/branch" . "v1.1.16") - ("launchpad.net/project/series/sub/directory" . "v1.1.15") - ("launchpad.net/project/series" . "v1.1.14") - ("launchpad.net/project" . "v1.1.13") - ("bitbucket.org/user/project/sub/directory" . "v1.11.21") - ("bitbucket.org/user/project" . "v1.11.20") - ("k8s.io/kubernetes/subproject" . "v1.1.101") - ("github.com/user/project/sub/directory" . "v1.1.12") - ("github.com/user/project" . "v1.1.11") - ("github.com/go-check/check" . "v0.0.0-20140225173054-eb6ee6f84d0a")) - fixture-go-mod-complete) +(testing-parse-mod + "parse-go.mod-complete" + '(("github.com/corp/arbitrary-repo" "v0.0.2") + ("quoted.example.com/abitrary/repo" "v0.0.2") + ("one.example.com/abitrary/repo" "v1.1.111") + ("hub.jazz.net/git/user/project/sub/directory" "v1.1.19") + ("hub.jazz.net/git/user/project" "v1.1.18") + ("launchpad.net/~user/project/branch/sub/directory" "v1.1.17") + ("launchpad.net/~user/project/branch" "v1.1.16") + ("launchpad.net/project/series/sub/directory" "v1.1.15") + ("launchpad.net/project/series" "v1.1.14") + ("launchpad.net/project" "v1.1.13") + ("bitbucket.org/user/project/sub/directory" "v1.11.21") + ("bitbucket.org/user/project" "v1.11.20") + ("k8s.io/kubernetes/subproject" "v1.1.101") + ("github.com/user/project/sub/directory" "v1.1.12") + ("github.com/user/project" "v1.1.11") + ("github.com/go-check/check" "v0.0.0-20140225173054-eb6ee6f84d0a")) + fixture-go-mod-complete) ;;; End-to-end tests for (guix import go) (define (mock-http-fetch testcase) From patchwork Mon Mar 22 04:20:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 28012 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 97CBA27BC57; Mon, 22 Mar 2021 04:27:25 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,SPF_HELO_PASS, T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 347EF27BC58 for ; Mon, 22 Mar 2021 04:27:24 +0000 (GMT) Received: from localhost ([::1]:44916 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOC9z-00044G-Cf for patchwork@mira.cbaines.net; Mon, 22 Mar 2021 00:27:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOC3s-0007HO-3e for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44286) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOC3q-000124-Qv for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOC3q-0005tq-Mg for guix-patches@gnu.org; Mon, 22 Mar 2021 00:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47310] [PATCH 4/4] import: go: Append version to symbol name in the pinned version mode. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 22 Mar 2021 04:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47310 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47310@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 47310-submit@debbugs.gnu.org id=B47310.161638684622615 (code B ref 47310); Mon, 22 Mar 2021 04:21:02 +0000 Received: (at 47310) by debbugs.gnu.org; 22 Mar 2021 04:20:46 +0000 Received: from localhost ([127.0.0.1]:55828 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3Z-0005sb-QG for submit@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:46 -0400 Received: from mail-qv1-f42.google.com ([209.85.219.42]:38448) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOC3U-0005ru-OO for 47310@debbugs.gnu.org; Mon, 22 Mar 2021 00:20:42 -0400 Received: by mail-qv1-f42.google.com with SMTP id t5so8065714qvs.5 for <47310@debbugs.gnu.org>; Sun, 21 Mar 2021 21:20:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5wPZpTUdLlzl7ct1KPNgprUvMeIORWb5BGSuBXVERgI=; b=uMVj5Wd4Cpss0TrGPYmWIDcrHXxSH9ERrKRxUHKlRe7rzOQbPRv1Y5wCJVdKyojYqm 4VlnZaiDcCOIWynIoYBc5v5G8zk1zpirSaIEDvRZ80pedHOHxRSgm5PYh9miU3vX0sHp 8geSA3PiC2FnG2rk+nCnr1DsYEzwlfFF1xBw76spaIjs9FNCJHUwRkNNi2ALPPVtV3HW 1KhXLnGTpfetuloQHzrQ3hClmAcykSpkyfTTChhsmMbxR7jr9378nyOwgHi0rMNfdM0C 6txRIH+eK1D8nZsEBg9vqORkpKRakXsjvtntn+T/ZEY7u3j3dwsi2tcyp4NzCeS3sGfj lyHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5wPZpTUdLlzl7ct1KPNgprUvMeIORWb5BGSuBXVERgI=; b=VdOwg/QZAzlsuV98mnWCKJhyQTzQFYhJUVRy/HOSA55XMEo5a1TRy32Fj9E7wsoIuV IEl8lG6CJ/kE90rRcdxzE2BuSIpbimSHC3KQ2lDgIgVR81t2PIuiKm2iJ8peaTyb1Pau uhmZLZLHoZBSyRWkzgjcT561uJQxxqEp6mL5Ylvfcqcdb6hg4wN/2K1ngVCN+mSBa4Lj WZI6wEgJnlICmshcWr2m6KwdCyIqDx2SaUQyIkIpZoVj/hnLzjqZ6KaOBQEtaQz+fTUX uM8ei50Kv+9AuaVOslJaGmGd0uay1op7Cr3acOLXDlslTItkC+ObJOuNY1jKnsra+bWb 9NHw== X-Gm-Message-State: AOAM531agiE6oSaKLhItLZ1onhauiG2uVjv37nECClKeYqeHqLwH89jo h21O9j32cAH8IKQe8STSVRPNUnjUkVg= X-Google-Smtp-Source: ABdhPJwZLmDK9PbUkdTPKMYECfXaoIZaUUtK1ozFK3PGS7jIV2kO6/K95GdUh5oRuYgIqLE+EUuuJA== X-Received: by 2002:ad4:584d:: with SMTP id de13mr19764048qvb.17.1616386835016; Sun, 21 Mar 2021 21:20:35 -0700 (PDT) Received: from localhost.localdomain (dsl-10-134-34.b2b2c.ca. [72.10.134.34]) by smtp.gmail.com with ESMTPSA id z8sm8340478qtn.12.2021.03.21.21.20.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Mar 2021 21:20:34 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 22 Mar 2021 00:20:22 -0400 Message-Id: <20210322042022.17385-4-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210322042022.17385-1-maxim.cournoyer@gmail.com> References: <20210322042022.17385-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches This allows importing packages with complicated version specific dependency chains without the package symbol names colliding. * doc/guix.texi (Invoking guix import): Document the --pin-versions option. Mention that a specific version can be imported. Remove the experimental warning. * guix/import/go.scm (go-module->guix-package-name)[version]: Add optional argument. (go-module->guix-package): Conditionally use dependencies whose symbol include their version, based no the value of the PIN-VERSIONS? argument. * guix/import/utils.scm (package->definition): Add a new case where the full version string is appended to the package symbol. * guix/scripts/import.scm (guix-import): Correctly print forms starting with '(define-public [...]'. * guix/scripts/import/go.scm (guix-import-go): Conditionally include the version in the package symbols defined. --- doc/guix.texi | 14 +++++++++++--- guix/import/go.scm | 34 +++++++++++++++++++++++----------- guix/import/utils.scm | 7 +++++-- guix/scripts/import.scm | 3 ++- guix/scripts/import/go.scm | 17 ++++++++++------- 5 files changed, 51 insertions(+), 24 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 386169b2a5..be20215638 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11520,13 +11520,13 @@ Select the given repository (a repository name). Possible values include: Import metadata for a Go module using @uref{https://proxy.golang.org, proxy.golang.org}. -This importer is highly experimental. See the source code for more info -about the current state. - @example guix import go gopkg.in/yaml.v2 @end example +It is possible to use a package specification with a @code{@VERSION} +suffix to import a specific version. + Additional options include: @table @code @@ -11535,6 +11535,14 @@ Additional options include: Traverse the dependency graph of the given upstream package recursively and generate package expressions for all those packages that are not yet in Guix. +@item --pin-versions +When using this option, the importer preserves the exact versions of the +Go modules dependencies instead of using their latest available +versions. This can be useful when attempting to import packages that +recursively depend on former versions of themselves to build. When +using this mode, the symbol of the package is made by appending the +version to its name, so that multiple versions of the same package can +coexist. @end table @end table diff --git a/guix/import/go.scm b/guix/import/go.scm index 2376336265..d8b6fddac5 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -427,13 +427,17 @@ hence the need to derive this information." (vcs-qualified-module-path->root-repo-url module-path) module-path)) -(define (go-module->guix-package-name module-path) - "Converts a module's path to the canonical Guix format for Go packages." +(define* (go-module->guix-package-name module-path #:optional version) + "Converts a module's path to the canonical Guix format for Go packages. +Optionally include a VERSION string to append to the name." (string-downcase (string-append "go-" (string-replace-substring (string-replace-substring module-path "." "-") - "/" "-")))) + "/" "-") + (if version + (string-append "-" version) + "")))) (define (strip-.git-suffix/maybe repo-url) "Strip a repository URL '.git' suffix from REPO-URL if hosted at GitHub." @@ -572,6 +576,8 @@ When VERSION is unspecified, the latest version available is used." (let* ((available-versions (go-module-available-versions goproxy module-path)) (version* (or version (go-module-version-string goproxy module-path))) ;latest + ;; Elide the "v" prefix Go uses. + (strip-v-prefix (cut string-trim <> #\v)) ;; Pseudo-versions do not appear in the versions list; skip the ;; following check. (_ (unless (or (go-pseudo-version? version*) @@ -581,7 +587,9 @@ hint: use one of the following available versions ~a\n" version* available-versions)))) (content (fetch-go.mod goproxy module-path version*)) (dependencies+versions (parse-go.mod content)) - (dependencies (map car dependencies+versions)) + (dependencies (if pin-versions? + dependencies+versions + (map car dependencies+versions))) (guix-name (go-module->guix-package-name module-path)) (root-module-path (module-path->repository-root module-path)) ;; The VCS type and URL are not included in goproxy information. For @@ -595,23 +603,27 @@ hint: use one of the following available versions ~a\n" (values `(package (name ,guix-name) - ;; Elide the "v" prefix Go uses - (version ,(string-trim version* #\v)) + (version ,(strip-v-prefix version*)) (source ,(vcs->origin vcs-type vcs-repo-url version*)) (build-system go-build-system) (arguments '(#:import-path ,root-module-path)) - ,@(maybe-propagated-inputs (map go-module->guix-package-name - dependencies)) + ,@(maybe-propagated-inputs + (map (match-lambda + ((name version) + (go-module->guix-package-name name (strip-v-prefix version))) + (name + (go-module->guix-package-name name))) + dependencies)) (home-page ,(format #f "https://~a" root-module-path)) (synopsis ,synopsis) (description ,(and=> description beautify-description)) (license ,(match (list->licenses licenses) - (() #f) ;unknown license - ((license) ;a single license + (() #f) ;unknown license + ((license) ;a single license license) - ((license ...) ;a list of licenses + ((license ...) ;a list of licenses `(list ,@license))))) (if pin-versions? dependencies+versions diff --git a/guix/import/utils.scm b/guix/import/utils.scm index 21fafad388..21352841b8 100644 --- a/guix/import/utils.scm +++ b/guix/import/utils.scm @@ -273,8 +273,9 @@ snippet generated is for regular inputs." (maybe-inputs package-names output #:type 'propagated)) (define* (package->definition guix-package #:optional append-version?/string) - "If APPEND-VERSION?/STRING is #t, append the package's major+minor -version. If APPEND-VERSION?/string is a string, append this string." + "If APPEND-VERSION?/STRING is #t, append the package's major+minor version. +If it is the symbol 'full, append the package's complete version. If +APPEND-VERSION?/string is a string, append this string." (match guix-package ((or ('package ('name name) ('version version) . rest) @@ -286,6 +287,8 @@ version. If APPEND-VERSION?/string is a string, append this string." (string-append name "-" append-version?/string)) ((eq? append-version?/string #t) (string-append name "-" (version-major+minor version))) + ((eq? 'full append-version?/string) + (string-append name "-" version)) (else name))) ,guix-package)))) diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index 1d2b45d942..98554ef79b 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -119,7 +119,8 @@ Run IMPORTER with ARGS.\n")) (current-output-port)))))) (match (apply (resolve-importer importer) args) ((and expr (or ('package _ ...) - ('let _ ...))) + ('let _ ...) + ('define-public _ ...))) (print expr)) ((? list? expressions) (for-each (lambda (expr) diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm index 33d2470ce1..04b07f80cc 100644 --- a/guix/scripts/import/go.scm +++ b/guix/scripts/import/go.scm @@ -22,9 +22,11 @@ #:use-module (guix utils) #:use-module (guix scripts) #:use-module (guix import go) + #:use-module (guix import utils) #:use-module (guix scripts import) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) + #:use-module (srfi srfi-26) #:use-module (srfi srfi-37) #:use-module (ice-9 match) #:use-module (ice-9 format) @@ -94,7 +96,12 @@ that are not yet in Guix")) (('argument . value) value) (_ #f)) - (reverse opts)))) + (reverse opts))) + ;; Append the full version to the package symbol name when using + ;; pinned versions. + (package->definition* (if (assoc-ref opts 'pin-versions?) + (cut package->definition <> 'full) + package->definition))) (match args ((spec) ;e.g., github.com/golang/protobuf@v1.3.1 (receive (name version) @@ -106,18 +113,14 @@ that are not yet in Guix")) (assoc-ref opts 'pin-versions?)))) (if (assoc-ref opts 'recursive) ;; Recursive import. - (map (match-lambda - ((and ('package ('name name) . rest) pkg) - `(define-public ,(string->symbol name) - ,pkg)) - (_ #f)) + (map package->definition* (apply go-module-recursive-import arguments)) ;; Single import. (let ((sexp (apply go-module->guix-package arguments))) (unless sexp (leave (G_ "failed to download meta-data for module '~a'~%") module-name)) - sexp))))) + (package->definition* sexp)))))) (() (leave (G_ "too few arguments~%"))) ((many ...)