[bug#44178] Create importer for Go modules

This patch add a `guix import go` command.

It was tested with several big repositories and seems to mostly work for
the import part (because building Guix packages is an other story). There
is still bugs blocking e.g. use of any k8s.io modules.

* guix/import/go.scm: Created Go Importer
* guix/scripts/import.scm: Created Go Importer Subcommand
* guix/import/go.scm (importers): Added Go Importer Subcommand

Signed-off-by: Francois Joulaud <francois.joulaud@radiofrance.com>
---
The patch is a rebased and modified version of the one proposed by
Katherine Cox-Buday.

Notable modifications are :
- move from (guix json) to (json)
- new parse-go.mod with no "set!" and parsing some go.mod which were in
  error before
- adding comments (maybe too much comments)
- renamed SCS to VCS to be in accordance with vocabulary in use in Guix
  and in Go worlds
- replacing escape-capital-letters by Helio Machado's go-path-escape
- no pruning of major version in go module names as they are considered
  as completely different artefacts by Go programmers
- fixed recursive-import probably broken by the rebase
- force usage of url-fetch from (guix build download)

I would be happy to hear about problems and perspective for this patch and
will now focus on my next step which is actually building any package.

Hope I CCed the right persons, I am not really aware of applicable
netiquette here.

Interdiff :
  diff --git a/guix/import/go.scm b/guix/import/go.scm
  index 61009f3565..7f5f300f0a 100644
  --- a/guix/import/go.scm
  +++ b/guix/import/go.scm
  @@ -1,5 +1,7 @@
   ;;; GNU Guix --- Functional package management for GNU
   ;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com>
  +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com>
  +;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com>
   ;;;
   ;;; This file is part of GNU Guix.
   ;;;
  @@ -16,6 +18,21 @@
   ;;; You should have received a copy of the GNU General Public License
   ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

  +;;; (guix import golang) wants to make easier to create Guix package
  +;;; declaration for Go modules.
  +;;;
  +;;; Modules in Go are "collection of related Go packages" which are
  +;;; "the unit of source code interchange and versioning".
  +;;; Modules are generally hosted in a repository.
  +;;;
  +;;; At this point it should handle correctly modules which
  +;;; - have only Go dependencies;
  +;;; - use go.mod;
  +;;; - and are accessible from proxy.golang.org (or configured GOPROXY).
  +;;;
  +;;; We translate Go module paths  to a Guix package name under the
  +;;; assumption that there will be no collision.
  +
   (define-module (guix import go)
     #:use-module (ice-9 match)
     #:use-module (ice-9 rdelim)
  @@ -23,7 +40,7 @@
     #:use-module (ice-9 regex)
     #:use-module (srfi srfi-1)
     #:use-module (srfi srfi-9)
  -  #:use-module (guix json)
  +  #:use-module (json)
     #:use-module ((guix download) #:prefix download:)
     #:use-module (guix import utils)
     #:use-module (guix import json)
  @@ -33,88 +50,129 @@
     #:use-module ((guix licenses) #:prefix license:)
     #:use-module (guix base16)
     #:use-module (guix base32)
  -  #:use-module (guix build download)
  +  #:use-module ((guix build download) #:prefix build-download:)
     #:use-module (web uri)

     #:export (go-module->guix-package
               go-module-recursive-import
               infer-module-root))

  -(define (escape-capital-letters s)
  -  "To avoid ambiguity when serving from case-insensitive file systems, the
  -$module and $version elements are case-encoded by replacing every uppercase
  -letter with an exclamation mark followed by the corresponding lower-case
  -letter."
  -  (let ((escaped-string (string)))
  -    (string-for-each-index
  -     (lambda (i)
  -       (let ((c (string-ref s i)))
  -         (set! escaped-string
  -           (string-concatenate
  -            (list escaped-string
  -                  (if (char-upper-case? c) "!" "")
  -                  (string (char-downcase c)))))))
  -     s)
  -    escaped-string))
  +(define (go-path-escape path)
  +  "Escape a module path by replacing every uppercase letter with an exclamation
  +mark followed with its lowercase equivalent, as per the module Escaped Paths
  +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths"
  +  (define (escape occurrence)
  +    (string-append "!" (string-downcase (match:substring occurrence))))
  +  (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post))
  +

   (define (fetch-latest-version goproxy-url module-path)
     "Fetches the version number of the latest version for MODULE-PATH from the
   given GOPROXY-URL server."
     (assoc-ref
      (json-fetch (format #f "~a/~a/@latest" goproxy-url
  -                       (escape-capital-letters module-path)))
  +                       (go-path-escape module-path)))
      "Version"))

   (define (fetch-go.mod goproxy-url module-path version file)
     "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH
   and VERSION."
  -  (url-fetch (format #f "~a/~a/@v/~a.mod" goproxy-url
  -                     (escape-capital-letters module-path)
  -                     (escape-capital-letters version))
  -             file
  -             #:print-build-trace? #f))
  +  (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url
  +                     (go-path-escape module-path)
  +                     (go-path-escape version))))
  +    (parameterize ((current-output-port (current-error-port)))
  +      (build-download:url-fetch url
  +                                file
  +                                #:print-build-trace? #f))))

   (define (parse-go.mod go.mod-path)
  -  "Parses a go.mod file and returns an alist of module path to version."
  +  "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of
  +requirements from it."
  +  ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar
  +  ;; which we think necessary for our use case.
  +  (define (toplevel results)
  +    "Main parser, RESULTS is a pair of alist serving as accumulator for
  +     all encountered requirements and replacements."
  +    (let ((line (read-line)))
  +      (cond
  +       ((eof-object? line)
  +        ;; parsing ended, give back the result
  +        results)
  +       ((string=? line "require (")
  +        ;; a require block begins, delegate parsing to IN-REQUIRE
  +        (in-require results))
  +       ((string-prefix? "require " line)
  +        ;; a require directive by itself
  +        (let* ((stripped-line (string-drop line 8))
  +               (new-results (require-directive results stripped-line)))
  +          (toplevel new-results)))
  +       ((string-prefix? "replace " line)
  +        ;; a replace directive by itself
  +        (let* ((stripped-line (string-drop line 8))
  +               (new-results (replace-directive results stripped-line)))
  +          (toplevel new-results)))
  +       (#t
  +        ;; unrecognised line, ignore silently
  +        (toplevel results)))))
  +  (define (in-require results)
  +    (let ((line (read-line)))
  +      (cond
  +       ((eof-object? line)
  +        ;; this should never happen here but we ignore silently
  +        results)
  +       ((string=? line ")")
  +        ;; end of block, coming back to toplevel
  +        (toplevel results))
  +       (#t
  +        (in-require (require-directive results line))))))
  +  (define (replace-directive results line)
  +    "Extract replaced modules and new requirements from replace directive
  +    in LINE and add to RESULTS."
  +    ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline
  +    ;;             | ModulePath [ Version ] "=>" ModulePath Version newline .
  +    (let* ((requirements (car results))
  +           (replaced (cdr results))
  +           (re (string-concatenate
  +                '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"
  +                  "[[:blank:]]+" "=>" "[[:blank:]]+"
  +                  "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?")))
  +           (match (string-match re line))
  +           (module-path (match:substring match 1))
  +           (version (match:substring match 3))
  +           (new-module-path (match:substring match 4))
  +           (new-version (match:substring match 6))
  +           (new-replaced (acons module-path version replaced))
  +           (new-requirements
  +            (if (string-match "^\\.?\\./" new-module-path)
  +                requirements
  +                (acons new-module-path new-version requirements))))
  +      (cons new-requirements new-replaced)))
  +  (define (require-directive results line)
  +    "Extract requirement from LINE and add it to RESULTS."
  +    (let* ((requirements (car results))
  +           (replaced (cdr results))
  +           ;; A line in a require directive is composed of a module path and
  +           ;; a version separated by whitespace and an optionnal '//' comment at
  +           ;; the end.
  +           (re (string-concatenate
  +                '("^[[:blank:]]*"
  +                  "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)"
  +                  "([[:blank:]]+//.*)?")))
  +           (match (string-match re line))
  +           (module-path (match:substring match 1))
  +           (version (match:substring match 2)))
  +      (cons (acons module-path version requirements) replaced)))
     (with-input-from-file go.mod-path
       (lambda ()
  -      (let ((in-require? #f)
  -            (requirements (list)))
  -        (do ((line (read-line) (read-line)))
  -            ((eof-object? line))
  -          (set! line (string-trim line))
  -          ;; The parser is either entering, within, exiting, or after the
  -          ;; require block. The Go toolchain is trustworthy so edge-cases like
  -          ;; double-entry, etc. need not complect the parser.
  -          (cond
  -           ((string=? line "require (")
  -            (set! in-require? #t))
  -           ((and in-require? (string=? line ")"))
  -            (set! in-require? #f))
  -           (in-require?
  -            (let* ((requirement (string-split line #\space))
  -                   ;; Modules should be unquoted
  -                   (module-path (string-delete #\" (car requirement)))
  -                   (version (list-ref requirement 1)))
  -              (set! requirements (acons module-path version requirements))))
  -           ((string-prefix? "replace" line)
  -            (let* ((requirement (string-split line #\space))
  -                   (module-path (list-ref requirement 1))
  -                   (new-module-path (list-ref requirement 3))
  -                   (version (list-ref requirement 4)))
  -              (set! requirements (assoc-remove! requirements module-path))
  -              (set! requirements (acons new-module-path version requirements))))))
  -        requirements))))
  -
  -(define (module-path-without-major-version module-path)
  -  "Go modules can be appended with a major version indicator,
  -e.g. /v3. Sometimes it is desirable to work with the root module path. For
  -instance, for a module path github.com/foo/bar/v3 this function returns
  -github.com/foo/bar."
  -  (let ((m (string-match "(.*)\\/v[0-9]+$" module-path)))
  -    (if m
  -        (match:substring m 1)
  -        module-path)))
  +      (let* ((results (toplevel '(() . ())))
  +             (requirements (car results))
  +             (replaced (cdr results)))
  +        ;; At last we remove replaced modules from the requirements list
  +        (fold
  +         (lambda (replacedelem requirements)
  +             (alist-delete! (car replacedelem) requirements))
  +         requirements
  +         replaced)))))

   (define (infer-module-root module-path)
     "Go modules can be defined at any level of a repository's tree, but querying
  @@ -124,38 +182,42 @@ root path from its path. For a set of well-known forges, the pattern of what
   consists of a module's root page is known before hand."
     ;; See the following URL for the official Go equivalent:
     ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087
  -  (define-record-type <scs>
  -    (make-scs url-prefix root-regex type)
  -    scs?
  -    (url-prefix	scs-url-prefix)
  -    (root-regex scs-root-regex)
  -    (type	scs-type))
  -  (let* ((known-scs
  +  ;;
  +  ;; FIXME: handle module path with VCS qualifier as described in
  +  ;; https://golang.org/ref/mod#vcs-find and
  +  ;; https://golang.org/cmd/go/#hdr-Remote_import_paths
  +  (define-record-type <vcs>
  +    (make-vcs url-prefix root-regex type)
  +    vcs?
  +    (url-prefix vcs-url-prefix)
  +    (root-regex vcs-root-regex)
  +    (type vcs-type))
  +  (let* ((known-vcs
             (list
  -           (make-scs
  +           (make-vcs
               "github.com"
               "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$"
               'git)
  -           (make-scs
  +           (make-vcs
               "bitbucket.org"
               "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`"
               'unknown)
  -           (make-scs
  +           (make-vcs
               "hub.jazz.net/git/"
               "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$"
               'git)
  -           (make-scs
  +           (make-vcs
               "git.apache.org"
               "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$"
               'git)
  -           (make-scs
  +           (make-vcs
               "git.openstack.org"
               "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$"
               'git)))
  -         (scs (find (lambda (scs) (string-prefix? (scs-url-prefix scs) module-path))
  -                    known-scs)))
  -    (if scs
  -        (match:substring (string-match (scs-root-regex scs) module-path) 1)
  +         (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path))
  +                    known-vcs)))
  +    (if vcs
  +        (match:substring (string-match (vcs-root-regex vcs) module-path) 1)
           module-path)))

   (define (to-guix-package-name module-path)
  @@ -164,8 +226,7 @@ consists of a module's root page is known before hand."
      (string-append "go-"
                     (string-replace-substring
                      (string-replace-substring
  -                    ;; Guix has its own field for version
  -                    (module-path-without-major-version module-path)
  +                    module-path
                       "." "-")
                      "/" "-"))))

  @@ -173,7 +234,9 @@ consists of a module's root page is known before hand."
     "Fetches module meta-data from a module's landing page. This is necessary
   because goproxy servers don't currently provide all the information needed to
   build a package."
  -  (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get=1" module-path))))
  +  ;; FIXME: This code breaks on k8s.io which have a meta tag splitted
  +  ;; on several lines
  +  (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path))))
            (module-metadata #f)
            (meta-tag-prefix "<meta name=\"go-import\" content=\"")
            (meta-tag-prefix-length (string-length meta-tag-prefix)))
  @@ -185,7 +248,7 @@ build a package."
             (let* ((start (+ meta-tag-index meta-tag-prefix-length))
                    (end (string-index line #\" start)))
               (set! module-metadata
  -              (string-split (substring/shared line start end) #\space))))))
  +                  (string-split (substring/shared line start end) #\space))))))
       (close-port port)
       module-metadata))

  @@ -244,7 +307,7 @@ control system is being used."
               (dependencies (map car (parse-go.mod temp)))
               (guix-name (to-guix-package-name module-path))
               (root-module-path (infer-module-root module-path))
  -            ;; SCS type and URL are not included in goproxy information. For
  +            ;; VCS type and URL are not included in goproxy information. For
               ;; this we need to fetch it from the official module page.
               (meta-data (fetch-module-meta-data root-module-path))
               (scs-type (module-meta-data-scs meta-data))
  @@ -268,9 +331,10 @@ control system is being used."

   (define* (go-module-recursive-import package-name
                                        #:key (goproxy-url "https://proxy.golang.org"))
  -  (recursive-import package-name #f
  -                    #:repo->guix-package
  -                    (lambda (name _)
  -                      (go-module->guix-package name
  -                                               #:goproxy-url goproxy-url))
  -                    #:guix-name to-guix-package-name))
  +  (recursive-import
  +   package-name
  +   #:repo->guix-package (lambda* (name . _)
  +                          (go-module->guix-package
  +                           name
  +                           #:goproxy-url goproxy-url))
  +   #:guix-name to-guix-package-name))

 guix/import/go.scm         | 340 +++++++++++++++++++++++++++++++++++++
 guix/scripts/import.scm    |   2 +-
 guix/scripts/import/go.scm | 118 +++++++++++++
 3 files changed, 459 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/go.scm
 create mode 100644 guix/scripts/import/go.scm

Message ID	20210123212742.m2thdeuzdvgpkgeo@fjo-extia-HPdeb.example.avalenn.eu
State	Accepted
Headers	show Return-Path: <guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org> Subject: [bug#44178] [PATCH] Create importer for Go modules References: <87sga5kpdp.fsf@gmail.com> Resent-From: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= <Francois.JOULAUD@radiofrance.com> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org> Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 23 Jan 2021 23:24:01 +0000 Resent-Message-ID: <handler.44178.B44178.161144423125918@debbugs.gnu.org> Resent-Sender: help-debbugs@gnu.org Thread-Topic: [PATCH] Create importer for Go modules Thread-Index: AQHW8c+fh3EThKxKn0aE+FBPMWbUYQ== Date: Sat, 23 Jan 2021 21:35:08 +0000 Message-ID: <20210123212742.m2thdeuzdvgpkgeo@fjo-extia-HPdeb.example.avalenn.eu> In-Reply-To: <CANe01w5DxwrF+APLbcNMaUhs4h05EXC1HdRJ_YT8CRy6ULxUYw@mail.gmail.com> Accept-Language: fr-FR, en-US Content-Language: en-US Content-Type: text/plain; charset="iso-8859-1" Content-ID: <69EE4CB1F6B8DF45B593FA40B59EECD4@FRAP264.PROD.OUTLOOK.COM> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Precedence: list Cc: Helio Machado <0x2b3bfa0@gmail.com>, Katherine Cox-Buday <cox.katherine.e@gmail.com> Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" <guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org> Reply-to: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= <Francois.JOULAUD@radiofrance.com> From: guix-patches--- via <guix-patches@gnu.org> X-getmail-retrieved-from-mailbox: Patches
Series	[bug#44178] Create importer for Go modules \| expand [bug#44178] Create importer for Go modules

Context	Check	Description
cbaines/submitting builds	success
cbaines/comparison	success	View comparision
cbaines/git branch	success	View Git branch
cbaines/applying patch	success	View Laminar job
cbaines/issue	success	View issue

[bug#44178] Create importer for Go modules

Checks

Commit Message

Comments

Patch