Message ID | 20210123212742.m2thdeuzdvgpkgeo@fjo-extia-HPdeb.example.avalenn.eu |
---|---|
State | Accepted |
Headers | show |
Series | [bug#44178] Create importer for Go modules | expand |
Context | Check | Description |
---|---|---|
cbaines/submitting builds | success | |
cbaines/comparison | success | View comparision |
cbaines/git branch | success | View Git branch |
cbaines/applying patch | success | View Laminar job |
cbaines/issue | success | View issue |
Thanks so much for the patches, Helio, Joulaud! I apologize for the long delay before looking at this again. My time right now is extremely limited due to COVID-19 related childcare and activities. I was negligent and left a patch to bitrot on my computer[1]. This patch supersedes it. In addition to the things this patch corrects, I was/am working on a few other bugs: - There are valid Go Module paths which when queried will not serve the requisite meta tag. I had modified `fetch-module-meta-data` to recursively walk up the module path searching for a valid meta tag (see diff[1]). - I think Joulaud's patch covers this, but replacements relative to a module are a possibility. - For reasons Joulaud calls out, a simple line-parser of the HTML for a module is not sufficient. Since we are pulling pages down from the wider internet, we should fall back on a library made for parsing HTML so we handle any edge-cases (e.g. meta tags split across lines). I am currently looking at `sxml`, and if that doesn't pan out `htmlprag` - Some module homepages issue HTTP redirects. Last time I tested this, `http-fetch` does not handle this properly. I think that covers everything. I have pushed everything (including Joulaud's patch with appropriate attribution) here[2]. I am admittedly new at using email to organize code changes, but using a forge seems easier. [1] https://github.com/guix-mirror/guix/commit/cce35c6d68a9bddf9558e85d2cb88be323da9247 [2] https://github.com/kat-co/guix/tree/create-go-importer Can I suggest we coordinate there, or is that too much of an imposition? - Katherine JOULAUD François <Francois.JOULAUD@radiofrance.com> writes: > This patch add a `guix import go` command. > > It was tested with several big repositories and seems to mostly work for > the import part (because building Guix packages is an other story). There > is still bugs blocking e.g. use of any k8s.io modules. > > * guix/import/go.scm: Created Go Importer > * guix/scripts/import.scm: Created Go Importer Subcommand > * guix/import/go.scm (importers): Added Go Importer Subcommand > > Signed-off-by: Francois Joulaud <francois.joulaud@radiofrance.com> > --- > The patch is a rebased and modified version of the one proposed by > Katherine Cox-Buday. > > Notable modifications are : > - move from (guix json) to (json) > - new parse-go.mod with no "set!" and parsing some go.mod which were in > error before > - adding comments (maybe too much comments) > - renamed SCS to VCS to be in accordance with vocabulary in use in Guix > and in Go worlds > - replacing escape-capital-letters by Helio Machado's go-path-escape > - no pruning of major version in go module names as they are considered > as completely different artefacts by Go programmers > - fixed recursive-import probably broken by the rebase > - force usage of url-fetch from (guix build download) > > I would be happy to hear about problems and perspective for this patch and > will now focus on my next step which is actually building any package. > > Hope I CCed the right persons, I am not really aware of applicable > netiquette here. > > Interdiff : > diff --git a/guix/import/go.scm b/guix/import/go.scm > index 61009f3565..7f5f300f0a 100644 > --- a/guix/import/go.scm > +++ b/guix/import/go.scm > @@ -1,5 +1,7 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com> > +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> > +;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com> > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -16,6 +18,21 @@ > ;;; You should have received a copy of the GNU General Public License > ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. > > +;;; (guix import golang) wants to make easier to create Guix package > +;;; declaration for Go modules. > +;;; > +;;; Modules in Go are "collection of related Go packages" which are > +;;; "the unit of source code interchange and versioning". > +;;; Modules are generally hosted in a repository. > +;;; > +;;; At this point it should handle correctly modules which > +;;; - have only Go dependencies; > +;;; - use go.mod; > +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). > +;;; > +;;; We translate Go module paths to a Guix package name under the > +;;; assumption that there will be no collision. > + > (define-module (guix import go) > #:use-module (ice-9 match) > #:use-module (ice-9 rdelim) > @@ -23,7 +40,7 @@ > #:use-module (ice-9 regex) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-9) > - #:use-module (guix json) > + #:use-module (json) > #:use-module ((guix download) #:prefix download:) > #:use-module (guix import utils) > #:use-module (guix import json) > @@ -33,88 +50,129 @@ > #:use-module ((guix licenses) #:prefix license:) > #:use-module (guix base16) > #:use-module (guix base32) > - #:use-module (guix build download) > + #:use-module ((guix build download) #:prefix build-download:) > #:use-module (web uri) > > #:export (go-module->guix-package > go-module-recursive-import > infer-module-root)) > > -(define (escape-capital-letters s) > - "To avoid ambiguity when serving from case-insensitive file systems, the > -$module and $version elements are case-encoded by replacing every uppercase > -letter with an exclamation mark followed by the corresponding lower-case > -letter." > - (let ((escaped-string (string))) > - (string-for-each-index > - (lambda (i) > - (let ((c (string-ref s i))) > - (set! escaped-string > - (string-concatenate > - (list escaped-string > - (if (char-upper-case? c) "!" "") > - (string (char-downcase c))))))) > - s) > - escaped-string)) > +(define (go-path-escape path) > + "Escape a module path by replacing every uppercase letter with an exclamation > +mark followed with its lowercase equivalent, as per the module Escaped Paths > +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" > + (define (escape occurrence) > + (string-append "!" (string-downcase (match:substring occurrence)))) > + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) > + > > (define (fetch-latest-version goproxy-url module-path) > "Fetches the version number of the latest version for MODULE-PATH from the > given GOPROXY-URL server." > (assoc-ref > (json-fetch (format #f "~a/~a/@latest" goproxy-url > - (escape-capital-letters module-path))) > + (go-path-escape module-path))) > "Version")) > > (define (fetch-go.mod goproxy-url module-path version file) > "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH > and VERSION." > - (url-fetch (format #f "~a/~a/@v/~a.mod" goproxy-url > - (escape-capital-letters module-path) > - (escape-capital-letters version)) > - file > - #:print-build-trace? #f)) > + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url > + (go-path-escape module-path) > + (go-path-escape version)))) > + (parameterize ((current-output-port (current-error-port))) > + (build-download:url-fetch url > + file > + #:print-build-trace? #f)))) > > (define (parse-go.mod go.mod-path) > - "Parses a go.mod file and returns an alist of module path to version." > + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of > +requirements from it." > + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar > + ;; which we think necessary for our use case. > + (define (toplevel results) > + "Main parser, RESULTS is a pair of alist serving as accumulator for > + all encountered requirements and replacements." > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; parsing ended, give back the result > + results) > + ((string=? line "require (") > + ;; a require block begins, delegate parsing to IN-REQUIRE > + (in-require results)) > + ((string-prefix? "require " line) > + ;; a require directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (require-directive results stripped-line))) > + (toplevel new-results))) > + ((string-prefix? "replace " line) > + ;; a replace directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (replace-directive results stripped-line))) > + (toplevel new-results))) > + (#t > + ;; unrecognised line, ignore silently > + (toplevel results))))) > + (define (in-require results) > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; this should never happen here but we ignore silently > + results) > + ((string=? line ")") > + ;; end of block, coming back to toplevel > + (toplevel results)) > + (#t > + (in-require (require-directive results line)))))) > + (define (replace-directive results line) > + "Extract replaced modules and new requirements from replace directive > + in LINE and add to RESULTS." > + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline > + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + (re (string-concatenate > + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" > + "[[:blank:]]+" "=>" "[[:blank:]]+" > + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 3)) > + (new-module-path (match:substring match 4)) > + (new-version (match:substring match 6)) > + (new-replaced (acons module-path version replaced)) > + (new-requirements > + (if (string-match "^\\.?\\./" new-module-path) > + requirements > + (acons new-module-path new-version requirements)))) > + (cons new-requirements new-replaced))) > + (define (require-directive results line) > + "Extract requirement from LINE and add it to RESULTS." > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + ;; A line in a require directive is composed of a module path and > + ;; a version separated by whitespace and an optionnal '//' comment at > + ;; the end. > + (re (string-concatenate > + '("^[[:blank:]]*" > + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" > + "([[:blank:]]+//.*)?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 2))) > + (cons (acons module-path version requirements) replaced))) > (with-input-from-file go.mod-path > (lambda () > - (let ((in-require? #f) > - (requirements (list))) > - (do ((line (read-line) (read-line))) > - ((eof-object? line)) > - (set! line (string-trim line)) > - ;; The parser is either entering, within, exiting, or after the > - ;; require block. The Go toolchain is trustworthy so edge-cases like > - ;; double-entry, etc. need not complect the parser. > - (cond > - ((string=? line "require (") > - (set! in-require? #t)) > - ((and in-require? (string=? line ")")) > - (set! in-require? #f)) > - (in-require? > - (let* ((requirement (string-split line #\space)) > - ;; Modules should be unquoted > - (module-path (string-delete #\" (car requirement))) > - (version (list-ref requirement 1))) > - (set! requirements (acons module-path version requirements)))) > - ((string-prefix? "replace" line) > - (let* ((requirement (string-split line #\space)) > - (module-path (list-ref requirement 1)) > - (new-module-path (list-ref requirement 3)) > - (version (list-ref requirement 4))) > - (set! requirements (assoc-remove! requirements module-path)) > - (set! requirements (acons new-module-path version requirements)))))) > - requirements)))) > - > -(define (module-path-without-major-version module-path) > - "Go modules can be appended with a major version indicator, > -e.g. /v3. Sometimes it is desirable to work with the root module path. For > -instance, for a module path github.com/foo/bar/v3 this function returns > -github.com/foo/bar." > - (let ((m (string-match "(.*)\\/v[0-9]+$" module-path))) > - (if m > - (match:substring m 1) > - module-path))) > + (let* ((results (toplevel '(() . ()))) > + (requirements (car results)) > + (replaced (cdr results))) > + ;; At last we remove replaced modules from the requirements list > + (fold > + (lambda (replacedelem requirements) > + (alist-delete! (car replacedelem) requirements)) > + requirements > + replaced))))) > > (define (infer-module-root module-path) > "Go modules can be defined at any level of a repository's tree, but querying > @@ -124,38 +182,42 @@ root path from its path. For a set of well-known forges, the pattern of what > consists of a module's root page is known before hand." > ;; See the following URL for the official Go equivalent: > ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 > - (define-record-type <scs> > - (make-scs url-prefix root-regex type) > - scs? > - (url-prefix scs-url-prefix) > - (root-regex scs-root-regex) > - (type scs-type)) > - (let* ((known-scs > + ;; > + ;; FIXME: handle module path with VCS qualifier as described in > + ;; https://golang.org/ref/mod#vcs-find and > + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths > + (define-record-type <vcs> > + (make-vcs url-prefix root-regex type) > + vcs? > + (url-prefix vcs-url-prefix) > + (root-regex vcs-root-regex) > + (type vcs-type)) > + (let* ((known-vcs > (list > - (make-scs > + (make-vcs > "github.com" > "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" > 'git) > - (make-scs > + (make-vcs > "bitbucket.org" > "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`" > 'unknown) > - (make-scs > + (make-vcs > "hub.jazz.net/git/" > "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" > 'git) > - (make-scs > + (make-vcs > "git.apache.org" > "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" > 'git) > - (make-scs > + (make-vcs > "git.openstack.org" > "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" > 'git))) > - (scs (find (lambda (scs) (string-prefix? (scs-url-prefix scs) module-path)) > - known-scs))) > - (if scs > - (match:substring (string-match (scs-root-regex scs) module-path) 1) > + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) > + known-vcs))) > + (if vcs > + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) > module-path))) > > (define (to-guix-package-name module-path) > @@ -164,8 +226,7 @@ consists of a module's root page is known before hand." > (string-append "go-" > (string-replace-substring > (string-replace-substring > - ;; Guix has its own field for version > - (module-path-without-major-version module-path) > + module-path > "." "-") > "/" "-")))) > > @@ -173,7 +234,9 @@ consists of a module's root page is known before hand." > "Fetches module meta-data from a module's landing page. This is necessary > because goproxy servers don't currently provide all the information needed to > build a package." > - (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) > + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted > + ;; on several lines > + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) > (module-metadata #f) > (meta-tag-prefix "<meta name=\"go-import\" content=\"") > (meta-tag-prefix-length (string-length meta-tag-prefix))) > @@ -185,7 +248,7 @@ build a package." > (let* ((start (+ meta-tag-index meta-tag-prefix-length)) > (end (string-index line #\" start))) > (set! module-metadata > - (string-split (substring/shared line start end) #\space)))))) > + (string-split (substring/shared line start end) #\space)))))) > (close-port port) > module-metadata)) > > @@ -244,7 +307,7 @@ control system is being used." > (dependencies (map car (parse-go.mod temp))) > (guix-name (to-guix-package-name module-path)) > (root-module-path (infer-module-root module-path)) > - ;; SCS type and URL are not included in goproxy information. For > + ;; VCS type and URL are not included in goproxy information. For > ;; this we need to fetch it from the official module page. > (meta-data (fetch-module-meta-data root-module-path)) > (scs-type (module-meta-data-scs meta-data)) > @@ -268,9 +331,10 @@ control system is being used." > > (define* (go-module-recursive-import package-name > #:key (goproxy-url "https://proxy.golang.org")) > - (recursive-import package-name #f > - #:repo->guix-package > - (lambda (name _) > - (go-module->guix-package name > - #:goproxy-url goproxy-url)) > - #:guix-name to-guix-package-name)) > + (recursive-import > + package-name > + #:repo->guix-package (lambda* (name . _) > + (go-module->guix-package > + name > + #:goproxy-url goproxy-url)) > + #:guix-name to-guix-package-name)) > > guix/import/go.scm | 340 +++++++++++++++++++++++++++++++++++++ > guix/scripts/import.scm | 2 +- > guix/scripts/import/go.scm | 118 +++++++++++++ > 3 files changed, 459 insertions(+), 1 deletion(-) > create mode 100644 guix/import/go.scm > create mode 100644 guix/scripts/import/go.scm > > diff --git a/guix/import/go.scm b/guix/import/go.scm > new file mode 100644 > index 0000000000..7f5f300f0a > --- /dev/null > +++ b/guix/import/go.scm > @@ -0,0 +1,340 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com> > +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> > +;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com> > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. > + > +;;; (guix import golang) wants to make easier to create Guix package > +;;; declaration for Go modules. > +;;; > +;;; Modules in Go are "collection of related Go packages" which are > +;;; "the unit of source code interchange and versioning". > +;;; Modules are generally hosted in a repository. > +;;; > +;;; At this point it should handle correctly modules which > +;;; - have only Go dependencies; > +;;; - use go.mod; > +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). > +;;; > +;;; We translate Go module paths to a Guix package name under the > +;;; assumption that there will be no collision. > + > +(define-module (guix import go) > + #:use-module (ice-9 match) > + #:use-module (ice-9 rdelim) > + #:use-module (ice-9 receive) > + #:use-module (ice-9 regex) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-9) > + #:use-module (json) > + #:use-module ((guix download) #:prefix download:) > + #:use-module (guix import utils) > + #:use-module (guix import json) > + #:use-module (guix packages) > + #:use-module (guix upstream) > + #:use-module (guix utils) > + #:use-module ((guix licenses) #:prefix license:) > + #:use-module (guix base16) > + #:use-module (guix base32) > + #:use-module ((guix build download) #:prefix build-download:) > + #:use-module (web uri) > + > + #:export (go-module->guix-package > + go-module-recursive-import > + infer-module-root)) > + > +(define (go-path-escape path) > + "Escape a module path by replacing every uppercase letter with an exclamation > +mark followed with its lowercase equivalent, as per the module Escaped Paths > +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" > + (define (escape occurrence) > + (string-append "!" (string-downcase (match:substring occurrence)))) > + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) > + > + > +(define (fetch-latest-version goproxy-url module-path) > + "Fetches the version number of the latest version for MODULE-PATH from the > +given GOPROXY-URL server." > + (assoc-ref > + (json-fetch (format #f "~a/~a/@latest" goproxy-url > + (go-path-escape module-path))) > + "Version")) > + > +(define (fetch-go.mod goproxy-url module-path version file) > + "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH > +and VERSION." > + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url > + (go-path-escape module-path) > + (go-path-escape version)))) > + (parameterize ((current-output-port (current-error-port))) > + (build-download:url-fetch url > + file > + #:print-build-trace? #f)))) > + > +(define (parse-go.mod go.mod-path) > + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of > +requirements from it." > + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar > + ;; which we think necessary for our use case. > + (define (toplevel results) > + "Main parser, RESULTS is a pair of alist serving as accumulator for > + all encountered requirements and replacements." > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; parsing ended, give back the result > + results) > + ((string=? line "require (") > + ;; a require block begins, delegate parsing to IN-REQUIRE > + (in-require results)) > + ((string-prefix? "require " line) > + ;; a require directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (require-directive results stripped-line))) > + (toplevel new-results))) > + ((string-prefix? "replace " line) > + ;; a replace directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (replace-directive results stripped-line))) > + (toplevel new-results))) > + (#t > + ;; unrecognised line, ignore silently > + (toplevel results))))) > + (define (in-require results) > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; this should never happen here but we ignore silently > + results) > + ((string=? line ")") > + ;; end of block, coming back to toplevel > + (toplevel results)) > + (#t > + (in-require (require-directive results line)))))) > + (define (replace-directive results line) > + "Extract replaced modules and new requirements from replace directive > + in LINE and add to RESULTS." > + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline > + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + (re (string-concatenate > + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" > + "[[:blank:]]+" "=>" "[[:blank:]]+" > + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 3)) > + (new-module-path (match:substring match 4)) > + (new-version (match:substring match 6)) > + (new-replaced (acons module-path version replaced)) > + (new-requirements > + (if (string-match "^\\.?\\./" new-module-path) > + requirements > + (acons new-module-path new-version requirements)))) > + (cons new-requirements new-replaced))) > + (define (require-directive results line) > + "Extract requirement from LINE and add it to RESULTS." > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + ;; A line in a require directive is composed of a module path and > + ;; a version separated by whitespace and an optionnal '//' comment at > + ;; the end. > + (re (string-concatenate > + '("^[[:blank:]]*" > + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" > + "([[:blank:]]+//.*)?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 2))) > + (cons (acons module-path version requirements) replaced))) > + (with-input-from-file go.mod-path > + (lambda () > + (let* ((results (toplevel '(() . ()))) > + (requirements (car results)) > + (replaced (cdr results))) > + ;; At last we remove replaced modules from the requirements list > + (fold > + (lambda (replacedelem requirements) > + (alist-delete! (car replacedelem) requirements)) > + requirements > + replaced))))) > + > +(define (infer-module-root module-path) > + "Go modules can be defined at any level of a repository's tree, but querying > +for the meta tag usually can only be done at the webpage at the root of the > +repository. Therefore, it is sometimes necessary to try and derive a module's > +root path from its path. For a set of well-known forges, the pattern of what > +consists of a module's root page is known before hand." > + ;; See the following URL for the official Go equivalent: > + ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 > + ;; > + ;; FIXME: handle module path with VCS qualifier as described in > + ;; https://golang.org/ref/mod#vcs-find and > + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths > + (define-record-type <vcs> > + (make-vcs url-prefix root-regex type) > + vcs? > + (url-prefix vcs-url-prefix) > + (root-regex vcs-root-regex) > + (type vcs-type)) > + (let* ((known-vcs > + (list > + (make-vcs > + "github.com" > + "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" > + 'git) > + (make-vcs > + "bitbucket.org" > + "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`" > + 'unknown) > + (make-vcs > + "hub.jazz.net/git/" > + "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" > + 'git) > + (make-vcs > + "git.apache.org" > + "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" > + 'git) > + (make-vcs > + "git.openstack.org" > + "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" > + 'git))) > + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) > + known-vcs))) > + (if vcs > + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) > + module-path))) > + > +(define (to-guix-package-name module-path) > + "Converts a module's path to the canonical Guix format for Go packages." > + (string-downcase > + (string-append "go-" > + (string-replace-substring > + (string-replace-substring > + module-path > + "." "-") > + "/" "-")))) > + > +(define (fetch-module-meta-data module-path) > + "Fetches module meta-data from a module's landing page. This is necessary > +because goproxy servers don't currently provide all the information needed to > +build a package." > + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted > + ;; on several lines > + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) > + (module-metadata #f) > + (meta-tag-prefix "<meta name=\"go-import\" content=\"") > + (meta-tag-prefix-length (string-length meta-tag-prefix))) > + (do ((line (read-line port) (read-line port))) > + ((or (eof-object? line) > + module-metadata)) > + (let ((meta-tag-index (string-contains line meta-tag-prefix))) > + (when meta-tag-index > + (let* ((start (+ meta-tag-index meta-tag-prefix-length)) > + (end (string-index line #\" start))) > + (set! module-metadata > + (string-split (substring/shared line start end) #\space)))))) > + (close-port port) > + module-metadata)) > + > +(define (module-meta-data-scs meta-data) > + "Return the source control system specified by a module's meta-data." > + (string->symbol (list-ref meta-data 1))) > + > +(define (module-meta-data-repo-url meta-data goproxy-url) > + "Return the URL where the fetcher which will be used can download the source > +control." > + (if (member (module-meta-data-scs meta-data) '(fossil mod)) > + goproxy-url > + (list-ref meta-data 2))) > + > +(define (source-uri scs-type scs-repo-url file) > + "Generate the `origin' block of a package depending on what type of source > +control system is being used." > + (case scs-type > + ((git) > + `(origin > + (method git-fetch) > + (uri (git-reference > + (url ,scs-repo-url) > + (commit (string-append "v" version)))) > + (file-name (git-file-name name version)) > + (sha256 > + (base32 > + ,(guix-hash-url file))))) > + ((hg) > + `(origin > + (method hg-fetch) > + (uri (hg-reference > + (url ,scs-repo-url) > + (changeset ,version))) > + (file-name (format #f "~a-~a-checkout" name version)))) > + ((svn) > + `(origin > + (method svn-fetch) > + (uri (svn-reference > + (url ,scs-repo-url) > + (revision (string->number version)) > + (recursive? #f))) > + (file-name (format #f "~a-~a-checkout" name version)) > + (sha256 > + (base32 > + ,(guix-hash-url file))))) > + (else > + (raise-exception (format #f "unsupported scs type: ~a" scs-type))))) > + > +(define* (go-module->guix-package module-path #:key (goproxy-url "https://proxy.golang.org")) > + (call-with-temporary-output-file > + (lambda (temp port) > + (let* ((latest-version (fetch-latest-version goproxy-url module-path)) > + (go.mod-path (fetch-go.mod goproxy-url module-path latest-version > + temp)) > + (dependencies (map car (parse-go.mod temp))) > + (guix-name (to-guix-package-name module-path)) > + (root-module-path (infer-module-root module-path)) > + ;; VCS type and URL are not included in goproxy information. For > + ;; this we need to fetch it from the official module page. > + (meta-data (fetch-module-meta-data root-module-path)) > + (scs-type (module-meta-data-scs meta-data)) > + (scs-repo-url (module-meta-data-repo-url meta-data goproxy-url))) > + (values > + `(package > + (name ,guix-name) > + ;; Elide the "v" prefix Go uses > + (version ,(string-trim latest-version #\v)) > + (source > + ,(source-uri scs-type scs-repo-url temp)) > + (build-system go-build-system) > + ,@(maybe-inputs (map to-guix-package-name dependencies)) > + ;; TODO(katco): It would be nice to make an effort to fetch this > + ;; from known forges, e.g. GitHub > + (home-page ,(format #f "https://~a" root-module-path)) > + (synopsis "A Go package") > + (description ,(format #f "~a is a Go package." guix-name)) > + (license #f)) > + dependencies))))) > + > +(define* (go-module-recursive-import package-name > + #:key (goproxy-url "https://proxy.golang.org")) > + (recursive-import > + package-name > + #:repo->guix-package (lambda* (name . _) > + (go-module->guix-package > + name > + #:goproxy-url goproxy-url)) > + #:guix-name to-guix-package-name)) > diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm > index 0a3863f965..1d2b45d942 100644 > --- a/guix/scripts/import.scm > +++ b/guix/scripts/import.scm > @@ -77,7 +77,7 @@ rather than \\n." > ;;; > > (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem" > - "cran" "crate" "texlive" "json" "opam")) > + "go" "cran" "crate" "texlive" "json" "opam")) > > (define (resolve-importer name) > (let ((module (resolve-interface > diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm > new file mode 100644 > index 0000000000..fde7555973 > --- /dev/null > +++ b/guix/scripts/import/go.scm > @@ -0,0 +1,118 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com> > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. > + > +(define-module (guix scripts import go) > + #:use-module (guix ui) > + #:use-module (guix utils) > + #:use-module (guix scripts) > + #:use-module (guix import go) > + #:use-module (guix scripts import) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-11) > + #:use-module (srfi srfi-37) > + #:use-module (ice-9 match) > + #:use-module (ice-9 format) > + #:export (guix-import-go)) > + > + > +;;; > +;;; Command-line options. > +;;; > + > +(define %default-options > + '()) > + > +(define (show-help) > + (display (G_ "Usage: guix import go PACKAGE-PATH > +Import and convert the Go module for PACKAGE-PATH.\n")) > + (display (G_ " > + -h, --help display this help and exit")) > + (display (G_ " > + -V, --version display version information and exit")) > + (display (G_ " > + -r, --recursive generate package expressions for all Go modules\ > + that are not yet in Guix")) > + (display (G_ " > + -p, --goproxy=GOPROXY specify which goproxy server to use")) > + (newline) > + (show-bug-report-information)) > + > +(define %options > + ;; Specification of the command-line options. > + (cons* (option '(#\h "help") #f #f > + (lambda args > + (show-help) > + (exit 0))) > + (option '(#\V "version") #f #f > + (lambda args > + (show-version-and-exit "guix import go"))) > + (option '(#\r "recursive") #f #f > + (lambda (opt name arg result) > + (alist-cons 'recursive #t result))) > + (option '(#\p "goproxy") #t #f > + (lambda (opt name arg result) > + (alist-cons 'goproxy > + (string->symbol arg) > + (alist-delete 'goproxy result)))) > + %standard-import-options)) > + > + > +;;; > +;;; Entry point. > +;;; > + > +(define (guix-import-go . args) > + (define (parse-options) > + ;; Return the alist of option values. > + (args-fold* args %options > + (lambda (opt name arg result) > + (leave (G_ "~A: unrecognized option~%") name)) > + (lambda (arg result) > + (alist-cons 'argument arg result)) > + %default-options)) > + > + (let* ((opts (parse-options)) > + (args (filter-map (match-lambda > + (('argument . value) > + value) > + (_ #f)) > + (reverse opts)))) > + (match args > + ((module-name) > + (if (assoc-ref opts 'recursive) > + (map (match-lambda > + ((and ('package ('name name) . rest) pkg) > + `(define-public ,(string->symbol name) > + ,pkg)) > + (_ #f)) > + (go-module-recursive-import module-name > + #:goproxy-url > + (or (assoc-ref opts 'goproxy) > + "https://proxy.golang.org"))) > + (let ((sexp (go-module->guix-package module-name > + #:goproxy-url > + (or (assoc-ref opts 'goproxy) > + "https://proxy.golang.org")))) > + (unless sexp > + (leave (G_ "failed to download meta-data for module '~a'~%") > + module-name)) > + sexp))) > + (() > + (leave (G_ "too few arguments~%"))) > + ((many ...) > + (leave (G_ "too many arguments~%"))))))
Hello, On Sat, Jan 23, 2021 at 04:41:18PM -0600, Katherine Cox-Buday wrote: > Thanks so much for the patches, Helio, Joulaud! You're welcome! As a side note I prefer to be adressed as "François" ;-) > I have pushed everything (including Joulaud's patch with appropriate > attribution) here[2]. I am admittedly new at using email to organize > code changes, but using a forge seems easier. > Can I suggest we coordinate there, or is that too much of an imposition? I have no problem to coordinate in a forge and can push in a shared branch if you give me access. Even if I must say I found it refreshing to be able to work with mails. It has the nice property of fully-offline and asynchronous communication and it helped me to better articulate my problems. Best regards, François
JOULAUD François <Francois.JOULAUD@radiofrance.com> writes: > Hello, > > On Sat, Jan 23, 2021 at 04:41:18PM -0600, Katherine Cox-Buday wrote: >> Thanks so much for the patches, Helio, Joulaud! > > You're welcome! As a side note I prefer to be adressed as "François" > ;-) I'm very sorry, François! >> I have pushed everything (including Joulaud's patch with appropriate >> attribution) here[2]. I am admittedly new at using email to organize >> code changes, but using a forge seems easier. > >> Can I suggest we coordinate there, or is that too much of an imposition? > > I have no problem to coordinate in a forge and can push in a shared > branch if you give me access. > > Even if I must say I found it refreshing to be able to work with mails. It > has the nice property of fully-offline and asynchronous communication > and it helped me to better articulate my problems. OK, how about we stick to the preferred Guix approach then and coordinate here. Maybe you can teach me some things along the way :) I do like the idea of an email-based forge! The main problems I've had thus far are: - The inter-diff patch you sent was not formatted correctly because it contained some extra leading whitespace. I found myself juggling a few different tools just to understand your changes. - The conversation about this seems to be happening in 3 different places: here, help-guix, and guix-devel. I guess we should be centralizing here. - I don't have any experience coordinating with people in this email-style forge. I don't know how the inter-diff patches work with attribution, nor how to get everyone on the same page. It seems like multiple people are making conflicting changes at once (maybe that's just my perception).
I checked out https://github.com/kat-co/guix/tree/create-go-importer: $ ./pre-inst-env guix import go -r github.com/coredns/coredns ;;; note: source file /s/timmy/guix/guix/import/go.scm ;;; newer than compiled /home/timmy/.cache/guile/ccache/3.0-LE-8-4.3/s/timmy/guix/guix/import/go.scm.go Starting download of /tmp/guix-file.jkncVo From https://proxy.golang.org/github.com/coredns/coredns/@v/v1.8.1.mod... v1.8.1.mod 2KiB 870KiB/s 00:00 [##################] 100.0% Starting download of /tmp/guix-file.GtI9fs From https://proxy.golang.org/k8s.io/klog/@v/v1.0.0.mod... v1.0.0.mod 68B 189KiB/s 00:00 [##################] 100.0% Backtrace: In ice-9/boot-9.scm: 1736:10 13 (with-exception-handler _ _ #:unwind? _ # _) In unknown file: 12 (apply-smob/0 #<thunk 7fdc57f964e0>) In ice-9/boot-9.scm: 718:2 11 (call-with-prompt _ _ #<procedure default-prompt-handle?>) In ice-9/eval.scm: 619:8 10 (_ #(#(#<directory (guile-user) 7fdc57be3f00>))) In guix/ui.scm: 2154:12 9 (run-guix-command _ . _) In guix/scripts/import.scm: 120:11 8 (guix-import . _) In ice-9/eval.scm: 159:9 7 (_ _) In guix/import/utils.scm: 464:27 6 (recursive-import _ #:repo->guix-package _ #:guix-name _ ?) In srfi/srfi-1.scm: 586:17 5 (map1 (("k8s.io/klog" #f) ("k8s.io/client-go" #f) (?) ?)) In guix/import/utils.scm: 453:33 4 (lookup-node "k8s.io/klog" #f) In guix/utils.scm: 700:8 3 (call-with-temporary-output-file #<procedure 7fdc464b03?>) In ice-9/eval.scm: 293:34 2 (_ #(#(#(#(#(#(#(#(#<directory ?> ?) ?) ?) ?) ?) ?) ?) ?)) 155:9 1 (_ #(#(#<directory (guix import go) 7fdc55b65f00>) #f)) In unknown file: 0 (list-ref #f 1) ERROR: In procedure list-ref: In procedure list-ref: Wrong type argument in position 1: #f This is due to: (go-module->guix-package "k8s.io/klog") The temp file looks like this: module k8s.io/klog go 1.12 require github.com/go-logr/logr v0.1.0 -> (fetch-module-meta-data '("github.com/go-logr/logr")) -> (string->uri (format #f "https://~a?go-get=1" module-path)) -> #f -> (http-fetch #f) Is there a better way to debug this? `guix import` kicked me back to the cmd line instead of the guile debugger, which is understandable for users, but the stacktrace is missing a lot of information. I opened emacs and geiser and had to eval a bunch of things to narrow it down. It feels like I'm doing it wrong. (I don't have much experience with scheme)
Hello! Katherine Cox-Buday <cox.katherine.e@gmail.com> skribis: > - The conversation about this seems to be happening in 3 different > places: here, help-guix, and guix-devel. I guess we should be > centralizing here. +1 > - I don't have any experience coordinating with people in this > email-style forge. I don't know how the inter-diff patches work with > attribution, nor how to get everyone on the same page. It seems like > multiple people are making conflicting changes at once (maybe that's > just my perception). I think whoever among you is available to work on it these days could take the lead and prepare a final version of the patches. It looks like it’s approaching a first “committable” version (perhaps just missing an addition to doc/guix.texi and test cases like we have ‘tests/cpan.scm’ & co.) For attribution, I’d keep Katherine as the commit author and add a ‘Co-authored-by’ line for François and for Helio (that’s how we usually handle that given that Git assumes each commit has a single author). When that first version is committed, you can all submit patches for improvements. For now, the focus should be on getting the first version in. :-) My 2¢, Ludo’.
Hello! On Thu, Jan 28, 2021 at 02:27:57PM +0100, Ludovic Courtès wrote: > I think whoever among you is available to work on it these days could > take the lead and prepare a final version of the patches. It looks like > it’s approaching a first “committable” version (perhaps just missing an > addition to doc/guix.texi and test cases like we have ‘tests/cpan.scm’ & > co.) I thought I would be able to send a working v2 of this patch today but it seems I was too optimistic. I found that some go.mod out there uses quoted string which our ad-hoc parser don't know how to parse. cf. https://github.com/go-yaml/yaml/blob/496545a6307b2a7d7a710fd516e5e16e8ab62dbc/go.mod I don't know if this is a blocker for a merge or not. Apart from that I don't know how to add guile-lib to the dependencies of Guix (in order to use htmlprag). Help needed. I tested it recursively with github.com/hashicorp/consul (which was one of those with the most dependencies I found) and it mostly works. Regards, François
Hi, JOULAUD François <Francois.JOULAUD@radiofrance.com> skribis: > On Thu, Jan 28, 2021 at 02:27:57PM +0100, Ludovic Courtès wrote: >> I think whoever among you is available to work on it these days could >> take the lead and prepare a final version of the patches. It looks like >> it’s approaching a first “committable” version (perhaps just missing an >> addition to doc/guix.texi and test cases like we have ‘tests/cpan.scm’ & >> co.) > > I thought I would be able to send a working v2 of this patch today but > it seems I was too optimistic. > > I found that some go.mod out there uses quoted string > which our ad-hoc parser don't know how to parse. cf. > https://github.com/go-yaml/yaml/blob/496545a6307b2a7d7a710fd516e5e16e8ab62dbc/go.mod > > I don't know if this is a blocker for a merge or not. Your call; if it’s an infrequent problem, we could commit it and leave a FIXME in the code. We could also use guile-yaml (or maybe some Go code?) to parse it correctly. > Apart from that I don't know how to add guile-lib to the dependencies of > Guix (in order to use htmlprag). Help needed. So ‘xml->sxml’ isn’t good enough? (If we can avoid the guile-lib dependency, the better.) To depend on Guile-Lib, you would: 1. Add it to (guix self) — this is the code used by ‘guix pull’; 2. Add it to the ‘inputs’ field of the ‘guix’ package; 3. Maybe add a configure check in ‘configure.ac’, though it would be best if we could arrange to make it an optional dependency. > I tested it recursively with github.com/hashicorp/consul (which was one > of those with the most dependencies I found) and it mostly works. Yay, sounds promising! Thanks, Ludo’.
Hello, I will send a v3 of the patch very soon which I hope will be mergeable. It is still very experimental and I noted it as such in documentation. I had problems with the hash in origin which did not work as expected. I prefered to drop this completely for now and replaced it by a full 0 placeholder. At least it is consistently and conspicuously bad. I have a working version of something which download git repo and generates guix hash for it but I'd rather to push this in a subsequent patch as it is very rough for now. On Sun, Jan 31, 2021 at 05:23:59PM +0100, Ludovic Courtès wrote: > JOULAUD François <Francois.JOULAUD@radiofrance.com> skribis: > > I found that some go.mod out there uses quoted string > > which our ad-hoc parser don't know how to parse. cf. > > Your call; if it’s an infrequent problem, we could commit it and leave a > FIXME in the code. We could also use guile-yaml (or maybe some Go > code?) to parse it correctly. I found a way to work around the problem. Indeed using "go mod" to parse the go.mod file could perhaps be easier and have been explored[1]. It works for now with the ad-hoc parser. Let's revisit the choice later if neeeded. > > Apart from that I don't know how to add guile-lib to the dependencies of > > Guix (in order to use htmlprag). Help needed. > > So ‘xml->sxml’ isn’t good enough? (If we can avoid the guile-lib > dependency, the better.) HTML is not well-formed XML (and the hopes given by XHTML have faded) so no, xml->sxml is unfortunately not good enough. > To depend on Guile-Lib, you would: > > 1. Add it to (guix self) — this is the code used by ‘guix pull’; > > 2. Add it to the ‘inputs’ field of the ‘guix’ package; Done 1 and 2 in the patch. Mainly by copy-paste without understanding anything. I hope it will work. > 3. Maybe add a configure check in ‘configure.ac’, though it would be > best if we could arrange to make it an optional dependency. I did not touch to configure.ac which is a strange beast to me. Hope that for optional dependency (only used in "guix import go") it is sufficient. > > I tested it recursively with github.com/hashicorp/consul (which was one > > of those with the most dependencies I found) and it mostly works. > > Yay, sounds promising! Promising but not there. I have now several recursive dependencies between generated packages I must investigate. Still, 234 package definitions generated on one recursive import (even if I had to retry because of intermittent failure of fetching from proxy.golang.org), so Yay! [1]: https://git.sr.ht/~elais/orange/tree/master/item/guix/import/go.scm#L78
diff --git a/guix/import/go.scm b/guix/import/go.scm new file mode 100644 index 0000000000..7f5f300f0a --- /dev/null +++ b/guix/import/go.scm @@ -0,0 +1,340 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com> +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> +;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +;;; (guix import golang) wants to make easier to create Guix package +;;; declaration for Go modules. +;;; +;;; Modules in Go are "collection of related Go packages" which are +;;; "the unit of source code interchange and versioning". +;;; Modules are generally hosted in a repository. +;;; +;;; At this point it should handle correctly modules which +;;; - have only Go dependencies; +;;; - use go.mod; +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). +;;; +;;; We translate Go module paths to a Guix package name under the +;;; assumption that there will be no collision. + +(define-module (guix import go) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) + #:use-module (json) + #:use-module ((guix download) #:prefix download:) + #:use-module (guix import utils) + #:use-module (guix import json) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix base16) + #:use-module (guix base32) + #:use-module ((guix build download) #:prefix build-download:) + #:use-module (web uri) + + #:export (go-module->guix-package + go-module-recursive-import + infer-module-root)) + +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an exclamation +mark followed with its lowercase equivalent, as per the module Escaped Paths +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + + +(define (fetch-latest-version goproxy-url module-path) + "Fetches the version number of the latest version for MODULE-PATH from the +given GOPROXY-URL server." + (assoc-ref + (json-fetch (format #f "~a/~a/@latest" goproxy-url + (go-path-escape module-path))) + "Version")) + +(define (fetch-go.mod goproxy-url module-path version file) + "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH +and VERSION." + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url + (go-path-escape module-path) + (go-path-escape version)))) + (parameterize ((current-output-port (current-error-port))) + (build-download:url-fetch url + file + #:print-build-trace? #f)))) + +(define (parse-go.mod go.mod-path) + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of +requirements from it." + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar + ;; which we think necessary for our use case. + (define (toplevel results) + "Main parser, RESULTS is a pair of alist serving as accumulator for + all encountered requirements and replacements." + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; parsing ended, give back the result + results) + ((string=? line "require (") + ;; a require block begins, delegate parsing to IN-REQUIRE + (in-require results)) + ((string-prefix? "require " line) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (require-directive results stripped-line))) + (toplevel new-results))) + ((string-prefix? "replace " line) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (replace-directive results stripped-line))) + (toplevel new-results))) + (#t + ;; unrecognised line, ignore silently + (toplevel results))))) + (define (in-require results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-require (require-directive results line)))))) + (define (replace-directive results line) + "Extract replaced modules and new requirements from replace directive + in LINE and add to RESULTS." + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . + (let* ((requirements (car results)) + (replaced (cdr results)) + (re (string-concatenate + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" + "[[:blank:]]+" "=>" "[[:blank:]]+" + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 3)) + (new-module-path (match:substring match 4)) + (new-version (match:substring match 6)) + (new-replaced (acons module-path version replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (acons new-module-path new-version requirements)))) + (cons new-requirements new-replaced))) + (define (require-directive results line) + "Extract requirement from LINE and add it to RESULTS." + (let* ((requirements (car results)) + (replaced (cdr results)) + ;; A line in a require directive is composed of a module path and + ;; a version separated by whitespace and an optionnal '//' comment at + ;; the end. + (re (string-concatenate + '("^[[:blank:]]*" + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" + "([[:blank:]]+//.*)?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 2))) + (cons (acons module-path version requirements) replaced))) + (with-input-from-file go.mod-path + (lambda () + (let* ((results (toplevel '(() . ()))) + (requirements (car results)) + (replaced (cdr results))) + ;; At last we remove replaced modules from the requirements list + (fold + (lambda (replacedelem requirements) + (alist-delete! (car replacedelem) requirements)) + requirements + replaced))))) + +(define (infer-module-root module-path) + "Go modules can be defined at any level of a repository's tree, but querying +for the meta tag usually can only be done at the webpage at the root of the +repository. Therefore, it is sometimes necessary to try and derive a module's +root path from its path. For a set of well-known forges, the pattern of what +consists of a module's root page is known before hand." + ;; See the following URL for the official Go equivalent: + ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 + ;; + ;; FIXME: handle module path with VCS qualifier as described in + ;; https://golang.org/ref/mod#vcs-find and + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths + (define-record-type <vcs> + (make-vcs url-prefix root-regex type) + vcs? + (url-prefix vcs-url-prefix) + (root-regex vcs-root-regex) + (type vcs-type)) + (let* ((known-vcs + (list + (make-vcs + "github.com" + "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "bitbucket.org" + "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`" + 'unknown) + (make-vcs + "hub.jazz.net/git/" + "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "git.apache.org" + "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "git.openstack.org" + "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" + 'git))) + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) + known-vcs))) + (if vcs + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) + module-path))) + +(define (to-guix-package-name module-path) + "Converts a module's path to the canonical Guix format for Go packages." + (string-downcase + (string-append "go-" + (string-replace-substring + (string-replace-substring + module-path + "." "-") + "/" "-")))) + +(define (fetch-module-meta-data module-path) + "Fetches module meta-data from a module's landing page. This is necessary +because goproxy servers don't currently provide all the information needed to +build a package." + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted + ;; on several lines + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) + (module-metadata #f) + (meta-tag-prefix "<meta name=\"go-import\" content=\"") + (meta-tag-prefix-length (string-length meta-tag-prefix))) + (do ((line (read-line port) (read-line port))) + ((or (eof-object? line) + module-metadata)) + (let ((meta-tag-index (string-contains line meta-tag-prefix))) + (when meta-tag-index + (let* ((start (+ meta-tag-index meta-tag-prefix-length)) + (end (string-index line #\" start))) + (set! module-metadata + (string-split (substring/shared line start end) #\space)))))) + (close-port port) + module-metadata)) + +(define (module-meta-data-scs meta-data) + "Return the source control system specified by a module's meta-data." + (string->symbol (list-ref meta-data 1))) + +(define (module-meta-data-repo-url meta-data goproxy-url) + "Return the URL where the fetcher which will be used can download the source +control." + (if (member (module-meta-data-scs meta-data) '(fossil mod)) + goproxy-url + (list-ref meta-data 2))) + +(define (source-uri scs-type scs-repo-url file) + "Generate the `origin' block of a package depending on what type of source +control system is being used." + (case scs-type + ((git) + `(origin + (method git-fetch) + (uri (git-reference + (url ,scs-repo-url) + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + ((hg) + `(origin + (method hg-fetch) + (uri (hg-reference + (url ,scs-repo-url) + (changeset ,version))) + (file-name (format #f "~a-~a-checkout" name version)))) + ((svn) + `(origin + (method svn-fetch) + (uri (svn-reference + (url ,scs-repo-url) + (revision (string->number version)) + (recursive? #f))) + (file-name (format #f "~a-~a-checkout" name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + (else + (raise-exception (format #f "unsupported scs type: ~a" scs-type))))) + +(define* (go-module->guix-package module-path #:key (goproxy-url "https://proxy.golang.org")) + (call-with-temporary-output-file + (lambda (temp port) + (let* ((latest-version (fetch-latest-version goproxy-url module-path)) + (go.mod-path (fetch-go.mod goproxy-url module-path latest-version + temp)) + (dependencies (map car (parse-go.mod temp))) + (guix-name (to-guix-package-name module-path)) + (root-module-path (infer-module-root module-path)) + ;; VCS type and URL are not included in goproxy information. For + ;; this we need to fetch it from the official module page. + (meta-data (fetch-module-meta-data root-module-path)) + (scs-type (module-meta-data-scs meta-data)) + (scs-repo-url (module-meta-data-repo-url meta-data goproxy-url))) + (values + `(package + (name ,guix-name) + ;; Elide the "v" prefix Go uses + (version ,(string-trim latest-version #\v)) + (source + ,(source-uri scs-type scs-repo-url temp)) + (build-system go-build-system) + ,@(maybe-inputs (map to-guix-package-name dependencies)) + ;; TODO(katco): It would be nice to make an effort to fetch this + ;; from known forges, e.g. GitHub + (home-page ,(format #f "https://~a" root-module-path)) + (synopsis "A Go package") + (description ,(format #f "~a is a Go package." guix-name)) + (license #f)) + dependencies))))) + +(define* (go-module-recursive-import package-name + #:key (goproxy-url "https://proxy.golang.org")) + (recursive-import + package-name + #:repo->guix-package (lambda* (name . _) + (go-module->guix-package + name + #:goproxy-url goproxy-url)) + #:guix-name to-guix-package-name)) diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index 0a3863f965..1d2b45d942 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -77,7 +77,7 @@ rather than \\n." ;;; (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem" - "cran" "crate" "texlive" "json" "opam")) + "go" "cran" "crate" "texlive" "json" "opam")) (define (resolve-importer name) (let ((module (resolve-interface diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm new file mode 100644 index 0000000000..fde7555973 --- /dev/null +++ b/guix/scripts/import/go.scm @@ -0,0 +1,118 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (guix scripts import go) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (guix scripts) + #:use-module (guix import go) + #:use-module (guix scripts import) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-37) + #:use-module (ice-9 match) + #:use-module (ice-9 format) + #:export (guix-import-go)) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '()) + +(define (show-help) + (display (G_ "Usage: guix import go PACKAGE-PATH +Import and convert the Go module for PACKAGE-PATH.\n")) + (display (G_ " + -h, --help display this help and exit")) + (display (G_ " + -V, --version display version information and exit")) + (display (G_ " + -r, --recursive generate package expressions for all Go modules\ + that are not yet in Guix")) + (display (G_ " + -p, --goproxy=GOPROXY specify which goproxy server to use")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (cons* (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix import go"))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive #t result))) + (option '(#\p "goproxy") #t #f + (lambda (opt name arg result) + (alist-cons 'goproxy + (string->symbol arg) + (alist-delete 'goproxy result)))) + %standard-import-options)) + + +;;; +;;; Entry point. +;;; + +(define (guix-import-go . args) + (define (parse-options) + ;; Return the alist of option values. + (args-fold* args %options + (lambda (opt name arg result) + (leave (G_ "~A: unrecognized option~%") name)) + (lambda (arg result) + (alist-cons 'argument arg result)) + %default-options)) + + (let* ((opts (parse-options)) + (args (filter-map (match-lambda + (('argument . value) + value) + (_ #f)) + (reverse opts)))) + (match args + ((module-name) + (if (assoc-ref opts 'recursive) + (map (match-lambda + ((and ('package ('name name) . rest) pkg) + `(define-public ,(string->symbol name) + ,pkg)) + (_ #f)) + (go-module-recursive-import module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org"))) + (let ((sexp (go-module->guix-package module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org")))) + (unless sexp + (leave (G_ "failed to download meta-data for module '~a'~%") + module-name)) + sexp))) + (() + (leave (G_ "too few arguments~%"))) + ((many ...) + (leave (G_ "too many arguments~%"))))))
This patch add a `guix import go` command. It was tested with several big repositories and seems to mostly work for the import part (because building Guix packages is an other story). There is still bugs blocking e.g. use of any k8s.io modules. * guix/import/go.scm: Created Go Importer * guix/scripts/import.scm: Created Go Importer Subcommand * guix/import/go.scm (importers): Added Go Importer Subcommand Signed-off-by: Francois Joulaud <francois.joulaud@radiofrance.com> --- The patch is a rebased and modified version of the one proposed by Katherine Cox-Buday. Notable modifications are : - move from (guix json) to (json) - new parse-go.mod with no "set!" and parsing some go.mod which were in error before - adding comments (maybe too much comments) - renamed SCS to VCS to be in accordance with vocabulary in use in Guix and in Go worlds - replacing escape-capital-letters by Helio Machado's go-path-escape - no pruning of major version in go module names as they are considered as completely different artefacts by Go programmers - fixed recursive-import probably broken by the rebase - force usage of url-fetch from (guix build download) I would be happy to hear about problems and perspective for this patch and will now focus on my next step which is actually building any package. Hope I CCed the right persons, I am not really aware of applicable netiquette here. Interdiff : diff --git a/guix/import/go.scm b/guix/import/go.scm index 61009f3565..7f5f300f0a 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -1,5 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020 Katherine Cox-Buday <cox.katherine.e@gmail.com> +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> +;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com> ;;; ;;; This file is part of GNU Guix. ;;; @@ -16,6 +18,21 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. +;;; (guix import golang) wants to make easier to create Guix package +;;; declaration for Go modules. +;;; +;;; Modules in Go are "collection of related Go packages" which are +;;; "the unit of source code interchange and versioning". +;;; Modules are generally hosted in a repository. +;;; +;;; At this point it should handle correctly modules which +;;; - have only Go dependencies; +;;; - use go.mod; +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). +;;; +;;; We translate Go module paths to a Guix package name under the +;;; assumption that there will be no collision. + (define-module (guix import go) #:use-module (ice-9 match) #:use-module (ice-9 rdelim) @@ -23,7 +40,7 @@ #:use-module (ice-9 regex) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) - #:use-module (guix json) + #:use-module (json) #:use-module ((guix download) #:prefix download:) #:use-module (guix import utils) #:use-module (guix import json) @@ -33,88 +50,129 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix base16) #:use-module (guix base32) - #:use-module (guix build download) + #:use-module ((guix build download) #:prefix build-download:) #:use-module (web uri) #:export (go-module->guix-package go-module-recursive-import infer-module-root)) -(define (escape-capital-letters s) - "To avoid ambiguity when serving from case-insensitive file systems, the -$module and $version elements are case-encoded by replacing every uppercase -letter with an exclamation mark followed by the corresponding lower-case -letter." - (let ((escaped-string (string))) - (string-for-each-index - (lambda (i) - (let ((c (string-ref s i))) - (set! escaped-string - (string-concatenate - (list escaped-string - (if (char-upper-case? c) "!" "") - (string (char-downcase c))))))) - s) - escaped-string)) +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an exclamation +mark followed with its lowercase equivalent, as per the module Escaped Paths +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + (define (fetch-latest-version goproxy-url module-path) "Fetches the version number of the latest version for MODULE-PATH from the given GOPROXY-URL server." (assoc-ref (json-fetch (format #f "~a/~a/@latest" goproxy-url - (escape-capital-letters module-path))) + (go-path-escape module-path))) "Version")) (define (fetch-go.mod goproxy-url module-path version file) "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH and VERSION." - (url-fetch (format #f "~a/~a/@v/~a.mod" goproxy-url - (escape-capital-letters module-path) - (escape-capital-letters version)) - file - #:print-build-trace? #f)) + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url + (go-path-escape module-path) + (go-path-escape version)))) + (parameterize ((current-output-port (current-error-port))) + (build-download:url-fetch url + file + #:print-build-trace? #f)))) (define (parse-go.mod go.mod-path) - "Parses a go.mod file and returns an alist of module path to version." + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of +requirements from it." + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar + ;; which we think necessary for our use case. + (define (toplevel results) + "Main parser, RESULTS is a pair of alist serving as accumulator for + all encountered requirements and replacements." + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; parsing ended, give back the result + results) + ((string=? line "require (") + ;; a require block begins, delegate parsing to IN-REQUIRE + (in-require results)) + ((string-prefix? "require " line) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (require-directive results stripped-line))) + (toplevel new-results))) + ((string-prefix? "replace " line) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (replace-directive results stripped-line))) + (toplevel new-results))) + (#t + ;; unrecognised line, ignore silently + (toplevel results))))) + (define (in-require results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-require (require-directive results line)))))) + (define (replace-directive results line) + "Extract replaced modules and new requirements from replace directive + in LINE and add to RESULTS." + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . + (let* ((requirements (car results)) + (replaced (cdr results)) + (re (string-concatenate + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" + "[[:blank:]]+" "=>" "[[:blank:]]+" + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 3)) + (new-module-path (match:substring match 4)) + (new-version (match:substring match 6)) + (new-replaced (acons module-path version replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (acons new-module-path new-version requirements)))) + (cons new-requirements new-replaced))) + (define (require-directive results line) + "Extract requirement from LINE and add it to RESULTS." + (let* ((requirements (car results)) + (replaced (cdr results)) + ;; A line in a require directive is composed of a module path and + ;; a version separated by whitespace and an optionnal '//' comment at + ;; the end. + (re (string-concatenate + '("^[[:blank:]]*" + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" + "([[:blank:]]+//.*)?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 2))) + (cons (acons module-path version requirements) replaced))) (with-input-from-file go.mod-path (lambda () - (let ((in-require? #f) - (requirements (list))) - (do ((line (read-line) (read-line))) - ((eof-object? line)) - (set! line (string-trim line)) - ;; The parser is either entering, within, exiting, or after the - ;; require block. The Go toolchain is trustworthy so edge-cases like - ;; double-entry, etc. need not complect the parser. - (cond - ((string=? line "require (") - (set! in-require? #t)) - ((and in-require? (string=? line ")")) - (set! in-require? #f)) - (in-require? - (let* ((requirement (string-split line #\space)) - ;; Modules should be unquoted - (module-path (string-delete #\" (car requirement))) - (version (list-ref requirement 1))) - (set! requirements (acons module-path version requirements)))) - ((string-prefix? "replace" line) - (let* ((requirement (string-split line #\space)) - (module-path (list-ref requirement 1)) - (new-module-path (list-ref requirement 3)) - (version (list-ref requirement 4))) - (set! requirements (assoc-remove! requirements module-path)) - (set! requirements (acons new-module-path version requirements)))))) - requirements)))) - -(define (module-path-without-major-version module-path) - "Go modules can be appended with a major version indicator, -e.g. /v3. Sometimes it is desirable to work with the root module path. For -instance, for a module path github.com/foo/bar/v3 this function returns -github.com/foo/bar." - (let ((m (string-match "(.*)\\/v[0-9]+$" module-path))) - (if m - (match:substring m 1) - module-path))) + (let* ((results (toplevel '(() . ()))) + (requirements (car results)) + (replaced (cdr results))) + ;; At last we remove replaced modules from the requirements list + (fold + (lambda (replacedelem requirements) + (alist-delete! (car replacedelem) requirements)) + requirements + replaced))))) (define (infer-module-root module-path) "Go modules can be defined at any level of a repository's tree, but querying @@ -124,38 +182,42 @@ root path from its path. For a set of well-known forges, the pattern of what consists of a module's root page is known before hand." ;; See the following URL for the official Go equivalent: ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 - (define-record-type <scs> - (make-scs url-prefix root-regex type) - scs? - (url-prefix scs-url-prefix) - (root-regex scs-root-regex) - (type scs-type)) - (let* ((known-scs + ;; + ;; FIXME: handle module path with VCS qualifier as described in + ;; https://golang.org/ref/mod#vcs-find and + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths + (define-record-type <vcs> + (make-vcs url-prefix root-regex type) + vcs? + (url-prefix vcs-url-prefix) + (root-regex vcs-root-regex) + (type vcs-type)) + (let* ((known-vcs (list - (make-scs + (make-vcs "github.com" "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" 'git) - (make-scs + (make-vcs "bitbucket.org" "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`" 'unknown) - (make-scs + (make-vcs "hub.jazz.net/git/" "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" 'git) - (make-scs + (make-vcs "git.apache.org" "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" 'git) - (make-scs + (make-vcs "git.openstack.org" "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" 'git))) - (scs (find (lambda (scs) (string-prefix? (scs-url-prefix scs) module-path)) - known-scs))) - (if scs - (match:substring (string-match (scs-root-regex scs) module-path) 1) + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) + known-vcs))) + (if vcs + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) module-path))) (define (to-guix-package-name module-path) @@ -164,8 +226,7 @@ consists of a module's root page is known before hand." (string-append "go-" (string-replace-substring (string-replace-substring - ;; Guix has its own field for version - (module-path-without-major-version module-path) + module-path "." "-") "/" "-")))) @@ -173,7 +234,9 @@ consists of a module's root page is known before hand." "Fetches module meta-data from a module's landing page. This is necessary because goproxy servers don't currently provide all the information needed to build a package." - (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted + ;; on several lines + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) (module-metadata #f) (meta-tag-prefix "<meta name=\"go-import\" content=\"") (meta-tag-prefix-length (string-length meta-tag-prefix))) @@ -185,7 +248,7 @@ build a package." (let* ((start (+ meta-tag-index meta-tag-prefix-length)) (end (string-index line #\" start))) (set! module-metadata - (string-split (substring/shared line start end) #\space)))))) + (string-split (substring/shared line start end) #\space)))))) (close-port port) module-metadata)) @@ -244,7 +307,7 @@ control system is being used." (dependencies (map car (parse-go.mod temp))) (guix-name (to-guix-package-name module-path)) (root-module-path (infer-module-root module-path)) - ;; SCS type and URL are not included in goproxy information. For + ;; VCS type and URL are not included in goproxy information. For ;; this we need to fetch it from the official module page. (meta-data (fetch-module-meta-data root-module-path)) (scs-type (module-meta-data-scs meta-data)) @@ -268,9 +331,10 @@ control system is being used." (define* (go-module-recursive-import package-name #:key (goproxy-url "https://proxy.golang.org")) - (recursive-import package-name #f - #:repo->guix-package - (lambda (name _) - (go-module->guix-package name - #:goproxy-url goproxy-url)) - #:guix-name to-guix-package-name)) + (recursive-import + package-name + #:repo->guix-package (lambda* (name . _) + (go-module->guix-package + name + #:goproxy-url goproxy-url)) + #:guix-name to-guix-package-name)) guix/import/go.scm | 340 +++++++++++++++++++++++++++++++++++++ guix/scripts/import.scm | 2 +- guix/scripts/import/go.scm | 118 +++++++++++++ 3 files changed, 459 insertions(+), 1 deletion(-) create mode 100644 guix/import/go.scm create mode 100644 guix/scripts/import/go.scm