From patchwork Sat Jan 23 21:35:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "vasilii.smirnov--- via Guix-patches\" via" X-Patchwork-Id: 26589 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 380B727BC17; Sat, 23 Jan 2021 23:24:17 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS, T_DKIM_INVALID,URIBL_BLOCKED,URIBL_SBL,URIBL_SBL_A autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 609E527BC18 for ; Sat, 23 Jan 2021 23:24:11 +0000 (GMT) Received: from localhost ([::1]:60214 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l3SGI-00028q-FI for patchwork@mira.cbaines.net; Sat, 23 Jan 2021 18:24:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33870) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l3SGA-00028e-Af for guix-patches@gnu.org; Sat, 23 Jan 2021 18:24:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:52286) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l3SGA-0000MC-2P for guix-patches@gnu.org; Sat, 23 Jan 2021 18:24:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l3SG9-0006kM-VH for guix-patches@gnu.org; Sat, 23 Jan 2021 18:24:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#44178] [PATCH] Create importer for Go modules References: <87sga5kpdp.fsf@gmail.com> Resent-From: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 23 Jan 2021 23:24:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44178 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: "44178@debbugs.gnu.org" <44178@debbugs.gnu.org> Received: via spool by 44178-submit@debbugs.gnu.org id=B44178.161144423125918 (code B ref 44178); Sat, 23 Jan 2021 23:24:01 +0000 Received: (at 44178) by debbugs.gnu.org; 23 Jan 2021 23:23:51 +0000 Received: from localhost ([127.0.0.1]:35599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l3SFx-0006jw-Vq for submit@debbugs.gnu.org; Sat, 23 Jan 2021 18:23:51 -0500 Received: from mx08-00115501.pphosted.com ([91.207.212.23]:30758) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l3QYq-0001wR-T0 for 44178@debbugs.gnu.org; Sat, 23 Jan 2021 16:35:15 -0500 Received: from pps.filterd (m0030078.ppops.net [127.0.0.1]) by mx08-00115501.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 10NLUv5S024623; Sat, 23 Jan 2021 22:35:11 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=radiofrance.com; h=from : to : cc : subject : date : message-id : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=radiofrance20190306; bh=c/E7epY40VpNm3Ov0BETlgEjFfGd5m1P1cmMca8kwXs=; b=YWMs10D9sjJkknF1SiAFuXdp1e3SMgXV/8cANWHS1OF97CFt4U4IcZKDENduheDwezq5 vwGMwefubpxojjzmST345FdlGffGTFqPpaPo+aIzo3Bi74wr/ClM7h9P8Iqi33Is3YBg PZ9WEhmZdFMwc9yovrp4F4Hc+QRhRh7faXpAiYSubJwAnXLTysJpUEiqAbWQjNYpvKyt CCJq+3qVF7lH+1VTYibbF6S36kWnrRDD9fzQiPHwo+YptWFqHAbgZz4fE0cdgxp/u7xK HE5MlfWL6Iab6sdHMPaWzoe+HknfToer2HH4cEwZoTRun8shWmk4F0auUHgUtKxh1jNE Zg== Received: from fra01-pr2-obe.outbound.protection.outlook.com (mail-pr2fra01lp0105.outbound.protection.outlook.com [104.47.24.105]) by mx08-00115501.pphosted.com with ESMTP id 3688byh1xc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 23 Jan 2021 22:35:11 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LfVN/cdBq/yLjAPbyIBXqyyY5clJH4b5NoGWgJFTO65w1ZL3V1jWZhv6FLroKmFkzVsgd/4C79VsO3g3WyPr3Pv4i5x6YS1b2KwiSd8KUp3buNtid3tJ8a5JrENgGiwutWC2jqUTvBOb1tiSPFy3HzIbhneL+Y16XAd0RG1WKaY6Fmf/1McPeeQ1Hf+ZBFA6qRe9jx445ftMkLF4qvEclDLFJO7fyE/eHoNB9UaGyZx0cAggnb8Tku7nujS2Cybu1d4LFN5MclioIuAIG2SKAUlkDOpXkRe5de6wf48St+aiiaTgwOiCPqgQp7RW9Te6ys7D6PR9RdP9wPaBR9hC9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c/E7epY40VpNm3Ov0BETlgEjFfGd5m1P1cmMca8kwXs=; b=iSwmKoYHubU3ILVJ0HzJxGrYHAG3WRUgfAwGOb+Kvvhcw23v9yM0ejgjTM3v2LR4jWj7ZeSObXSZPO8hR2f+wDBu/E1yGEcVy0ReEJTu3XXJLxA4brGxPYS9x3Ehdm5QtdDPN68hJGc5sluLo9wMIwxBtHhgtTrVOXUl718e7/fcXaRMcXs62nG4hgKb54a7gOaQ7buioOn6RUS/fJGtEBR/hZXB0HCnGdmp8eNGeUBKOS8W2WYatjctVEaz1FyyC6NChyc2MEjkHJdn4NV/MgfErTxEB469t2k1GiaEDFqs0e+70xkJxjFbpP7SDLwFxlQYqsafB84Z+315gFJ4Ng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=radiofrance.com; dmarc=pass action=none header.from=radiofrance.com; dkim=pass header.d=radiofrance.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RFonline.onmicrosoft.com; s=selector2-RFonline-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c/E7epY40VpNm3Ov0BETlgEjFfGd5m1P1cmMca8kwXs=; b=HRLN3aYCqz0zydCI1cH38ZD1cmNSSgGMGcPrh9vq9Y1VgOMqdg0TCssQyCgXnJ4CfUxNiEoyi8ts9iO3pIM9rNae/OqzFOzeaSPd7o4uxlyne+FS4/tu4z+yYYIBS5WMYs60cSdN/pVZX77hXEUPHi6F5HWAru4u5l9p02a3/6c= Received: from PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:b::20) by PR0P264MB0492.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.14; Sat, 23 Jan 2021 21:35:09 +0000 Received: from PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM ([fe80::9b7:6491:8aa7:550b]) by PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM ([fe80::9b7:6491:8aa7:550b%3]) with mapi id 15.20.3763.016; Sat, 23 Jan 2021 21:35:08 +0000 Thread-Topic: [PATCH] Create importer for Go modules Thread-Index: AQHW8c+fh3EThKxKn0aE+FBPMWbUYQ== Date: Sat, 23 Jan 2021 21:35:08 +0000 Message-ID: <20210123212742.m2thdeuzdvgpkgeo@fjo-extia-HPdeb.example.avalenn.eu> In-Reply-To: Accept-Language: fr-FR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: debbugs.gnu.org; dkim=none (message not signed) header.d=none;debbugs.gnu.org; dmarc=none action=none header.from=radiofrance.com; x-originating-ip: [88.126.13.52] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b4cff3e2-da60-4670-2bdf-08d8bfe6c1e3 x-ms-traffictypediagnostic: PR0P264MB0492: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4502; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: rnEERXYaoW2OULCXw/mUlKraibknnUUws9gOkVlVHpMh+RJUhmV4vekN2QYY8I19i78maZw+A+Cb8frsDKk5SSXFeP/5qcTyozvPK2/ax0EjT8okxxY/T+BZSd5asOqzNZVZ+DKiTjf0Gq1wLAJOpDE+PU/wkRz+QhbBdbv421ZwxFgRyBEoCA82tF8vBbN47Rrgh0+0M95zVM3+JB0e3W1XZTngcy0AXcWFFE3Uk3Ee0jH0jOV3TMCGcuY4bRnl6Ujjz4PVtUK7ZnJAZDjD1HKiMjVbHzVhHltj2ltalS8zZgLPPQAzXqL6mg6kXX0eV8BrnhMyiK6Qbr/JQClRonzu3TRGYmtsB37n7Zxu/tv2ZibcRTB1+h1dGfb4RoX+wIEYT8o7FDesILxdaAtRRNRt50ZmMxhkHeiNNjZ+rOybdltobaziRSzM+E7GRFY+Ba0ewGOJfv51nG+XR5r5y4ZzWB0Jum78B6wdBYFbTikcLchbLS/0JK1joNeQMqVxFqIg0IjUCjWhFOtQIAlGVwPPSVaptrrOaQ9nW1bboFTjWfArQZ3ANHaLmj/NJAtqDb2lgXnivzDLfqDcZCdMMvgeDQKG7nCN4Ser9Q+TXHs1ewg8/NEF/J5NMKwbG8GOyo65dyuyc4ExeMvdTD0KBw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(4636009)(376002)(346002)(39850400004)(136003)(366004)(396003)(8936002)(6506007)(186003)(5660300002)(1076003)(26005)(316002)(4326008)(6486002)(8676002)(64756008)(66446008)(66556008)(66476007)(66946007)(76116006)(2906002)(86362001)(966005)(54906003)(6916009)(83380400001)(9686003)(478600001)(71200400001)(30864003)(66574015)(6512007)(2004002)(579004)(559001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: gfjjLxBpUS5W8JxwSwrheKPOMubV6kb1Nsnf5ct/xZMf/rIINPxNtnItdGgNgLgY/sSoD3kxFxVw2caCjgCbI93lLtVp+wxJAk+CVEeCd86YttxaPt1Yc1yvbR2J0v5wY7jS3Q8xDYaOWPp0LSvxaa996hpTEbMjqXGkoBvAke0IEJNxgSYbAS5XDXDZk9DNlIB3p6BVOnJy3RXdOjO2EJnIiP1glKH2pI1BUTlCH33HOffUe4JOg8W+lkL/MP2vhL5KGLY4F6X4Jm5jWM2o8zpzI3DIVDKXnmysRzJEyN+CRLKEze6h/UL89l8aUuF+yUTZ7NhxGpXusIYzvbsxhXysPjb6EtJZW/Br61O0y661MAJN/BnKnmkVvrxR6TtR1i67nC5qXABWhsaOiKy3AFseGZ9oAeETOV9SXSw9SeynhdEPvh2shhdCZW5Kn31ASZ4z+mfnT8bVMOrnIVIGxK8KlibQwWkJ/YxOab8vzTcDKijhOFbvm5KHyIvIGuH78VArRKmSKXMaM8tZ1ekI/StFAEwI1xfDUN3+nrDDlxmhJ9VRxWSjHNxezbo/fVNrZjQEm+p2Fks456Vmw2gko0MCa8tEzV582xk38PUskqPYJJ0Y7J0Hr1weKcfDn5S+Ru1q/jWY20votRwFyYyzJNtMdh++FAczxI0x1jyqx9Ej3EqKKpxyx2/eYGmBj45FzHaoWMP40/Tx6tJzeHcyhSVrWzvjJ/C1BoEARwSUTELBqR8bCIqMhnUzERqs1uqkYyIQl/wWFAtIKLgtA/GQLYsd4Pd49568Cip4JhKvRcXMGr6M8KpkO5bIcbgVRGeZf/cV6KoehNQjif9ZBu3G6NAIu0aUqjYG65AdK+ZdCKNnh45pyaEp1nP2Ho4oTW93TUvMPrLXAnffZM6i+QsAAL2JgG8PKODtw7c17fiut0ZOgfx8xKTfVU8ZWY7RwT+1Zy2XSbf6y9e7RZE0AUolWBgiJKoffyn+BbKyR69e3eQPYFBKtAUzFfqw9lp2QusOGfB9MaRl6qM7TGaOHg3xoWEATgY3YNals5n8IqymBj/jt2YZLrAI5yPdsd2gJYU/ Content-ID: <69EE4CB1F6B8DF45B593FA40B59EECD4@FRAP264.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-OriginatorOrg: radiofrance.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: b4cff3e2-da60-4670-2bdf-08d8bfe6c1e3 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jan 2021 21:35:08.7889 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 1d643b07-8cf5-4e2d-ad1e-86d6b948fc3b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: SVkCT04BH8E7Oy7MxTI9EMda5NS5POdgtEKmebBf6xr+G/eqEjW056NAehlPZFFUdDDeFgd6ufviX19+ijlEJ5i869l+qdQIAmeFpe+eOWn+bLgMkq8LhhXGgb/uRzYY X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR0P264MB0492 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343, 18.0.737 definitions=2021-01-23_12:2021-01-22, 2021-01-23 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 clxscore=1011 impostorscore=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 bulkscore=0 adultscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101230127 X-Mailman-Approved-At: Sat, 23 Jan 2021 18:23:48 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Helio Machado <0x2b3bfa0@gmail.com>, Katherine Cox-Buday Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" Reply-to: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= X-ACL-Warn: , JOULAUD =?utf-8?b?RnJhbsOnb2lz?= via Guix-patches X-Patchwork-Original-From: guix-patches--- via From: "vasilii.smirnov--- via Guix-patches\" via" X-getmail-retrieved-from-mailbox: Patches This patch add a `guix import go` command. It was tested with several big repositories and seems to mostly work for the import part (because building Guix packages is an other story). There is still bugs blocking e.g. use of any k8s.io modules. * guix/import/go.scm: Created Go Importer * guix/scripts/import.scm: Created Go Importer Subcommand * guix/import/go.scm (importers): Added Go Importer Subcommand Signed-off-by: Francois Joulaud --- The patch is a rebased and modified version of the one proposed by Katherine Cox-Buday. Notable modifications are : - move from (guix json) to (json) - new parse-go.mod with no "set!" and parsing some go.mod which were in error before - adding comments (maybe too much comments) - renamed SCS to VCS to be in accordance with vocabulary in use in Guix and in Go worlds - replacing escape-capital-letters by Helio Machado's go-path-escape - no pruning of major version in go module names as they are considered as completely different artefacts by Go programmers - fixed recursive-import probably broken by the rebase - force usage of url-fetch from (guix build download) I would be happy to hear about problems and perspective for this patch and will now focus on my next step which is actually building any package. Hope I CCed the right persons, I am not really aware of applicable netiquette here. Interdiff : diff --git a/guix/import/go.scm b/guix/import/go.scm index 61009f3565..7f5f300f0a 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -1,5 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020 Katherine Cox-Buday +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> +;;; Copyright © 2021 François Joulaud ;;; ;;; This file is part of GNU Guix. ;;; @@ -16,6 +18,21 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see . +;;; (guix import golang) wants to make easier to create Guix package +;;; declaration for Go modules. +;;; +;;; Modules in Go are "collection of related Go packages" which are +;;; "the unit of source code interchange and versioning". +;;; Modules are generally hosted in a repository. +;;; +;;; At this point it should handle correctly modules which +;;; - have only Go dependencies; +;;; - use go.mod; +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). +;;; +;;; We translate Go module paths to a Guix package name under the +;;; assumption that there will be no collision. + (define-module (guix import go) #:use-module (ice-9 match) #:use-module (ice-9 rdelim) @@ -23,7 +40,7 @@ #:use-module (ice-9 regex) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) - #:use-module (guix json) + #:use-module (json) #:use-module ((guix download) #:prefix download:) #:use-module (guix import utils) #:use-module (guix import json) @@ -33,88 +50,129 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix base16) #:use-module (guix base32) - #:use-module (guix build download) + #:use-module ((guix build download) #:prefix build-download:) #:use-module (web uri) #:export (go-module->guix-package go-module-recursive-import infer-module-root)) -(define (escape-capital-letters s) - "To avoid ambiguity when serving from case-insensitive file systems, the -$module and $version elements are case-encoded by replacing every uppercase -letter with an exclamation mark followed by the corresponding lower-case -letter." - (let ((escaped-string (string))) - (string-for-each-index - (lambda (i) - (let ((c (string-ref s i))) - (set! escaped-string - (string-concatenate - (list escaped-string - (if (char-upper-case? c) "!" "") - (string (char-downcase c))))))) - s) - escaped-string)) +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an exclamation +mark followed with its lowercase equivalent, as per the module Escaped Paths +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + (define (fetch-latest-version goproxy-url module-path) "Fetches the version number of the latest version for MODULE-PATH from the given GOPROXY-URL server." (assoc-ref (json-fetch (format #f "~a/~a/@latest" goproxy-url - (escape-capital-letters module-path))) + (go-path-escape module-path))) "Version")) (define (fetch-go.mod goproxy-url module-path version file) "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH and VERSION." - (url-fetch (format #f "~a/~a/@v/~a.mod" goproxy-url - (escape-capital-letters module-path) - (escape-capital-letters version)) - file - #:print-build-trace? #f)) + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url + (go-path-escape module-path) + (go-path-escape version)))) + (parameterize ((current-output-port (current-error-port))) + (build-download:url-fetch url + file + #:print-build-trace? #f)))) (define (parse-go.mod go.mod-path) - "Parses a go.mod file and returns an alist of module path to version." + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of +requirements from it." + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar + ;; which we think necessary for our use case. + (define (toplevel results) + "Main parser, RESULTS is a pair of alist serving as accumulator for + all encountered requirements and replacements." + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; parsing ended, give back the result + results) + ((string=? line "require (") + ;; a require block begins, delegate parsing to IN-REQUIRE + (in-require results)) + ((string-prefix? "require " line) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (require-directive results stripped-line))) + (toplevel new-results))) + ((string-prefix? "replace " line) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (replace-directive results stripped-line))) + (toplevel new-results))) + (#t + ;; unrecognised line, ignore silently + (toplevel results))))) + (define (in-require results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-require (require-directive results line)))))) + (define (replace-directive results line) + "Extract replaced modules and new requirements from replace directive + in LINE and add to RESULTS." + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . + (let* ((requirements (car results)) + (replaced (cdr results)) + (re (string-concatenate + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" + "[[:blank:]]+" "=>" "[[:blank:]]+" + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 3)) + (new-module-path (match:substring match 4)) + (new-version (match:substring match 6)) + (new-replaced (acons module-path version replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (acons new-module-path new-version requirements)))) + (cons new-requirements new-replaced))) + (define (require-directive results line) + "Extract requirement from LINE and add it to RESULTS." + (let* ((requirements (car results)) + (replaced (cdr results)) + ;; A line in a require directive is composed of a module path and + ;; a version separated by whitespace and an optionnal '//' comment at + ;; the end. + (re (string-concatenate + '("^[[:blank:]]*" + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" + "([[:blank:]]+//.*)?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 2))) + (cons (acons module-path version requirements) replaced))) (with-input-from-file go.mod-path (lambda () - (let ((in-require? #f) - (requirements (list))) - (do ((line (read-line) (read-line))) - ((eof-object? line)) - (set! line (string-trim line)) - ;; The parser is either entering, within, exiting, or after the - ;; require block. The Go toolchain is trustworthy so edge-cases like - ;; double-entry, etc. need not complect the parser. - (cond - ((string=? line "require (") - (set! in-require? #t)) - ((and in-require? (string=? line ")")) - (set! in-require? #f)) - (in-require? - (let* ((requirement (string-split line #\space)) - ;; Modules should be unquoted - (module-path (string-delete #\" (car requirement))) - (version (list-ref requirement 1))) - (set! requirements (acons module-path version requirements)))) - ((string-prefix? "replace" line) - (let* ((requirement (string-split line #\space)) - (module-path (list-ref requirement 1)) - (new-module-path (list-ref requirement 3)) - (version (list-ref requirement 4))) - (set! requirements (assoc-remove! requirements module-path)) - (set! requirements (acons new-module-path version requirements)))))) - requirements)))) - -(define (module-path-without-major-version module-path) - "Go modules can be appended with a major version indicator, -e.g. /v3. Sometimes it is desirable to work with the root module path. For -instance, for a module path github.com/foo/bar/v3 this function returns -github.com/foo/bar." - (let ((m (string-match "(.*)\\/v[0-9]+$" module-path))) - (if m - (match:substring m 1) - module-path))) + (let* ((results (toplevel '(() . ()))) + (requirements (car results)) + (replaced (cdr results))) + ;; At last we remove replaced modules from the requirements list + (fold + (lambda (replacedelem requirements) + (alist-delete! (car replacedelem) requirements)) + requirements + replaced))))) (define (infer-module-root module-path) "Go modules can be defined at any level of a repository's tree, but querying @@ -124,38 +182,42 @@ root path from its path. For a set of well-known forges, the pattern of what consists of a module's root page is known before hand." ;; See the following URL for the official Go equivalent: ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 - (define-record-type - (make-scs url-prefix root-regex type) - scs? - (url-prefix scs-url-prefix) - (root-regex scs-root-regex) - (type scs-type)) - (let* ((known-scs + ;; + ;; FIXME: handle module path with VCS qualifier as described in + ;; https://golang.org/ref/mod#vcs-find and + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths + (define-record-type + (make-vcs url-prefix root-regex type) + vcs? + (url-prefix vcs-url-prefix) + (root-regex vcs-root-regex) + (type vcs-type)) + (let* ((known-vcs (list - (make-scs + (make-vcs "github.com" "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" 'git) - (make-scs + (make-vcs "bitbucket.org" "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`" 'unknown) - (make-scs + (make-vcs "hub.jazz.net/git/" "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" 'git) - (make-scs + (make-vcs "git.apache.org" "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" 'git) - (make-scs + (make-vcs "git.openstack.org" "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" 'git))) - (scs (find (lambda (scs) (string-prefix? (scs-url-prefix scs) module-path)) - known-scs))) - (if scs - (match:substring (string-match (scs-root-regex scs) module-path) 1) + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) + known-vcs))) + (if vcs + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) module-path))) (define (to-guix-package-name module-path) @@ -164,8 +226,7 @@ consists of a module's root page is known before hand." (string-append "go-" (string-replace-substring (string-replace-substring - ;; Guix has its own field for version - (module-path-without-major-version module-path) + module-path "." "-") "/" "-")))) @@ -173,7 +234,9 @@ consists of a module's root page is known before hand." "Fetches module meta-data from a module's landing page. This is necessary because goproxy servers don't currently provide all the information needed to build a package." - (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted + ;; on several lines + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) (module-metadata #f) (meta-tag-prefix "guix-package - (lambda (name _) - (go-module->guix-package name - #:goproxy-url goproxy-url)) - #:guix-name to-guix-package-name)) + (recursive-import + package-name + #:repo->guix-package (lambda* (name . _) + (go-module->guix-package + name + #:goproxy-url goproxy-url)) + #:guix-name to-guix-package-name)) guix/import/go.scm | 340 +++++++++++++++++++++++++++++++++++++ guix/scripts/import.scm | 2 +- guix/scripts/import/go.scm | 118 +++++++++++++ 3 files changed, 459 insertions(+), 1 deletion(-) create mode 100644 guix/import/go.scm create mode 100644 guix/scripts/import/go.scm diff --git a/guix/import/go.scm b/guix/import/go.scm new file mode 100644 index 0000000000..7f5f300f0a --- /dev/null +++ b/guix/import/go.scm @@ -0,0 +1,340 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2020 Katherine Cox-Buday +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> +;;; Copyright © 2021 François Joulaud +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; (guix import golang) wants to make easier to create Guix package +;;; declaration for Go modules. +;;; +;;; Modules in Go are "collection of related Go packages" which are +;;; "the unit of source code interchange and versioning". +;;; Modules are generally hosted in a repository. +;;; +;;; At this point it should handle correctly modules which +;;; - have only Go dependencies; +;;; - use go.mod; +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). +;;; +;;; We translate Go module paths to a Guix package name under the +;;; assumption that there will be no collision. + +(define-module (guix import go) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) + #:use-module (json) + #:use-module ((guix download) #:prefix download:) + #:use-module (guix import utils) + #:use-module (guix import json) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix base16) + #:use-module (guix base32) + #:use-module ((guix build download) #:prefix build-download:) + #:use-module (web uri) + + #:export (go-module->guix-package + go-module-recursive-import + infer-module-root)) + +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an exclamation +mark followed with its lowercase equivalent, as per the module Escaped Paths +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + + +(define (fetch-latest-version goproxy-url module-path) + "Fetches the version number of the latest version for MODULE-PATH from the +given GOPROXY-URL server." + (assoc-ref + (json-fetch (format #f "~a/~a/@latest" goproxy-url + (go-path-escape module-path))) + "Version")) + +(define (fetch-go.mod goproxy-url module-path version file) + "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH +and VERSION." + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url + (go-path-escape module-path) + (go-path-escape version)))) + (parameterize ((current-output-port (current-error-port))) + (build-download:url-fetch url + file + #:print-build-trace? #f)))) + +(define (parse-go.mod go.mod-path) + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of +requirements from it." + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar + ;; which we think necessary for our use case. + (define (toplevel results) + "Main parser, RESULTS is a pair of alist serving as accumulator for + all encountered requirements and replacements." + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; parsing ended, give back the result + results) + ((string=? line "require (") + ;; a require block begins, delegate parsing to IN-REQUIRE + (in-require results)) + ((string-prefix? "require " line) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (require-directive results stripped-line))) + (toplevel new-results))) + ((string-prefix? "replace " line) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (replace-directive results stripped-line))) + (toplevel new-results))) + (#t + ;; unrecognised line, ignore silently + (toplevel results))))) + (define (in-require results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-require (require-directive results line)))))) + (define (replace-directive results line) + "Extract replaced modules and new requirements from replace directive + in LINE and add to RESULTS." + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . + (let* ((requirements (car results)) + (replaced (cdr results)) + (re (string-concatenate + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" + "[[:blank:]]+" "=>" "[[:blank:]]+" + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 3)) + (new-module-path (match:substring match 4)) + (new-version (match:substring match 6)) + (new-replaced (acons module-path version replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (acons new-module-path new-version requirements)))) + (cons new-requirements new-replaced))) + (define (require-directive results line) + "Extract requirement from LINE and add it to RESULTS." + (let* ((requirements (car results)) + (replaced (cdr results)) + ;; A line in a require directive is composed of a module path and + ;; a version separated by whitespace and an optionnal '//' comment at + ;; the end. + (re (string-concatenate + '("^[[:blank:]]*" + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" + "([[:blank:]]+//.*)?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 2))) + (cons (acons module-path version requirements) replaced))) + (with-input-from-file go.mod-path + (lambda () + (let* ((results (toplevel '(() . ()))) + (requirements (car results)) + (replaced (cdr results))) + ;; At last we remove replaced modules from the requirements list + (fold + (lambda (replacedelem requirements) + (alist-delete! (car replacedelem) requirements)) + requirements + replaced))))) + +(define (infer-module-root module-path) + "Go modules can be defined at any level of a repository's tree, but querying +for the meta tag usually can only be done at the webpage at the root of the +repository. Therefore, it is sometimes necessary to try and derive a module's +root path from its path. For a set of well-known forges, the pattern of what +consists of a module's root page is known before hand." + ;; See the following URL for the official Go equivalent: + ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 + ;; + ;; FIXME: handle module path with VCS qualifier as described in + ;; https://golang.org/ref/mod#vcs-find and + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths + (define-record-type + (make-vcs url-prefix root-regex type) + vcs? + (url-prefix vcs-url-prefix) + (root-regex vcs-root-regex) + (type vcs-type)) + (let* ((known-vcs + (list + (make-vcs + "github.com" + "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "bitbucket.org" + "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$`" + 'unknown) + (make-vcs + "hub.jazz.net/git/" + "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "git.apache.org" + "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "git.openstack.org" + "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" + 'git))) + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) + known-vcs))) + (if vcs + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) + module-path))) + +(define (to-guix-package-name module-path) + "Converts a module's path to the canonical Guix format for Go packages." + (string-downcase + (string-append "go-" + (string-replace-substring + (string-replace-substring + module-path + "." "-") + "/" "-")))) + +(define (fetch-module-meta-data module-path) + "Fetches module meta-data from a module's landing page. This is necessary +because goproxy servers don't currently provide all the information needed to +build a package." + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted + ;; on several lines + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) + (module-metadata #f) + (meta-tag-prefix "symbol (list-ref meta-data 1))) + +(define (module-meta-data-repo-url meta-data goproxy-url) + "Return the URL where the fetcher which will be used can download the source +control." + (if (member (module-meta-data-scs meta-data) '(fossil mod)) + goproxy-url + (list-ref meta-data 2))) + +(define (source-uri scs-type scs-repo-url file) + "Generate the `origin' block of a package depending on what type of source +control system is being used." + (case scs-type + ((git) + `(origin + (method git-fetch) + (uri (git-reference + (url ,scs-repo-url) + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + ((hg) + `(origin + (method hg-fetch) + (uri (hg-reference + (url ,scs-repo-url) + (changeset ,version))) + (file-name (format #f "~a-~a-checkout" name version)))) + ((svn) + `(origin + (method svn-fetch) + (uri (svn-reference + (url ,scs-repo-url) + (revision (string->number version)) + (recursive? #f))) + (file-name (format #f "~a-~a-checkout" name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + (else + (raise-exception (format #f "unsupported scs type: ~a" scs-type))))) + +(define* (go-module->guix-package module-path #:key (goproxy-url "https://proxy.golang.org")) + (call-with-temporary-output-file + (lambda (temp port) + (let* ((latest-version (fetch-latest-version goproxy-url module-path)) + (go.mod-path (fetch-go.mod goproxy-url module-path latest-version + temp)) + (dependencies (map car (parse-go.mod temp))) + (guix-name (to-guix-package-name module-path)) + (root-module-path (infer-module-root module-path)) + ;; VCS type and URL are not included in goproxy information. For + ;; this we need to fetch it from the official module page. + (meta-data (fetch-module-meta-data root-module-path)) + (scs-type (module-meta-data-scs meta-data)) + (scs-repo-url (module-meta-data-repo-url meta-data goproxy-url))) + (values + `(package + (name ,guix-name) + ;; Elide the "v" prefix Go uses + (version ,(string-trim latest-version #\v)) + (source + ,(source-uri scs-type scs-repo-url temp)) + (build-system go-build-system) + ,@(maybe-inputs (map to-guix-package-name dependencies)) + ;; TODO(katco): It would be nice to make an effort to fetch this + ;; from known forges, e.g. GitHub + (home-page ,(format #f "https://~a" root-module-path)) + (synopsis "A Go package") + (description ,(format #f "~a is a Go package." guix-name)) + (license #f)) + dependencies))))) + +(define* (go-module-recursive-import package-name + #:key (goproxy-url "https://proxy.golang.org")) + (recursive-import + package-name + #:repo->guix-package (lambda* (name . _) + (go-module->guix-package + name + #:goproxy-url goproxy-url)) + #:guix-name to-guix-package-name)) diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index 0a3863f965..1d2b45d942 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -77,7 +77,7 @@ rather than \\n." ;;; (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem" - "cran" "crate" "texlive" "json" "opam")) + "go" "cran" "crate" "texlive" "json" "opam")) (define (resolve-importer name) (let ((module (resolve-interface diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm new file mode 100644 index 0000000000..fde7555973 --- /dev/null +++ b/guix/scripts/import/go.scm @@ -0,0 +1,118 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2020 Katherine Cox-Buday +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix scripts import go) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (guix scripts) + #:use-module (guix import go) + #:use-module (guix scripts import) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-37) + #:use-module (ice-9 match) + #:use-module (ice-9 format) + #:export (guix-import-go)) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '()) + +(define (show-help) + (display (G_ "Usage: guix import go PACKAGE-PATH +Import and convert the Go module for PACKAGE-PATH.\n")) + (display (G_ " + -h, --help display this help and exit")) + (display (G_ " + -V, --version display version information and exit")) + (display (G_ " + -r, --recursive generate package expressions for all Go modules\ + that are not yet in Guix")) + (display (G_ " + -p, --goproxy=GOPROXY specify which goproxy server to use")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (cons* (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix import go"))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive #t result))) + (option '(#\p "goproxy") #t #f + (lambda (opt name arg result) + (alist-cons 'goproxy + (string->symbol arg) + (alist-delete 'goproxy result)))) + %standard-import-options)) + + +;;; +;;; Entry point. +;;; + +(define (guix-import-go . args) + (define (parse-options) + ;; Return the alist of option values. + (args-fold* args %options + (lambda (opt name arg result) + (leave (G_ "~A: unrecognized option~%") name)) + (lambda (arg result) + (alist-cons 'argument arg result)) + %default-options)) + + (let* ((opts (parse-options)) + (args (filter-map (match-lambda + (('argument . value) + value) + (_ #f)) + (reverse opts)))) + (match args + ((module-name) + (if (assoc-ref opts 'recursive) + (map (match-lambda + ((and ('package ('name name) . rest) pkg) + `(define-public ,(string->symbol name) + ,pkg)) + (_ #f)) + (go-module-recursive-import module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org"))) + (let ((sexp (go-module->guix-package module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org")))) + (unless sexp + (leave (G_ "failed to download meta-data for module '~a'~%") + module-name)) + sexp))) + (() + (leave (G_ "too few arguments~%"))) + ((many ...) + (leave (G_ "too many arguments~%"))))))