From patchwork Fri Feb 19 16:21:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?JOULAUD_Fran=C3=A7ois?= X-Patchwork-Id: 27143 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3A5FB27BC48; Fri, 19 Feb 2021 16:22:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_PASS,T_DKIM_INVALID,URIBL_BLOCKED, URIBL_SBL,URIBL_SBL_A autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id BAF0427BC49 for ; Fri, 19 Feb 2021 16:22:13 +0000 (GMT) Received: from localhost ([::1]:40186 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lD8Xk-0003Wn-Sa for patchwork@mira.cbaines.net; Fri, 19 Feb 2021 11:22:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:57242) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lD8Xa-0003Tr-5B for guix-patches@gnu.org; Fri, 19 Feb 2021 11:22:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:38692) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lD8XZ-0001W7-Qs for guix-patches@gnu.org; Fri, 19 Feb 2021 11:22:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lD8XZ-000867-Mf for guix-patches@gnu.org; Fri, 19 Feb 2021 11:22:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#44178] [PATCHv3] Create importer for Go modules References: <87sga5kpdp.fsf@gmail.com> Resent-From: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 19 Feb 2021 16:22:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44178 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: "44178@debbugs.gnu.org" <44178@debbugs.gnu.org> Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Katherine Cox-Buday Received: via spool by 44178-submit@debbugs.gnu.org id=B44178.161375168231081 (code B ref 44178); Fri, 19 Feb 2021 16:22:01 +0000 Received: (at 44178) by debbugs.gnu.org; 19 Feb 2021 16:21:22 +0000 Received: from localhost ([127.0.0.1]:50238 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lD8Wk-000855-UK for submit@debbugs.gnu.org; Fri, 19 Feb 2021 11:21:21 -0500 Received: from mx08-00115501.pphosted.com ([91.207.212.23]:21438) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lD8Wg-00084r-RQ for 44178@debbugs.gnu.org; Fri, 19 Feb 2021 11:21:10 -0500 Received: from pps.filterd (m0030078.ppops.net [127.0.0.1]) by mx08-00115501.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 11JG8oqv001253; Fri, 19 Feb 2021 17:21:05 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=radiofrance.com; h=from : to : cc : subject : date : message-id : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=radiofrance20190306; bh=CKrfpACY0kDCjEUnxl+mArrdWrKUYbuonE2HAGwIKdI=; b=rUIG2JJxEYpzRrn2pKFAdzxGYcFNmQbtAt74b+fyYoGNnd9S6hNf2JjZlV8TXTQn0kzB fau0Mib9Dc1CBZu+fR/gWfhFoGvusb4C8xAaekByJPQkF4jRxa2G0mbf7AzQVQ50v1P6 MvAn4hWY4QIQsuXQ1i2CtMw1anQPcx1+RdBN7ZgbNqfRf+oLG9YMByKoYHtyrBMKvk1G lS6IzsY77ezwLmMeC5fClBTNe5r1UbRHYZP7b0UqaYAX1s2zjAr2t+3fX3ZSPjtIcnll CDfhQENtFZj9pldqUA84kwnAjUzNskAi2cAV3Ab4tnnJwXtewdkuLDNQCTSUqD38eAqG Rg== Received: from fra01-mr2-obe.outbound.protection.outlook.com (mail-mr2fra01on0106.outbound.protection.outlook.com [104.47.25.106]) by mx08-00115501.pphosted.com with ESMTP id 36rd7ap5qb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 19 Feb 2021 17:21:05 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y2rCDCB6QGVR2gNikZ9m6pjeAU0ChYEPZKe3gv/AgvZ+HpIough1BEe88iQjyNgT6Ns+LyoAOstmQq757hwJ80cY0ldfruSObzgDfbj/uBWuXEZ+QJ+uihH9dmuSN0kN25MTwmFCm0wzRcjw4JrOEWB2ajpCRQO+LCIyBlnr+z9+P+3ND2xNPLJClFLDntpZtJNUP4vO4j6H3vynjtiypCmPJ8hXOoRQrhdqaFIcKD+ZfKatWeKcQpOCJ1ocD6Ilopxoo0I7AHoH1ywCXh+mqHWKoiwS5WewVklG1m3OPhreLbSyFYD51QnzB0UPJYfsdq0mQL7FaZu2yMs2b8hMIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CKrfpACY0kDCjEUnxl+mArrdWrKUYbuonE2HAGwIKdI=; b=cvYzAcufVz56I0F2GizcbEciZMM3v+iXPazPCTPiYfu3hAGFy92yFgpwbGwmEeqLad3FkKJOhhL7zoVToSPteTHSM/PPyVMZeQ5xgeGEI7+noTSLOC+0jxtJ5cFu39xc59ahyE3Oo6WKa0fnwsENqw530dJ2smd9qdg6oYe1hAXW9+ztYnYZXr0IVx2opySkSv8sVpjH9x/YbqEC0R8FI47uCeB9lpI1/u/hJjlzSIKZtUvveemf1VCG1DqdVaR8SmqnoQUFzni3eJgVCdFeqRdNW/vSZ/vVvVVN9p0fe3zjzIwBbdcyJ6YaI2jrTxhhyC3fO2+4mJtSzmXnfGZ5MQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=radiofrance.com; dmarc=pass action=none header.from=radiofrance.com; dkim=pass header.d=radiofrance.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RFonline.onmicrosoft.com; s=selector2-RFonline-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CKrfpACY0kDCjEUnxl+mArrdWrKUYbuonE2HAGwIKdI=; b=s02xI/0aGA604swqvx5IEDKDGGWeJJKaU0KUNm+mwX29ArWhssnfMWyh8rGW8kgstifiNFpa5s4Xl0mpfRCcqLHU23SCe0UgIDOPZQ1cf+Rrvmuu1Rx6/W9PqLbxBQvH2+92+ArZgmHRfoXtBOKvMkZiRFLagzkXjt693YbndN0= Received: from PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:b::20) by PR0P264MB1177.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:165::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3846.36; Fri, 19 Feb 2021 16:21:03 +0000 Received: from PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM ([fe80::51bc:289d:ff1a:6b58]) by PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM ([fe80::51bc:289d:ff1a:6b58%6]) with mapi id 15.20.3825.040; Fri, 19 Feb 2021 16:21:03 +0000 Thread-Topic: [PATCHv3] Create importer for Go modules Thread-Index: AQHXBts3rF0o2Aqs1kGgq7srmbRqZg== Date: Fri, 19 Feb 2021 16:21:03 +0000 Message-ID: <20210219161737.4l266imcd24gqxwn@fjo-extia-HPdeb.example.avalenn.eu> In-Reply-To: <20210219154028.z5aoyozf7qsrz3mt@fjo-extia-HPdeb.example.avalenn.eu> Accept-Language: fr-FR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: debbugs.gnu.org; dkim=none (message not signed) header.d=none;debbugs.gnu.org; dmarc=none action=none header.from=radiofrance.com; x-originating-ip: [88.126.13.52] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0499643d-03cc-4534-dffb-08d8d4f25a48 x-ms-traffictypediagnostic: PR0P264MB1177: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4502; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: WN/sjV8lE8njcy0PPHY/iKljGD0OnvPmUL5kNw3x5QwGUsZbaiXJu2sLFZJr4Qpxh1tqciDrJFd8snW3DYJIx8NYPGm93KiEdpwQChoM5/zu/Ms69WfrbxZjFuY0Cmz4d1masJkpCGiiVmmgu7h1mHdQ34jB35NULZxALcjczCzZ+t+3rchfnkdl9mht/uB4+0nKIMjLm0ov/8jzpBcH/nkD9MuRi0/nV9eOeyRAy06THQ12Idglj5/S6Po/+P3PfBJURWz6ejW+V+mgPYqkMrr40lvVR1p2n5/9E0WmAh2XqfXlyScU5sfvlyv1DMQUfDXkBm6x9yagteyFNmBFL0YriLG1SOLDynrHbQ/G3rnHtDqZJ018TSQvBExoUFmI1r8HkQywctpW/Bk8Pa/8Bp2Yr8LhBxQTHitOautMlP0uFUm2i9RowVNAyk9J9GYdSO/mvMEVtSkNBkOrBzCP/G6Uj4OFc0dEv2ukC7xqZOCDmMDTVGzfg1SKRvNZSCB+omGQnwzjY9ELBZXrVU/QwiUVe2kIhr3nwVmoi3GLlr8ZXH7vWRmat2fMmYkNro0K03NfeVHTs5BcE3HnlvdJRJKmzyusKdtUN7WcX9Q8cmGMWPoIan8JCTmIrEiiuboKvAe2tUz264qSK0kGCeunbQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(4636009)(346002)(136003)(396003)(39850400004)(376002)(366004)(30864003)(1076003)(76116006)(4326008)(6486002)(478600001)(5660300002)(71200400001)(54906003)(8936002)(66574015)(6506007)(6916009)(9686003)(66946007)(6512007)(316002)(66446008)(966005)(64756008)(66556008)(66476007)(2906002)(86362001)(26005)(8676002)(83380400001)(186003)(2004002)(559001)(579004); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: bG3Dz3h4Qrno3yKHxzkj3f9X8LR2ShhpRqR+MmE2XN4qKOgQN5amBoTpsFPS+07gLONx+y8EOx9/3isGtrDxjsM8zULZgw04I2RkIJOd9dVraq0ad+q27k5KJQi3x3pWuuosAgAniHlHZeCoH00Jt9nVkEgYt7+1XuSvuTZXYMlXfz/YbAE/LEYWmUhNdZMdtCO1GKUrlUNWnmaezbyMVN78fp4pvb18w+0O0Ljz/4xY6ppkbsMk0CNSsb0EwKkXGOQHD9LhWqdjf0oRoWH8DMOkCYBrTLaAePXkXMtkjoVZsTdiJLm1LAOUnEpS6w+Wceg+32TTUgZ2B/8fweqlFkZFlQRB0f6GnBeJs/hMXvLDAwA2AyRUONemP4XEtQALS+a5I++9I0DkYVjec2aXIfDRwYAFzFRDBoLzJG4sdGGZoWdNGLxYYh9J64/oZDlUNibgK7REnX8G7x9I3tWJ8qa6zllbUkYHflarMSOr50ws2ZRJgAbrwFEaE0QHheYpbJgNkZVQLCxsfgb9JsVkSAgYV9hyYnbo5LpyUsXOUOqkGkDXrwxgUPMCjCURPGabRQZULAtO9rV0ODqKIJz/YEZtRQ6Sx7EypXVoyvGGoEhtz6e37thpA6ks7t++f2UVZSO6FZlDXZXbYh3ff2vsMUsyYpwb+geOtld8sKcLy+IHXEt7wwtRLjiZD+V+Tf8HNAGVGuLH0FC2aI6uJwfENh5q3nHBMuRYqzw+ewrZLo4TWsq0tE2tGCwpR5op6cZPSEoDhIgoAOSTKecXNnlUsqFZrIgFG2SoEBAQcvE07UbWEDOxImSUd5FXaqKQrWi8vJbgIDt7qrBQR6xN4xAOvUKVuVtWk2Jyhzs64mbg/7DarLpcy8RHo1FQtA/oYXDhmGqV6pqQOiCqMWbYMxO/RclKTCdfNHawmSYK7GGjlUzOzjSYeKraFSSa6t8p851Uz+EN1Ys5tPf408RWov1idziHgYJdSRP5sEOtW2ZDltE+gcB4Uk0JGiRgmb4YdmkmdA6fQUJifWya5N8H5xBx/o3ft3YCyOqWc3cpQdkUSzrFGLtgEBXSpRovpndQbirNBgIzrC0uxBJHSeQ+HohAbRrrXQGC1PCs01rm6LemMf+21zF0TvJ81b4+tOaSWRLTMC0sUKsNV3bkJfFcRyFicK/M/XDgQ3gDlFJ/zYw53oOn/YZj1s7duzcA3PQjBE/DVHHy5GcIgfaU2af7mCgMjf00IF3oJZY42y+mw9tSkqGJnR2AqeYvwZBG862aCXjLKj+lEnhjxvkGYFdG246mj6W6NGF5ZmPYqzbd8qxH7YmNtaj1inyISg2BcR77l/1b Content-ID: <39CB4C40A3BD9748AB5C20E71470A75C@FRAP264.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-OriginatorOrg: radiofrance.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 0499643d-03cc-4534-dffb-08d8d4f25a48 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Feb 2021 16:21:03.4046 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 1d643b07-8cf5-4e2d-ad1e-86d6b948fc3b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3g082Rkc7ekGSMqH3X9t03ZXrzeeyX/ZQ4qs6SIK5czZUezKNqskVPLp/qMY6buyC5lSFjiniYrDYttIqDHISVtDBDayEporELyKIRNRB1GQtyv5eRjnmaGGKMkpo+Pe X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR0P264MB1177 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-02-19_07:2021-02-18, 2021-02-19 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 clxscore=1015 priorityscore=1501 spamscore=0 mlxscore=0 suspectscore=0 lowpriorityscore=0 mlxlogscore=999 impostorscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102190127 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" Reply-to: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= X-ACL-Warn: , JOULAUD =?utf-8?b?RnJhbsOnb2lz?= via Guix-patches X-Patchwork-Original-From: JOULAUD =?utf-8?b?RnJhbsOnb2lz?= via Guix-patches via From: =?utf-8?q?JOULAUD_Fran=C3=A7ois?= X-getmail-retrieved-from-mailbox: Patches This patch add a `guix import go` command. It was tested with several big repositories and mostly works. Several features are lacking (see TODO in source code) but we will do the improvments step-by-step in future patches. * doc/guix.texi: doc about go importer and guile-lib dependency * gnu/packages/package-management.scm: added guile-lib dependency * guix/self.scm: add guile-lib dependency * guix/build-system/go.scm: go-version->git-ref function * guix/import/go.scm: Created Go importer * guix/scripts/import/go.scm: Subcommand for Go importer * guix/scripts/import.scm: Declare subcommand guix import go * tests/import-go.scm: Tests for parse-go.mod procedure Signed-off-by: Francois Joulaud --- doc/guix.texi | 26 ++ gnu/packages/package-management.scm | 2 + guix/build-system/go.scm | 35 ++- guix/import/go.scm | 416 ++++++++++++++++++++++++++++ guix/scripts/import.scm | 2 +- guix/scripts/import/go.scm | 118 ++++++++ guix/self.scm | 5 +- tests/import-go.scm | 144 ++++++++++ 8 files changed, 745 insertions(+), 3 deletions(-) create mode 100644 guix/import/go.scm create mode 100644 guix/scripts/import/go.scm create mode 100644 tests/import-go.scm diff --git a/doc/guix.texi b/doc/guix.texi index 5d28fca837..89c8abd261 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -861,6 +861,10 @@ substitutes (@pxref{Invoking guix publish}). @uref{https://ngyro.com/software/guile-semver.html, Guile-Semver} for the @code{crate} importer (@pxref{Invoking guix import}). +@item +@uref{https://www.nongnu.org/guile-lib/doc/ref/htmlprag/, guile-lib} for +the @code{crate} importer (@pxref{Invoking guix import}). + @item When @url{http://www.bzip.org, libbz2} is available, @command{guix-daemon} can use it to compress build logs. @@ -11493,6 +11497,28 @@ Select the given repository (a repository name). Possible values include: of coq packages. @end itemize @end table + +@item go +@cindex go +Import metadata for a Go module using +@uref{https://proxy.golang.org, proxy.golang.org}. + +This importer is highly experimental. See the source code for more info +about the current state. + +@example +guix import go gopkg.in/yaml.v2 +@end example + +Additional options include: + +@table @code +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. +@end table @end table The structure of the @command{guix import} code is modular. It would be diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 9fb8c40a31..06bb5bd2df 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -304,6 +304,7 @@ $(prefix)/etc/init.d\n"))) '((assoc-ref inputs "guile")))) (avahi (assoc-ref inputs "guile-avahi")) (gcrypt (assoc-ref inputs "guile-gcrypt")) + (guile-lib (assoc-ref inputs "guile-lib")) (json (assoc-ref inputs "guile-json")) (sqlite (assoc-ref inputs "guile-sqlite3")) (zlib (assoc-ref inputs "guile-zlib")) @@ -367,6 +368,7 @@ $(prefix)/etc/init.d\n"))) `(("guile-avahi" ,guile-avahi))) ("guile-gcrypt" ,guile-gcrypt) ("guile-json" ,guile-json-4) + ("guile-lib" ,guile-lib) ("guile-sqlite3" ,guile-sqlite3) ("guile-zlib" ,guile-zlib) ("guile-lzlib" ,guile-lzlib) diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm index f8ebaefb27..594e0cb4f3 100644 --- a/guix/build-system/go.scm +++ b/guix/build-system/go.scm @@ -26,9 +26,42 @@ #:use-module (guix build-system gnu) #:use-module (guix packages) #:use-module (ice-9 match) + #:use-module (ice-9 regex) #:export (%go-build-system-modules go-build - go-build-system)) + go-build-system + + go-version->git-ref)) + +(define (go-version->git-ref version) + "GO-VERSION->GIT-REF parse pseudo-versions and extract the commit + hash from it, defaulting to full VERSION if we don't recognise a + pseudo-version pattern." + ;; A module version like v1.2.3 is introduced by tagging a revision in + ;; the underlying source repository. Untagged revisions can be referred + ;; to using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef, + ;; where the time is the commit time in UTC and the final suffix is the + ;; prefix of the commit hash. + ;; cf. https://golang.org/cmd/go/#hdr-Pseudo_versions + (let* ((version + ;; if a source code repository has a v2.0.0 or later tag for + ;; a file tree with no go.mod, the version is considered to be + ;; part of the v1 module's available versions and is given an + ;; +incompatible suffix + ;; https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning + (if (string-suffix? "+incompatible" version) + (string-drop-right version 13) + version)) + (re (string-concatenate + (list + "(v?[0-9]\\.[0-9]\\.[0-9])" ; "v" prefix can be omitted in version prefix + "(-|-pre\\.0\\.|-0\\.)" ; separator + "([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])-" ; timestamp + "([0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f])"))) ; commit hash + (match (string-match re version))) + (if match + (match:substring match 4) + version))) ;; Commentary: ;; diff --git a/guix/import/go.scm b/guix/import/go.scm new file mode 100644 index 0000000000..fead355bd2 --- /dev/null +++ b/guix/import/go.scm @@ -0,0 +1,416 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2020 Katherine Cox-Buday +;;; Copyright © 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> +;;; Copyright © 2021 François Joulaud +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; (guix import golang) wants to make easier to create Guix package +;;; declaration for Go modules. +;;; +;;; Modules in Go are "collection of related Go packages" which are +;;; "the unit of source code interchange and versioning". +;;; Modules are generally hosted in a repository. +;;; +;;; At this point it should handle correctly modules which +;;; have only Go dependencies and are accessible from proxy.golang.org +;;; (or configured GOPROXY). +;;; +;;; We want it to work more or less this way: +;;; - get latest version for the module from GOPROXY +;;; - infer VCS root repo from which we will check-out source by +;;; + recognising known patterns (like github.com) +;;; + or (TODO) recognising .vcs suffix +;;; + or parsing meta tag in html served at the URL +;;; + or (TODO) if nothing else works by using zip file served by GOPROXY +;;; - get go.mod from GOPROXY (which is able to synthetize one if needed) +;;; - extract list of dependencies from this go.mod +;;; +;;; We translate Go module paths to a Guix package name under the +;;; assumption that there will be no collision. + +;;; TODO list +;;; - get correct hash in vcs->origin +;;; - print partial result during recursive imports (need to catch +;;; exceptions) +;;; - infer repo from module path with VCS qualifier +;;; (e.g. site.example/my/path/to/repo.git/and/subdir/module) +;;; - don't print fetch messages to stdout +;;; - pre-fill synopsis, description and license + +(define-module (guix import go) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) + #:use-module (ice-9 regex) + #:use-module (guix build-system go) + #:use-module (htmlprag) + #:use-module (sxml xpath) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) + #:use-module (srfi srfi-11) + #:use-module (json) + #:use-module ((guix download) #:prefix download:) + #:use-module (guix git) + #:use-module (guix import utils) + #:use-module (guix import json) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix base16) + #:use-module (guix base32) + #:use-module (guix memoization) + #:use-module ((guix build download) #:prefix build-download:) + #:use-module (web uri) + + #:export (go-module->guix-package + go-module-recursive-import + infer-module-root-repo)) + + +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an exclamation +mark followed with its lowercase equivalent, as per the module Escaped Paths +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths" + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + + +(define (go-module-latest-version goproxy-url module-path) + "Fetches the version number of the latest version for MODULE-PATH from the +given GOPROXY-URL server." + (assoc-ref + (json-fetch (format #f "~a/~a/@latest" goproxy-url + (go-path-escape module-path))) + "Version")) + +(define go-module-latest-version* (memoize go-module-latest-version)) + +(define (fetch-go.mod goproxy-url module-path version file) + "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-PATH +and VERSION." + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url + (go-path-escape module-path) + (go-path-escape version)))) + (parameterize ((current-output-port (current-error-port))) + (build-download:url-fetch url + file + #:print-build-trace? #f)))) + +(define (parse-go.mod go.mod-path) + (parse-go.mod-port (open-input-file go.mod-path))) + +(define (parse-go.mod-port go.mod-port) + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of +requirements from it." + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar + ;; which we think necessary for our use case. + (define (toplevel results) + "Main parser, RESULTS is a pair of alist serving as accumulator for + all encountered requirements and replacements." + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; parsing ended, give back the result + results) + ((string=? line "require (") + ;; a require block begins, delegate parsing to IN-REQUIRE + (in-require results)) + ((string=? line "replace (") + ;; a replace block begins, delegate parsing to IN-REPLACE + (in-replace results)) + ((string-prefix? "require " line) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (require-directive results stripped-line))) + (toplevel new-results))) + ((string-prefix? "replace " line) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (replace-directive results stripped-line))) + (toplevel new-results))) + (#t + ;; unrecognised line, ignore silently + (toplevel results))))) + (define (in-require results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-require (require-directive results line)))))) + (define (in-replace results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-replace (replace-directive results line)))))) + (define (replace-directive results line) + "Extract replaced modules and new requirements from replace directive + in LINE and add to RESULTS." + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . + (let* ((requirements (car results)) + (replaced (cdr results)) + (re (string-concatenate + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" + "[[:blank:]]+" "=>" "[[:blank:]]+" + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 3)) + (new-module-path (match:substring match 4)) + (new-version (match:substring match 6)) + (new-replaced (acons module-path version replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (acons new-module-path new-version requirements)))) + (cons new-requirements new-replaced))) + (define (require-directive results line) + "Extract requirement from LINE and add it to RESULTS." + (let* ((requirements (car results)) + (replaced (cdr results)) + ;; A line in a require directive is composed of a module path and + ;; a version separated by whitespace and an optionnal '//' comment at + ;; the end. + (re (string-concatenate + '("^[[:blank:]]*" + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" + "([[:blank:]]+//.*)?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + ;; we saw double-quoted string in the wild without escape + ;; sequences so we just trim the quotes + (module-path (string-trim-both module-path #\")) + (version (match:substring match 2))) + (cons (acons module-path version requirements) replaced))) + (with-input-from-port go.mod-port + (lambda () + (let* ((results (toplevel '(() . ()))) + (requirements (car results)) + (replaced (cdr results))) + ;; At last we remove replaced modules from the requirements list + (fold + (lambda (replacedelem requirements) + (alist-delete! (car replacedelem) requirements)) + requirements + replaced))))) + +(define (infer-module-root-repo module-path) + "Go modules can be defined at any level of a repository's tree, but querying +for the meta tag usually can only be done at the webpage at the root of the +repository. Therefore, it is sometimes necessary to try and derive a module's +root path from its path. For a set of well-known forges, the pattern of what +consists of a module's root page is known before hand." + ;; See the following URL for the official Go equivalent: + ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 + ;; + ;; TODO: handle module path with VCS qualifier as described in + ;; https://golang.org/ref/mod#vcs-find and + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths + (define-record-type + (make-vcs url-prefix root-regex type) + vcs? + (url-prefix vcs-url-prefix) + (root-regex vcs-root-regex) + (type vcs-type)) + (let* ((known-vcs + (list + (make-vcs + "github.com" + "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "bitbucket.org" + "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-Za-z0-9_.\\-]+)*$" + 'unknown) + (make-vcs + "hub.jazz.net/git/" + "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "git.apache.org" + "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+)*$" + 'git) + (make-vcs + "git.openstack.org" + "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" + 'git))) + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) module-path)) + known-vcs))) + (if vcs + (match:substring (string-match (vcs-root-regex vcs) module-path) 1) + module-path))) + +(define (go-module->guix-package-name module-path) + "Converts a module's path to the canonical Guix format for Go packages." + (string-downcase + (string-append "go-" + (string-replace-substring + (string-replace-substring + module-path + "." "-") + "/" "-")))) + +(define-record-type + (make-module-meta import-prefix vcs repo-root) + module-meta? + (import-prefix module-meta-import-prefix) + ;; VCS field is a symbol + (vcs module-meta-vcs) + (repo-root module-meta-repo-root)) + +(define (fetch-module-meta-data module-path) + "Fetches module meta-data from a module's landing page. This is + necessary because goproxy servers don't currently provide all the + information needed to build a package." + ;; + (define (meta-go-import->module-meta text) + "Takes the content of the go-import meta tag as TEXT and gives back + a MODULE-META record" + (define (get-component s start) + (let* + ((start (string-skip s char-set:whitespace start)) + (end (string-index s char-set:whitespace start)) + (end (if end end (string-length s))) + (result (substring s start end))) + (values result end))) + (let*-values (((import-prefix end) (get-component text 0)) + ((vcs end) (get-component text end)) + ((repo-root end) (get-component text end))) + (make-module-meta import-prefix (string->symbol vcs) repo-root))) + (define (html->meta-go-import port) + "Read PORT with HTML content. Find the go-import meta tag and gives + back its content as a string." + (let* ((parsedhtml (html->sxml port)) + (extract-content (node-join + (select-kids (node-typeof? 'html)) + (select-kids (node-typeof? 'head)) + (select-kids (node-typeof? 'meta)) + (select-kids (node-typeof? '@)) + (node-self + (node-join + (select-kids (node-typeof? 'name)) + (select-kids (node-equal? "go-import")))) + (select-kids (node-typeof? 'content)) + (select-kids (lambda (_) #t)))) + (content (car (extract-content parsedhtml)))) + content)) + (let* ((port (build-download:http-fetch (string->uri (format #f "https://~a?go-get=1" module-path)))) + (meta-go-import (html->meta-go-import port)) + (module-metadata (meta-go-import->module-meta meta-go-import))) + (close-port port) + module-metadata)) + +(define (module-meta-data-repo-url meta-data goproxy-url) + "Return the URL where the fetcher which will be used can download the source +control." + (if (member (module-meta-vcs meta-data)'(fossil mod)) + goproxy-url + (module-meta-repo-root meta-data))) + +(define (vcs->origin vcs-type vcs-repo-url version file) + "Generate the `origin' block of a package depending on what type of source +control system is being used." + (case vcs-type + ((git) + `(origin + (method git-fetch) + (uri (git-reference + (url ,vcs-repo-url) + (commit (go-version->git-ref version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + ;; FIXME: get hash for git repo checkout + "0000000000000000000000000000000000000000000000000000")))) + ((hg) + `(origin + (method hg-fetch) + (uri (hg-reference + (url ,vcs-repo-url) + (changeset ,version))) + (file-name (format #f "~a-~a-checkout" name version)))) + ((svn) + `(origin + (method svn-fetch) + (uri (svn-reference + (url ,vcs-repo-url) + (revision (string->number version)) + (recursive? #f))) + (file-name (format #f "~a-~a-checkout" name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + (else + (raise-exception (format #f "unsupported vcs type: ~a" vcs-type))))) + +(define* (go-module->guix-package module-path #:key (goproxy-url "https://proxy.golang.org")) + (call-with-temporary-output-file + (lambda (temp port) + (let* ((latest-version (go-module-latest-version* goproxy-url module-path)) + (go.mod-path (fetch-go.mod goproxy-url module-path latest-version + temp)) + (dependencies (map car (parse-go.mod temp))) + (guix-name (go-module->guix-package-name module-path)) + (root-module-path (infer-module-root-repo module-path)) + ;; VCS type and URL are not included in goproxy information. For + ;; this we need to fetch it from the official module page. + (meta-data (fetch-module-meta-data root-module-path)) + (vcs-type (module-meta-vcs meta-data)) + (vcs-repo-url (module-meta-data-repo-url meta-data goproxy-url))) + (values + `(package + (name ,guix-name) + ;; Elide the "v" prefix Go uses + (version ,(string-trim latest-version #\v)) + (source + ,(vcs->origin vcs-type vcs-repo-url latest-version temp)) + (build-system go-build-system) + (arguments + '(#:import-path ,root-module-path)) + ,@(maybe-inputs (map go-module->guix-package-name dependencies)) + ;; TODO(katco): It would be nice to make an effort to fetch this + ;; from known forges, e.g. GitHub + (home-page ,(format #f "https://~a" root-module-path)) + (synopsis "A Go package") + (description ,(format #f "~a is a Go package." guix-name)) + (license #f)) + dependencies))))) + +(define go-module->guix-package* (memoize go-module->guix-package)) + +(define* (go-module-recursive-import package-name + #:key (goproxy-url "https://proxy.golang.org")) + (recursive-import + package-name + #:repo->guix-package (lambda* (name . _) + (go-module->guix-package* + name + #:goproxy-url goproxy-url)) + #:guix-name go-module->guix-package-name)) diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index 0a3863f965..1d2b45d942 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -77,7 +77,7 @@ rather than \\n." ;;; (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" "gem" - "cran" "crate" "texlive" "json" "opam")) + "go" "cran" "crate" "texlive" "json" "opam")) (define (resolve-importer name) (let ((module (resolve-interface diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm new file mode 100644 index 0000000000..fde7555973 --- /dev/null +++ b/guix/scripts/import/go.scm @@ -0,0 +1,118 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2020 Katherine Cox-Buday +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix scripts import go) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (guix scripts) + #:use-module (guix import go) + #:use-module (guix scripts import) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-37) + #:use-module (ice-9 match) + #:use-module (ice-9 format) + #:export (guix-import-go)) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '()) + +(define (show-help) + (display (G_ "Usage: guix import go PACKAGE-PATH +Import and convert the Go module for PACKAGE-PATH.\n")) + (display (G_ " + -h, --help display this help and exit")) + (display (G_ " + -V, --version display version information and exit")) + (display (G_ " + -r, --recursive generate package expressions for all Go modules\ + that are not yet in Guix")) + (display (G_ " + -p, --goproxy=GOPROXY specify which goproxy server to use")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (cons* (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix import go"))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive #t result))) + (option '(#\p "goproxy") #t #f + (lambda (opt name arg result) + (alist-cons 'goproxy + (string->symbol arg) + (alist-delete 'goproxy result)))) + %standard-import-options)) + + +;;; +;;; Entry point. +;;; + +(define (guix-import-go . args) + (define (parse-options) + ;; Return the alist of option values. + (args-fold* args %options + (lambda (opt name arg result) + (leave (G_ "~A: unrecognized option~%") name)) + (lambda (arg result) + (alist-cons 'argument arg result)) + %default-options)) + + (let* ((opts (parse-options)) + (args (filter-map (match-lambda + (('argument . value) + value) + (_ #f)) + (reverse opts)))) + (match args + ((module-name) + (if (assoc-ref opts 'recursive) + (map (match-lambda + ((and ('package ('name name) . rest) pkg) + `(define-public ,(string->symbol name) + ,pkg)) + (_ #f)) + (go-module-recursive-import module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org"))) + (let ((sexp (go-module->guix-package module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org")))) + (unless sexp + (leave (G_ "failed to download meta-data for module '~a'~%") + module-name)) + sexp))) + (() + (leave (G_ "too few arguments~%"))) + ((many ...) + (leave (G_ "too many arguments~%")))))) diff --git a/guix/self.scm b/guix/self.scm index 35fba1152d..ed5ee9ddea 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -814,6 +814,9 @@ itself." (define guile-ssh (specification->package "guile-ssh")) + (define guile-lib + (specification->package "guile-lib")) + (define guile-git (specification->package "guile-git")) @@ -842,7 +845,7 @@ itself." (append-map transitive-package-dependencies (list guile-gcrypt gnutls guile-git guile-avahi guile-json guile-semver guile-ssh guile-sqlite3 - guile-zlib guile-lzlib guile-zstd))) + guile-lib guile-zlib guile-lzlib guile-zstd))) (define *core-modules* (scheme-node "guix-core" diff --git a/tests/import-go.scm b/tests/import-go.scm new file mode 100644 index 0000000000..ad4185684c --- /dev/null +++ b/tests/import-go.scm @@ -0,0 +1,144 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 François Joulaud +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; Summary +;; Tests for guix/import/go.scm + +(define-module (test-import-go) + #:use-module (guix import go) + #:use-module (guix base32) + #:use-module (ice-9 iconv) + #:use-module (ice-9 match) + #:use-module (srfi srfi-64)) + +(define fixture-go-mod-simple + "module my/thing +go 1.12 +require other/thing v1.0.2 +require new/thing/v2 v2.3.4 +exclude old/thing v1.2.3 +replace bad/thing v1.4.5 => good/thing v1.4.5 +") + +(define fixture-go-mod-with-block + "module M + +require ( + A v1 + B v1.0.0 + C v1.0.0 + D v1.2.3 + E dev +) + +exclude D v1.2.3 +") + + +(define fixture-go-mod-complete + "module M + +go 1.13 + +replace github.com/myname/myproject/myapi => ./api + +replace github.com/mymname/myproject/thissdk => ../sdk + +replace launchpad.net/gocheck => github.com/go-check/check v0.0.0-20140225173054-eb6ee6f84d0a + +require ( + github.com/user/project v1.1.11 + github.com/user/project/sub/directory v1.1.12 + bitbucket.org/user/project v1.11.20 + bitbucket.org/user/project/sub/directory v1.11.21 + launchpad.net/project v1.1.13 + launchpad.net/project/series v1.1.14 + launchpad.net/project/series/sub/directory v1.1.15 + launchpad.net/~user/project/branch v1.1.16 + launchpad.net/~user/project/branch/sub/directory v1.1.17 + hub.jazz.net/git/user/project v1.1.18 + hub.jazz.net/git/user/project/sub/directory v1.1.19 + k8s.io/kubernetes/subproject v1.1.101 + one.example.com/abitrary/repo v1.1.111 + two.example.com/abitrary/repo v0.0.2 + \"quoted.example.com/abitrary/repo\" v0.0.2 +) + +replace two.example.com/abitrary/repo => github.com/corp/arbitrary-repo v0.0.2 + +replace ( + golang.org/x/sys => golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a // pinned to release-branch.go1.13 + golang.org/x/tools => golang.org/x/tools v0.0.0-20190821162956-65e3620a7ae7 // pinned to release-branch.go1.13 +) + +") + +(test-begin "import go") + +(test-equal "go-path-escape" + "github.com/!azure/!avere" + ((@@ (guix import go) go-path-escape) "github.com/Azure/Avere")) + + + +;; We define a function for all similar tests with different go.mod files +(define (testing-parse-mod name expected input) + (define (inf? p1 p2) + (string