From patchwork Fri Nov 15 21:11:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?J=C3=B8rgen_Kvalsvik?= X-Patchwork-Id: 70424 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 5684827BBE9; Fri, 15 Nov 2024 21:12:35 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id D1D2527BBE2 for ; Fri, 15 Nov 2024 21:12:33 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tC3bp-0002E2-4a; Fri, 15 Nov 2024 16:12:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bn-0002Do-VC for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tC3bn-00069Y-NI for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=yPzaXCYSdaziVreq7nuo+jL0yuX+QD9obeGjigv5Dl8=; b=nBk+w72PtVN7TSuWhj8DqPmjrf3cc+qj/9Cif+dRMp6iwurtH+3wWagK0zKppENMhKrAhST7XmntuX9H8bNwc4ndczfJKmAQR7CTEoY4vVa5dNH9ap8Ntf3o2MMLiH1fXCtDrP1bNus8SzWRr8SuiEtffqqTtykUb0jrG9scg1jM6e2GTtLy+qpdmd3ZMHKMQf0pn4J2N4k7HpYUeTih2Bel/Pv+LTg2pf2XIz0Z8i0Geo79k1ZUlnDyP8cC5fOMculSfWKjiuCNtJLC0xxSZ3WqSUhtv4OTEQwycsAmJlYvKarqNQAyorCQ34eCgAnhkwSX1u5Vz+GfMHrOhdkzUQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tC3bn-0000GM-HI for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#74372] [PATCH 1/4] guix: Add go module fetcher Resent-From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 15 Nov 2024 21:12:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 74372 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 74372@debbugs.gnu.org Cc: =?utf-8?q?J=C3=B8rgen?= Kvalsvik X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.1731705105925 (code B ref -1); Fri, 15 Nov 2024 21:12:03 +0000 Received: (at submit) by debbugs.gnu.org; 15 Nov 2024 21:11:45 +0000 Received: from localhost ([127.0.0.1]:51077 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bT-0000Eb-OG for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:44 -0500 Received: from lists.gnu.org ([209.51.188.17]:48708) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bQ-0000Dz-6V for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bN-0002Bt-Ui for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:37 -0500 Received: from mx.kolabnow.com ([212.103.80.155]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bE-00066P-1c for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:32 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id E0B863069A9E for ; Fri, 15 Nov 2024 22:11:18 +0100 (CET) Authentication-Results: ext-mx-out013.mykolab.com (amavis); dkim=pass (2048-bit key) reason="pass (just generated, assumed good)" header.d=kolabnow.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:date:subject:subject:from:from:received :received:received; s=dkim20240523; t=1731705075; x=1733519476; bh=yPzaXCYSdaziVreq7nuo+jL0yuX+QD9obeGjigv5Dl8=; b=qu3YTNjntF4G 6zNBfOomZ50I2sPOO87FGl6ELIQCWzksH1Ed3GElueeX8/1/PtBiNicqyrQt1U0o 3jPsSse2n5j6Hf6AKiKWkTwG1kuNHfjtxXYj6BTaTM0o5tffzSb/PDS308UAXJKA XZtrzQBj/fn3M+3dQeOSKgYkOgQkYI9G41MpRTsbgAlbFUbKTp/1dPikN6L3ilZz U90nz4A8kfELN6nrIrY98DywtrBkx6ZNwn8PnrkjGehZrDreXrfc7x/CYxL3qAEB 9fwvL3MIr5RTKa5+1JATlpm/Z+V1WN7sI055GQ7PVHQ7CJdQN7F5W0vAl2XxQM6v aVEyLkYE6Q== X-Virus-Scanned: amavis at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out013.mykolab.com [127.0.0.1]) (amavis, port 10024) with ESMTP id ZWUaZ5VU-YZZ for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from int-mx011.mykolab.com (unknown [10.9.13.11]) by mx.kolabnow.com (Postfix) with ESMTPS id 20A6B3069A98 for ; Fri, 15 Nov 2024 22:11:14 +0100 (CET) Received: from ext-subm010.mykolab.com (unknown [10.9.6.10]) by int-mx011.mykolab.com (Postfix) with ESMTPS id E5A1D30EDB4A for ; Fri, 15 Nov 2024 22:11:14 +0100 (CET) From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Date: Fri, 15 Nov 2024 22:11:03 +0100 Message-Id: <20241115211106.2759121-2-j@lambda.is> In-Reply-To: <20241115211106.2759121-1-j@lambda.is> References: <20241115211106.2759121-1-j@lambda.is> MIME-Version: 1.0 Received-SPF: pass client-ip=212.103.80.155; envelope-from=j@lambda.is; helo=mx.kolabnow.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Add a new fetcher for go programs, based on go mod download. This is largely how go programs *want* to be fetched, and some pseudo-private dependencies [1] are difficult to wrangle without letting go do the work. This patch only adds support for git as the root module fetch. The approach is conceptually simple: 1. Fetch the root package to build (usually a git repo) 2. Find all the go.mod files in this tree and run go mod download 3. Store the sources of the package, its direct- and transitive dependencies as the source. The source is in source/, the dependencies in go/pkg as encouraged by the toolchain. Go tooling is generally very opinionated on how things are supposed to be done, and fighting it usually brings pain. This approach is not without drawbacks. Go programs tend to be very liberal with dependencies and specific with versions, which will in practice leads to a **large** set of sources. At a system level there will be a lot of duplicated sources, and a lot of slightly different versions which could be compatible. This is not guix specific but rather how go programs design their environments. Another is libraries. Go wants to statically link everything, and generally likes to work with source, and every program pins dependencies to different revisions. The go-mod-fetch getting the source of all libraries the libraries breaks the general packaing approach of packaging libraries separately, but this was never really leveraged by go anyway. This switches the focus to packaging applications rather than libraries. Finally, the package.source needs two hashes - one for the direct origin (e.g. git), and one for the sum of libraries fetched by go.mod. [1] github.com/bufbuild/buf 1.46 requires buf.build/gen/go/bufbuild/bufplugin/protocolbuffers/go v1.35.1-20241031151143-70f632351282.1 which is only available as a .zip and has a layout that go mod understands, but is alien to the go-build-system. * guix/go-mod-download.scm: New file. * gnu/local.mk: Register it. Change-Id: I84c00df07393a9978124667e3e2497aec7009252 --- Makefile.am | 1 + guix/go-mod-download.scm | 146 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 guix/go-mod-download.scm diff --git a/Makefile.am b/Makefile.am index 3a35b7becd..fc00947f4f 100644 --- a/Makefile.am +++ b/Makefile.am @@ -102,6 +102,7 @@ MODULES = \ guix/android-repo-download.scm \ guix/bzr-download.scm \ guix/git-download.scm \ + guix/go-mod-download.scm \ guix/hg-download.scm \ guix/hash.scm \ guix/swh.scm \ diff --git a/guix/go-mod-download.scm b/guix/go-mod-download.scm new file mode 100644 index 0000000000..1a7ffbb6ac --- /dev/null +++ b/guix/go-mod-download.scm @@ -0,0 +1,146 @@ +;;; GNU Guix --- Functional package management for GNU +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; TODOs: +;;; 1. Support non-git root repositories +;;; 2. Store/cache individual module downloads + +(define-module (guix go-mod-download) + #:use-module (guix build utils) + #:use-module (guix derivations) + #:use-module (guix packages) + #:use-module (guix gexp) + #:use-module (guix modules) + #:use-module (guix monads) + #:use-module (guix records) + #:use-module (guix store) + #:use-module (guix git-download) + + #:export (go-mod-reference + go-mod-reference? + go-mod-source + go-mod-go + + go-mod-fetch)) + +(define (go-package) + "Return the default Go package." + (let ((distro (resolve-interface '(gnu packages golang)))) + (module-ref distro 'go))) + +(define (nss-certs-package) + "Return the default nss-certs package." + (let ((distro (resolve-interface '(gnu packages certs)))) + (module-ref distro 'nss-certs))) + +;; The source key accepts the same kinds as the package record, but mostly go +;; use git. The go key specifies which go version to use, which might be +;; necessary if any module sets a newer toolchain in go.mod +(define-record-type* + go-mod-reference make-go-mod-reference + go-mod-reference? + (source go-mod-source) + (go go-mod-go (default (go-package)) (thunked))) + +;; Fetch all direct and indirect dependencies of the go modules in the source +;; tree (usually a git repo) using go mod download. +(define* (go-mod-fetch source hash-algo hash + #:optional name + #:key (system (%current-system)) + (guile (default-guile)) + (go (go-package)) + (nss-certs (nss-certs-package))) + (define guile-json + (module-ref (resolve-interface '(gnu packages guile)) 'guile-json-4)) + + (define* (build source go) + (with-imported-modules (source-module-closure + '((guix build utils) + (guix build download) + (srfi srfi-34))) + (with-extensions (list guile-json) + #~(begin + (use-modules + (guix build download) + (guix build utils) + (srfi srfi-34)) + (let* ((cert-dir (string-append #$nss-certs "/etc/ssl/certs")) + (src-dir (string-append #$output "/source")) + (mod-cache (string-append #$output "/go/pkg")) + (xgo (string-append #$go "/bin/go"))) + ;; go.mod files can specify a minimum required toolchain which could + ;; cause go mod download to fetch and install a newer compiler + ;; if invoked with an older one. + (setenv "GOTOOLCHAIN" (string-append "go" (getenv "go toolchain"))) + (setenv "SSL_CERT_DIR" cert-dir) + (setenv "GOCACHE" "/homeless-shelter") + (setenv "GOPATH" "/homeless-shelter") + (setenv "GOMODCACHE" mod-cache) + + (mkdir-p src-dir) + (mkdir-p mod-cache) + (copy-recursively #$source src-dir) + (let* ((go-mods (find-files src-dir "go.mod"))) + ;; go mod will update the go.mod with transitive dependencies if + ;; they are not set, and fail with an error if the file is not + ;; writable. + (for-each make-file-writable go-mods) + (with-throw-handler + #t + (lambda _ + (for-each + (lambda (go-mod) + (with-directory-excursion (dirname go-mod) + ;; go mod download must be run twice - the first + ;; fetches direct dependencies and *records* + ;; transitive dependencies, the second run fetches + ;; the transitive dependencies. + (and + (invoke xgo "mod" "download") + (invoke xgo "mod" "download")))) + go-mods) + #t) + (lambda (key . args) + (display (string-append "Fetching modules failed.\n" + "Here are the results of `go env`:\n")) + (invoke xgo "env"))))))))) + + (let* ((mod-source (go-mod-source source)) + (mod-ref (origin-uri mod-source)) + (mod-go (go-mod-go source)) + (mod-hash (origin-hash mod-source)) + (mod-hash-value (content-hash-value mod-hash)) + (mod-hash-algo (content-hash-algorithm mod-hash))) + + (mlet* %store-monad ((guile-for-build (package->derivation guile system)) + (git-source (git-fetch mod-ref mod-hash-algo + mod-hash-value + #:system system + #:guile guile-for-build))) + (gexp->derivation (or name "go-mod-fetch") (build git-source mod-go) + #:script-name "go-mod-fetch" + #:env-vars + `(("go toolchain" . ,(package-version mod-go))) + #:leaked-env-vars '("http_proxy" "https_proxy" + "LC_ALL" "LC_MESSAGES" "LANG" + "COLUMNS") + #:system system + #:local-build? #t ;don't offload repo cloning + #:recursive? #t + #:hash-algo hash-algo + #:hash hash + #:guile-for-build guile-for-build)))) From patchwork Fri Nov 15 21:11:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?J=C3=B8rgen_Kvalsvik?= X-Patchwork-Id: 70427 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id F21B827BBEA; Fri, 15 Nov 2024 21:12:41 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id C466B27BBE2 for ; Fri, 15 Nov 2024 21:12:40 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tC3bs-0002FL-9H; Fri, 15 Nov 2024 16:12:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bp-0002EA-CW for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tC3bp-00069p-3q for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=/Tam8bTEpdah4I/zAuQauL0yU7P6UfKtDyyJgKinirU=; b=Vdg2fJXd+3JR7awo/MAtLltyCzl1xWdKknzjMEiL9l9ywfs2loGTnT5z6lezgsGIEfbUe55g7GMityZBSgHKrnWEC7DtVWbIB8a0ON3f7YldQxGEqJoB8w6xzwgEjUYruJxXtDs0y3CWSzADLGV2Ly31lzorP24wLIUpFz+5oMGsU8J5VqQ7+3/N7cNPxRnXk7wJ2mGYaZK0BLNoO5o3eGNc9/X0tp+TyWIJs5e8AZTGpCOnRfnvaWx9sZm5TKR3Y/OXLwVg7W5g6/zCW/y4c92sD53IRZ+byWgHrH9k9Ks4Kuv4e5karat4I0p2gjYCtj4bkCHdjlYPiLsDcmNueQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tC3bo-0000Gt-UA for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#74374] [PATCH 2/4] guix: add go module aware build system Resent-From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 15 Nov 2024 21:12:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 74374 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 74374@debbugs.gnu.org Cc: =?utf-8?q?J=C3=B8rgen?= Kvalsvik X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.1731705111950 (code B ref -1); Fri, 15 Nov 2024 21:12:04 +0000 Received: (at submit) by debbugs.gnu.org; 15 Nov 2024 21:11:51 +0000 Received: from localhost ([127.0.0.1]:51081 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3ba-0000F9-O7 for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:51 -0500 Received: from lists.gnu.org ([209.51.188.17]:48722) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bT-0000EX-2l for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bN-0002Br-Tm for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:37 -0500 Received: from mx.kolabnow.com ([212.103.80.155]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bE-00066J-1T for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:31 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id 76A9220B3656 for ; Fri, 15 Nov 2024 22:11:18 +0100 (CET) Authentication-Results: ext-mx-out011.mykolab.com (amavis); dkim=pass (2048-bit key) reason="pass (just generated, assumed good)" header.d=kolabnow.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:date:subject:subject:from:from:received :received:received; s=dkim20240523; t=1731705075; x=1733519476; bh=/Tam8bTEpdah4I/zAuQauL0yU7P6UfKtDyyJgKinirU=; b=mlQ13lq4hd9z dhcXLTT4yfyXal1Stck3vN3Ql6bL90IBQsc35Pa8jQlVMUqkFk4TYQBeSVxxp8Yq 2FGBUBYeYWnS7Sdwhl9PM7zakYe2BZn54+XnerflwqfIlZWIaD5uaEckEjyqy3KU xcW0cnKAvi7pdPN+y/YeVRNs8wKlECaQLC1e44L2svemIsKl9SrIJkNZCBZwfush cTiQeGjULmcdS+L1vXYdG0ZWXrXd8udfG7TeUqsCh+ifjEKIkGY0tU9KjZ5Int81 XBBOBF0OOrjd5qstCHBiXGe7So8QkJRnguWBropXUsk4wMGeRNWxm/wmc3V8QADC KKnieru8cQ== X-Virus-Scanned: amavis at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out011.mykolab.com [127.0.0.1]) (amavis, port 10024) with ESMTP id TrsSp77ntU42 for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from int-mx009.mykolab.com (unknown [10.9.13.9]) by mx.kolabnow.com (Postfix) with ESMTPS id 5DCA820B3658 for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from ext-subm010.mykolab.com (unknown [10.9.6.10]) by int-mx009.mykolab.com (Postfix) with ESMTPS id 3141D2097183 for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Date: Fri, 15 Nov 2024 22:11:04 +0100 Message-Id: <20241115211106.2759121-3-j@lambda.is> In-Reply-To: <20241115211106.2759121-1-j@lambda.is> References: <20241115211106.2759121-1-j@lambda.is> MIME-Version: 1.0 Received-SPF: pass client-ip=212.103.80.155; envelope-from=j@lambda.is; helo=mx.kolabnow.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Add a go module aware build system, and make it available through build-system/go.scm. The go-mod-build and supporting functions is largely a copy-and-paste job of the go-build and could probably be refactored. The build process when using go modules is slightly different from the non-module version, and relies on sources already being fetched with go-mod-fetch. This revision does not do anything clever with reusing compiled packages, but it might be possible to store precompiled modules to the store path without adding explicit entries in golang-*.scm * guix/build-system/go.scm (%go-mod-build-system-modules): New define. (mod-lower): New function. (go-mod-build): New function. (go-mod-build-system): New function. * guix/build-system/go-mod-build-system.scm: New file. * gnu/local.mk: Register it. Change-Id: I394089073b894e8cf9da5aa18759c939fca45a31 --- Makefile.am | 1 + guix/build-system/go.scm | 120 ++++++++++++++++++++++ guix/build/go-mod-build-system.scm | 154 +++++++++++++++++++++++++++++ 3 files changed, 275 insertions(+) create mode 100644 guix/build/go-mod-build-system.scm diff --git a/Makefile.am b/Makefile.am index fc00947f4f..5768b721aa 100644 --- a/Makefile.am +++ b/Makefile.am @@ -226,6 +226,7 @@ MODULES = \ guix/build/minify-build-system.scm \ guix/build/font-build-system.scm \ guix/build/go-build-system.scm \ + guix/build/go-mod-build-system.scm \ guix/build/android-repo.scm \ guix/build/asdf-build-system.scm \ guix/build/bzr.scm \ diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm index 226688f2d2..1e60fd9471 100644 --- a/guix/build-system/go.scm +++ b/guix/build-system/go.scm @@ -38,6 +38,8 @@ (define-module (guix build-system go) #:export (%go-build-system-modules go-build go-build-system + go-mod-build + go-mod-build-system go-pseudo-version? go-target @@ -117,6 +119,12 @@ (define %go-build-system-modules (guix build union) ,@%default-gnu-imported-modules)) +(define %go-mod-build-system-modules + ;; Build-side modules imported and used by default. + `((guix build go-mod-build-system) + (guix build union) + ,@%default-gnu-imported-modules)) + (define (default-go) ;; Lazily resolve the binding to avoid a circular dependency. (let ((go (resolve-interface '(gnu packages golang)))) @@ -181,6 +189,57 @@ (define inputs-with-cache (build (if target go-cross-build go-build)) (arguments (strip-keyword-arguments private-keywords arguments)))) +(define* (mod-lower name + #:key source inputs native-inputs outputs system target + (go (if (supported-package? (default-go)) + (default-go) + (default-gccgo))) + #:allow-other-keys + #:rest arguments) + "Return a bag for NAME." + (define private-keywords + '(#:target #:go #:inputs #:native-inputs)) + + (define inputs-with-cache + ;; XXX: Avoid a circular dependency. This should be rewritten with + ;; 'package-mapping' or similar. + (let ((go-std-name (string-append (package-name go) "-std"))) + (if (string-prefix? go-std-name name) + inputs + (cons `(,go-std-name ,((make-go-std) go)) inputs)))) + + (bag + (name name) + (system system) + (target target) + (build-inputs `(,@(if source + `(("source" ,source)) + '()) + ,@`(("go" ,go)) + ,@native-inputs + ,@(if target '() inputs-with-cache) + ,@(if target + ;; Use the standard cross inputs of + ;; 'gnu-build-system'. + (standard-cross-packages target 'host) + '()) + ;; Keep the standard inputs of 'gnu-build-system'. + ,@(standard-packages))) + (host-inputs (if target inputs-with-cache '())) + + ;; The cross-libc is really a target package, but for bootstrapping + ;; reasons, we can't put it in 'host-inputs'. Namely, 'cross-gcc' is a + ;; native package, so it would end up using a "native" variant of + ;; 'cross-libc' (built with 'gnu-build'), whereas all the other packages + ;; would use a target variant (built with 'gnu-cross-build'.) + (target-inputs (if target + (standard-cross-packages target 'target) + '())) + + (outputs outputs) + (build go-mod-build) + (arguments (strip-keyword-arguments private-keywords arguments)))) + (define* (go-build name inputs #:key source @@ -310,9 +369,70 @@ (define %outputs #:substitutable? substitutable? #:guile-for-build guile))) +(define* (go-mod-build name inputs + #:key + source + (phases '%standard-phases) + (outputs '("out")) + (search-paths '()) + (install-source? #t) + (import-path "") + (unpack-path "") + (build-flags ''()) + (tests? #t) + (parallel-build? #t) + (parallel-tests? #t) + (allow-go-reference? #f) + (system (%current-system)) + (goarch #f) + (goos #f) + (guile #f) + (imported-modules %go-mod-build-system-modules) + (modules '((guix build go-mod-build-system) + (guix build union) + (guix build utils))) + (substitutable? #t)) + (define builder + (with-imported-modules imported-modules + #~(begin + (use-modules #$@modules) + (go-build #:name #$name + #:source #+source + #:system #$system + #:phases #$phases + #:outputs #$(outputs->gexp outputs) + #:substitutable? #$substitutable? + #:goarch #$goarch + #:goos #$goos + #:search-paths '#$(sexp->gexp + (map search-path-specification->sexp + search-paths)) + #:install-source? #$install-source? + #:import-path #$import-path + #:unpack-path #$unpack-path + #:build-flags #$build-flags + #:tests? #$tests? + #:parallel-build? #$parallel-build? + #:parallel-tests? #$parallel-tests? + #:allow-go-reference? #$allow-go-reference? + #:inputs #$(input-tuples->gexp inputs))))) + + (mlet %store-monad ((guile (package->derivation (or guile (default-guile)) + system #:graft? #f))) + (gexp->derivation name builder + #:system system + #:guile-for-build guile))) + (define go-build-system (build-system (name 'go) (description "Build system for Go programs") (lower lower))) + +(define go-mod-build-system + (build-system + (name 'go) + (description + "Build system for Go programs, module aware") + (lower mod-lower))) diff --git a/guix/build/go-mod-build-system.scm b/guix/build/go-mod-build-system.scm new file mode 100644 index 0000000000..80a43a6a60 --- /dev/null +++ b/guix/build/go-mod-build-system.scm @@ -0,0 +1,154 @@ +;;; GNU Guix --- Functional package management for GNU +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix build go-mod-build-system) + #:use-module ((guix build gnu-build-system) #:prefix gnu:) + #:use-module (guix build utils) + #:use-module (ice-9 match) + #:export (%standard-phases + go-build)) + +;; Commentary: +;; +;; Build procedures for Go packages, using go modules. This is the +;; builder-side code. +;; +;; Software written in Go is either a 'package' (i.e. library) or 'command' +;; (i.e. executable). The module approach is currently heavily biased towards +;; building executables. +;; +;; Unlike the go build system, this builder does not rely on the workspace +;; or GOPATH, but instead assumes all modules are a part of the input source +;; (otherwise, go build tries to download it which would fail). Go projects +;; rigidly specify dependencies which is handled by the sources being resolved +;; and downloaded together. The compiler is fast enough that building +;; everything from source on a per-package basis is not a great speed loss, +;; and the benefit from precompiling libraries is reduced by go statically +;; linking everything anyway. +;; +;; TODO: +;; * Re-use compiled packages +;; +;; Code: + +(define* (setup-go-environment #:key inputs outputs import-path goos goarch + #:allow-other-keys) + "Prepare a Go build environment. We need to tell go to use the specific +toolchain even if a module specifies a (slightly) newer one. We must also +tell the go build command where to find downloaded packages (go/pkg) and +where executables (\"commands\") are installed to." + (let* ((mod-cache (string-append (getcwd) "/go/pkg")) + (src-dir (string-append (getcwd) "/source")) + (go-dir (assoc-ref inputs "go")) + (out-dir (assoc-ref outputs "out"))) + + ;; TODO: Get toolchain from the program itself or package.version, not the + ;; store path + (setenv "GOTOOLCHAIN" (string-delete #\- (strip-store-file-name go-dir))) + (setenv "GOMODCACHE" mod-cache) + (setenv "GOCACHE" (string-append (getcwd) "/go/cache")) + (setenv "GOBIN" (string-append out-dir "/bin")) + (setenv "GO111MODULE" "on") + (setenv "GOARCH" (or goarch (getenv "GOHOSTARCH"))) + (setenv "GOOS" (or goos (getenv "GOHOSTOS"))) + (match goarch + ("arm" + (setenv "GOARM" "7")) + ((or "mips" "mipsel") + (setenv "GOMIPS" "hardfloat")) + ((or "mips64" "mips64le") + (setenv "GOMIPS64" "hardfloat")) + ((or "ppc64" "ppc64le") + (setenv "GOPPC64" "power8")) + (_ #t)))) + +(define* (build #:key import-path build-flags (parallel-build? #t) + #:allow-other-keys) + "Build the package named by IMPORT-PATH." + (let* ((njobs (if parallel-build? (parallel-job-count) 1))) + (setenv "GOMAXPROCS" (number->string njobs))) + + (with-throw-handler + #t + (lambda _ + ;; TODO: This should maybe support list to install multiple commands + ;; from the same project in the same package + (with-directory-excursion (string-append "source/" import-path) + (apply invoke "go" "build" + "-v" ; print the name of packages as they are compiled + "-x" ; print each command as it is invoked + ;; Respectively, strip the symbol table and debug + ;; information, and the DWARF symbol table. + "-ldflags=-s -w" + `(,@build-flags)))) + (lambda (key . args) + (display (string-append "Building '" import-path "' failed.\n" + "Here are the results of `go env`:\n")) + (invoke "go" "env")))) + +(define* (check #:key tests? import-path (parallel-tests? #t) + #:allow-other-keys) + "Run the tests for the package named by IMPORT-PATH." + (when tests? + (let* ((njobs (if parallel-tests? (parallel-job-count) 1))) + (setenv "GOMAXPROCS" (number->string njobs))) + (with-directory-excursion (string-append "source/" import-path) + (invoke "go" "test"))) + #t) + +(define* (install #:key install-source? source outputs import-path + #:allow-other-keys) + (with-directory-excursion (string-append "source/" import-path) + (display "INSTALLING PROGRAM\n") + (invoke "go" "install" + "-v" ; print the name of packages as they are compiled + "-x" ; print each command as it is invoked + ;; Respectively, strip the symbol table and debug + ;; information, and the DWARF symbol table. + "-ldflags=-s -w")) + + ;; TODO: This is probably less interesting when using the go-mod builder + (when install-source? + (let* ((out (assoc-ref outputs "out")) + (src (string-append source "/source")) + (dest (string-append out "/src"))) + (mkdir-p dest) + (copy-recursively src dest #:keep-mtime? #t))) + #t) + +(define* (install-license-files #:rest args) + "Install license files matching LICENSE-FILE-REGEXP to 'share/doc'. Adjust +the standard install-license-files phase to first enter the correct directory." + (with-directory-excursion "source" + (apply (assoc-ref gnu:%standard-phases 'install-license-files) args))) + + +(define %standard-phases + (modify-phases gnu:%standard-phases + (delete 'bootstrap) + (delete 'configure) + (delete 'patch-generated-file-shebangs) + (add-before 'build 'setup-go-environment setup-go-environment) + (replace 'build build) + (replace 'check check) + (replace 'install install) + (replace 'install-license-files install-license-files))) + +(define* (go-build #:key inputs (phases %standard-phases) + #:allow-other-keys #:rest args) + "Build the given Go package, applying all of PHASES in order." + (apply gnu:gnu-build #:inputs inputs #:phases phases args)) From patchwork Fri Nov 15 21:11:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?J=C3=B8rgen_Kvalsvik?= X-Patchwork-Id: 70425 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 904EA27BBEA; Fri, 15 Nov 2024 21:12:37 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E331027BBE2 for ; Fri, 15 Nov 2024 21:12:36 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tC3bp-0002Dy-0R; Fri, 15 Nov 2024 16:12:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bn-0002Df-Ae for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tC3bn-00069Q-1z for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=YaH4k+uq5KG+RAgyIBwT+Nc4HUwgRX/GbVCcqH+6cO4=; b=NwQgVFZU+t/XSccXu04oJC1Qj5uiJTqyxRFsXHdEJuQxzTQtau8W0Y1snEGf3+D7LqyPKPfiAieZRmzmgMDUBsbkWqj2tENaU/OntvA3MF6WI0SqE7B9CEScL6XC+CL51heI8TvqRsSK5wqEd2LzB8jWwGL+R8wZqxRBLzbofVCpYnoQudR0OB+kR7hecXBp5tVDR14YOjfLKb2orEjfbzaqyNpQXZVOg5ikFxXcHJZBUyccusTxK/q3Xq1yzy6/2GNt9O/2y6ekGt5cWe8R6kVW2kYASZhzaPqwZXHFld2k62DnnDsbBSIGZFM3H3z5IbqcFAArigXkKg7+Y5AgNA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tC3bm-0000G7-S2 for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#74371] [PATCH 3/4] guix: Add module aware 'guix import go' Resent-From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 15 Nov 2024 21:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 74371 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 74371@debbugs.gnu.org Cc: =?utf-8?q?J=C3=B8rgen?= Kvalsvik X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.1731705104912 (code B ref -1); Fri, 15 Nov 2024 21:12:02 +0000 Received: (at submit) by debbugs.gnu.org; 15 Nov 2024 21:11:44 +0000 Received: from localhost ([127.0.0.1]:51074 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bS-0000EP-O3 for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:43 -0500 Received: from lists.gnu.org ([209.51.188.17]:48698) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bQ-0000Du-7I for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bN-0002Bq-Sr for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:37 -0500 Received: from mx.kolabnow.com ([212.103.80.155]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bD-00066X-V5 for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:32 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id 4F5653069A98 for ; Fri, 15 Nov 2024 22:11:19 +0100 (CET) Authentication-Results: ext-mx-out013.mykolab.com (amavis); dkim=pass (2048-bit key) reason="pass (just generated, assumed good)" header.d=kolabnow.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:date:subject:subject:from:from:received :received:received; s=dkim20240523; t=1731705075; x=1733519476; bh=YaH4k+uq5KG+RAgyIBwT+Nc4HUwgRX/GbVCcqH+6cO4=; b=nwUM5Gb+Q3O2 OFgQNulbdmbi5Kt7KXG3Tftna1cLWmx7AN0+AujHRKl+qmMH9wZG3Lv3rK2JPqAy jn2FMLYK1EdZUEDlDQve2rDGD4Y9co4lgRlu+T/61oCnoIZeob7mPG45yFLiE/8g Y1tWiJPJ2AQwHKUjZ6LMAozTwy2cOA12OSj1bPvlFfh2yla4DvZQLPk8/WkdRgMt MD4aAKe2bgsr7QV+yHiVvh85Z5YNSoAHp2z+A84T54rYGVAbpJOEqEjKgGxKFSxE EVrHv7BWTM8DhbK5/V57hz1jMNB7LWp4V9ZALpIiZ1tPK3VgBVV4BYeVPsSfJvvf 0GNnZbxIHA== X-Virus-Scanned: amavis at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out013.mykolab.com [127.0.0.1]) (amavis, port 10024) with ESMTP id MNYmo643scnZ for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from int-mx009.mykolab.com (unknown [10.9.13.9]) by mx.kolabnow.com (Postfix) with ESMTPS id BD0103069A9C for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from ext-subm010.mykolab.com (unknown [10.9.6.10]) by int-mx009.mykolab.com (Postfix) with ESMTPS id 762582097185 for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Date: Fri, 15 Nov 2024 22:11:05 +0100 Message-Id: <20241115211106.2759121-4-j@lambda.is> In-Reply-To: <20241115211106.2759121-1-j@lambda.is> References: <20241115211106.2759121-1-j@lambda.is> MIME-Version: 1.0 Received-SPF: pass client-ip=212.103.80.155; envelope-from=j@lambda.is; helo=mx.kolabnow.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_FILL_THIS_FORM_SHORT=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Emit module aware package go packages. It does not compute the hash of the full module yet, and only supports git, but goes a long way towards making it easy to package new go programs using the module-aware build system. * guix/import/go.scm (go-module->guix-package): Add go.mod awareness * guix/scripts/import/go.scm (show-help): Document -m, --mod (%options): Accept them. Change-Id: I4efd7260d69276279940e21698ecc7eb57232a67 --- guix/import/go.scm | 88 ++++++++++++++++++++++++-------------- guix/scripts/import/go.scm | 6 +++ 2 files changed, 61 insertions(+), 33 deletions(-) diff --git a/guix/import/go.scm b/guix/import/go.scm index dd9298808d..967aa54d58 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -615,6 +615,7 @@ (define (validate-version version available-versions module-path) (define* (go-module->guix-package module-path #:key (goproxy "https://proxy.golang.org") version + go-mod? pin-versions? #:allow-other-keys) "Return the package S-expression corresponding to MODULE-PATH at VERSION, a Go package. @@ -641,43 +642,62 @@ (define* (go-module->guix-package module-path #:key (meta-data (fetch-module-meta-data root-module-path)) (vcs-type (module-meta-vcs meta-data)) (vcs-repo-url (module-meta-data-repo-url meta-data goproxy)) + (home-page (format #f "https://~a" root-module-path)) (synopsis (go-package-synopsis module-path)) - (description (go-package-description module-path)) - (licenses (go-package-licenses module-path))) - (values - `(package - (name ,guix-name) - (version ,(strip-v-prefix version*)) - (source - ,(vcs->origin vcs-type vcs-repo-url version*)) - (build-system go-build-system) - (arguments - (list ,@(if (version>? min-go-version (package-version (go-package))) - `(#:go ,(string->number min-go-version)) - '()) - #:import-path ,module-path - ,@(if (string=? module-path-sans-suffix root-module-path) - '() - `(#:unpack-path ,root-module-path)))) - ,@(maybe-propagated-inputs - (map (match-lambda - ((name version) - (go-module->guix-package-name name (strip-v-prefix version))) - (name - (go-module->guix-package-name name))) - dependencies)) - (home-page ,(format #f "https://~a" root-module-path)) - (synopsis ,synopsis) - (description ,(and=> description beautify-description)) - (license ,(match (list->licenses licenses) + (description (and=> (go-package-description module-path) beautify-description)) + (licenses (go-package-licenses module-path)) + (license (match (list->licenses licenses) (() #f) ;unknown license - ((license) ;a single license - license) + ((license) license) ;a single license ((license ...) ;a list of licenses `(list ,@license))))) - (if pin-versions? - dependencies+versions - dependencies)))) + (if go-mod? + (values + `(package + (name ,guix-name) + (version ,(strip-v-prefix version*)) + (source + (origin + (method go-mod-fetch) + (uri (go-mod-reference + (source ,(vcs->origin vcs-type vcs-repo-url version*)))) + (sha256 + (base32 + ;; FIXME: fetch & compute checksum + "0000000000000000000000000000000000000000000000000000")))) + (build-system go-mod-build-system) + (home-page ,home-page) + (synopsis ,synopsis) + (description ,description) + (license ,license))) + (values + `(package + (name ,guix-name) + (version ,(strip-v-prefix version*)) + (source ,(vcs->origin vcs-type vcs-repo-url version*)) + (build-system go-build-system) + (arguments + (list ,@(if (version>? min-go-version (package-version (go-package))) + `(#:go ,(string->number min-go-version)) + '()) + #:import-path ,module-path + ,@(if (string=? module-path-sans-suffix root-module-path) + '() + `(#:unpack-path ,root-module-path)))) + ,@(maybe-propagated-inputs + (map (match-lambda + ((name version) + (go-module->guix-package-name name (strip-v-prefix version))) + (name + (go-module->guix-package-name name))) + dependencies)) + (home-page ,home-page) + (synopsis ,synopsis) + (description ,description) + (license ,license) + (if pin-versions? + dependencies+versions + dependencies)))))) (define go-module->guix-package* (lambda args @@ -699,6 +719,7 @@ (define go-module->guix-package* (define* (go-module-recursive-import package-name #:key (goproxy "https://proxy.golang.org") version + go-mod? pin-versions?) (recursive-import @@ -709,6 +730,7 @@ (define* (go-module-recursive-import package-name (receive (package-sexp dependencies) (go-module->guix-package* name #:goproxy goproxy #:version version + #:go-mod? go-mod? #:pin-versions? pin-versions?) (values package-sexp dependencies)))) #:guix-name go-module->guix-package-name diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm index b90c6ac72f..a12c3a9b2f 100644 --- a/guix/scripts/import/go.scm +++ b/guix/scripts/import/go.scm @@ -50,6 +50,8 @@ (define (show-help) (display (G_ " -h, --help display this help and exit")) (display (G_ " + -m, --mod generate go-module based packages")) + (display (G_ " -r, --recursive generate package expressions for all Go modules that are not yet in Guix")) (display (G_ " @@ -65,6 +67,9 @@ (define %options (lambda args (show-help) (exit 0))) + (option '(#\m "mod") #f #f + (lambda (opt name arg result) + (alist-cons 'go-mod? #t result))) (option '(#\r "recursive") #f #f (lambda (opt name arg result) (alist-cons 'recursive #t result))) @@ -106,6 +111,7 @@ (define (parse-options) (let ((arguments (list name #:goproxy (assoc-ref opts 'goproxy) #:version version + #:go-mod? (assoc-ref opts 'go-mod?) #:pin-versions? (assoc-ref opts 'pin-versions?)))) (if (assoc-ref opts 'recursive) From patchwork Fri Nov 15 21:11:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?J=C3=B8rgen_Kvalsvik?= X-Patchwork-Id: 70426 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id AF2CE27BBEA; Fri, 15 Nov 2024 21:12:38 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id DC1BC27BBE2 for ; Fri, 15 Nov 2024 21:12:37 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tC3bs-0002FM-TX; Fri, 15 Nov 2024 16:12:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bo-0002Dx-N7 for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:04 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tC3bo-00069e-D3 for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=sOLzUXbmn++WwRYEDzd7Oaokx8dkLM6oP0xdf0EMoaQ=; b=LeV61NXTaHFJL6p2m6lwwGO2ULRxw+nn0HOyp7UkixHfeVE912goYuB3+FyNnKChBFq9zrt5xW7NHxhX+CEtcbilmjBmUJVS5fN2jQ/8vnyNJzpzp/9sNn1TAhrovSa3wjZ/vi8m0x+NML0bqmkn7tQaQr4z5JIFL80I8Xl/JvCYM3l+douOrfHvSU2QtgVUhvyyCIFov7SxLvap8sCKbulL9++NiAald9WEag75gk7yCoZiEown0MXymtJ8vLgmnBcSDqnR8ppUR1KVoFLfdD+Oj+aEltLbfqe+FeQudM2Nvp13iO31sGgKy6bGeGSvMpNxqjq23SFzEpvBHjOZfw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tC3bo-0000Gc-7h for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#74373] [PATCH 4/4] gnu: Add go-buf. Resent-From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 15 Nov 2024 21:12:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 74373 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 74373@debbugs.gnu.org Cc: =?utf-8?q?J=C3=B8rgen?= Kvalsvik X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.1731705110942 (code B ref -1); Fri, 15 Nov 2024 21:12:04 +0000 Received: (at submit) by debbugs.gnu.org; 15 Nov 2024 21:11:50 +0000 Received: from localhost ([127.0.0.1]:51079 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3ba-0000F7-2I for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:50 -0500 Received: from lists.gnu.org ([209.51.188.17]:48712) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bR-0000EH-SQ for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bN-0002Bp-St for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:37 -0500 Received: from mx.kolabnow.com ([212.103.80.155]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bD-00066A-V6 for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:30 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id 5219B20B365A for ; Fri, 15 Nov 2024 22:11:17 +0100 (CET) Authentication-Results: ext-mx-out011.mykolab.com (amavis); dkim=pass (2048-bit key) reason="pass (just generated, assumed good)" header.d=kolabnow.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:date:subject:subject:from:from:received :received:received; s=dkim20240523; t=1731705076; x=1733519477; bh=sOLzUXbmn++WwRYEDzd7Oaokx8dkLM6oP0xdf0EMoaQ=; b=I6oYhkSqUfz4 yOCH6Iw//zQZdxY7JDLNv6hXLJxP+pqSaYhpI9Qk8lIvMipNnOXB6gQeqr0HlWiD Vbw0NDNXlENy1BYe5e17+97vF9ierLwqMOT6PCURI2UadI4faNEc37Q+XLP0EDee KlYj1HlxP0OpkQQV4+2aL03jzod4t5cWQ2q/kYXMPOLnrR6maj59B8zhRDl8iUv3 lZPNmWv6h3HxHCEeIb0WtIO/lUh5xiDcnKnF3IbedYtMxpu2fNEUNcJ7HOM35orl 2wHPJhjps5WhOjOA28Sb+L69y9VmpBvA4dGYLErV8TXdVB3bCTW8a/dNsrSi/oSS SAEnXvORZw== X-Virus-Scanned: amavis at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out011.mykolab.com [127.0.0.1]) (amavis, port 10024) with ESMTP id PuqmV_SQ8Pxs for ; Fri, 15 Nov 2024 22:11:16 +0100 (CET) Received: from int-mx009.mykolab.com (unknown [10.9.13.9]) by mx.kolabnow.com (Postfix) with ESMTPS id 0A3CE20B3659 for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from ext-subm010.mykolab.com (unknown [10.9.6.10]) by int-mx009.mykolab.com (Postfix) with ESMTPS id BB74E2097187 for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Date: Fri, 15 Nov 2024 22:11:06 +0100 Message-Id: <20241115211106.2759121-5-j@lambda.is> In-Reply-To: <20241115211106.2759121-1-j@lambda.is> References: <20241115211106.2759121-1-j@lambda.is> MIME-Version: 1.0 Received-SPF: pass client-ip=212.103.80.155; envelope-from=j@lambda.is; helo=mx.kolabnow.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * gnu/packages/golang-xyz.scm (go-buf): New variable. Change-Id: I2ff7ee23bcf5413ad3b4b4b9cbfc8a007b0b9b80 --- gnu/packages/golang-xyz.scm | 63 ++++++++++++++++++++++++++++--------- 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/gnu/packages/golang-xyz.scm b/gnu/packages/golang-xyz.scm index 0d6debcac4..0d8147e484 100644 --- a/gnu/packages/golang-xyz.scm +++ b/gnu/packages/golang-xyz.scm @@ -68,6 +68,7 @@ (define-module (gnu packages golang-xyz) #:use-module (guix build-system copy) #:use-module (guix gexp) #:use-module (guix git-download) + #:use-module (guix go-mod-download) #:use-module (guix packages) #:use-module (guix utils) #:use-module (gnu packages) @@ -7591,31 +7592,65 @@ (define-public glua #:import-path "github.com/yuin/gopher-lua/cmd/glua" #:unpack-path "github.com/yuin/gopher-lua")))) +(define-public go-buf + (package + (name "go-buf") + (version "1.46.0") + (source + (origin + (method go-mod-fetch) + (uri (go-mod-reference + (go go-1.23) + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/bufbuild/buf") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "049rp4f2i759xmws6pqrw3208x1v0zdcmi3ymjbxmaq4hijl9daw")))))) + (sha256 + (base32 "1sk2jhazaa1y2y5m1nmd753s0gr52c8967xr6j9pf2d675fzi7q0")))) + (arguments + (list + #:go go-1.23 + #:import-path "cmd/buf")) + (build-system go-mod-build-system) + (home-page "https://github.com/bufbuild/buf") + (synopsis "Buf") + (description + "The @@url{https://buf.build,(code buf)} CLI is the best tool for working with +@@url{https://protobuf.dev,Protocol Buffers}. It provides:.") + (license license:asl2.0))) + (define-public go-chroma (package (name "go-chroma") (version "2.14.0") (source (origin - (method git-fetch) - (uri (git-reference - (url "https://github.com/alecthomas/chroma") - (commit (string-append "v" version)))) - (file-name (git-file-name name version)) + (method go-mod-fetch) + (uri (go-mod-reference + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/alecthomas/chroma") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "1qgr4gywjks869sc85wb8nby612b8wvsa1dwpsbanjsljq7wq7mp")))))) (sha256 (base32 - "1qgr4gywjks869sc85wb8nby612b8wvsa1dwpsbanjsljq7wq7mp")))) - (build-system go-build-system) + "1xq2v3qr71mbjm328a6snqvba83nq70z6qqjfgh5mp25bwkwy6bl")))) + (build-system go-mod-build-system) (arguments (list #:install-source? #f - #:import-path "github.com/alecthomas/chroma/cmd/chroma")) - (native-inputs - (list go-github-com-alecthomas-assert-v2 - go-github-com-alecthomas-chroma-v2 - go-github-com-alecthomas-kong - go-github-com-mattn-go-colorable - go-github-com-mattn-go-isatty)) + #:import-path "cmd/chroma")) (home-page "https://github.com/alecthomas/chroma") (synopsis "General purpose syntax highlighter in pure Golang") (description