From patchwork Tue May 13 09:55:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?J=C3=B8rgen_Kvalsvik?= X-Patchwork-Id: 42575 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id D803727BC4C; Tue, 13 May 2025 10:56:40 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 1FC1027BC49 for ; Tue, 13 May 2025 10:56:38 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uEmMr-00060B-LU; Tue, 13 May 2025 05:56:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uEmMl-0005zU-Iz for guix-patches@gnu.org; Tue, 13 May 2025 05:56:04 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1uEmMk-0006Mp-Hw for guix-patches@gnu.org; Tue, 13 May 2025 05:56:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=vedhp9stXQG41Bh0LJxMKQwnccRwhfu8Xyg/ClcQQZA=; b=swfrSJPXAvDzytrygK3HXbppzMfvPBFk2Iq6szWmSeuU6gbwXsQ558iTVfTPCWayw82T9xnOGdpX+W9FVBZKyYObP+FeJwVutlH0LaJXcdBT0rmKYrAydtoYAFJZ+i0sAEXSeYP9sPIMhF9di8rtRj90hDNwNwaCZwsT45C/9SJ0IGngW8j8F+X8u5rhKXxtJ82AB9F28Fi+EfE4lqqdbmDTko7VCXSP67q0x0A1S2RKoiHF/01RadDYgtNMoB+qDGTxd6pq11eT2xwI5pNbOZy0yLZo7KTzXxL7xxxx6rXyaT7xb4/9yfDwD9xaRI2oQxjiVWGqxvrzWPfrLCZdxQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1uEmMk-0000RM-6p for guix-patches@gnu.org; Tue, 13 May 2025 05:56:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#78404] [PATCH 2/2] guix: Add module-aware build system for go Resent-From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 13 May 2025 09:56:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 78404 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 78404@debbugs.gnu.org Cc: =?utf-8?q?J=C3=B8rgen?= Kvalsvik , steve@futurile.net Received: via spool by 78404-submit@debbugs.gnu.org id=B78404.17471301431644 (code B ref 78404); Tue, 13 May 2025 09:56:02 +0000 Received: (at 78404) by debbugs.gnu.org; 13 May 2025 09:55:43 +0000 Received: from localhost ([127.0.0.1]:58110 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uEmMP-0000QO-2b for submit@debbugs.gnu.org; Tue, 13 May 2025 05:55:42 -0400 Received: from mx.kolabnow.com ([212.103.80.154]:52794) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uEmMK-0000Pw-Fo for 78404@debbugs.gnu.org; Tue, 13 May 2025 05:55:38 -0400 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id 9DB5F20A9F51; Tue, 13 May 2025 11:55:30 +0200 (CEST) Authentication-Results: ext-mx-out011.mykolab.com (amavis); dkim=pass reason="pass (just generated, assumed good)" header.d=lambda.is DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lambda.is; h= content-transfer-encoding:content-type:content-type:mime-version :references:in-reply-to:message-id:date:date:subject:subject :from:from:received:received:received; s=dkim2; t=1747130129; x= 1748944530; bh=vedhp9stXQG41Bh0LJxMKQwnccRwhfu8Xyg/ClcQQZA=; b=O xK5sKQd4ZhWR+IPfgxHcuuWUKzy/vLyIKYEKHZEpZufF3FnXv1gh98e+vHmqWOYr rr6vG55lWLy9HLjoTqiSJoVPkdPYD1X4ye4warXj8OhWCrhjhjNnW138Nga32U8Y P3v/DFyRXXjCKJClPWHsymXaDs8xwv+IsmO0uTSaSPoly36OtWyxodgZ+NIEjxKx BYWLfr6aXin1lbNLCr+pAbuZDrc4lgWo4uUCxUkANc0Qt2Zdz+MCCzt66GyaOaDt qxOkhrKRuZaldRN+P+0EZEk3ko/b8CwRZXTWTMQNNUF4H6hlqP829uiXzoXv2kD6 n3Dl7pdOTzXF1BjNLSGTQ== X-Virus-Scanned: amavis at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out011.mykolab.com [127.0.0.1]) (amavis, port 10024) with ESMTP id Q1YHtEZ6uccZ; Tue, 13 May 2025 11:55:29 +0200 (CEST) Received: from int-mx011.mykolab.com (unknown [10.9.13.11]) by mx.kolabnow.com (Postfix) with ESMTPS id 22C8420B2741; Tue, 13 May 2025 11:55:28 +0200 (CEST) Received: from ext-subm010.mykolab.com (unknown [10.9.6.10]) by int-mx011.mykolab.com (Postfix) with ESMTPS id E48F8323EF35; Tue, 13 May 2025 11:55:28 +0200 (CEST) From: =?utf-8?q?J=C3=B8rgen?= Kvalsvik Date: Tue, 13 May 2025 11:55:22 +0200 Message-Id: <20250513095522.4313-2-j@lambda.is> In-Reply-To: <20250513095522.4313-1-j@lambda.is> References: <20250513095522.4313-1-j@lambda.is> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Add a new build system for go, using go modules. This build system is partly compatible with go-build-system; they can both be used as build inputs to each other, but their options are incompatible. The main departure from go-build-system is that go-build-system tries to build a workspace [1], where go-module-build-system builds a goproxy + go.mod file and lets `go build` do what it wants to. Most go libraries should be straight forward to build. For example, this is the package definition for golang.org/x/sync@0.12: (define-public go-golang-org-x-sync (package (name "go-golang-org-x-sync") (version "0.12.0") (source (origin (method go-mod-fetch) (uri (go-mod-reference (path "golang.org/x/sync") (version version))) (sha256 (base32 "00pd84ah4xd5sjax8rxv98xbnwrvkk8clazl3kq1xrbkmvjq2m53")))) (build-system go-module-build-system) (home-page "https://golang.org/x/sync") (synopsis "Go Sync") (description "This repository provides Go concurrency primitives in addition to the ones provided by the language and \"sync\" and \"sync/atomic\" packages.") (license license:bsd-3))) The build system also supports higher resolution build-, test-, and install targets, re-use of compiled files, and options for common build tweaks. [1] * guix/build/go-module-build-system.scm: New file. * guix/build-system/go-module.scm: New file. * Makefile.am (MODULES): Add them. Change-Id: I47a028ab8f95fd3a338036480dbad6677e9c50a5 --- Makefile.am | 2 + guix/build-system/go-module.scm | 268 +++++++++++++++ guix/build/go-module-build-system.scm | 459 ++++++++++++++++++++++++++ 3 files changed, 729 insertions(+) create mode 100644 guix/build-system/go-module.scm create mode 100644 guix/build/go-module-build-system.scm diff --git a/Makefile.am b/Makefile.am index b5fb81f412..12446e6bb4 100644 --- a/Makefile.am +++ b/Makefile.am @@ -168,6 +168,7 @@ MODULES = \ guix/build-system/glib-or-gtk.scm \ guix/build-system/gnu.scm \ guix/build-system/go.scm \ + guix/build-system/go-module.scm \ guix/build-system/guile.scm \ guix/build-system/haskell.scm \ guix/build-system/julia.scm \ @@ -227,6 +228,7 @@ MODULES = \ guix/build/minify-build-system.scm \ guix/build/font-build-system.scm \ guix/build/go-build-system.scm \ + guix/build/go-module-build-system.scm \ guix/build/android-repo.scm \ guix/build/asdf-build-system.scm \ guix/build/bzr.scm \ diff --git a/guix/build-system/go-module.scm b/guix/build-system/go-module.scm new file mode 100644 index 0000000000..5692e318d3 --- /dev/null +++ b/guix/build-system/go-module.scm @@ -0,0 +1,268 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2025 Jørgen Kvalsvik +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix build-system go-module) + #:use-module (guix gexp) + #:use-module (guix monads) + #:use-module (guix packages) + #:use-module (guix store) + #:use-module (guix utils) + #:use-module (guix search-paths) + #:use-module (guix build-system) + #:use-module (guix build-system gnu) + #:use-module ((guix build-system go) #:prefix go-build:) + #:use-module (srfi srfi-1) + #:export (%go-module-build-system-modules + go-module-build + go-module-build-system)) + +;;; Commentary: +;;; +;;; Build procedure for packages using the module aware Go build system. +;;; +;;; Code: + +(define %go-module-build-system-modules + ;; Build-side modules imported by default. + `((guix build go-module-build-system) + (guix build union) + ,@%default-gnu-imported-modules)) + +(define (default-go) + ;; Lazily resolve the binding to avoid a circular dependency. + (let ((go (resolve-interface '(gnu packages golang)))) + (module-ref go 'go))) + +(define (default-gccgo) + ;; Lazily resolve the binding to avoid a circular dependency. + (let ((gcc (resolve-interface '(gnu packages gcc)))) + (module-ref gcc 'gccgo-12))) + +(define (default-zip) + "Return the 'zip' package. This is a lazy reference so that we don't +depend on (gnu packages compression)." + (let ((distro (resolve-interface '(gnu packages compression)))) + (module-ref distro 'zip))) + +(define* (lower name + #:key source inputs native-inputs outputs system target + (go (if (supported-package? (default-go)) + (default-go) + (default-gccgo))) + (zip (default-zip)) + #:allow-other-keys + #:rest arguments) + "Return a bag for NAME." + (define private-keywords + '(#:target #:inputs #:native-inputs #:go #:zip)) + + (bag + (name name) + (system system) + (target target) + (build-inputs `(,@(if source + `(("source" ,source)) + '()) + ,@`(("go" ,go) ("zip" ,zip)) + ,@inputs + ,@native-inputs + ,@(if target (standard-cross-packages target 'host) '()) + ;; Keep the standard inputs of 'gnu-build-system'. + ,@(standard-packages))) + (target-inputs (if target (standard-cross-packages target 'target) '())) + (outputs outputs) + (build (if target go-cross-module-build go-module-build)) + (arguments (strip-keyword-arguments private-keywords arguments)))) + +(define* (go-module-build name inputs + #:key + source + (phases '%standard-phases) + (outputs '("out")) + (search-paths '()) + (go-flags '()) + (ld-flags '("-s" "-w")) + (tags '()) + (build-targets '("./...")) + (test-targets '("./...")) + (install-targets '()) + (test-flags '()) + (module-path #f) + (trimpath? #t) + (cgo? #f) + (tests? #t) + (build-output-dir? #f) + (skip-build? #f) + (install-source? #t) + (install-cache? #t) + (parallel-build? #t) + (parallel-tests? #t) + (environment-variables '()) + (system (%current-system)) + (goarch #f) + (goos #f) + (guile #f) + (substitutable? #t) + (imported-modules %go-module-build-system-modules) + (modules '((guix build go-module-build-system) + (guix build utils)))) + + (define builder + (with-imported-modules + imported-modules + #~(begin + (use-modules #$@(sexp->gexp modules)) + (go-module-build #:name #$name + #:source #+source + #:system #$system + #:go-flags '#$go-flags + #:ld-flags '#$ld-flags + #:tags '#$tags + #:build-targets '#$build-targets + #:test-targets '#$test-targets + #:install-targets '#$install-targets + #:test-flags '#$test-flags + #:module-path '#$module-path + #:trimpath? #$trimpath? + #:cgo? '#$cgo? + #:tests? #$tests? + #:build-output-dir? #$build-output-dir? + #:skip-build? #$skip-build? + #:install-source? #$install-source? + #:install-cache? #$install-cache? + #:parallel-build? #$parallel-build? + #:parallel-tests? #$parallel-tests? + #:environment-variables '#$environment-variables + #:goarch #$goarch + #:goos #$goos + #:phases #$phases + #:outputs #$(outputs->gexp outputs) + #:search-paths '#$(map + search-path-specification->sexp + search-paths) + #:inputs #$(input-tuples->gexp inputs))))) + + (mlet %store-monad ((guile (package->derivation (or guile (default-guile)) + system #:graft? #f))) + (gexp->derivation name builder + #:system system + #:guile-for-build guile))) + +(define* (go-cross-module-build name + #:key + source target + build-inputs target-inputs host-inputs + (phases '%standard-phases) + (outputs '("out")) + (search-paths '()) + (native-search-paths '()) + (go-flags '()) + (ld-flags '("-s" "-w")) + (tags '()) + (build-targets '("./...")) + (test-targets '()) + (install-targets '()) + (tests? #f) ; nothing can be done + (test-flags '()) + (module-path #f) + (trimpath? #t) + (cgo? #f) + (build-output-dir? #f) + (skip-build? #f) + (install-source? #t) + (install-cache? #t) + (parallel-build? #t) + (parallel-tests? #f) + (environment-variables '()) + (system (%current-system)) + (goarch (if target (first (go-build:go-target target)) #f)) + (goos (if target (last (go-build:go-target target)) #f)) + (guile #f) + (imported-modules %go-module-build-system-modules) + (modules '((guix build go-module-build-system) + (guix build utils))) + (substitutable? #t)) + + (define builder + (with-imported-modules + imported-modules + #~(begin + (use-modules #$@(sexp->gexp modules)) + + (define %build-host-inputs + #+(input-tuples->gexp build-inputs)) + + (define %build-target-inputs + (append #$(input-tuples->gexp host-inputs) + #+(input-tuples->gexp target-inputs))) + + (define %build-inputs + (append %build-host-inputs %build-target-inputs)) + + (go-module-build #:name #$name + #:source #+source + #:system #$system + #:go-flags '#$go-flags + #:ld-flags '#$ld-flags + #:tags '#$tags + #:build-targets '#$build-targets + #:test-targets '#$test-targets + #:install-targets '#$install-targets + #:test-flags '#$test-flags + #:module-path '#$module-path + #:trimpath? #$trimpath? + #:cgo? '#$cgo? + #:tests? #$tests? + #:build-output-dir? #$build-output-dir? + #:skip-build? #$skip-build? + #:install-source? #$install-source? + #:install-cache? #$install-cache? + #:parallel-build? #$parallel-build? + #:parallel-tests? #$parallel-tests? + #:environment-variables '#$environment-variables + #:target #$target + #:goarch #$goarch + #:goos #$goos + #:phases #$phases + #:outputs #$(outputs->gexp outputs) + #:make-dynamic-linker-cache? #f + #:search-paths '#$(map + search-path-specification->sexp + search-paths) + #:native-search-paths '#$(map + search-path-specification->sexp + native-search-paths) + #:native-inputs %build-host-inputs + #:inputs %build-inputs)))) + + (mlet %store-monad ((guile (package->derivation (or guile (default-guile)) + system #:graft? #f))) + (gexp->derivation name builder + #:system system + #:target target + #:graft? #f + #:substitutable? substitutable? + #:guile-for-build guile))) + +(define go-module-build-system + (build-system + (name 'go-module) + (description "Go Module Build System") + (lower lower))) + +;;; go-module.scm ends here diff --git a/guix/build/go-module-build-system.scm b/guix/build/go-module-build-system.scm new file mode 100644 index 0000000000..8eeaac426c --- /dev/null +++ b/guix/build/go-module-build-system.scm @@ -0,0 +1,459 @@ +(define-module (guix build go-module-build-system) + #:use-module ((guix build gnu-build-system) #:prefix gnu:) + #:use-module (guix build union) + #:use-module (guix build utils) + #:use-module (srfi srfi-71) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (ice-9 match) + #:export (%standard-phases + go-module-build)) + +;;; Commentary: +;;; +;;; Build procedure for packages using the module aware Go build +;;; system. The go build system aggressively tries to fetch dependencies +;;; or even compiler toolchains. While it may be possible to convince it to +;;; not do that, we opt for not fighting it, and instead let it fetch +;;; everything it wants to, served from the local filesystem in directories we +;;; populate. +;;; +;;; The GOPROXY protocol [1] permits using file:// urls. From the manual: +;;; +;;; A module proxy is an HTTP server that can respond to GET requests for +;;; paths specified below. The requests have no query parameters, and no +;;; specific headers are required, so even a site serving from a fixed file +;;; system (including a file:// URL) can be a module proxy. +;;; +;;; Go dependencies tend to be rigidly specified to very specific versions, +;;; with hashes, which the go build tooling will figure out. This does not +;;; work too well with guix' model, where we want to specify dependencies more +;;; fludily (e.g. with input substitutions). Go modules also tend to specify +;;; (minimum) toolchains which is not strictly necessary from a language +;;; feature perspective, which breaks builds with older compilers. +;;; +;;; To address these problems, we always write a fresh go.mod file based on +;;; the build-inputs. There is no guarantee that there even is a go.mod file +;;; in the source, especially for older projects. Go build uses this file to +;;; "download" from our just-assembled goproxy, which makes it happy. This +;;; also clears any toolchain directive which makes the build accept the go +;;; compiler through build-inputs. We populate the goproxy with just-in-time +;;; built zips, version, and info files. This is a separate phase so that +;;; additional build steps can be added between building the proxy and running go +;;; build. +;;; +;;; The build system is compatible with go-build-system, in the sense that +;;; go-build-system can be used as build-inputs, and vice versa, because they +;;; both use the same $out/source/. +;;; +;;; We re-used compiled packages. The Go build system creates a +;;; content-addressable build cache, which we install into build output, and +;;; use to seed downstream builds. Go programs are (mostly) statically +;;; linked, so this is roughly equivalent of installing lib.a. Note that this +;;; only works when the build-input is built with go-module-build-system. +;;; +;;; [1] https://go.dev/ref/mod#goproxy-protocol +;;; +;;; Code: + +(define (find-single-file dir regex) + "Find the file in DIR matching the REGEX, and fail unless there is +exactly one match." + (let ((files (find-files dir regex #:directories? #f))) + (unless (eq? 1 (length files)) + (error "Expected exactly one file matching pattern, found:" files)) + (car files))) + +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an +exclamation mark followed with its lowercase equivalent, as per the module +Escaped Paths specification (see: +https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths)." + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + +(define (call-with-append-file path f) + "call-with-output-file, but appends to the file if it exists rather +than truncating it" + (let ((file (open-file path "a"))) + (f file) + (close file))) + +(define (set-cache-action-epoch f) + "Set go build cache action entry timestamp to 0 + +The go build cache action entries (xxxx-a) record a timestamp, which +would break reproducibility of the build cache. Set it to all-zeros." + ;; The file has 5 columns, tand the timestamp is the rightmost one + ;; + ;; + ;; The timestamp seems to be in nanoseconds since epoch. We use + ;; replacement to avoid potential problems with whitespace + ;; sensitivity. + (let* ((file-line (read-first-line f)) + (start-timestamp (string-skip-right file-line char-set:digit)) + (end-timestamp (string-length file-line)) + (base-line (substring file-line 0 start-timestamp)) + (zero-timestamp (make-string (- end-timestamp start-timestamp) #\0))) + (call-with-output-file f + (lambda (port) + (format port "~a~a~%" base-line zero-timestamp))))) + +(define (copy-nonlink-recursively src-dir dst-dir) + "Recursively install files from src/ to dst/, skipping symlinks" + (copy-recursively src-dir dst-dir + #:log #f + #:copy-file + (lambda (src dst) + (unless (symbolic-link? src) + (copy-file src dst))))) + +(define (make-tags tags) + "Construct a -tags argument list. + +We accept tags both as a single string and a list of tags. go expects +-tags tag1,tag2,..." + (cond ((and (list? tags) (not (null? tags))) + (list "-tags" (string-join tags ","))) + ((string? tags) (list "-tags" tags)) + (else '()))) + +(define (re-init-module module-path) + "Create a fresh go.mod file, replacing an old one if it exists." + ;; Wipe the go.mod if it exists, then create a new one. We might + ;; not use the exact same input set (versions or even modules) as + ;; upstream, e.g. when splitting an upstream package into multiple + ;; parts. + ;; + ;; Delete all go.sum files, if they exist. We do our own + ;; checksums so there is no safety here, and since our packages + ;; are differently sourced (and maybe differently versioned) they + ;; won't match upstream checksums. + (when (file-exists? "go.mod") (delete-file "go.mod")) + (when (file-exists? "go.sum") (delete-file "go.sum")) + (invoke/quiet "go" "mod" "init" module-path)) + +(define (read-first-line f) + "Read the first line in a file" + (call-with-input-file f read-line)) + +(define (filter-go-inputs inputs) + "Return the store paths of go library inputs. + +Inputs is a list of ('pkg' 'store-path') pairs, and returns a list of +store paths. + +((zip . /gnu/store/x1c9w6dnmk23mpdfg08zyq379q26nd88-zip-3.0) + (unzip . /gnu/store/fmli224wbxrz1n0i2lhz6gy8a1ydcbp3-unzip-6.0) + (go-github-com-stretchr-testify . /gnu/store/7p6zk3zka35g3699b9kfl0njzwykimjm-go-github-com-stretchr-testify-1.10.0) + (go-golang-org-x-tools . /gnu/store/ax54x3d7fyywbppqvf0gmavsmxkz0h03-go-golang-org-x-tools-0.25.0)) + +-> + +(/gnu/store/7p6zk3zka35g3699b9kfl0njzwykimjm-go-github-com-stretchr-testify-1.10.0 + /gnu/store/ax54x3d7fyywbppqvf0gmavsmxkz0h03-go-golang-org-x-tools-0.25.0) + +Sources installed with go-build-system and go-module-build-system have +a /src directory. Packages realistically have dozens of inputs (the go +compiler, coreutils, etc.) so this filtering is very much necessary." + (map cdr + (filter + (lambda (input) + (and (string-prefix? "go-" (car input)) + (directory-exists? (string-append (cdr input) "/src")))) + inputs))) + +(define* (infer-module-path-from-dir dir #:optional (subdir "src")) + "Infer the go module 'github.com/user/module' from a store path. + +DIR should be a a store path / + +//src/github.com/user/module/... -> github.com/user/module + +By default, this function assumes the store path is a go-build-system +or go-module-build-system installed package with the sources installed +under DIR/SUBDIR." + (with-directory-excursion (string-append dir "/" subdir) + (string-trim + (car + (sort! (map dirname (find-files "." #:fail-on-error? #t)) + (lambda (x y) (< (string-length x) (string-length y))))) + char-set:punctuation))) + +(define* (module-path-from-file #:optional (go.mod "go.mod")) + "Read the module-path from a go.mod file. + +This assumes that go.mod exists and is well-defined. This can not be +assumed in general until after the build stage, in which case go.mod +should always have been generated in the root dir." + (cadr (string-split (read-first-line go.mod) #\space))) + +(define* (prepare-sources #:key module-path #:allow-other-keys) + ;; Remove any go.mod and go.sum files. The builder will write its + ;; own based on the build inputs, and any upstream checksums will be + ;; wrong. This has to be done before setup-proxy as it would detect + ;; the source dir's go.mod and go.sum files and fail because of + ;; checksum mismatches. It is in its own phase so that it can be + ;; overriden, if necessary. + ;; + ;; We want to init it with go init module so that it records the go + ;; version in order to not fall back to too old go versions: + ;; function instantiation requires go1.18 or later (-lang was set + ;; to go1.16; check go.mod) + ;; + ;; If the module path is explicitly set, use it. Otherwise, + ;; infer it from the go.mod file. If the go.mod file does not + ;; exist and module-path is not specified, fail the build. +(let ((module-path (or module-path (module-path-from-file)))) + (for-each delete-file (find-files "." "go\\.mod$")) + (for-each delete-file (find-files "." "go\\.sum$")) + (re-init-module module-path))) + +(define* (setup-go-env #:key outputs cgo? environment-variables + goarch goos #:allow-other-keys) + (let* ((go-proxy (string-append (getcwd) "/guix-go/proxy")) + (go-cache (string-append (getcwd) "/guix-go/cache")) + (go-mod-cache (string-append (getcwd) "/guix-go/modcache"))) + (setenv "GOSUMDB" "off") + (setenv "GOPROXY" (string-append "file://" go-proxy)) + (setenv "GOCACHE" go-cache) + (setenv "GOMODCACHE" go-mod-cache) + (setenv "GOBIN" (string-append (assoc-ref outputs "out") "/bin")) + (when cgo? + (setenv "CGO_ENABLED" "1")) + + (setenv "GOARCH" (or goarch (getenv "GOHOSTARCH"))) + (setenv "GOOS" (or goos (getenv "GOHOSTOS"))) + (match goarch + ("arm" + (setenv "GOARM" "7")) + ((or "mips" "mipsel") + (setenv "GOMIPS" "hardfloat")) + ((or "mips64" "mips64le") + (setenv "GOMIPS64" "hardfloat")) + ((or "ppc64" "ppc64le") + (setenv "GOPPC64" "power8")) + (_ #t)) + + (for-each + (lambda (var) (setenv (car var) (cdr var))) + environment-variables))) + +(define (module-version-or-synthesized mod-input-path module-path) + "Resolve module or synthesize module version. + +Figure out what module version to use for go get to resolve, either +the package version or a special syntesized one. + +This addresses a quirk of go module paths and the go module system. Go +expects that if a package has a version >= v2.x.y, the module path is +module/v2. When splitting a large package into smaller libraries that +share import prefix, the module path no longer ends with the major +version, and go get complains: + + invalid version: should be v0 or v1, not v4 + +The actual package versions used during the build matters little, and +is an implementation detail for the builder. For the packages with +version >= 2 with an \"unversioned\" module path, synthesize the special +version 0.0.1-guix. + +MOD-INPUT-PATH should be the store path of the module as returned by +filter-go-inputs. MODULE-PATH should be the module path as it is +written in the go.mod, for example: + +(module-version-or-synthesized + \"/gnu/store/2v69cskzdjininks376wlw9cq3dv2gd1-go-github-com-stretchr-objx-0.5.2\" + \"github.com/stretchr/objx\") +" + (let* ((_ pkg-version (package-name->name+version + (strip-store-file-name mod-input-path))) + (dir-version (basename module-path))) + (if (and (string-match "^[2-9]+\\." pkg-version) + (not (string-match "v[2-9]+$" dir-version))) + "0.0.1-guix" + pkg-version))) + +(define* (setup-goproxy #:key inputs #:allow-other-keys) + (let* (;; Remove file:// from the goproxy, we want the dir + (go-proxy (substring (getenv "GOPROXY") 7))) + (mkdir-p go-proxy) + (for-each + (lambda (mod-input-path) + (let* ((store-path (string-append mod-input-path "/src")) + (module-path (infer-module-path-from-dir mod-input-path)) + (source-dir (string-append store-path "/" module-path)) + (version (module-version-or-synthesized mod-input-path module-path)) + (module-dir (format #f "~a@v~a" module-path version)) + (proxy-dir (format #f "~a/~a/@v" go-proxy (go-path-escape module-path))) + (proxy/mod (string-append proxy-dir "/v" version ".mod")) + (proxy/zip (string-append proxy-dir "/v" version ".zip")) + (tmp-dir "guix-tmp") + (tmp-module (string-append tmp-dir "/" module-dir)) + (tmp-mod (string-append tmp-module "/go.mod"))) + + ;; In some cases a module will show up twice, e.g. when + ;; breaking cyclic dependencies. In that case, don't install + ;; the second version. In that case the inputs have different + ;; store paths, and we can't rely at all on the package name. + (unless (file-exists? (string-append proxy-dir "/v" version ".mod")) + (mkdir-p tmp-module) + (copy-recursively source-dir tmp-module #:log #f) + + ;; Delete all go.mod and go.sum files, and re-write the + ;; root one without dependencies and toolchain directives. + ;; Note that we cannot use re-init-module here. If there is + ;; a go.mod in the project root, it would be detected by + ;; re-init-module, and if it happened to contain a + ;; toolchain directive it would infect this module here. + (for-each delete-file (find-files tmp-module "go\\.mod$")) + (for-each delete-file (find-files tmp-module "go\\.sum$")) + (with-directory-excursion tmp-module + (re-init-module module-path)) + (for-each + (lambda (f) (utime f 0 0 0 0 AT_SYMLINK_NOFOLLOW)) + (find-files tmp-dir #:directories? #t)) + + ;; We need the -D flag, because go mod fails hard on any path + ;; that does not begin with $module@version, even if for + ;; sub-prefixes of $module + (mkdir-p proxy-dir) + (with-directory-excursion tmp-dir + (invoke "zip" "-r" "-q" "-o" "-D" "-X" proxy/zip ".")) + + (copy-file tmp-mod proxy/mod) + (delete-file-recursively tmp-dir) + (call-with-output-file (string-append proxy-dir "/v" version ".info") + (lambda (f) (format f "{~s:\"v~a\"}~%" "Version" version))) + + (call-with-append-file (string-append proxy-dir "/list") + (lambda (f) (format f "v~a~%" version)))))) + (filter-go-inputs inputs)))) + +;; These paths must be consistent across different stages, so use +;; symbols for them to ensure they're consistent +(define guix-install-cache "guix-out-cache") +(define var-build-cache "/var/cache/go/build") + +(define* (setup-gocache #:key inputs #:allow-other-keys) + (define (search-input-directories dir go-inputs) + (filter directory-exists? + (map (lambda (store) (string-append store "/" dir)) + go-inputs))) + + (union-build (getenv "GOCACHE") + (search-input-directories var-build-cache + (filter-go-inputs inputs)) + ;; Creating all directories isn't that bad, because + ;; there are only ever 256 of them. + #:create-all-directories? #t + #:log-port (%make-void-port "w"))) + +(define* (build #:key inputs go-flags tags build-targets + install-targets trimpath? build-output-dir? skip-build? + install-cache? install-source? parallel-build? + #:allow-other-keys) + (setenv "GOMAXPROCS" + (number->string + (if parallel-build? (parallel-job-count) 1))) + + (for-each + (lambda (store-path) + (let* ((module (infer-module-path-from-dir store-path)) + (version (module-version-or-synthesized store-path module))) + (invoke "go" "get" (string-append module "@v" version)))) + (filter-go-inputs inputs)) + ;; go.mod and go.sum have had their timestamps updated by go get, + ;; which will be snapshotted in the build cache and break it. From + ;; here on out these files should not need to change, so fix the + ;; timestamp. The sum will only exist if there are any + ;; dependencies. + (utime "go.mod" 0 0 0 0) + (when (file-exists? "go.sum") + (utime "go.sum" 0 0 0 0)) + + ;; If -o is used it must be the first flag to build. This flag is + ;; necessary when there is a command (program) with the same name + ;; as its directory, e.g. info/main.go instead of cmd/info.go + (unless skip-build? + (apply invoke "go" "build" + (append + (if build-output-dir? + (list "-o" (string-append (or (getenv "TMP") "/tmp") + "/go-build/")) '()) + go-flags (make-tags tags) + (if trimpath? '("-trimpath") '()) + build-targets install-targets))) + + ;; Snapshot the cache before running tests. It is not interesting + ;; to snapshot test artifacts, and they may pollute the cache with + ;; non-reproducible artifacts. + (when (and install-cache? (not skip-build?)) + (mkdir-p guix-install-cache) + (with-directory-excursion guix-install-cache + (copy-nonlink-recursively (getenv "GOCACHE") ".") + (delete-file "trim.txt") + (delete-file "README") + (for-each set-cache-action-epoch (find-files "." "-a$"))))) + +(define* (check #:key inputs tests? go-flags tags test-flags + test-targets module-path parallel-tests? trimpath? + #:allow-other-keys) + (when tests? + (let ((njobs (if parallel-tests? (parallel-job-count) 1))) + (setenv "GOMAXPROCS" (number->string njobs)) + (for-each + (lambda target + (apply invoke (append + (list "go" "test") go-flags (make-tags tags) + (if trimpath? '("-trimpath") '()) + test-flags target))) + test-targets)))) + +(define* (install #:key source inputs outputs go-flags tags + install-targets install-cache? install-source? + trimpath? #:allow-other-keys) + (for-each + (lambda target + (apply invoke (append (list "go" "install") go-flags (make-tags tags) + (if trimpath? '("-trimpath") '()) + target))) + install-targets) + + (when (directory-exists? guix-install-cache) + (with-directory-excursion guix-install-cache + (copy-recursively + "." (string-append (assoc-ref outputs "out") var-build-cache) + #:log #f))) + + (when install-source? + (let* ((out (assoc-ref outputs "out")) + (name version (package-name->name+version (assoc-ref outputs "out"))) + (module-line (read-first-line "go.mod")) + (module-path (cadr (string-split module-line #\space))) + (dst (format #f "~a/src/~a" out module-path))) + (copy-recursively source dst #:log #f) + (when (file-exists? (string-append dst "/go.mod")) + (make-file-writable (string-append dst "/go.mod"))) + (install-file "go.mod" dst)))) + +(define %standard-phases + (modify-phases gnu:%standard-phases + (delete 'bootstrap) + (delete 'configure) + (delete 'patch-generated-file-shebangs) + (add-after 'unpack 'prepare-sources prepare-sources) + (add-before 'build 'setup-go-env setup-go-env) + (add-after 'setup-go-env 'setup-goproxy setup-goproxy) + (add-after 'setup-goproxy 'setup-gocache setup-gocache) + (replace 'build build) + (replace 'check check) + (replace 'install install))) + +(define* (go-module-build #:key inputs (phases %standard-phases) + #:allow-other-keys #:rest args) + "Go Module Build System" + (apply gnu:gnu-build #:inputs inputs #:phases phases args)) + +;;; go-module-build-system.scm ends here