mbox series

[bug#50227,0/3] go-build-system and GOPATH improvements

Message ID 20210827151052.12611-1-marius@gnu.org
Headers show
Series go-build-system and GOPATH improvements | expand

Message

Marius Bakke Aug. 27, 2021, 3:10 p.m. UTC
These patches adjust the Go build system to use Guix's regular
native-search-paths mechanism instead of ad-hoc GOPATH trickery.

The context is that I needed to hack on a Go package, and was somewhat
surprised that my usual workflow of "guix environment PKG" did not work.

It still does not work "out of the box", but these patches bring it a
step further.  Now "all" that is needed is to:

  $ cd ~/src/go-foo
  $ guix environment go-example-com-foo
  $ MYGOPATH="$HOME/tmp/go"
  $ NAMESPACE="$MYGOPATH/src/example.com/foo"
  $ mkdir -p $(dirname $NAMESPACE)
  $ ln -s $PWD $NAMESPACE    # or git worktree add $NAMESPACE
  $ export GOPATH="$MYGOPATH:$GOPATH"
  $ go build                 # no 'go get' necessary!

I don't know how feasible it is to avoid making a local directory and
symlinking the project to the expected namespace.  Still a complete Go
newbie, but this approach feels more natural and idiomatic Guix-wise.

Marius Bakke (3):
  build-system/go: Use a native-search-path for GOPATH.
  gnu: hyperledger-fabric: Do not assume GOPATH contains a single entry.
  gnu: go-gotest-tools-assert: Provide internal inputs with the source.

 gnu/packages/golang.scm        | 49 +++++++++++++++++++++++-----------
 gnu/packages/hyperledger.scm   |  6 ++++-
 guix/build/go-build-system.scm | 37 +++----------------------
 3 files changed, 43 insertions(+), 49 deletions(-)

Comments

Sarah Morgensen Aug. 28, 2021, 2:16 a.m. UTC | #1
Hello Marius,

(Apologies in advance for the length of this treatise! I did not have
the time to be concise.)

Marius Bakke <marius@gnu.org> writes:

> These patches adjust the Go build system to use Guix's regular
> native-search-paths mechanism instead of ad-hoc GOPATH trickery.

I have been working on overhauling the Go build system behind the
scenes; I expected to have a patch ready last week, but I fell down the
"modules" rabbit hole after learning GOPATH is expected to be deprecated
as soon as 1.18.  Sorry for the duplicated work!  (I also have a Go 1.17
ready to launch, but I've been attempting to nail down Go build system
changes first so I didn't introduce anything incompatible.)

In any case, I hadn't thought of using search paths, that's quite
clever!  I like it.

Before falling down the "modules" rabbit-hole, here is what was working
for my GOPATH-based Go build system:

1. Install source in "out/share/go/src" rather than "out/src", and then
simply create a directory union of "/share/go/src" from inputs.  This
avoids accidentally including non-go packages with a "/src" directory.
If you did this, then you could make the GOPATH search path be
"/share/go/..".

2. Split the GOPATH as you have done (except with only two components;
one for the package we're building and one for the union in 1).

3. Reuse build artifacts by copying the $GOPATH[0]/pkg directory to
"out/share/go/pkg" in the install phase.  They will be transparently
used since they will be in GOPATH.  (You can use "out/lib/go/pkg", but
you must set 'strip-directories' to avoid stripping Go archives, and
then include "/lib/go/.." in the GOPATH search path.)  However, "go
install" will eventually deprecate installing archives [0], perhaps even
before GOPATH is deprecated.

4. Use -trimpath instead of remove-go-references, as you did.  Also, to
avoid rebuilding the standard library with '-trimpath' for every package
(since the Go build cache does not persist between build environments):

  a) modify the Go package to build standard libraries with -trimpath,
     which would unfortunately mean most users of the Go package would
     find that ~180MB of space wasted; or
  
  b) build a '-trimpath' version of the standard library separately and
     use it with '-pkgdir' (which would prevent #3 from working) or by
     building a directory union of Go and Go-std-library-with-trimpath
     and setting GOROOT=/path/to/union.

Personally, I'm partial to a), along with removing the pre-compiled
standard library from the Go package since it ends up recompiled more
often than not, is very fast to recompile, and it will eventually no
longer be distributed or used by Go [0].

>
> The context is that I needed to hack on a Go package, and was somewhat
> surprised that my usual workflow of "guix environment PKG" did not work.
>
> It still does not work "out of the box", but these patches bring it a
> step further.  Now "all" that is needed is to:
>
>   $ cd ~/src/go-foo
>   $ guix environment go-example-com-foo
>   $ MYGOPATH="$HOME/tmp/go"
>   $ NAMESPACE="$MYGOPATH/src/example.com/foo"
>   $ mkdir -p $(dirname $NAMESPACE)
>   $ ln -s $PWD $NAMESPACE    # or git worktree add $NAMESPACE
>   $ export GOPATH="$MYGOPATH:$GOPATH"
>   $ go build                 # no 'go get' necessary!

Interesting.  I hadn't thought of the use-case for actually hacking on go
packages like this!  I'll have to think of how modules-mode can be made
to work with this.

(A digression: the current issue with fully implementing module-aware
mode is that Go really wants a specific version for each dependency.  If
we just populate the module cache with the versions we have, it will
inevitable complain when a package we try to build wants a version we do
not have.  I see a few solutions:

1. Put all dependencies in the module cache, and rewrite the main
module's go.mod (that is, add replace directives) to replace all
dependencies with the versions we have.

2. Rewrite the go.mod to replace all dependencies with the local
filesystem path of the versions we have.

3. Put all dependencies in the vendor/ directory, and use -mod=vendor.
Any pre-existing vendor directory must be handled properly.

These three solutions fail to allow re-using the build cache (and
therefore build artifacts), because Go computes the build cache keys
differently for main and non-main modules.  Building in Go is generally
fast, so we probably shouldn't compromise much to enable reusing the
build cache, but a few ideas for doing so:

4. Set up a dummy go.mod out of the source tree, which 'replace's all
dependencies AND the module we're building like in 1) or 2).  This may
have to account for replace directives in the go.mod of the module we're
building, though.

5. Put the module we're building in the module cache, and build it with
"go install module@version".  The same caveat as in 4) applies, as well
as that "go install module@version" only works for main packages (that
is, packages which produce an executable).)

>
> I don't know how feasible it is to avoid making a local directory and
> symlinking the project to the expected namespace.  Still a complete Go
> newbie, but this approach feels more natural and idiomatic Guix-wise.

My intuition is that if you're working in GOPATH-mode, you already have
a ~/go/src directory or similar, and your project is probably under
~/go/src/my-project. Then, in order to hack on it Guix-like, you would

  $ cd ~/go/src/my-project
  $ guix environment go-github-com-me-my-project
  $ export GOPATH=~/go:$GOPATH
  $ go build

I'm not sure what a similar idiom for Guix-like hacking in module-aware
mode would be; we'd have to set GOMODCACHE or something, but it would be
very easy for Go to overwrite (or fail to overwrite) things without
GOPROXY=off.  Alternatively, if we make a "full" go proxy directory
layout, we can do

  GOPROXY=file://path/to/gomodcache

or even a search path like

  GOPROXY=file:///gnu/store/p1/gomodcache|file://gnu/store/p2/gomodcache

though I'm not sure how well that would scale w.r.t. number of packages.

Both of these GOPROXY methods have the advantage over setting GOMODCACHE
that the user could modify GOPROXY to include the default proxy, and
would still be able to get packages and versions not packaged by Guix.

I suppose there's no reason we couldn't set both GOPATH and
e.g. GOPROXY.

[0] https://github.com/golang/go/issues/47257

--
Sarah
Marius Bakke Aug. 28, 2021, 2:52 p.m. UTC | #2
Hi Sarah,

Sarah Morgensen <iskarian@mgsn.dev> skriver:

> Hello Marius,
>
> (Apologies in advance for the length of this treatise! I did not have
> the time to be concise.)

No hurries, the insightful feedback is much appreciated.

> Marius Bakke <marius@gnu.org> writes:
>
>> These patches adjust the Go build system to use Guix's regular
>> native-search-paths mechanism instead of ad-hoc GOPATH trickery.
>
> I have been working on overhauling the Go build system behind the
> scenes; I expected to have a patch ready last week, but I fell down the
> "modules" rabbit hole after learning GOPATH is expected to be deprecated
> as soon as 1.18.  Sorry for the duplicated work!  (I also have a Go 1.17
> ready to launch, but I've been attempting to nail down Go build system
> changes first so I didn't introduce anything incompatible.)

That is excellent news, thank you!  I briefly looked into "modules"
while researching this, but realized that the rabbit hole would be too
deep for me.  I did not notice that GOPATH was being deprecated however.

> In any case, I hadn't thought of using search paths, that's quite
> clever!  I like it.
>
> Before falling down the "modules" rabbit-hole, here is what was working
> for my GOPATH-based Go build system:
>
> 1. Install source in "out/share/go/src" rather than "out/src", and then
> simply create a directory union of "/share/go/src" from inputs.  This
> avoids accidentally including non-go packages with a "/src" directory.
> If you did this, then you could make the GOPATH search path be
> "/share/go/..".
>
> 2. Split the GOPATH as you have done (except with only two components;
> one for the package we're building and one for the union in 1).

Installing to $out/share/go is much nicer indeed.

> 3. Reuse build artifacts by copying the $GOPATH[0]/pkg directory to
> "out/share/go/pkg" in the install phase.  They will be transparently
> used since they will be in GOPATH.  (You can use "out/lib/go/pkg", but
> you must set 'strip-directories' to avoid stripping Go archives, and
> then include "/lib/go/.." in the GOPATH search path.)  However, "go
> install" will eventually deprecate installing archives [0], perhaps even
> before GOPATH is deprecated.

Right.  The churn in Go's build tooling is surprising, I would expect it
to slow down at this point!  I suppose we can ignore the "pkg" artifacts
for now, or keep them in "share/go" as a stop-gap, since they will be
deprecated soon.

> 4. Use -trimpath instead of remove-go-references, as you did.  Also, to
> avoid rebuilding the standard library with '-trimpath' for every package
> (since the Go build cache does not persist between build environments):
>
>   a) modify the Go package to build standard libraries with -trimpath,
>      which would unfortunately mean most users of the Go package would
>      find that ~180MB of space wasted; or
>   
>   b) build a '-trimpath' version of the standard library separately and
>      use it with '-pkgdir' (which would prevent #3 from working) or by
>      building a directory union of Go and Go-std-library-with-trimpath
>      and setting GOROOT=/path/to/union.
>
> Personally, I'm partial to a), along with removing the pre-compiled
> standard library from the Go package since it ends up recompiled more
> often than not, is very fast to recompile, and it will eventually no
> longer be distributed or used by Go [0].

Removing the compiled libraries sounds fine to me.  I suppose we'll
still need -trimpath for executables ("main.go")?

>> The context is that I needed to hack on a Go package, and was somewhat
>> surprised that my usual workflow of "guix environment PKG" did not work.
>>
>> It still does not work "out of the box", but these patches bring it a
>> step further.  Now "all" that is needed is to:
>>
>>   $ cd ~/src/go-foo
>>   $ guix environment go-example-com-foo
>>   $ MYGOPATH="$HOME/tmp/go"
>>   $ NAMESPACE="$MYGOPATH/src/example.com/foo"
>>   $ mkdir -p $(dirname $NAMESPACE)
>>   $ ln -s $PWD $NAMESPACE    # or git worktree add $NAMESPACE
>>   $ export GOPATH="$MYGOPATH:$GOPATH"
>>   $ go build                 # no 'go get' necessary!
>
> Interesting.  I hadn't thought of the use-case for actually hacking on go
> packages like this!  I'll have to think of how modules-mode can be made
> to work with this.

It's not a very important feature, and I'm happy to scratch that itch
again once gomodules are first-class.  :-)

> (A digression: the current issue with fully implementing module-aware
> mode is that Go really wants a specific version for each dependency.  If
> we just populate the module cache with the versions we have, it will
> inevitable complain when a package we try to build wants a version we do
> not have.  I see a few solutions:
>
> 1. Put all dependencies in the module cache, and rewrite the main
> module's go.mod (that is, add replace directives) to replace all
> dependencies with the versions we have.
>
> 2. Rewrite the go.mod to replace all dependencies with the local
> filesystem path of the versions we have.
>
> 3. Put all dependencies in the vendor/ directory, and use -mod=vendor.
> Any pre-existing vendor directory must be handled properly.
>
> These three solutions fail to allow re-using the build cache (and
> therefore build artifacts), because Go computes the build cache keys
> differently for main and non-main modules.  Building in Go is generally
> fast, so we probably shouldn't compromise much to enable reusing the
> build cache, but a few ideas for doing so:
>
> 4. Set up a dummy go.mod out of the source tree, which 'replace's all
> dependencies AND the module we're building like in 1) or 2).  This may
> have to account for replace directives in the go.mod of the module we're
> building, though.
>
> 5. Put the module we're building in the module cache, and build it with
> "go install module@version".  The same caveat as in 4) applies, as well
> as that "go install module@version" only works for main packages (that
> is, packages which produce an executable).)

Thank you for this analysis.  The vendoring option is compelling, if it
does not require patching the go.mod files, and can work also for
packages where unbundling is not feasible (or downstream channels with
less strict packaging policies).

For reusing build artifacts, perhaps we can piggy-back on whatever is
implemented for Bazel as mentioned in [0].

>> I don't know how feasible it is to avoid making a local directory and
>> symlinking the project to the expected namespace.  Still a complete Go
>> newbie, but this approach feels more natural and idiomatic Guix-wise.
>
> My intuition is that if you're working in GOPATH-mode, you already have
> a ~/go/src directory or similar, and your project is probably under
> ~/go/src/my-project. Then, in order to hack on it Guix-like, you would
>
>   $ cd ~/go/src/my-project
>   $ guix environment go-github-com-me-my-project
>   $ export GOPATH=~/go:$GOPATH
>   $ go build

That is much easier.  You can tell I never hacked on Go before.  :-)

> I'm not sure what a similar idiom for Guix-like hacking in module-aware
> mode would be; we'd have to set GOMODCACHE or something, but it would be
> very easy for Go to overwrite (or fail to overwrite) things without
> GOPROXY=off.  Alternatively, if we make a "full" go proxy directory
> layout, we can do
>
>   GOPROXY=file://path/to/gomodcache
>
> or even a search path like
>
>   GOPROXY=file:///gnu/store/p1/gomodcache|file://gnu/store/p2/gomodcache
>
> though I'm not sure how well that would scale w.r.t. number of packages.
>
> Both of these GOPROXY methods have the advantage over setting GOMODCACHE
> that the user could modify GOPROXY to include the default proxy, and
> would still be able to get packages and versions not packaged by Guix.
>
> I suppose there's no reason we couldn't set both GOPATH and
> e.g. GOPROXY.

GOPROXY seems like a great middle ground for local development.

Given the conflicting work here, what do you think we should do?  I'm
happy to scrap this PR as it was largely an exercise to learn
go-build-system, in addition to scratching a very minor itch.

Is the reduced complexity worth it while waiting for the gomodules
rewrite, and if so, are there parts that can be merged with your work
such as using $out/share/go?

Let me know if I can be of assistance.  :-)

> [0] https://github.com/golang/go/issues/47257
Sarah Morgensen Aug. 29, 2021, 6:17 a.m. UTC | #3
Hello,

Marius Bakke <marius@gnu.org> writes:

> * gnu/packages/golang.scm (go-gotest-tools-assert)[inputs]: Add
> GO-GOTEST-TOOLS-INTERNAL-FORMAT, GO-GOTEST-TOOLS-INTERNAL-DIFFLIB, and
> GO-GOTEST-TOOLS-INTERNAL-SOURCE.
> [arguments]: Add phase to install a union of the above inputs.
> * gnu/packages/golang.scm (gotestsum)[native-inputs]: Don't add the above
> mentioned inputs.
> ---
>  gnu/packages/golang.scm | 45 +++++++++++++++++++++++++++--------------
>  1 file changed, 30 insertions(+), 15 deletions(-)
>
> diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
> index 3a5c6ddc3f..295b442a2a 100644
> --- a/gnu/packages/golang.scm
> +++ b/gnu/packages/golang.scm
> @@ -20,7 +20,7 @@
>  ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
>  ;;; Copyright © 2020 Nicolas Goaziou <mail@nicolasgoaziou.com>
>  ;;; Copyright © 2020 Ryan Prior <rprior@protonmail.com>
> -;;; Copyright © 2020 Marius Bakke <marius@gnu.org>
> +;;; Copyright © 2020, 2021 Marius Bakke <marius@gnu.org>
>  ;;; Copyright © 2020 raingloom <raingloom@riseup.net>
>  ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
>  ;;; Copyright © 2021 Ricardo Wurmus <rekado@elephly.net>
> @@ -5945,9 +5945,35 @@ gotest-tools.")))
>      (arguments
>       `(#:tests? #f ; Test failure concerning message formatting (FIXME)
>         #:import-path "gotest.tools/assert"
> -       #:unpack-path "gotest.tools"))
> -    ;(propagated-inputs
> -    ; `(("go-gotest-tools-internal-format" ,go-gotest-tools-internal-format)))
> +       #:unpack-path "gotest.tools"
> +       #:modules ((ice-9 match)
> +                  (srfi srfi-26)
> +                  ,@%go-build-system-modules)
> +       #:phases
> +       (modify-phases (@ (guix build go-build-system) %standard-phases)
> +         (add-before 'install 'install-internal-inputs
> +           (lambda* (#:key inputs outputs #:allow-other-keys)
> +             (let ((out (assoc-ref outputs "out")))
> +               ;; The Go compiler does not permit importing libraries with
> +               ;; "internal" in the path from anywhere except below the
> +               ;; package that uses them.  Thus, install these inputs
> +               ;; alongside this package.
> +               (union-build
> +                out
> +                (match (filter (lambda (input)
> +                                 (string-prefix? "go-gotest-tools-internal"
> +                                                 (car input)))
> +                               inputs)
> +                  (((names . directories) ...) directories))
> +                #:create-all-directories? #t
> +                #:log-port (%make-void-port "w"))))))))
> +    (inputs
> +     `(("go-gotest-tools-internal-format"
> +        ,go-gotest-tools-internal-format)
> +       ("go-gotest-tools-internal-difflib"
> +        ,go-gotest-tools-internal-difflib)
> +       ("go-gotest-tools-internal-source"
> +        ,go-gotest-tools-internal-source)))
>      (native-inputs
>       `(("go-github-com-pkg-errors" ,go-github-com-pkg-errors)
>         ("go-github-com-google-go-cmp-cmp"
> @@ -5985,17 +6011,6 @@ test when a comparison fails.")
>          ,go-github-com-jonboulle-clockwork)
>         ("go-golang-org-x-crypto" ,go-golang-org-x-crypto)
>         ("go-gotest-tools-assert" ,go-gotest-tools-assert)
> -       ("go-github-com-google-go-cmp-cmp"
> -        ,go-github-com-google-go-cmp-cmp)
> -       ;; TODO: This would be better as a propagated-input of
> -       ;; go-gotest-tools-assert, but that does not work for
> -       ;; some reason.
> -       ("go-gotest-tools-internal-format"
> -        ,go-gotest-tools-internal-format)
> -       ("go-gotest-tools-internal-difflib"
> -        ,go-gotest-tools-internal-difflib)
> -       ("go-gotest-tools-internal-source"
> -        ,go-gotest-tools-internal-source)
>         ("go-github-com-google-go-cmp-cmp"
>          ,go-github-com-google-go-cmp-cmp)))
>      (synopsis "Go test runner with output optimized for humans")

Just piggybacking off this to add that I believe the "correct" way
forward to handle packages like these is to put all of them in a single
go-gotest-tools package, and modify the build system to build them all.

I've tested a proof-of-concept of this, based off of what Debian does
[0].  Essentially:

1. Add two arguments to the build system, GO-PACKAGES-INCLUDE and
GO-PACKAGES-EXCLUDE.  GO-PACKAGES-INCLUDE defaults to something like
'("IMPORT-PATH/...") and GO-PACKAGES-EXCLUDE defaults to the empty list.

2. Run "go list GO-PACKAGES-INCLUDE", which lists all packages matching
GO-PACKAGES-INCLUDE.

3. Remove any packages matching GO-PACKAGES-EXCLUDE (should this be a
regex? I'm not sure), leaving us with GO-PACKAGES.

4. Run "go install ... GO-PACKAGES"

From my testing, this causes a LOT of packages to need edits, because it
surfaces a lot of hidden bugs.  For example, suppose we have a Guix
package "go-B-tool" with import path "B/tool" and another Guix package
"go-use-B" which imports "B/tool/extra".  If "B/tool/extra" is not
imported by "B/tool", we will not have actually built or tested
"B/tool/extra" so we will only encounter issues when building
"go-use-B", even those the actual issue should be addressed in
"go-B-tool".

In the case of the above package, we would merge all go-gotest-tools
packages into a go-gotest-tools-v3 package, with the import path
"gotest.tools/v3", which is what its go.mod states.  (Note that none of
the sub-packages have their own go.mod, so it would cause issues down
the road with the module system to have each of those sub-packages be a
Guix package.)  With the above build-system changes, all of the
previously-separate packages would be built and tested together.  (If we
wanted to exclude a problematic package which we didn't need, we could
remove the directory in a snippet.)

Depending on how many packages are affected, perhaps this part will
warrant a wip-go-build-system branch?

[0] https://manpages.debian.org/testing/dh-golang/Debian::Debhelper::Buildsystem::golang.3pm.en.html

--
Sarah
Sarah Morgensen Sept. 3, 2021, 10:55 p.m. UTC | #4
Hello,

Marius Bakke <marius@gnu.org> writes:

> That is excellent news, thank you!  I briefly looked into "modules"
> while researching this, but realized that the rabbit hole would be too
> deep for me.  I did not notice that GOPATH was being deprecated however.

The Go 1.16 release notes actually stated that a deprecation notice
would be printed in Go 1.17 but then they didn't do that!  I think
they're waiting for the "workspaces" feature [0] to be implemented
first.

[0] https://github.com/golang/go/issues/45713

>> 4. Use -trimpath instead of remove-go-references, as you did.  Also, to
>> avoid rebuilding the standard library with '-trimpath' for every package
>> (since the Go build cache does not persist between build environments):
>>
>>   a) modify the Go package to build standard libraries with -trimpath,
>>      which would unfortunately mean most users of the Go package would
>>      find that ~180MB of space wasted; or
>>   
>>   b) build a '-trimpath' version of the standard library separately and
>>      use it with '-pkgdir' (which would prevent #3 from working) or by
>>      building a directory union of Go and Go-std-library-with-trimpath
>>      and setting GOROOT=/path/to/union.
>>
>> Personally, I'm partial to a), along with removing the pre-compiled
>> standard library from the Go package since it ends up recompiled more
>> often than not, is very fast to recompile, and it will eventually no
>> longer be distributed or used by Go [0].
>
> Removing the compiled libraries sounds fine to me.  I suppose we'll
> still need -trimpath for executables ("main.go")?

Yes, we'll need '-trimpath' in all invocations of build tools, including
'test' as well, or it will actually recompile everything without
'-trimpath' and then test that!

>> (A digression: the current issue with fully implementing module-aware
>> mode is that Go really wants a specific version for each dependency.  If
>> we just populate the module cache with the versions we have, it will
>> inevitable complain when a package we try to build wants a version we do
>> not have.  I see a few solutions:
>>
>> 1. Put all dependencies in the module cache, and rewrite the main
>> module's go.mod (that is, add replace directives) to replace all
>> dependencies with the versions we have.
>>
>> 2. Rewrite the go.mod to replace all dependencies with the local
>> filesystem path of the versions we have.
>>
>> 3. Put all dependencies in the vendor/ directory, and use -mod=vendor.
>> Any pre-existing vendor directory must be handled properly.
>>
>> These three solutions fail to allow re-using the build cache (and
>> therefore build artifacts), because Go computes the build cache keys
>> differently for main and non-main modules.  Building in Go is generally
>> fast, so we probably shouldn't compromise much to enable reusing the
>> build cache, but a few ideas for doing so:
>>
>> 4. Set up a dummy go.mod out of the source tree, which 'replace's all
>> dependencies AND the module we're building like in 1) or 2).  This may
>> have to account for replace directives in the go.mod of the module we're
>> building, though.
>>
>> 5. Put the module we're building in the module cache, and build it with
>> "go install module@version".  The same caveat as in 4) applies, as well
>> as that "go install module@version" only works for main packages (that
>> is, packages which produce an executable).)
>
> Thank you for this analysis.  The vendoring option is compelling, if it
> does not require patching the go.mod files, and can work also for
> packages where unbundling is not feasible (or downstream channels with
> less strict packaging policies).

Yes, and it has the upside that '-mod=vendor' automatically disables
network access for most tools.

On the other hand, I think the only solution that properly moves in the
direction of allowing hacking on Go packages is the GOPROXY search-path
approach.

This requires some more thought, for sure.

> For reusing build artifacts, perhaps we can piggy-back on whatever is
> implemented for Bazel as mentioned in [0].

Bazel is an entire build system replacing cmd/go, which (presumably)
constructs its own build graph, etc, so probably a little heavy-handed
for our use-case.  We probably want to stay as close as possible to
vanilla tooling.

Probably the best we can do is saving and re-using the build cache,
since they seem insistent on moving everything into the cache.

> Given the conflicting work here, what do you think we should do?  I'm
> happy to scrap this PR as it was largely an exercise to learn
> go-build-system, in addition to scratching a very minor itch.
>
> Is the reduced complexity worth it while waiting for the gomodules
> rewrite, and if so, are there parts that can be merged with your work
> such as using $out/share/go?
>
> Let me know if I can be of assistance.  :-)
>
>> [0] https://github.com/golang/go/issues/47257

I think I would suggest breaking up the Go build system changes into:

1. Make the changes roughly included in your patch, along with making a
"go-std-cache-for-build" package (hidden?) which will be an implicit
input in the Go build system (perhaps non-substitutable? it will be
faster to build than download on nearly all machines), and seeding the
build cache with it in 'setup-go-environment'.  We skip re-using build
artifacts.

2. Update Go build system to use Go 1.16, leaving only docker with Go
1.14 (via "#:go ,go-1.14").  We should be ready to do this as soon as
[0] is fixed [1].

3. Enable building multiple Go packages in one Guix package, and merge
all Guix packages such that one Guix package == one module path.

4. Make gomodules changes.

5. Release Go 1.18, which will bootstrap from Go 1.16 or Gccgo 11 [2].

6. Update Go build system to use Go 1.18.

[0] https://issues.guix.gnu.org/49921
[1] https://logs.guix.gnu.org/guix/2021-08-11.log#213401
[2] <https://github.com/golang/go/issues/44505> build: Adopt Go 11.6 as
bootstrap toolchain for Go 1.18

--
Sarah
Maxim Cournoyer Jan. 14, 2022, 3:13 a.m. UTC | #5
Hello,

[...]

>> I'm not sure what a similar idiom for Guix-like hacking in module-aware
>> mode would be; we'd have to set GOMODCACHE or something, but it would be
>> very easy for Go to overwrite (or fail to overwrite) things without
>> GOPROXY=off.  Alternatively, if we make a "full" go proxy directory
>> layout, we can do
>>
>>   GOPROXY=file://path/to/gomodcache
>>
>> or even a search path like
>>
>>   GOPROXY=file:///gnu/store/p1/gomodcache|file://gnu/store/p2/gomodcache
>>
>> though I'm not sure how well that would scale w.r.t. number of packages.
>>
>> Both of these GOPROXY methods have the advantage over setting GOMODCACHE
>> that the user could modify GOPROXY to include the default proxy, and
>> would still be able to get packages and versions not packaged by Guix.
>>
>> I suppose there's no reason we couldn't set both GOPATH and
>> e.g. GOPROXY.
>
> GOPROXY seems like a great middle ground for local development.
>
> Given the conflicting work here, what do you think we should do?  I'm
> happy to scrap this PR as it was largely an exercise to learn
> go-build-system, in addition to scratching a very minor itch.
>
> Is the reduced complexity worth it while waiting for the gomodules
> rewrite, and if so, are there parts that can be merged with your work
> such as using $out/share/go?
>
> Let me know if I can be of assistance.  :-)

I'm confused by the status of this series :-) Is there something to
salvage, or do we scrap it, awaiting for a proper gomodules solution?

Thanks!

Maxim