[bug#70492] bug#66866: Grafting breaks cross-compilation

Hi,

dan <i@dan.games> skribis:

> I spent some time digging into the rabbit hole.  After changing 
> the lower function of the copy-build-system to look more like the 
> lower function of the gnu-build-system, I'm able to cross compile 
> alsa-lib without the --no-grafts flag.  The changes I made are 
> like:
>
> diff --git a/guix/build-system/copy.scm 
> b/guix/build-system/copy.scm
> index d58931b33c..74304b4bfb 100644
> --- a/guix/build-system/copy.scm
> +++ b/guix/build-system/copy.scm
> @@ -66,13 +66,13 @@ (define* (lower name
>    (bag
>      (name name)
>      (system system)
> -    (host-inputs `(,@(if source
> +    (build-inputs `(,@(if source
>                           `(("source" ,source))
>                           '())
> -                   ,@inputs
> +                   ,@native-inputs
>                     ;; Keep the standard inputs of 
>                     'gnu-build-system'.
>                     ,@(standard-packages)))
> -    (build-inputs native-inputs)
> +    (host-inputs inputs)
>      (outputs outputs)
>      (build copy-build)
>      (arguments (strip-keyword-arguments private-keywords 
>      arguments))))
>
> Can we put everything inside build-inputs?  From my understanding, 
> copy-build-system shouldn't care about cross-compilation at all.

Intuitively, if ‘copy-build-system’ is about copying
architecture-independent files, then it should do the same thing whether
or not we are cross-compiling.

However, users can and do add phases whose result is
architecture-dependent.  Small sample:

  • ‘desec-certbot-hook’ captures a reference to curl, so it would get
    the wrong one when cross-compiling if we assumed build-inputs =
    host-inputs.

  • ‘chez-scheme-for-racket-bootstrap-bootfiles’ builds stuff when
    cross-compiling.  Philip, could you explain the intent and what you
    expect here?

So it would seem we can’t just assume everything is a native input like
https://issues.guix.gnu.org/70492 does.

Now, as David and you found out, the use of inputs in
build-system/copy.scm:lower is bogus.  It seems that it can be fixed by
following the intended definition of build/host inputs, as David
suggested:
But wait! That’s all theoretical because the bag always has (target #f)
and ‘copy-build’ bundles build and host inputs together, as if doing a
native build.

So it seems like https://issues.guix.gnu.org/70492 (putting everything
in ‘build-inputs’) is OK, after all.

But still, there seem to be some expectation that ‘copy-build-system’
can support cross-compilation for real, so maybe we should add a
‘copy-cross-build’ procedure in addition to the patch above.

Thoughts?

Ludo’.
  

Hello,

I think I finally understand why the problem only occurs with grafts if
a dependency uses copy-build-system. It is actually somewhat
complicated.

When a package is lowered into a derivation by 'package->derivation' in
guix/packages.scm, a list of potentially applicable grafts is created by
the 'bag-grafts' procedure.
This works by first traversing the bag for native packages (where
bag-build-inputs and bag-target-inputs are followed recursively; and
also bag-host-inputs if not cross building) and building the native
derivations of the grafted versions of the packages which have a
replacement (such as glibc at the time of writing). Building the grafted
package (in the 'input-graft' procedure) is done again with
'package->derivation', which eventually calls 'graft-derivation' in
guix/grafts.scm, which in turn calls 'non-self-references', where the
ungrafted package is actually built (ignoring the store monad
indirection), not just its derivation.
In the case of cross builds, the bag is also traversed for the target
dependencies, where the bag-host-inputs are followed recursively.
Analogously, this causes the ungrafted packages and their ungrafted
replacement to be cross built to compute the grafted derivations.

As dan found out, the 'lower' procedure in guix/build-system/copy.scm
incorrectly puts standard-packages into the host inputs. They contain
the glibc-final package in gnu/packages/commencement.scm (which inherits
the replacement of glibc). Because copy-build-system puts them into
bag-host-inputs, the package replacement of the glibc-final package is
then cross built without grafts when calculating its grafted derivation,
which it does not support. Note that the glibc-final package and its
package replacement are distinct from the glibc package and its
replacement in gnu/packages/base.scm. I confirmed this by running
--8<---------------cut here---------------start------------->8---
,use (guix)
(with-store store (run-with-store store (package->cross-derivation (package-replacement (@@ (gnu packages commencement) glibc-final)) "i686-linux-gnu" #:graft? #f)))
--8<---------------cut here---------------end--------------->8---
in the REPL, which returns the same incorrect glibc derivation that is
attempted to be built as a dependency when running
`guix build alsa-lib --target=i686-linux-gnu`.

It is actually possible to compute the graft derivations only when
needed by a store reference (which does not change the derivation of the
grafted package). Then, this problem does not occur even without the fix
for the copy-build-system, as the invalid derivations are never actually
built. With substitutes or after running GC, this may even prevent
building a package with replacement when the requested package output
which (transitively) depends on it has no (transitive) store reference
to it. I made a patch for that here: https://issues.guix.gnu.org/70895

Regarding the changes to the copy-build-system:

Ludovic Courtès <ludo@gnu.org> writes:

> But still, there seem to be some expectation that ‘copy-build-system’
> can support cross-compilation for real, so maybe we should add a
> ‘copy-cross-build’ procedure in addition to the patch above.

In guix/build-system/copy.scm, it is described as an extension of
gnu-build-system and the manual says "It adds much of the
gnu-build-system packages to the set of inputs. Because of this, the
copy-build-system does not require all the boilerplate code often needed
for the trivial-build-system."
Therefore, I think it makes sense to add the copy-cross-build procedure
so that copy-build-system actually behaves like gnu-build-system.

Cheers,
David