diff mbox series

[bug#49565,core-updates] gnu: bootstrap: Use %current-target-system to decide bootstrap path

Message ID 20210716013143.901859-1-bauermann@kolabnow.com
State New
Headers show
Series [bug#49565,core-updates] gnu: bootstrap: Use %current-target-system to decide bootstrap path | expand

Checks

Context Check Description
cbaines/comparison success View comparision
cbaines/git branch success View Git branch
cbaines/applying patch success View Laminar job
cbaines/issue success View issue

Commit Message

Thiago Jung Bauermann July 16, 2021, 1:31 a.m. UTC 
When cross-building a non-x86 gcc-toolchain on an x86 machine,
‘%current-target’ has the x86 system value and Guix tries to use the
mesboot packages for the bootstrap – which don't work because they’re
currently x86-only.

Therefore look at ‘%current-target-system’ first, which has the non-x86
system value.

* gnu/packages/bootstrap.scm (%bootstrap-inputs): Check
‘%current-target-system’ before ‘%current-system’ to decide whether to use
the mesboot path.
* gnu/packages/commencement.scm (%bootstrap-inputs+toolchain): Ditto.
---
 gnu/packages/bootstrap.scm    | 2 +-
 gnu/packages/commencement.scm | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Hi,

As Maxime Devos mentined previously, glibc-mesboot is only supported on
i686-linux and x86_64-linux and not powerpc64le-linux so it’s strange that
it’s being built on my system. Investigating further, I believe the the
problem is the one described on the commit message above.

With the patch applied on core-updates, I can now get the same error that
the CI is getting when trying to cross-compile a gcc-toolchain package, as
seen here:

https://ci.guix.gnu.org/build/667452/details

But I can’t explain why I see this problem on my machine while it doesn’t
happen on the CI one. Perhaps because I’m running Guix on a foreign distro
(Ubuntu), while the CI presumably uses Guix System?

Here are the derivations that are built now:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build \
	--target=powerpc64le-linux-gnu \
	gcc-toolchain@11
substitute: atualizando substitutos de "https://ci.guix.gnu.org"...  100.0%
substitute: atualizando substitutos de "https://bordeaux.guix.gnu.org"... 100.0%
The following derivations will be built:
   /gnu/store/3smglg4didq9bizfa6mpgr3mwyl025bg-gcc-toolchain-11.1.0.drv
   /gnu/store/bqs0df2b3zalz3f6cs3blx3r13f8agwx-linux-libre-headers-5.10.35.drv
   /gnu/store/adi4bagfjzja2cy0xd4zsjrhrx4l8zzh-gcc-bootstrap-0.drv
   /gnu/store/lw6xvva92vb6yrg5s73c6m507mmg4wpn-gcc-11.1.0.drv
   /gnu/store/qm1lgdvlc6wkch2z6qiglqgfjrc9dl52-binutils-2.36.1.drv
   /gnu/store/23fsjsynp3ryb6npgiipx327ag7dd394-file-boot0-5.39.drv
   /gnu/store/m69xqa8k199gnphplml2pc4qbl0w07pj-make-boot0-4.3.drv
   /gnu/store/35v7glpl1p0sbkmyyls4nlcvfdq80bp4-findutils-boot0-4.8.0.drv
   /gnu/store/3lsjr0fpxx07h7kz42sqqkgqi76wr92w-diffutils-boot0-3.7.drv
   /gnu/store/8v38wv9nd9wa1v0xmb38la68pck43xsr-sed-boot0-4.8.drv
   /gnu/store/dxkjjafij00vda2a515clxq2s75l0pxy-patch-boot0-2.7.6.drv
   /gnu/store/g7zi4asrxwqjy2zskgisw098vs49smhn-tar-boot0-1.34.drv
   /gnu/store/k4p3hhphaw59afaxrhyzc58041736qp9-gawk-boot0-5.1.0.drv
   /gnu/store/qr4hzgwji76cr18c2b4ii10wciyplis3-coreutils-boot0-8.32.drv
   /gnu/store/r16wrs5pfad62pp104m58d6z3r010mq6-glibc-2.33.drv
   /gnu/store/crdznbwd3dq9xrmrxzjk1jl80fb2d1ws-gcc-cross-boot0-10.3.0.drv
   /gnu/store/i337vhnb6gh510s0cprbww1l3qila759-libstdc++-boot0-4.9.4.drv
   /gnu/store/vvkqxj9c4710jajddrqd7qbq1vhy1b2q-bzip2-boot0-1.0.8.drv
   /gnu/store/wry17xmsdh3dg33f4397a609kix5492q-binutils-cross-boot0-2.36.1.drv
   /gnu/store/f2yibv3n2ya2s2hcv2n103bmfb39q0q1-bash-static-5.1.8.drv
   /gnu/store/rn9xwjxjxdr6chwr5bdm0law300120yj-ld-wrapper-boot0-0.drv
   /gnu/store/vnaw38hssdshzsy17vmmy7x7zh3ni7zi-glibc-intermediate-2.33.drv
   /gnu/store/qzhgyd8qnwhy46ajsr1156zap6sryxvf-ld-wrapper-0.drv
   /gnu/store/5g8dn0w3k3m2famxndzxywxl6v3r79cm-libunistring-0.9.10.drv
   /gnu/store/0ycvzp0zmlqac5k3b32a7kb9y5z8sz09-gcc-10.3.0.drv
   /gnu/store/dh10c2zsxyv0i3j43q81xfmkhsnh03sh-libstdc++-10.3.0.drv
   /gnu/store/xwifg8mb9zh0azyxccls9q6gcb56p14h-ld-wrapper-boot3-0.drv
   /gnu/store/yljl3dspmzk6256sdl56swsq2r694nsi-zlib-1.2.11.drv
   /gnu/store/w87mmrp0c5irjmqhjj76m27i39zgj3xf-bash-minimal-5.1.8.drv
   /gnu/store/aag6c2sy2w3xyzs4nm0nfwsqw9c1xkvi-libgc-8.0.4.drv
   /gnu/store/vvljjz5dj38yqfd813z6z6xmwsk32jbg-libatomic-ops-7.6.10.drv
   /gnu/store/k4rzmyxidag567qvf7rkpwl0h041vczm-guile-3.0.7.drv
   /gnu/store/d608rkzh04rlaivcm9y0hsj4icanhdyp-libffi-3.3.drv
   /gnu/store/r16wrs5pfad62pp104m58d6z3r010mq6-glibc-2.33.drv
--8<---------------cut here---------------end--------------->8---

Comments

M July 16, 2021, 8:20 a.m. UTC | #1 
Hi,


> diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
> index 5a8028a4652c..b14dc63062e8 100644
> --- a/gnu/packages/bootstrap.scm
> +++ b/gnu/packages/bootstrap.scm
> @@ -927,7 +927,7 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
>  (define (%bootstrap-inputs)
>    ;; The initial, pre-built inputs.  From now on, we can start building our
>    ;; own packages.
> -  (match (%current-system)
> +  (match (or (%current-target-system) (%current-system))
>      ((or "i686-linux" "x86_64-linux")
>       `(("linux-libre-headers" ,%bootstrap-linux-libre-headers)
>         ("bootstrap-mescc-tools" ,%bootstrap-mescc-tools)

This makes sense to me.
However, you may need to modify 'package-from-tarball' as well,
replacing
("tarball" ,(bootstrap-origin (source (%current-system))))))
with
("tarball" ,(bootstrap-origin (source (or (%current-target-system) (%current-system)))))))

Could you verify the architecture of the binaries in the output
of "./pre-inst-env guix build gcc-toolchain@11 --target=powerpc64le-linux-gnu",
and its references (you can use "guix gc --references /gnu/store/..." and objdump)?

Greetings,
Maxime.
Thiago Jung Bauermann July 16, 2021, 8:01 p.m. UTC | #2 
Hello,

Em sexta-feira, 16 de julho de 2021, às 05:20:20 -03, Maxime Devos
escreveu:
> Hi,
> 
> > diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
> > index 5a8028a4652c..b14dc63062e8 100644
> > --- a/gnu/packages/bootstrap.scm
> > +++ b/gnu/packages/bootstrap.scm
> > @@ -927,7 +927,7 @@ exec ~a/bin/.gcc-wrapped -B~a/lib \
> > 
> >  (define (%bootstrap-inputs)
> >  
> >    ;; The initial, pre-built inputs.  From now on, we can start
> >    building our ;; own packages.
> > 
> > -  (match (%current-system)
> > +  (match (or (%current-target-system) (%current-system))
> > 
> >      ((or "i686-linux" "x86_64-linux")
> >      
> >       `(("linux-libre-headers" ,%bootstrap-linux-libre-headers)
> >       
> >         ("bootstrap-mescc-tools" ,%bootstrap-mescc-tools)
> 
> This makes sense to me.
> However, you may need to modify 'package-from-tarball' as well,
> replacing
> ("tarball" ,(bootstrap-origin (source (%current-system))))))
> with
> ("tarball" ,(bootstrap-origin (source (or (%current-target-system)
> (%current-system)))))))

Thanks! I did that but it doesn’t work in this case because the ‘source’
functions expect a Nix system string and ‘%current-target-system’ is a
GNU triplet string. After I defined a function which calls
‘gnu-triplet->nix-system’ on it, then it worked.

This made me realize that all places which do
`(or (%current-target-system) (%current-system))` have this inconsistency.
I’m currently preparing a couple of patches to clean them up.

> Could you verify the architecture of the binaries in the output
> of "./pre-inst-env guix build gcc-toolchain@11
> --target=powerpc64le-linux-gnu", and its references (you can use "guix
> gc --references /gnu/store/..." and objdump)?

I can’t do that on the output because the GCC build fails – which is what
I originally set out to investigate when I stumbled on this problem. :-)

I did a `guix build -K …` and this is the result on the build directory:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build \
	-K \
	--target=powerpc64le-linux-gnu \
	gcc-toolchain@11
    ⋮
<snip lots of output>
    ⋮
make[1]: *** [Makefile:11850: all-target-libstdc++-v3] Error 2
make[1]: Leaving directory '/tmp/guix-build-gcc-11.1.0.drv-0/build'
make: *** [Makefile:962: all] Error 2
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-j" "8" "CFLAGS=-g0 -O2") exit-status: 2 term-signal: #f stop-signal: #f>
phase `build' failed after 810.7 seconds
command "make" "-j" "8" "CFLAGS=-g0 -O2" failed with status 2
note: keeping build directory `/tmp/guix-build-gcc-11.1.0.drv-0'
builder for `/gnu/store/lw6xvva92vb6yrg5s73c6m507mmg4wpn-gcc-11.1.0.drv' failed with exit code 1
build of /gnu/store/lw6xvva92vb6yrg5s73c6m507mmg4wpn-gcc-11.1.0.drv failed
View build log at '/var/log/guix/drvs/lw/6xvva92vb6yrg5s73c6m507mmg4wpn-gcc-11.1.0.drv.bz2'.
cannot build derivation `/gnu/store/gnsdxz7r9l6d9wafwcms2dgz6hhi6gvn-gcc-toolchain-11.1.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/gnsdxz7r9l6d9wafwcms2dgz6hhi6gvn-gcc-toolchain-11.1.0.drv' failed
$ find /tmp/guix-build-gcc-11.1.0.drv-0 -name '*.o' -exec file '{}' ';' | cut -d: -f2 | sort | uniq -c
      3  ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (GNU/Linux), with debug_info, not stripped
    235  ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped
   1496  ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), with debug_info, not stripped
    118  ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
     14  ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), with debug_info, not stripped
--8<---------------cut here---------------end--------------->8---

The vast majority of the files are ppc64le. Of the x86-64 ones, 87 are in
/tmp/guix-build-gcc-11.1.0.drv-0/build/build-x86_64-unknown-linux-gnu/ and
45 are in /tmp/guix-build-gcc-11.1.0.drv-0/build/gcc/build/.

I’m not very familiar with GCC’s build system, so I can’t say whether it’s
expected to have it create these x86-64 objects, but I wouldn’t be surprised
if it needed to build some native auxiliaryprograms for the build process.

Because there’s no finished output, I wasn’t able to check for references.

I can make a more conclusive test when this GCC cross build problem is fixed.
M July 18, 2021, 4:10 p.m. UTC | #3 
Thiago Jung Bauermann schreef op vr 16-07-2021 om 17:01 [-0300]:
> Thanks! I did that but it doesn’t work in this case because the ‘source’
> functions expect a Nix system string and ‘%current-target-system’ is a
> GNU triplet string. After I defined a function which calls
> ‘gnu-triplet->nix-system’ on it, then it worked.
> 
> This made me realize that all places which do
> `(or (%current-target-system) (%current-system))` have this inconsistency.
> I’m currently preparing a couple of patches to clean them up.

There are some places where it doesn't matter if it's the GNU triplet
or Nix system string (e.g. libflame, tlsdate) and there are some places
where the difference does matter (e.g. the definition of libpasastro
seems buggy o me).

> The vast majority of the files are ppc64le. Of the x86-64 ones, 87 are in
> /tmp/guix-build-gcc-11.1.0.drv-0/build/build-x86_64-unknown-linux-gnu/ and
> 45 are in /tmp/guix-build-gcc-11.1.0.drv-0/build/gcc/build/.
> 
> I’m not very familiar with GCC’s build system, so I can’t say whether it’s
> expected to have it create these x86-64 objects, but I wouldn’t be surprised
> if it needed to build some native auxiliaryprograms for the build process.

When compiling GCC (version M) with GCC (version N), first version M is compiled
using version N, then the resulting gcc is used to compile GCC (version M) again.
As I understand it, the idea is to let the end result be independent from the
compiler one started out with.

> Because there’s no finished output, I wasn’t able to check for references.
> 
> I can make a more conclusive test when this GCC cross build problem is fixed.

Ok.

Greetings,
Maxime.
Thiago Jung Bauermann July 18, 2021, 4:45 p.m. UTC | #4 
Em domingo, 18 de julho de 2021, às 13:10:40 -03, Maxime Devos escreveu:
> Thiago Jung Bauermann schreef op vr 16-07-2021 om 17:01 [-0300]:
> > Thanks! I did that but it doesn’t work in this case because the
> > ‘source’
> > functions expect a Nix system string and ‘%current-target-system’ is a
> > GNU triplet string. After I defined a function which calls
> > ‘gnu-triplet->nix-system’ on it, then it worked.
> > 
> > This made me realize that all places which do
> > `(or (%current-target-system) (%current-system))` have this
> > inconsistency. I’m currently preparing a couple of patches to clean
> > them up.
> 
> There are some places where it doesn't matter if it's the GNU triplet
> or Nix system string (e.g. libflame, tlsdate) and there are some places
> where the difference does matter (e.g. the definition of libpasastro
> seems buggy o me).

True. I’m going through the places using ‘%current-target-system’ and 
fixing the ones that seem buggy to me. I changed libpasastro here. I should 
have something to send in the next few days.

> > The vast majority of the files are ppc64le. Of the x86-64 ones, 87 are
> > in
> > /tmp/guix-build-gcc-11.1.0.drv-0/build/build-x86_64-unknown-linux-gnu/
> > and 45 are in /tmp/guix-build-gcc-11.1.0.drv-0/build/gcc/build/.
> > 
> > I’m not very familiar with GCC’s build system, so I can’t say whether
> > it’s expected to have it create these x86-64 objects, but I wouldn’t
> > be surprised if it needed to build some native auxiliaryprograms for
> > the build process.
> When compiling GCC (version M) with GCC (version N), first version M is
> compiled using version N, then the resulting gcc is used to compile GCC
> (version M) again. As I understand it, the idea is to let the end result
> be independent from the compiler one started out with.

Makes sense. Thanks for clarifying!
diff mbox series

Patch

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index 5a8028a4652c..b14dc63062e8 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -927,7 +927,7 @@  exec ~a/bin/.gcc-wrapped -B~a/lib \
 (define (%bootstrap-inputs)
   ;; The initial, pre-built inputs.  From now on, we can start building our
   ;; own packages.
-  (match (%current-system)
+  (match (or (%current-target-system) (%current-system))
     ((or "i686-linux" "x86_64-linux")
      `(("linux-libre-headers" ,%bootstrap-linux-libre-headers)
        ("bootstrap-mescc-tools" ,%bootstrap-mescc-tools)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index fb028390a260..2af3dead3c73 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -2396,7 +2396,7 @@  exec " gcc "/bin/" program
 (define (%bootstrap-inputs+toolchain)
   ;; The traditional bootstrap-inputs.  For the i686-linux, x86_64-linux
   ;; Scheme-only bootstrap the actual reduced set with bootstrapped toolchain.
-  (match (%current-system)
+  (match (or (%current-target-system) (%current-system))
     ((or "i686-linux" "x86_64-linux")
      (%boot-mesboot6-inputs))
     (_