[bug#38390,bug,#38390] Building bootstrap Gash and Gash-Utils

Hi Ludo,

(CC’ing Jan and bug#38390.)

We’re nearing a release for Gash-Utils, and while working on it I made
some changes to how the bootstrap Gash and Gash-Utils are built (on the
wip-bootstrap branch.)  Jan suggested that we get your opinion on the
changes, as you are behind the current '%bootstrap-guile+guild'
approach.

The very short story is that I found that the Scheme GZip code is being
maintained and works out-of-the-box on Guile 2.0.9 [1].  This made me
unsure about copying the code into Gash-Utils when maybe we could just
use it as a dependency.  Then, I realized that the library included a
simple Tar reader, and wondered how simple a program could be that could
handle 'tar xvf gash.tar.gz'.  It turns out pretty simple!  So I put the
program and all its dependencies in an a-list with a little loop that
writes them to disk, and made a self-extracting Scheme script that can
unpack compressed tarballs [2].

[1] https://github.com/weinholt/compression
[2] https://git.ngyro.com/bootar/

The other thing is that I always intended for Gash and Gash-Utils to be
built with a loop calling “compile-file”.  This avoids the need for
“guild” which in turn avoids “bash” (AIUI).

These patches update Gash-Utils, replace the binary “tar” with my
self-extracting Scheme implementation, and replace “guild” with
“compile-file”.  I think this simplifies the bootstrap processes, and I
really like getting rid of the references to “tar”, “bash”, and “xz”
(even though they are needed for the bootstrap Guile, it feels nice to
quarantine them there).

WDYT?


-- Tim

Hello Timothy!

Timothy Sample <samplet@ngyro.com> skribis:

> We’re nearing a release for Gash-Utils, and while working on it I made
> some changes to how the bootstrap Gash and Gash-Utils are built (on the
> wip-bootstrap branch.)  Jan suggested that we get your opinion on the
> changes, as you are behind the current '%bootstrap-guile+guild'
> approach.

Thanks for reaching out to me!

> The very short story is that I found that the Scheme GZip code is being
> maintained and works out-of-the-box on Guile 2.0.9 [1].  This made me
> unsure about copying the code into Gash-Utils when maybe we could just
> use it as a dependency.  Then, I realized that the library included a
> simple Tar reader, and wondered how simple a program could be that could
> handle 'tar xvf gash.tar.gz'.  It turns out pretty simple!  So I put the
> program and all its dependencies in an a-list with a little loop that
> writes them to disk, and made a self-extracting Scheme script that can
> unpack compressed tarballs [2].

Woow, that is very nice!  So all of a sudden there’s an extra bunch of
binary seeds we can get rid of, woohoo!  I didn’t expect that even bzip2
and xz would be implemented.

> [2] https://git.ngyro.com/bootar/

Clean, short, and elegant; brilliant!

> The other thing is that I always intended for Gash and Gash-Utils to be
> built with a loop calling “compile-file”.  This avoids the need for
> “guild” which in turn avoids “bash” (AIUI).
>
> These patches update Gash-Utils, replace the binary “tar” with my
> self-extracting Scheme implementation, and replace “guild” with
> “compile-file”.  I think this simplifies the bootstrap processes, and I
> really like getting rid of the references to “tar”, “bash”, and “xz”
> (even though they are needed for the bootstrap Guile, it feels nice to
> quarantine them there).
>
> WDYT?

I think it’s perfect.

One comment that can be addressed later:

> From 2cec50928a4ff67df363322d2adfb6aaa5aedc83 Mon Sep 17 00:00:00 2001
> From: Timothy Sample <samplet@ngyro.com>
> Date: Mon, 3 Feb 2020 10:51:07 -0500
> Subject: [PATCH 2/3] Simplify bootstrap Gash and Gash-Utils.
>
> This change does three things.  First Gash-Utils is updated to
> 0.1.0-preview.  Then, the bootstrap Gash and Gash-Utils packages are
> arranged to be built without using 'guild'.  Finally, instead of
> using a binary 'tar' via 'bootstrap-executable' to extract Gash and
> Gash-Utils, a self-extracting Scheme implementation of 'tar' and
> 'gzip' is used instead.

[...]

> +(define (make-bootstrap-phases version scripts modules)
> +  "Create a form that modifies the standard GNU build phases so that
> +they build simple Guile programs using only the bootstrap Guile.  The
> +'.in' files in the directory MODULES are configured with VERSION, the
> +'.in' files in the directory SCRIPTS are configured with the bootstrap
> +Guile and its module and object directories, and the Scheme files in the
> +directory MODULES are compiled and installed."
> +  `(modify-phases %standard-phases
> +     (replace 'configure

Should this be factorized out in a (guix build gnu-bootstrap) module or
similar?  That would keep build-side code separate and would avoid
making ‘commencement.scm’ bigger.

I guess the only thing that remains to be done is changing the temporary
URLs to the self-extracting script & co., right?

Thanks a lot!

Ludo’.

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Timothy Sample <samplet@ngyro.com> skribis:
>
> [...]
>
>> So I put the program and all its dependencies in an a-list with a
>> little loop that writes them to disk, and made a self-extracting
>> Scheme script that can unpack compressed tarballs [2].
>
> Woow, that is very nice!  So all of a sudden there’s an extra bunch of
> binary seeds we can get rid of, woohoo!  I didn’t expect that even bzip2
> and xz would be implemented.

I don’t think we could get rid of any binaries just yet.  We need Guile
to run Bootar and we can’t get Guile without the statically linked
“tar”, “xz”, “mkdir”, and “bash”.  This just removes some references to
them.  Although, it occurred to me that we could get something like
Bootar (perhaps further simplified) to run on Mes, in which case we
could use a statically linked Mes to unpack and wrap Guile.  That would
let us get rid of “%bootstrap-executables”.  (Note however that I tried
running Bootar in Mes for fun, and the extractor script – once it was
simplified – caused a segfault.)

BZip2 and XZ just barely work, by the way.  I implemented BZip2, but I
skipped over all the CRC checking.  XZ has a bug when called with input
from stdin, which is how Gash-Utils tries to call it.

>> [2] https://git.ngyro.com/bootar/
>
> Clean, short, and elegant; brilliant!

Thanks!

>> [...]
>>
>> WDYT?
>
> I think it’s perfect.

\o/

> One comment that can be addressed later:
>
>> From 2cec50928a4ff67df363322d2adfb6aaa5aedc83 Mon Sep 17 00:00:00 2001
>> From: Timothy Sample <samplet@ngyro.com>
>> Date: Mon, 3 Feb 2020 10:51:07 -0500
>> Subject: [PATCH 2/3] Simplify bootstrap Gash and Gash-Utils.
>>
>> This change does three things.  First Gash-Utils is updated to
>> 0.1.0-preview.  Then, the bootstrap Gash and Gash-Utils packages are
>> arranged to be built without using 'guild'.  Finally, instead of
>> using a binary 'tar' via 'bootstrap-executable' to extract Gash and
>> Gash-Utils, a self-extracting Scheme implementation of 'tar' and
>> 'gzip' is used instead.
>
> [...]
>
>> +(define (make-bootstrap-phases version scripts modules)
>> +  "Create a form that modifies the standard GNU build phases so that
>> +they build simple Guile programs using only the bootstrap Guile.  The
>> +'.in' files in the directory MODULES are configured with VERSION, the
>> +'.in' files in the directory SCRIPTS are configured with the bootstrap
>> +Guile and its module and object directories, and the Scheme files in the
>> +directory MODULES are compiled and installed."
>> +  `(modify-phases %standard-phases
>> +     (replace 'configure
>
> Should this be factorized out in a (guix build gnu-bootstrap) module or
> similar?  That would keep build-side code separate and would avoid
> making ‘commencement.scm’ bigger.

I would be happy to do that.  It’s nice having everything in one place,
but having a bootstrap build system would certainly make the packages
clearer.  I suppose it could also get rid of the implicit inputs for us
and use “%bootstrap-guile” by default.

> I guess the only thing that remains to be done is changing the temporary
> URLs to the self-extracting script & co., right?

Yup.  I’ll release both packages soon.

> Thanks a lot!

My pleasure!


-- Tim

Hello!

Timothy Sample <samplet@ngyro.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Woow, that is very nice!  So all of a sudden there’s an extra bunch of
>> binary seeds we can get rid of, woohoo!  I didn’t expect that even bzip2
>> and xz would be implemented.
>
> I don’t think we could get rid of any binaries just yet.  We need Guile
> to run Bootar and we can’t get Guile without the statically linked
> “tar”, “xz”, “mkdir”, and “bash”.  This just removes some references to
> them.  Although, it occurred to me that we could get something like
> Bootar (perhaps further simplified) to run on Mes, in which case we
> could use a statically linked Mes to unpack and wrap Guile.  That would
> let us get rid of “%bootstrap-executables”.  (Note however that I tried
> running Bootar in Mes for fun, and the extractor script – once it was
> simplified – caused a segfault.)

Oh cool.  Being able to run Guile code on Mes sounds like a worthy goal
longer-term anyway, so hopefully we’ll get there!

> BZip2 and XZ just barely work, by the way.  I implemented BZip2, but I
> skipped over all the CRC checking.  XZ has a bug when called with input
> from stdin, which is how Gash-Utils tries to call it.

Heh, not so bad.

>>> +(define (make-bootstrap-phases version scripts modules)
>>> +  "Create a form that modifies the standard GNU build phases so that
>>> +they build simple Guile programs using only the bootstrap Guile.  The
>>> +'.in' files in the directory MODULES are configured with VERSION, the
>>> +'.in' files in the directory SCRIPTS are configured with the bootstrap
>>> +Guile and its module and object directories, and the Scheme files in the
>>> +directory MODULES are compiled and installed."
>>> +  `(modify-phases %standard-phases
>>> +     (replace 'configure
>>
>> Should this be factorized out in a (guix build gnu-bootstrap) module or
>> similar?  That would keep build-side code separate and would avoid
>> making ‘commencement.scm’ bigger.
>
> I would be happy to do that.  It’s nice having everything in one place,
> but having a bootstrap build system would certainly make the packages
> clearer.  I suppose it could also get rid of the implicit inputs for us
> and use “%bootstrap-guile” by default.

I doesn’t have to be a full-blown build system, because there might be
as much boilerplate as actual code, but simply moving the definitions of
phases in a separate file could help keep commencement.scm clear.

>> I guess the only thing that remains to be done is changing the temporary
>> URLs to the self-extracting script & co., right?
>
> Yup.  I’ll release both packages soon.

Great, thank you!

Ludo’.

Ludovic Courtès writes:

Hello!

I have rebased wip-bootstrap on core-updates that include your
commencements updates.

With your explanations on irc and studying your patch, I was able to
keep using your cleanups and do without the most ugly 'setenv stages.

So, wip-bootstrap on savannah is reset once again.

> Timothy Sample <samplet@ngyro.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:

>> I would be happy to do that.  It’s nice having everything in one place,
>> but having a bootstrap build system would certainly make the packages
>> clearer.  I suppose it could also get rid of the implicit inputs for us
>> and use “%bootstrap-guile” by default.
>
> I doesn’t have to be a full-blown build system, because there might be
> as much boilerplate as actual code, but simply moving the definitions of
> phases in a separate file could help keep commencement.scm clear.
>
>>> I guess the only thing that remains to be done is changing the temporary
>>> URLs to the self-extracting script & co., right?
>>
>> Yup.  I’ll release both packages soon.
>
> Great, thank you!

Yes, that's great!

@Timothy: I haven't included your previous patches on `wip-bootstrap',
feel free to push them to wip-bootstrap.

We are getting real close to merging this, I think.

Greetings,
janneke

Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> I have rebased wip-bootstrap on core-updates that include your
> commencements updates.
>
> With your explanations on irc and studying your patch, I was able to
> keep using your cleanups and do without the most ugly 'setenv stages.
>
> So, wip-bootstrap on savannah is reset once again.

Great, thanks for taking the time to rebase and adjust the code, and
apologies for all the trouble!  (Actually making this change to
commencement.scm wasn’t on my agenda, it’s one of the things that pop up
and give you a good excuse to avoid more boring stuff, I suppose…)

> @Timothy: I haven't included your previous patches on `wip-bootstrap',
> feel free to push them to wip-bootstrap.
>
> We are getting real close to merging this, I think.

This is exciting.  Let’s synchronize with Marius once the branch has
stabilized to see if this can go in ‘core-updates’ this time.

Thanks,
Ludo’.

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> @Timothy: I haven't included your previous patches on `wip-bootstrap',
>> feel free to push them to wip-bootstrap.
>>
>> We are getting real close to merging this, I think.
>
> This is exciting.  Let’s synchronize with Marius once the branch has
> stabilized to see if this can go in ‘core-updates’ this time.

Indeed!

I just pushed five commits to “wip-bootstrap”.  They are a little
different from the patches I sent.

For one, I followed Ludo’s advice and added a “gnu-bootstrap” module for
the build code.  That means I got rid of “make-bootstrap-phases” and
adjusted the packages accordingly.  It’s a little janky, but it does
clear up “commencement.scm”.

Bootar is released and saw some cosmetic improvements.  Notably, it
disables “escape-newlines” and uses “pretty-print”, making the SES file
easy to read for humans.  The SES file is still hosted on my server, but
at a stable URL.  Is it worrisome that it’s on my server?  I put a
mirror on Gitlab <https://gitlab.com/samplet/bootar/-/tags/v1>, but the
URL to the actual file is kinda ugly.  Thoughts?

Gash-Utils is released!  The Git repo and tarball are on Savannah.  I
updated how the base package builds a little bit, and fiddled with the
home-page, synopsis, and description.  The bootstrap version was
installing a useless “template” binary, which I fixed.

I noticed a little issue with “%boot-mesboot1-inputs”, so I fixed it and
made sure that “bootar” does not get propagated past that point.

I have not tested the full bootstrap yet, but I did test “hello-mesboot”
on basically this same code, and it was fine.  Since almost nothing
substantial has been changed, it should be OK, but you never know until
you try!  I’ll build the full thing as fast as my little computer can.
:)


-- Tim

Timothy Sample writes:

Hello,

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>> @Timothy: I haven't included your previous patches on `wip-bootstrap',
>>> feel free to push them to wip-bootstrap.
>>>
>>> We are getting real close to merging this, I think.
>>
>> This is exciting.  Let’s synchronize with Marius once the branch has
>> stabilized to see if this can go in ‘core-updates’ this time.
>
> Indeed!
>
> I just pushed five commits to “wip-bootstrap”.  They are a little
> different from the patches I sent.

Very nice.

> For one, I followed Ludo’s advice and added a “gnu-bootstrap” module for
> the build code.  That means I got rid of “make-bootstrap-phases” and
> adjusted the packages accordingly.  It’s a little janky, but it does
> clear up “commencement.scm”.

I like it!

> Bootar is released and saw some cosmetic improvements.  Notably, it
> disables “escape-newlines” and uses “pretty-print”, making the SES file
> easy to read for humans.  The SES file is still hosted on my server, but
> at a stable URL.  Is it worrisome that it’s on my server?  I put a
> mirror on Gitlab <https://gitlab.com/samplet/bootar/-/tags/v1>, but the
> URL to the actual file is kinda ugly.  Thoughts?
>
> Gash-Utils is released!  The Git repo and tarball are on Savannah.  I
> updated how the base package builds a little bit, and fiddled with the
> home-page, synopsis, and description.  The bootstrap version was
> installing a useless “template” binary, which I fixed.

Oh, congrats!

> I noticed a little issue with “%boot-mesboot1-inputs”, so I fixed it and
> made sure that “bootar” does not get propagated past that point.
>
> I have not tested the full bootstrap yet, but I did test “hello-mesboot”
> on basically this same code, and it was fine.  Since almost nothing
> substantial has been changed, it should be OK, but you never know until
> you try!  I’ll build the full thing as fast as my little computer can.
> :)

I found that the new file-boot0 5.38 does not build with the new bzip
feature; I added a squash! commit to disable that.

janneke

Hi Jan,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> I found that the new file-boot0 5.38 does not build with the new bzip
> feature; I added a squash! commit to disable that.

I managed to build GNU Hello, but I had to disable “bzlib” for the
“file” package in “%final-inputs” as well.  I assume that’s a problem.

You may have already noticed, but I thought I would mention it just in
case.


-- Tim

Hi Timothy,

Timothy Sample <samplet@ngyro.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>> @Timothy: I haven't included your previous patches on `wip-bootstrap',
>>> feel free to push them to wip-bootstrap.
>>>
>>> We are getting real close to merging this, I think.
>>
>> This is exciting.  Let’s synchronize with Marius once the branch has
>> stabilized to see if this can go in ‘core-updates’ this time.
>
> Indeed!
>
> I just pushed five commits to “wip-bootstrap”.  They are a little
> different from the patches I sent.

Awesome, thank you!

> For one, I followed Ludo’s advice and added a “gnu-bootstrap” module for
> the build code.  That means I got rid of “make-bootstrap-phases” and
> adjusted the packages accordingly.  It’s a little janky, but it does
> clear up “commencement.scm”.
>
> Bootar is released and saw some cosmetic improvements.  Notably, it
> disables “escape-newlines” and uses “pretty-print”, making the SES file
> easy to read for humans.  The SES file is still hosted on my server, but
> at a stable URL.  Is it worrisome that it’s on my server?  I put a
> mirror on Gitlab <https://gitlab.com/samplet/bootar/-/tags/v1>, but the
> URL to the actual file is kinda ugly.  Thoughts?

We can host a copy at <https://ftp.gnu.org/gnu/guix/mirror>, which will
be picked up by Software Heritage and archived for eternity.  :-)
Let’s do that when we merge the branch.

> Gash-Utils is released!  The Git repo and tarball are on Savannah.  I
> updated how the base package builds a little bit, and fiddled with the
> home-page, synopsis, and description.  The bootstrap version was
> installing a useless “template” binary, which I fixed.
>
> I noticed a little issue with “%boot-mesboot1-inputs”, so I fixed it and
> made sure that “bootar” does not get propagated past that point.
>
> I have not tested the full bootstrap yet, but I did test “hello-mesboot”
> on basically this same code, and it was fine.  Since almost nothing
> substantial has been changed, it should be OK, but you never know until
> you try!  I’ll build the full thing as fast as my little computer can.
> :)

Thank you for all the great work!

Ludo’.