diff mbox series

[bug#39655,core-updates] doc: Add 'Scheme-only Bootstrap' node.

Message ID 87367yhgva.fsf@gnu.org
State Accepted
Headers show
Series [bug#39655,core-updates] doc: Add 'Scheme-only Bootstrap' node. | expand

Checks

Context Check Description
cbaines/applying patch fail View Laminar job

Commit Message

Janneke Nieuwenhuizen May 17, 2020, 12:43 p.m. UTC 
Ludovic Courtès writes:

Hello Ludo, Timothy!

Finally another version of the new bootstrap situation.  Timothy, please
have a look and feel free to change/add some things!

It's also getting time to write a blog post on this, would you like to
help out with that?

>> * doc/guix.texi (@detailmenu): Add Bootstrapping menu.
>> (Scheme-only Bootstrap): New node.
>> * doc/images/gcc-core-mesboot0-scheme-only-graph.dot: New image.
>> * doc/local.mk (DOT_FILES): Add it.
>
> Great!  I wonder if we need a new section though.  Do you think we could
> just keep it all in a single bootstrapping section that would describe
> the current state of affairs, possibly with a “Bootstrapping History”
> section or similar that would describe the milestones that got us there?
[..]
> That way, the main bootstrapping section would simply describe the
> current status in a standalone fashion.

Thanks for your thoughts on this.  I started to rework this by adding a
History section but in the end I removed it again, describing the
current situation while mentioning the gcc-triplet reduction in Guix
v1.1.

>> +@c
>> +packages@footnote{@c
>> +gash-boot,
>> +gash-utils-boot,
>> +%bootstrap-mes-rewired,
>> +gzip-mesboot,
>
> I wouldn’t list them all here because (1) it’s hard to understand what
> these names refer to, unless reading commencement.scm is your favorite
> hobby ;-), and (2) it’s bound to become outdated.
>
> WDYT?

Yes, instead of adding more of these, I also removed the previous list.

>> +As Gash and Gash Utils mature, and GNU packages become more

@Timothy: this is possibly touching very lightly on what Gash has
brought us; do you want to share some more here?

>> +bootstrappable (e.g., new releases of GNU Sed will also ship in
[..]
>> +@image{images/gcc-core-mesboot0-scheme-only-graph,6in,,Dependency graph of gcc-core-mesboot0}
>
> It would be nice to have a couple of lines commenting the graph, like:

Sure!  Updated version attached.

Greetings,
Janneke

Comments

Timothy Sample May 18, 2020, 5:53 p.m. UTC | #1 
Hi janneke,

Overall it is very good!  I only have a one small suggestion.

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> [...]
>
>>> +As Gash and Gash Utils mature, and GNU packages become more
>
> @Timothy: this is possibly touching very lightly on what Gash has
> brought us; do you want to share some more here?

Honestly I think this says enough for now.  I have most of the code in
place so that Gash can build nearly anything.  (In particular, Awk is
nearly complete with respect to POSIX.)  It will take a while to clean
it up, but we can refresh this part when the new code lands.  :)

> [...]
>
> +The only significant binary bootstrap seeds that remain@footnote{
> +Ignoring the 68KB @code{mescc-tools}; that will be removed later,
> +together with @code{mes}.} are a Scheme intepreter and a Scheme
> +compiler: GNU Mes and GNU Guile.

Is it too confusing to mention the Bash, Tar, and XZ binaries used to
get Guile up and running?  IIRC, there are a few packages whose sources
are still patched via “bootstrap-origin”, too.  I understand that those
binaries are really insignificant, but maybe someone new to this whole
thing would be surprised to discover “secret” bootstrap binaries.
Perhaps their role in unpacking Guile could be put in the footnote with
messcc-tools.  WDYT?


-- Tim
Janneke Nieuwenhuizen May 22, 2020, 2:20 p.m. UTC | #2 
Hello!

Timothy Sample writes:

>> +The only significant binary bootstrap seeds that remain@footnote{
>> +Ignoring the 68KB @code{mescc-tools}; that will be removed later,
>> +together with @code{mes}.} are a Scheme intepreter and a Scheme
>> +compiler: GNU Mes and GNU Guile.
>
> Is it too confusing to mention the Bash, Tar, and XZ binaries used to
> get Guile up and running?  IIRC, there are a few packages whose sources
> are still patched via “bootstrap-origin”, too.  I understand that those
> binaries are really insignificant, but maybe someone new to this whole
> thing would be surprised to discover “secret” bootstrap binaries.
> Perhaps their role in unpacking Guile could be put in the footnote with
> messcc-tools.  WDYT?

Good question -- I've added this

compiler: GNU Mes and GNU Guile@footnote{Not shown in this graph are the
static binaries for @file{bash}, @code{tar}, and @code{xz} that are used
to get Guile running.}.

Efraim Flashner writes:

> > GNU Sed will also ship as gzipped tarballs again, as alternative to the
> > hard to bootstrap @code{xz}-compression), this set of added packages can
> > hopefully be reduced again.

> I have to ask since we're talking about bootstrapability; is it actually
> not bootstrappable or is it just harder/more complex? (The question is
> more about the wording than about the logistics of it.)

Good question, changed it to "hard to bootstrap", like so

GNU Sed will also ship as gzipped tarballs again, as alternative to the
hard to bootstrap @code{xz}-compression), this set of added packages can
hopefully be reduced again.

Thanks for your input, pushed to master!

janneke
diff mbox series

Patch

From b1341ecfc77345f1a423bbc66b6255a35613425e Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Tue, 18 Feb 2020 11:36:37 +0100
Subject: [PATCH] doc: Update 'Bootstrapping' for further binary seed
 reduction.

* doc/images/gcc-core-mesboot0-graph.dot: New image, replacing
* doc/images/gcc-mesboot0-bag-graph.dot: ... remove file.
* doc/local.mk (DOT_FILES): Update for new image file.
* doc/guix.texi (Reduced Binary Seed Bootstrap): Use it in updated description
of further reduction of the trusted computing base.
---
 doc/guix.texi                          | 114 +++++++++++++------------
 doc/images/gcc-core-mesboot0-graph.dot | 114 +++++++++++++++++++++++++
 doc/local.mk                           |   2 +-
 3 files changed, 175 insertions(+), 55 deletions(-)
 create mode 100644 doc/images/gcc-core-mesboot0-graph.dot

diff --git a/doc/guix.texi b/doc/guix.texi
index 90324ce291..833509860c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -35,7 +35,7 @@  Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
 Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@*
 Copyright @copyright{} 2016 John Darrington@*
 Copyright @copyright{} 2016, 2017 Nikita Gillmann@*
-Copyright @copyright{} 2016, 2017, 2018, 2019 Jan Nieuwenhuizen@*
+Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@*
 Copyright @copyright{} 2016 Julien Lepiller@*
 Copyright @copyright{} 2016 Alex ter Weele@*
 Copyright @copyright{} 2016, 2017, 2018, 2019 Christopher Baines@*
@@ -28606,10 +28606,8 @@  Binutils, libc, and the other packages mentioned above---the
 @dfn{bootstrap binaries}.
 
 These bootstrap binaries are ``taken for granted'', though we can also
-re-create them if needed (more on that later).
-
-For @code{i686-linux} and @code{x86_64-linux} the Guix bootstrap process is
-more elaborate, @pxref{Reduced Binary Seed Bootstrap}.
+re-create them if needed (see @pxref{Preparing to Use the Bootstrap
+Binaries}).
 
 @menu
 * Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
@@ -28625,60 +28623,68 @@  Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils, and the
 GNU C Library (@pxref{Bootstrapping}).  Usually, these bootstrap binaries are
 ``taken for granted.''
 
-Taking these binaries for granted means that we consider them to be a correct
-and trustworthy `seed' for building the complete system.  Therein lies a
-problem: the current combined size of these bootstrap binaries is about 250MB
-(@pxref{Bootstrappable Builds,,, mes, GNU Mes}).  Auditing or even inspecting
-these is next to impossible.
+Taking the bootstrap binaries for granted means that we consider them to
+be a correct and trustworthy `seed' for building the complete system.
+Therein lies a problem: the combined size of these bootstrap binaries is
+about 250MB (@pxref{Bootstrappable Builds,,, mes, GNU Mes}).  Auditing
+or even inspecting these is next to impossible.
 
-For @code{i686-linux} and @code{x86_64-linux}, Guix now features a ``Reduced
-Binary Seed'' bootstrap @footnote{We would like to say: ``Full Source
-Bootstrap'' and while we are working towards that goal it would be hyperbole
-to use that term for what we do now.}.
+For @code{i686-linux} and @code{x86_64-linux}, Guix now features a
+``Reduced Binary Seed'' bootstrap @footnote{We would like to say: ``Full
+Source Bootstrap'' and while we are working towards that goal it would
+be hyperbole to use that term for what we do now.}.
 
 The Reduced Binary Seed bootstrap removes the most critical tools---from a
 trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C
 Library are replaced by: @code{bootstrap-mescc-tools} (a tiny assembler and
 linker) and @code{bootstrap-mes} (a small Scheme Interpreter and a C compiler
-written in Scheme and the Mes C Library, built for TinyCC and for GCC).  Using
-these new binary seeds and a new set of
-@c
-packages@footnote{@c
-nyacc-boot,
-mes-boot,
-tcc-boot0,
-tcc-boot,
-make-mesboot0,
-diffutils-mesboot,
-binutils-mesboot0,
-gcc-core-mesboot,
-mesboot-headers,
-glibc-mesboot0,
-gcc-mesboot0,
-binutils-mesboot,
-make-mesboot,
-gcc-mesboot1,
-gcc-mesboot1-wrapper,
-glibc-headers-mesboot,
-glibc-mesboot,
-gcc-mesboot,
-and
-gcc-mesboot-wrapper.
-}
-@c
-the ``missing'' Binutils, GCC, and the GNU C Library are built from source.
-From here on the more traditional bootstrap process resumes.  This approach
-has reduced the bootstrap binaries in size to about 130MB.  Work is ongoing to
-reduce this further.  If you are interested, join us on @code{#bootstrappable}
-on the Freenode IRC network.
-
-@c ./pre-inst-env guix graph --type=bag -e '(begin (use-modules (guix packages)) (%current-system "i686-linux") (@@ (gnu packages commencement) gcc-mesboot))' > doc/images/gcc-mesboot-bag-graph.dot
-@c dot -T png doc/images/gcc-mesboot-bag-graph.dot > doc/images/gcc-mesboot-bag-graph.png
-
-Below is the generated dependency graph for @code{gcc-mesboot}, the bootstrap
-compiler used to build the rest of GuixSD.
-
-@image{images/gcc-mesboot-bag-graph,6in,,Dependency graph of the gcc-mesboot}
+written in Scheme and the Mes C Library, built for TinyCC and for GCC).
+
+Using these new binary seeds the ``missing'' Binutils, GCC, and the GNU
+C Library are built from source.  From here on the more traditional
+bootstrap process resumes.  This approach has reduced the bootstrap
+binaries in size to about 145MB in Guix v1.1.
+
+The next step that Guix has taken is to replace the shell and all its
+utilities with implementations in Guile Scheme, the @emph{Scheme-only
+bootstrap}.  Gash (@pxref{Gash,,, gash, The Gash manual}) is a
+POSIX-compatible shell that replaces Bash, and it comes with Gash Utils
+which has minimalist replacements for Awk, the GNU Core Utilities, Grep,
+Gzip, Sed, and Tar.  The rest of the bootstrap binary seeds that were
+removed are now built from source.
+
+Building the GNU System from source is currently only possibly by adding
+some historical GNU packages as intermediate steps@footnote{Packages
+such as @code{gcc-2.95.3}, @code{binutils-2.14}, @code{glibc-2.2.5},
+@code{gzip-1.2.4}, @code{tar-1.22}, and some others for details see
+@file{gnu/packages/commencement.scm}}.  As Gash and Gash Utils mature,
+and GNU packages become more bootstrappable again (e.g., new releases of
+GNU Sed will also ship in @code{tar.gz} format), this set of added
+packages can hopefully be reduced again.
+
+The graph below shows the resulting dependency graph for
+@code{gcc-core-mesboot0}, the bootstrap compiler used for the
+traditional bootstrap of the rest of the Guix System.
+
+@c ./pre-inst-env guix graph -e '(@@ (gnu packages commencement) gcc-core-mesboot0)' | sed -re 's,((bootstrap-mescc-tools|bootstrap-mes|guile-bootstrap).*shape =) box,\1 ellipse,' > doc/images/gcc-core-mesboot0-graph.dot
+@image{images/gcc-core-mesboot0-graph,6in,,Dependency graph of gcc-core-mesboot0}
+
+The only significant binary bootstrap seeds that remain@footnote{
+Ignoring the 68KB @code{mescc-tools}; that will be removed later,
+together with @code{mes}.} are a Scheme intepreter and a Scheme
+compiler: GNU Mes and GNU Guile.
+
+This further reduction has brought down the size of the binary seed to
+about 60MB for @code{i686-linux} and @code{x86_64-linux}.
+
+Work is ongoing to remove all binary blobs from our free software
+bootstrap stack, working towards a Full Source Bootstrap.  Also ongoing
+is work to bring these bootstraps to the @code{arm-linux} and
+@code{aarch64-linux} architectures and to the Hurd.
+
+If you are interested, join us on @code{#bootstrappable} on the Freenode
+IRC network or discuss on @code{bug-mes@@gnu.org} or
+@code{gash-devel@@nongnu.org}.
 
 @node Preparing to Use the Bootstrap Binaries
 @section Preparing to Use the Bootstrap Binaries
@@ -28698,7 +28704,7 @@  guix graph -t derivation \
   | dot -Tps > gcc.ps
 @end example
 
-or, for the Reduced Binary Seed bootstrap
+or, for the further Reduced Binary Seed bootstrap
 
 @example
 guix graph -t derivation \
diff --git a/doc/images/gcc-core-mesboot0-graph.dot b/doc/images/gcc-core-mesboot0-graph.dot
new file mode 100644
index 0000000000..191582ed4c
--- /dev/null
+++ b/doc/images/gcc-core-mesboot0-graph.dot
@@ -0,0 +1,114 @@ 
+digraph "Guix package" {
+  "139781154519456" [label = "gcc-core-mesboot0@2.95.3", shape = box, fontname = Helvetica];
+  "139781154519456" -> "139781154519632" [color = darkgoldenrod];
+  "139781154519456" -> "139781154520688" [color = darkgoldenrod];
+  "139781154519456" -> "139781034971312" [color = darkgoldenrod];
+  "139781154519456" -> "139781154520160" [color = darkgoldenrod];
+  "139781154519456" -> "139781034971664" [color = darkgoldenrod];
+  "139781154519456" -> "139781154519984" [color = darkgoldenrod];
+  "139781154519456" -> "139781154519808" [color = darkgoldenrod];
+  "139781154519456" -> "139781154520512" [color = darkgoldenrod];
+  "139781154519456" -> "139781034971488" [color = darkgoldenrod];
+  "139781154519456" -> "139781034972896" [color = darkgoldenrod];
+  "139781154519456" -> "139781034972368" [color = darkgoldenrod];
+  "139781154519456" -> "139781034973072" [color = darkgoldenrod];
+  "139781154519456" -> "139781131267872" [color = darkgoldenrod];
+  "139781154519632" [label = "binutils-mesboot0@2.14", shape = box, fontname = Helvetica];
+  "139781154519632" -> "139781154520688" [color = peachpuff4];
+  "139781154519632" -> "139781034971312" [color = peachpuff4];
+  "139781154519632" -> "139781154520160" [color = peachpuff4];
+  "139781154519632" -> "139781034971664" [color = peachpuff4];
+  "139781154519632" -> "139781154519984" [color = peachpuff4];
+  "139781154519632" -> "139781154519808" [color = peachpuff4];
+  "139781154519632" -> "139781154520512" [color = peachpuff4];
+  "139781154519632" -> "139781034971488" [color = peachpuff4];
+  "139781154519632" -> "139781034972896" [color = peachpuff4];
+  "139781154519632" -> "139781034972368" [color = peachpuff4];
+  "139781154519632" -> "139781034973072" [color = peachpuff4];
+  "139781154519632" -> "139781131267872" [color = peachpuff4];
+  "139781154520688" [label = "bash-mesboot0@2.05b", shape = box, fontname = Helvetica];
+  "139781154520688" -> "139781034971488" [color = magenta];
+  "139781154520688" -> "139781034971840" [color = magenta];
+  "139781154520688" -> "139781034972896" [color = magenta];
+  "139781154520688" -> "139781034972368" [color = magenta];
+  "139781154520688" -> "139781034973072" [color = magenta];
+  "139781154520688" -> "139781131267872" [color = magenta];
+  "139781034971488" [label = "make-mesboot0@3.80", shape = box, fontname = Helvetica];
+  "139781034971488" -> "139781034971840" [color = dimgrey];
+  "139781034971488" -> "139781034972896" [color = dimgrey];
+  "139781034971488" -> "139781034972368" [color = dimgrey];
+  "139781034971488" -> "139781034973072" [color = dimgrey];
+  "139781034971488" -> "139781131267872" [color = dimgrey];
+  "139781034971840" [label = "tcc-boot0@0.9.26-1103-g6e62e0e", shape = box, fontname = Helvetica];
+  "139781034971840" -> "139781034972016" [color = magenta];
+  "139781034971840" -> "139781131266816" [color = magenta];
+  "139781034971840" -> "139781034972896" [color = magenta];
+  "139781034971840" -> "139781034972368" [color = magenta];
+  "139781034971840" -> "139781034973072" [color = magenta];
+  "139781034971840" -> "139781131267872" [color = magenta];
+  "139781034972016" [label = "mes-boot@0.22", shape = box, fontname = Helvetica];
+  "139781034972016" -> "139781034972192" [color = cyan3];
+  "139781034972016" -> "139781131266816" [color = cyan3];
+  "139781034972016" -> "139781034972896" [color = cyan3];
+  "139781034972016" -> "139781034972368" [color = cyan3];
+  "139781034972016" -> "139781034973072" [color = cyan3];
+  "139781034972016" -> "139781131267872" [color = cyan3];
+  "139781034972192" [label = "bootstrap-mes-rewired@0.19", shape = ellipse, fontname = Helvetica];
+  "139781034972192" -> "139781131266640" [color = darkseagreen];
+  "139781034972192" -> "139781034972896" [color = darkseagreen];
+  "139781131266640" [label = "bootstrap-mes@0", shape = ellipse, fontname = Helvetica];
+  "139781034972896" [label = "gash-boot@0.2.0", shape = box, fontname = Helvetica];
+  "139781034972896" -> "139781034973072" [color = darkviolet];
+  "139781034972896" -> "139781131267872" [color = darkviolet];
+  "139781034973072" [label = "bootar@1", shape = box, fontname = Helvetica];
+  "139781034973072" -> "139781131267872" [color = darkviolet];
+  "139781131267872" [label = "guile-bootstrap@2.0", shape = ellipse, fontname = Helvetica];
+  "139781131266816" [label = "bootstrap-mescc-tools@0.5.2", shape = ellipse, fontname = Helvetica];
+  "139781034972368" [label = "gash-utils-boot@0.1.0", shape = box, fontname = Helvetica];
+  "139781034972368" -> "139781034973072" [color = darkseagreen];
+  "139781034972368" -> "139781034972896" [color = darkseagreen];
+  "139781034972368" -> "139781131267872" [color = darkseagreen];
+  "139781034971312" [label = "bzip2-mesboot@1.0.8", shape = box, fontname = Helvetica];
+  "139781034971312" -> "139781034971488" [color = cyan3];
+  "139781034971312" -> "139781034971840" [color = cyan3];
+  "139781034971312" -> "139781034972896" [color = cyan3];
+  "139781034971312" -> "139781034972368" [color = cyan3];
+  "139781034971312" -> "139781034973072" [color = cyan3];
+  "139781034971312" -> "139781131267872" [color = cyan3];
+  "139781154520160" [label = "diffutils-mesboot@2.7", shape = box, fontname = Helvetica];
+  "139781154520160" -> "139781034971488" [color = red];
+  "139781154520160" -> "139781034971840" [color = red];
+  "139781154520160" -> "139781034972896" [color = red];
+  "139781154520160" -> "139781034972368" [color = red];
+  "139781154520160" -> "139781034973072" [color = red];
+  "139781154520160" -> "139781131267872" [color = red];
+  "139781034971664" [label = "gzip-mesboot@1.2.4", shape = box, fontname = Helvetica];
+  "139781034971664" -> "139781034971840" [color = red];
+  "139781034971664" -> "139781034972896" [color = red];
+  "139781034971664" -> "139781034972368" [color = red];
+  "139781034971664" -> "139781034973072" [color = red];
+  "139781034971664" -> "139781131267872" [color = red];
+  "139781154519984" [label = "patch-mesboot@2.5.9", shape = box, fontname = Helvetica];
+  "139781154519984" -> "139781034971488" [color = blue];
+  "139781154519984" -> "139781034971840" [color = blue];
+  "139781154519984" -> "139781034972896" [color = blue];
+  "139781154519984" -> "139781034972368" [color = blue];
+  "139781154519984" -> "139781034973072" [color = blue];
+  "139781154519984" -> "139781131267872" [color = blue];
+  "139781154519808" [label = "sed-mesboot0@1.18", shape = box, fontname = Helvetica];
+  "139781154519808" -> "139781034971488" [color = red];
+  "139781154519808" -> "139781034971840" [color = red];
+  "139781154519808" -> "139781034972896" [color = red];
+  "139781154519808" -> "139781034972368" [color = red];
+  "139781154519808" -> "139781034973072" [color = red];
+  "139781154519808" -> "139781131267872" [color = red];
+  "139781154520512" [label = "tcc-boot@0.9.27", shape = box, fontname = Helvetica];
+  "139781154520512" -> "139781034971312" [color = darkviolet];
+  "139781154520512" -> "139781034971488" [color = darkviolet];
+  "139781154520512" -> "139781034971840" [color = darkviolet];
+  "139781154520512" -> "139781034972896" [color = darkviolet];
+  "139781154520512" -> "139781034972368" [color = darkviolet];
+  "139781154520512" -> "139781034973072" [color = darkviolet];
+  "139781154520512" -> "139781131267872" [color = darkviolet];
+
+}
diff --git a/doc/local.mk b/doc/local.mk
index f8709bb3a4..aca1958edd 100644
--- a/doc/local.mk
+++ b/doc/local.mk
@@ -39,7 +39,7 @@  DOT_FILES =					\
   %D%/images/bootstrap-packages.dot		\
   %D%/images/coreutils-graph.dot		\
   %D%/images/coreutils-bag-graph.dot		\
-  %D%/images/gcc-mesboot-bag-graph.dot		\
+  %D%/images/gcc-core-mesboot0-graph.dot	\
   %D%/images/service-graph.dot			\
   %D%/images/shepherd-graph.dot
 
-- 
2.26.0