From patchwork Sun May 17 12:43:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Janneke Nieuwenhuizen X-Patchwork-Id: 22123 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2A25227BBE1; Sun, 17 May 2020 13:44:17 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 9779427BBE3 for ; Sun, 17 May 2020 13:44:13 +0100 (BST) Received: from localhost ([::1]:41104 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jaIeL-0007ev-3u for patchwork@mira.cbaines.net; Sun, 17 May 2020 08:44:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56430) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jaIeA-0007du-Ke for guix-patches@gnu.org; Sun, 17 May 2020 08:44:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:59228) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jaIeA-0008Ie-C6 for guix-patches@gnu.org; Sun, 17 May 2020 08:44:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jaIeA-0006XF-Ai for guix-patches@gnu.org; Sun, 17 May 2020 08:44:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#39655] [PATCH core-updates] doc: Add 'Scheme-only Bootstrap' node. Resent-From: Jan Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 17 May 2020 12:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39655 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?utf-8?q?Court=C3=A8s?= Cc: Timothy Sample , 39655@debbugs.gnu.org Received: via spool by 39655-submit@debbugs.gnu.org id=B39655.158971940925063 (code B ref 39655); Sun, 17 May 2020 12:44:02 +0000 Received: (at 39655) by debbugs.gnu.org; 17 May 2020 12:43:29 +0000 Received: from localhost ([127.0.0.1]:42539 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jaIdV-0006W4-UH for submit@debbugs.gnu.org; Sun, 17 May 2020 08:43:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48918) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jaIdQ-0006VW-UV for 39655@debbugs.gnu.org; Sun, 17 May 2020 08:43:20 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58459) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jaIdJ-0008DJ-7I; Sun, 17 May 2020 08:43:09 -0400 Received: from [2001:980:1b4f:1:42d2:832d:bb59:862] (port=59406 helo=dundal.peder.onsbrabantnet.nl) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jaIdH-0000WD-D4; Sun, 17 May 2020 08:43:08 -0400 From: Jan Nieuwenhuizen Organization: AvatarAcademy.nl References: <20200218124411.20400-1-janneke@gnu.org> <87eeuqmuwb.fsf@gnu.org> X-Url: http://AvatarAcademy.nl Date: Sun, 17 May 2020 14:43:05 +0200 In-Reply-To: <87eeuqmuwb.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Wed, 19 Feb 2020 16:58:12 +0100") Message-ID: <87367yhgva.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Ludovic Courtès writes: Hello Ludo, Timothy! Finally another version of the new bootstrap situation. Timothy, please have a look and feel free to change/add some things! It's also getting time to write a blog post on this, would you like to help out with that? >> * doc/guix.texi (@detailmenu): Add Bootstrapping menu. >> (Scheme-only Bootstrap): New node. >> * doc/images/gcc-core-mesboot0-scheme-only-graph.dot: New image. >> * doc/local.mk (DOT_FILES): Add it. > > Great! I wonder if we need a new section though. Do you think we could > just keep it all in a single bootstrapping section that would describe > the current state of affairs, possibly with a “Bootstrapping History” > section or similar that would describe the milestones that got us there? [..] > That way, the main bootstrapping section would simply describe the > current status in a standalone fashion. Thanks for your thoughts on this. I started to rework this by adding a History section but in the end I removed it again, describing the current situation while mentioning the gcc-triplet reduction in Guix v1.1. >> +@c >> +packages@footnote{@c >> +gash-boot, >> +gash-utils-boot, >> +%bootstrap-mes-rewired, >> +gzip-mesboot, > > I wouldn’t list them all here because (1) it’s hard to understand what > these names refer to, unless reading commencement.scm is your favorite > hobby ;-), and (2) it’s bound to become outdated. > > WDYT? Yes, instead of adding more of these, I also removed the previous list. >> +As Gash and Gash Utils mature, and GNU packages become more @Timothy: this is possibly touching very lightly on what Gash has brought us; do you want to share some more here? >> +bootstrappable (e.g., new releases of GNU Sed will also ship in [..] >> +@image{images/gcc-core-mesboot0-scheme-only-graph,6in,,Dependency graph of gcc-core-mesboot0} > > It would be nice to have a couple of lines commenting the graph, like: Sure! Updated version attached. Greetings, Janneke From b1341ecfc77345f1a423bbc66b6255a35613425e Mon Sep 17 00:00:00 2001 From: Jan Nieuwenhuizen Date: Tue, 18 Feb 2020 11:36:37 +0100 Subject: [PATCH] doc: Update 'Bootstrapping' for further binary seed reduction. * doc/images/gcc-core-mesboot0-graph.dot: New image, replacing * doc/images/gcc-mesboot0-bag-graph.dot: ... remove file. * doc/local.mk (DOT_FILES): Update for new image file. * doc/guix.texi (Reduced Binary Seed Bootstrap): Use it in updated description of further reduction of the trusted computing base. --- doc/guix.texi | 114 +++++++++++++------------ doc/images/gcc-core-mesboot0-graph.dot | 114 +++++++++++++++++++++++++ doc/local.mk | 2 +- 3 files changed, 175 insertions(+), 55 deletions(-) create mode 100644 doc/images/gcc-core-mesboot0-graph.dot diff --git a/doc/guix.texi b/doc/guix.texi index 90324ce291..833509860c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -35,7 +35,7 @@ Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@* Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@* Copyright @copyright{} 2016 John Darrington@* Copyright @copyright{} 2016, 2017 Nikita Gillmann@* -Copyright @copyright{} 2016, 2017, 2018, 2019 Jan Nieuwenhuizen@* +Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@* Copyright @copyright{} 2016 Julien Lepiller@* Copyright @copyright{} 2016 Alex ter Weele@* Copyright @copyright{} 2016, 2017, 2018, 2019 Christopher Baines@* @@ -28606,10 +28606,8 @@ Binutils, libc, and the other packages mentioned above---the @dfn{bootstrap binaries}. These bootstrap binaries are ``taken for granted'', though we can also -re-create them if needed (more on that later). - -For @code{i686-linux} and @code{x86_64-linux} the Guix bootstrap process is -more elaborate, @pxref{Reduced Binary Seed Bootstrap}. +re-create them if needed (see @pxref{Preparing to Use the Bootstrap +Binaries}). @menu * Reduced Binary Seed Bootstrap:: A Bootstrap worthy of GNU. @@ -28625,60 +28623,68 @@ Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils, and the GNU C Library (@pxref{Bootstrapping}). Usually, these bootstrap binaries are ``taken for granted.'' -Taking these binaries for granted means that we consider them to be a correct -and trustworthy `seed' for building the complete system. Therein lies a -problem: the current combined size of these bootstrap binaries is about 250MB -(@pxref{Bootstrappable Builds,,, mes, GNU Mes}). Auditing or even inspecting -these is next to impossible. +Taking the bootstrap binaries for granted means that we consider them to +be a correct and trustworthy `seed' for building the complete system. +Therein lies a problem: the combined size of these bootstrap binaries is +about 250MB (@pxref{Bootstrappable Builds,,, mes, GNU Mes}). Auditing +or even inspecting these is next to impossible. -For @code{i686-linux} and @code{x86_64-linux}, Guix now features a ``Reduced -Binary Seed'' bootstrap @footnote{We would like to say: ``Full Source -Bootstrap'' and while we are working towards that goal it would be hyperbole -to use that term for what we do now.}. +For @code{i686-linux} and @code{x86_64-linux}, Guix now features a +``Reduced Binary Seed'' bootstrap @footnote{We would like to say: ``Full +Source Bootstrap'' and while we are working towards that goal it would +be hyperbole to use that term for what we do now.}. The Reduced Binary Seed bootstrap removes the most critical tools---from a trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C Library are replaced by: @code{bootstrap-mescc-tools} (a tiny assembler and linker) and @code{bootstrap-mes} (a small Scheme Interpreter and a C compiler -written in Scheme and the Mes C Library, built for TinyCC and for GCC). Using -these new binary seeds and a new set of -@c -packages@footnote{@c -nyacc-boot, -mes-boot, -tcc-boot0, -tcc-boot, -make-mesboot0, -diffutils-mesboot, -binutils-mesboot0, -gcc-core-mesboot, -mesboot-headers, -glibc-mesboot0, -gcc-mesboot0, -binutils-mesboot, -make-mesboot, -gcc-mesboot1, -gcc-mesboot1-wrapper, -glibc-headers-mesboot, -glibc-mesboot, -gcc-mesboot, -and -gcc-mesboot-wrapper. -} -@c -the ``missing'' Binutils, GCC, and the GNU C Library are built from source. -From here on the more traditional bootstrap process resumes. This approach -has reduced the bootstrap binaries in size to about 130MB. Work is ongoing to -reduce this further. If you are interested, join us on @code{#bootstrappable} -on the Freenode IRC network. - -@c ./pre-inst-env guix graph --type=bag -e '(begin (use-modules (guix packages)) (%current-system "i686-linux") (@@ (gnu packages commencement) gcc-mesboot))' > doc/images/gcc-mesboot-bag-graph.dot -@c dot -T png doc/images/gcc-mesboot-bag-graph.dot > doc/images/gcc-mesboot-bag-graph.png - -Below is the generated dependency graph for @code{gcc-mesboot}, the bootstrap -compiler used to build the rest of GuixSD. - -@image{images/gcc-mesboot-bag-graph,6in,,Dependency graph of the gcc-mesboot} +written in Scheme and the Mes C Library, built for TinyCC and for GCC). + +Using these new binary seeds the ``missing'' Binutils, GCC, and the GNU +C Library are built from source. From here on the more traditional +bootstrap process resumes. This approach has reduced the bootstrap +binaries in size to about 145MB in Guix v1.1. + +The next step that Guix has taken is to replace the shell and all its +utilities with implementations in Guile Scheme, the @emph{Scheme-only +bootstrap}. Gash (@pxref{Gash,,, gash, The Gash manual}) is a +POSIX-compatible shell that replaces Bash, and it comes with Gash Utils +which has minimalist replacements for Awk, the GNU Core Utilities, Grep, +Gzip, Sed, and Tar. The rest of the bootstrap binary seeds that were +removed are now built from source. + +Building the GNU System from source is currently only possibly by adding +some historical GNU packages as intermediate steps@footnote{Packages +such as @code{gcc-2.95.3}, @code{binutils-2.14}, @code{glibc-2.2.5}, +@code{gzip-1.2.4}, @code{tar-1.22}, and some others for details see +@file{gnu/packages/commencement.scm}}. As Gash and Gash Utils mature, +and GNU packages become more bootstrappable again (e.g., new releases of +GNU Sed will also ship in @code{tar.gz} format), this set of added +packages can hopefully be reduced again. + +The graph below shows the resulting dependency graph for +@code{gcc-core-mesboot0}, the bootstrap compiler used for the +traditional bootstrap of the rest of the Guix System. + +@c ./pre-inst-env guix graph -e '(@@ (gnu packages commencement) gcc-core-mesboot0)' | sed -re 's,((bootstrap-mescc-tools|bootstrap-mes|guile-bootstrap).*shape =) box,\1 ellipse,' > doc/images/gcc-core-mesboot0-graph.dot +@image{images/gcc-core-mesboot0-graph,6in,,Dependency graph of gcc-core-mesboot0} + +The only significant binary bootstrap seeds that remain@footnote{ +Ignoring the 68KB @code{mescc-tools}; that will be removed later, +together with @code{mes}.} are a Scheme intepreter and a Scheme +compiler: GNU Mes and GNU Guile. + +This further reduction has brought down the size of the binary seed to +about 60MB for @code{i686-linux} and @code{x86_64-linux}. + +Work is ongoing to remove all binary blobs from our free software +bootstrap stack, working towards a Full Source Bootstrap. Also ongoing +is work to bring these bootstraps to the @code{arm-linux} and +@code{aarch64-linux} architectures and to the Hurd. + +If you are interested, join us on @code{#bootstrappable} on the Freenode +IRC network or discuss on @code{bug-mes@@gnu.org} or +@code{gash-devel@@nongnu.org}. @node Preparing to Use the Bootstrap Binaries @section Preparing to Use the Bootstrap Binaries @@ -28698,7 +28704,7 @@ guix graph -t derivation \ | dot -Tps > gcc.ps @end example -or, for the Reduced Binary Seed bootstrap +or, for the further Reduced Binary Seed bootstrap @example guix graph -t derivation \ diff --git a/doc/images/gcc-core-mesboot0-graph.dot b/doc/images/gcc-core-mesboot0-graph.dot new file mode 100644 index 0000000000..191582ed4c --- /dev/null +++ b/doc/images/gcc-core-mesboot0-graph.dot @@ -0,0 +1,114 @@ +digraph "Guix package" { + "139781154519456" [label = "gcc-core-mesboot0@2.95.3", shape = box, fontname = Helvetica]; + "139781154519456" -> "139781154519632" [color = darkgoldenrod]; + "139781154519456" -> "139781154520688" [color = darkgoldenrod]; + "139781154519456" -> "139781034971312" [color = darkgoldenrod]; + "139781154519456" -> "139781154520160" [color = darkgoldenrod]; + "139781154519456" -> "139781034971664" [color = darkgoldenrod]; + "139781154519456" -> "139781154519984" [color = darkgoldenrod]; + "139781154519456" -> "139781154519808" [color = darkgoldenrod]; + "139781154519456" -> "139781154520512" [color = darkgoldenrod]; + "139781154519456" -> "139781034971488" [color = darkgoldenrod]; + "139781154519456" -> "139781034972896" [color = darkgoldenrod]; + "139781154519456" -> "139781034972368" [color = darkgoldenrod]; + "139781154519456" -> "139781034973072" [color = darkgoldenrod]; + "139781154519456" -> "139781131267872" [color = darkgoldenrod]; + "139781154519632" [label = "binutils-mesboot0@2.14", shape = box, fontname = Helvetica]; + "139781154519632" -> "139781154520688" [color = peachpuff4]; + "139781154519632" -> "139781034971312" [color = peachpuff4]; + "139781154519632" -> "139781154520160" [color = peachpuff4]; + "139781154519632" -> "139781034971664" [color = peachpuff4]; + "139781154519632" -> "139781154519984" [color = peachpuff4]; + "139781154519632" -> "139781154519808" [color = peachpuff4]; + "139781154519632" -> "139781154520512" [color = peachpuff4]; + "139781154519632" -> "139781034971488" [color = peachpuff4]; + "139781154519632" -> "139781034972896" [color = peachpuff4]; + "139781154519632" -> "139781034972368" [color = peachpuff4]; + "139781154519632" -> "139781034973072" [color = peachpuff4]; + "139781154519632" -> "139781131267872" [color = peachpuff4]; + "139781154520688" [label = "bash-mesboot0@2.05b", shape = box, fontname = Helvetica]; + "139781154520688" -> "139781034971488" [color = magenta]; + "139781154520688" -> "139781034971840" [color = magenta]; + "139781154520688" -> "139781034972896" [color = magenta]; + "139781154520688" -> "139781034972368" [color = magenta]; + "139781154520688" -> "139781034973072" [color = magenta]; + "139781154520688" -> "139781131267872" [color = magenta]; + "139781034971488" [label = "make-mesboot0@3.80", shape = box, fontname = Helvetica]; + "139781034971488" -> "139781034971840" [color = dimgrey]; + "139781034971488" -> "139781034972896" [color = dimgrey]; + "139781034971488" -> "139781034972368" [color = dimgrey]; + "139781034971488" -> "139781034973072" [color = dimgrey]; + "139781034971488" -> "139781131267872" [color = dimgrey]; + "139781034971840" [label = "tcc-boot0@0.9.26-1103-g6e62e0e", shape = box, fontname = Helvetica]; + "139781034971840" -> "139781034972016" [color = magenta]; + "139781034971840" -> "139781131266816" [color = magenta]; + "139781034971840" -> "139781034972896" [color = magenta]; + "139781034971840" -> "139781034972368" [color = magenta]; + "139781034971840" -> "139781034973072" [color = magenta]; + "139781034971840" -> "139781131267872" [color = magenta]; + "139781034972016" [label = "mes-boot@0.22", shape = box, fontname = Helvetica]; + "139781034972016" -> "139781034972192" [color = cyan3]; + "139781034972016" -> "139781131266816" [color = cyan3]; + "139781034972016" -> "139781034972896" [color = cyan3]; + "139781034972016" -> "139781034972368" [color = cyan3]; + "139781034972016" -> "139781034973072" [color = cyan3]; + "139781034972016" -> "139781131267872" [color = cyan3]; + "139781034972192" [label = "bootstrap-mes-rewired@0.19", shape = ellipse, fontname = Helvetica]; + "139781034972192" -> "139781131266640" [color = darkseagreen]; + "139781034972192" -> "139781034972896" [color = darkseagreen]; + "139781131266640" [label = "bootstrap-mes@0", shape = ellipse, fontname = Helvetica]; + "139781034972896" [label = "gash-boot@0.2.0", shape = box, fontname = Helvetica]; + "139781034972896" -> "139781034973072" [color = darkviolet]; + "139781034972896" -> "139781131267872" [color = darkviolet]; + "139781034973072" [label = "bootar@1", shape = box, fontname = Helvetica]; + "139781034973072" -> "139781131267872" [color = darkviolet]; + "139781131267872" [label = "guile-bootstrap@2.0", shape = ellipse, fontname = Helvetica]; + "139781131266816" [label = "bootstrap-mescc-tools@0.5.2", shape = ellipse, fontname = Helvetica]; + "139781034972368" [label = "gash-utils-boot@0.1.0", shape = box, fontname = Helvetica]; + "139781034972368" -> "139781034973072" [color = darkseagreen]; + "139781034972368" -> "139781034972896" [color = darkseagreen]; + "139781034972368" -> "139781131267872" [color = darkseagreen]; + "139781034971312" [label = "bzip2-mesboot@1.0.8", shape = box, fontname = Helvetica]; + "139781034971312" -> "139781034971488" [color = cyan3]; + "139781034971312" -> "139781034971840" [color = cyan3]; + "139781034971312" -> "139781034972896" [color = cyan3]; + "139781034971312" -> "139781034972368" [color = cyan3]; + "139781034971312" -> "139781034973072" [color = cyan3]; + "139781034971312" -> "139781131267872" [color = cyan3]; + "139781154520160" [label = "diffutils-mesboot@2.7", shape = box, fontname = Helvetica]; + "139781154520160" -> "139781034971488" [color = red]; + "139781154520160" -> "139781034971840" [color = red]; + "139781154520160" -> "139781034972896" [color = red]; + "139781154520160" -> "139781034972368" [color = red]; + "139781154520160" -> "139781034973072" [color = red]; + "139781154520160" -> "139781131267872" [color = red]; + "139781034971664" [label = "gzip-mesboot@1.2.4", shape = box, fontname = Helvetica]; + "139781034971664" -> "139781034971840" [color = red]; + "139781034971664" -> "139781034972896" [color = red]; + "139781034971664" -> "139781034972368" [color = red]; + "139781034971664" -> "139781034973072" [color = red]; + "139781034971664" -> "139781131267872" [color = red]; + "139781154519984" [label = "patch-mesboot@2.5.9", shape = box, fontname = Helvetica]; + "139781154519984" -> "139781034971488" [color = blue]; + "139781154519984" -> "139781034971840" [color = blue]; + "139781154519984" -> "139781034972896" [color = blue]; + "139781154519984" -> "139781034972368" [color = blue]; + "139781154519984" -> "139781034973072" [color = blue]; + "139781154519984" -> "139781131267872" [color = blue]; + "139781154519808" [label = "sed-mesboot0@1.18", shape = box, fontname = Helvetica]; + "139781154519808" -> "139781034971488" [color = red]; + "139781154519808" -> "139781034971840" [color = red]; + "139781154519808" -> "139781034972896" [color = red]; + "139781154519808" -> "139781034972368" [color = red]; + "139781154519808" -> "139781034973072" [color = red]; + "139781154519808" -> "139781131267872" [color = red]; + "139781154520512" [label = "tcc-boot@0.9.27", shape = box, fontname = Helvetica]; + "139781154520512" -> "139781034971312" [color = darkviolet]; + "139781154520512" -> "139781034971488" [color = darkviolet]; + "139781154520512" -> "139781034971840" [color = darkviolet]; + "139781154520512" -> "139781034972896" [color = darkviolet]; + "139781154520512" -> "139781034972368" [color = darkviolet]; + "139781154520512" -> "139781034973072" [color = darkviolet]; + "139781154520512" -> "139781131267872" [color = darkviolet]; + +} diff --git a/doc/local.mk b/doc/local.mk index f8709bb3a4..aca1958edd 100644 --- a/doc/local.mk +++ b/doc/local.mk @@ -39,7 +39,7 @@ DOT_FILES = \ %D%/images/bootstrap-packages.dot \ %D%/images/coreutils-graph.dot \ %D%/images/coreutils-bag-graph.dot \ - %D%/images/gcc-mesboot-bag-graph.dot \ + %D%/images/gcc-core-mesboot0-graph.dot \ %D%/images/service-graph.dot \ %D%/images/shepherd-graph.dot -- 2.26.0