| Message ID | 8ec588483525275aac1eb057543a903bea707ead.1695421391.git.ludo@gnu.org |
|---|---|
| State | New |
| Headers | show |
| Series | Add built-in builder for Git checkouts | expand |
Hi Ludo, On Sat, 23 Sep 2023 at 00:28, Ludovic Courtès <ludo@gnu.org> wrote: > * configure.ac: Check for ‘git’ and substitute ‘GIT’. > * guix/config.scm.in (%git): New variable. > * guix/self.scm (compiled-guix): Define ‘git’ and pass it to > ‘make-config.scm’. > (make-config.scm): Add #:git; emit a ‘%git’ variable. > * doc/guix.texi (Requirements): Add it. Moving the Git dependency to a daemon dependency tweaks a bit what we control when “bootstrapping”, no? Maybe I misread or misunderstand a point. Currently, the full bootstrap story requires the binary seed (well documented in the manual :-)), running a Linux kernel and a Guix daemon. And then, everything is built one after the other, resolving the chain of dependencies. And obviously, there is another chicken-or-the-egg problem barely discussed. For building something, we first need to communicate with the world for fetching the source code to build. :-) It is not Git specific and also happens with ’url-fetch’ – as reported very early (see #22774 from 2016). For instance, TLS is required before Guix has built GnuTLS. The chicken-or-the-egg had been solved by hiding this dependency as a dependency of the daemon. A hack. Therefore, it means that the Reduced Binary Seed bootstrap is somehow enlarged by what the Guix daemon depends on. I mean, breaking the dependency cycles by introducing a dependency on Git (as for fixing #63331) is not free of other dependencies. It is another hack when it could be avoided since the issue is about Guile-GnuTLS. It means we are doing a step back for a full bootstrap story, IMHO, having a “builtin:git-download” makes less clear what are the hard dependencies to what Guix is able to build from source starting from almost nothing. Cheers, simon
Hi, Simon Tournier <zimon.toutoune@gmail.com> skribis: > On Sat, 23 Sep 2023 at 00:28, Ludovic Courtès <ludo@gnu.org> wrote: >> * configure.ac: Check for ‘git’ and substitute ‘GIT’. >> * guix/config.scm.in (%git): New variable. >> * guix/self.scm (compiled-guix): Define ‘git’ and pass it to >> ‘make-config.scm’. >> (make-config.scm): Add #:git; emit a ‘%git’ variable. >> * doc/guix.texi (Requirements): Add it. > > Moving the Git dependency to a daemon dependency tweaks a bit what we > control when “bootstrapping”, no? Maybe I misread or misunderstand a > point. The model in Guix is that there’s a daemon to “emulate” a build “from scratch”. One can question whether the model matches reality, and this is what I and fellow hackers looked at during the 2019 R-B Summit: https://guix.gnu.org/en/blog/2019/reproducible-builds-summit-5th-edition/ (under “Extreme Bootstrapping”) (The ‘wip-system-bootstrap’ branch still exists!) Anyway, we’re drifting away from this patch series! Ludo’.
Hi Ludo, On Tue, 26 Sept 2023 at 16:05, Ludovic Courtès <ludo@gnu.org> wrote: > > Moving the Git dependency to a daemon dependency tweaks a bit what we > > control when “bootstrapping”, no? Maybe I misread or misunderstand a > > point. > > The model in Guix is that there’s a daemon to “emulate” a build “from > scratch”. Yes and that "emulate" will be bigger. > https://guix.gnu.org/en/blog/2019/reproducible-builds-summit-5th-edition/ > (under “Extreme Bootstrapping”) Thanks for the reference. I have forgotten it. Yes, that's it. :-) Adding Git as dependency to the daemon is adding Git in the Trusted Computing Base. It appears to me important to raise and to not hide under the carpet. :-) > (The ‘wip-system-bootstrap’ branch still exists!) Having a potential solution does not make pointless the current concern. ;-) > Anyway, we’re drifting away from this patch series! No, it is not drifting. The addition of Git in the trusting trust story cannot be dismissed, IMHO. It is not drifting to discuss for reaching some consensus about the "risk" of enlarging the trusting trust computing base. For example, is this "risk" worth the corner case of Guile-GnuTLS? As I said elsewhere, adding something is often much easier than removing something. Here the addition of Git has some implications (libgit2, trusted computing base, etc.) and it is always about the right balance. Do we have the right balance here? The discussion about concrete concerns for the addition of Git as dependency helps in reinforcing the consensus that this change is worth despite the downsides. To make it explicit: is this series worth the Guile-GnuTLS/Git circular dependency corner case? Maybe it is already all clear for you, and your answer is a big YES. :-) And perhaps it is the only answer. :-) But it does not mean the answer is fully clear for everybody, at least it is not necessary straightforward for me. Somehow, do we have a consensus about the way that this series is worth the Guile-GnuTLS/Git circular dependency corner case? And a consensus about the way that this series is The Right Thing for that circular dependency? Cheers, simon
Hi, Simon Tournier <zimon.toutoune@gmail.com> skribis: > To make it explicit: is this series worth the Guile-GnuTLS/Git > circular dependency corner case? Maybe it is already all clear for > you, and your answer is a big YES. :-) And perhaps it is the only > answer. :-) But it does not mean the answer is fully clear for > everybody, at least it is not necessary straightforward for me. > Somehow, do we have a consensus about the way that this series is > worth the Guile-GnuTLS/Git circular dependency corner case? And a > consensus about the way that this series is The Right Thing for that > circular dependency? One thing I probably didn’t explain clearly is that yes, the circular dependency issue is one we have to solve. For years, I hope we could avoid it but experience has shown that no, it’s a problem we did have to address. One example is Guile-GnuTLS being built from a Git checkout. Another one is Hurd packages in commencement.scm built from a Git checkout. We had to go to great lengths to avoid ‘git-fetch’: https://issues.guix.gnu.org/64708#6 More and more software is built from Git checkouts rather than tarballs. In the long run, we won’t be able to guarantee that none of the dependencies of ‘git-minimal’ takes its source via ‘git-fetch’. This is especially true if we want to move away from Autotools-generated tarballs and instead run ‘autoreconf’ ourselves on pristine source. Ludo’.
Hi Ludo, On Thu, 12 Oct 2023 at 12:54, Ludovic Courtès <ludo@gnu.org> wrote: >> To make it explicit: is this series worth the Guile-GnuTLS/Git >> circular dependency corner case? Maybe it is already all clear for >> you, and your answer is a big YES. :-) And perhaps it is the only >> answer. :-) But it does not mean the answer is fully clear for >> everybody, at least it is not necessary straightforward for me. >> Somehow, do we have a consensus about the way that this series is >> worth the Guile-GnuTLS/Git circular dependency corner case? And a >> consensus about the way that this series is The Right Thing for that >> circular dependency? > > One thing I probably didn’t explain clearly is that yes, the circular > dependency issue is one we have to solve. For years, I hope we could > avoid it but experience has shown that no, it’s a problem we did have to > address. There is different ways to solve this circular dependency. > One example is Guile-GnuTLS being built from a Git checkout. Another > one is Hurd packages in commencement.scm built from a Git checkout. We > had to go to great lengths to avoid ‘git-fetch’: > > https://issues.guix.gnu.org/64708#6 Somehow, I think it is the direction to explore: have some ’git-fetch-bootstrap’ which relies on some ’builtin:git-download’ and guix-daemon side code, and then that being used to have the packages required by some ’git-fetch’ without guix-daemon side code. Doing so, we would minimize the opaque – hard to audit – code, which means less power to guix-daemon, we keep the control, etc. It appears to me more consistent with the general approach elsewhere. Anyway. Rehashing all, your opinion was already made when you sent this patch. You wrote since the very beginning on 6 May 2023 for Guile-GnuTLS [1]: The longer-term solution is to add a “builtin:git-download” derivation builder, just like we have “builtin:download”. The implementation should be relatively easy, but we’ll have to be able to deal with daemons that lack this builtin possibly for several years. https://issues.guix.gnu.org/63331#0 Therefore, this patch was not an open discussion for some design but the review of some code for fixing concrete issues. No hard feeling, we need to make progress after all; the issue is pending since months. However, when giving a look at this patch, it was not my expectations. It is my own mistake to have not been enough active before with the issue. I had months to discuss some design for the circular dependencies of the fetching methods. Since I am spending some time reading about what is going on with Guix, I think we have a failure in some process. IMHO, we are missing a Request For Comments and I will send a proposal for some implementation details this week. Cheers, simon
diff --git a/configure.ac b/configure.ac index 92dede8014..d817f620cf 100644 --- a/configure.ac +++ b/configure.ac @@ -201,6 +201,13 @@ AC_SUBST([GZIP]) AC_SUBST([BZIP2]) AC_SUBST([XZ]) +dnl Git is now required for the "builtin:git-download" derivation builder. +AC_PATH_PROG([GIT], [git]) +if test "x$GIT" = "x"; then + AC_MSG_ERROR([Git is missing; please install it.]) +fi +AC_SUBST([GIT]) + LIBGCRYPT_LIBDIR="no" LIBGCRYPT_PREFIX="no" diff --git a/doc/guix.texi b/doc/guix.texi index 50c4984d71..8812e42e99 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -1011,6 +1011,7 @@ Requirements @item @uref{https://gitlab.com/guile-git/guile-git, Guile-Git}, version 0.5.0 or later; +@item @uref{https://git-scm.com, Git} (yes, both!); @item @uref{https://savannah.nongnu.org/projects/guile-json/, Guile-JSON} 4.3.0 or later; @item @url{https://www.gnu.org/software/make/, GNU Make}. diff --git a/guix/config.scm.in b/guix/config.scm.in index d582d91d74..62e15dd713 100644 --- a/guix/config.scm.in +++ b/guix/config.scm.in @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org> +;;; Copyright © 2012-2016, 2018-2019, 2021, 2023 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Caleb Ristvedt <caleb.ristvedt@cune.org> ;;; ;;; This file is part of GNU Guix. @@ -35,6 +35,7 @@ (define-module (guix config) %config-directory %system + %git %gzip %bzip2 %xz)) @@ -109,6 +110,9 @@ (define %config-directory (define %system "@guix_system@") +(define %git + "@GIT@") + (define %gzip "@GZIP@") diff --git a/guix/self.scm b/guix/self.scm index d2300052d8..9eaddc7a29 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -69,6 +69,7 @@ (define %packages ("gzip" . ,(ref 'compression 'gzip)) ("bzip2" . ,(ref 'compression 'bzip2)) ("xz" . ,(ref 'compression 'xz)) + ("git-minimal" . ,(ref 'version-control 'git-minimal)) ("po4a" . ,(ref 'gettext 'po4a)) ("gettext-minimal" . ,(ref 'gettext 'gettext-minimal)) ("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain)) @@ -826,6 +827,9 @@ (define* (compiled-guix source #:key (define guile-lzma (specification->package "guile-lzma")) + (define git + (specification->package "git-minimal")) + (define dependencies (append-map transitive-package-dependencies (list guile-gcrypt guile-gnutls guile-git guile-avahi @@ -999,6 +1003,7 @@ (define* (compiled-guix source #:key => ,(make-config.scm #:gzip gzip #:bzip2 bzip2 #:xz xz + #:git git #:package-name %guix-package-name #:package-version @@ -1104,7 +1109,7 @@ (define %default-config-variables (%storedir . "/gnu/store") (%sysconfdir . "/etc"))) -(define* (make-config.scm #:key gzip xz bzip2 +(define* (make-config.scm #:key gzip xz bzip2 git (package-name "GNU Guix") (package-version "0") (channel-metadata #f) @@ -1134,6 +1139,7 @@ (define* (make-config.scm #:key gzip xz bzip2 %state-directory %store-database-directory %config-directory + %git %gzip %bzip2 %xz)) @@ -1176,6 +1182,8 @@ (define* (make-config.scm #:key gzip xz bzip2 ;; information is used by (guix describe). '#$channel-metadata) + (define %git + #+(and git (file-append git "/bin/git"))) (define %gzip #+(and gzip (file-append gzip "/bin/gzip"))) (define %bzip2