| Message ID | 7ce3ce800ab5dde1cdaaf272a54738be6256c1b3.1692128648.git.maxim.cournoyer@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | None | expand |
Hi Maxim,
For the record, I have documented [1] the various roadblocks when using
“guix time-machine” rebuilding all from source. Time-bomb is one among
other annoyances – I have in mind the complete bootstrap.
Well, as Ludo pointed, the CI is currently building all the past
releases. Waiting the fixes for all the bugs, I suggest that we retain
the substitutes for the release. I mean that,
guix time-machine --commit=v1.X.0 -- help
just works when substitutes are available. If the project is lacking
disk space, the University of Montpellier (France) is proposing to store
some binary artifact outputs. At least, they were proposing back on
past September in 10 Years of Guix event. :-) There is discussion on
guix-sysadmin, I guess. Maybe we could resume this discussion and
complete the last steps. WDYT?
1: https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html
On Tue, 15 Aug 2023 at 15:44, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> * doc/guix.texi (Invoking guix time-machine): Document limitation.
> * guix/inferior.scm (cached-channel-instance): New VALIDATE-CHANNELS
> argument. Use it to validate channels when there are no cache hit.
> * guix/scripts/time-machine.scm
> (%options): Tag the given reference with 'tag-or-commit instead of 'commit.
> (%oldest-possible-commit): New variable.
> (guix-time-machine) <validate-guix-channel>: New nested procedure. Pass it to
> the 'cached-channel-instance' call.
> * tests/guix-time-machine.sh: New test.
> * Makefile.am (SH_TESTS): Register it.
Cool! That’s a nice usability improvement.
> diff --git a/doc/guix.texi b/doc/guix.texi
> index b50feed4c4..a3754b7019 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -5060,6 +5060,20 @@ Invoking guix time-machine
> large number of packages; the result is cached though and subsequent
> commands targeting the same commit are almost instantaneous.
>
> +Due to @command{guix time-machine} relying on the ``inferiors''
> +mechanism (@pxref{Inferiors}), the oldest commit it can travel to is
> +commit @samp{6298c3ff} (``v1.0.0''), dated May 1@sup{st}, 2019, which is
> +the first release that included the inferiors mechanism. An error is
> +returned when attempting to navigate to older commits.
There is also some issue with bootstrapping depending on your hardware.
About time-bomb, there are also gnutls and openssl or libgit2. It was
probably transparent for you because there are substitutable, I guess.
While Python 2 had probably been removed for some reasons.
Well, I would move the workaround to some dedicated block and move this
comment after the note about security
--8<---------------cut here---------------start------------->8---
@quotation Note
The history of Guix is immutable and @command{guix time-machine}
provides the exact same software as they are in a specific Guix
revision. Naturally, no security fixes are provided for old versions
of Guix or its channels. A careless use of @command{guix time-machine}
opens the door to security vulnerabilities. @xref{Invoking guix pull,
@option{--allow-downgrades}}.
@end quotation
+Due to @command{guix time-machine} relying on the ``inferiors''
+mechanism (@pxref{Inferiors}), the oldest commit it can travel to is
+commit @samp{6298c3ff} (``v1.0.0''), dated May 1@sup{st}, 2019, which is
+the first release that included the inferiors mechanism. An error is
+returned when attempting to navigate to older commits.
+
+@quotation Note
+Although it should technically be possible to travel to such an old
+revision, the ease to do so will largely depend on the availability of
+binary substitutes. When traveling to a distant past, some packages may
+not easily build from source anymore. One such example are old versions
+of Python 2 which had time bombs in its test suite, in the form of
+expiring SSL certificates. This particular problem can be worked around
+by setting the hardware clock to a value in the past before attempting
+the build.
+@end quotation
+
The general syntax is:
@example
--8<---------------cut here---------------end--------------->8---
> new file mode 100644
> index 0000000000..8b62ef75ea
> --- /dev/null
> +++ b/tests/guix-time-machine.sh
[...]
> +# Visiting a commit older than v1.0.0 fails.
> +! guix time-machine --commit=v0.15.0
Cool to add test. But this test needs a network access:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix time-machine --commit=v0.15.0 -- describe
guix time-machine: error: Git error: failed to resolve address for git.savannah.gnu.org: Name or service not known
--8<---------------cut here---------------end--------------->8---
It’s as I said elsewhere. :-) Well, I have not investigated more but I
guess a bug in some Git manipulation.
Cheers,
simon
Hi Simon, Simon Tournier <zimon.toutoune@gmail.com> writes: > Hi Maxim, > > For the record, I have documented [1] the various roadblocks when using > “guix time-machine” rebuilding all from source. Time-bomb is one among > other annoyances – I have in mind the complete bootstrap. > > Well, as Ludo pointed, the CI is currently building all the past > releases. Waiting the fixes for all the bugs, I suggest that we retain > the substitutes for the release. I mean that, > > guix time-machine --commit=v1.X.0 -- help > > just works when substitutes are available. If the project is lacking > disk space, the University of Montpellier (France) is proposing to store > some binary artifact outputs. At least, they were proposing back on > past September in 10 Years of Guix event. :-) There is discussion on > guix-sysadmin, I guess. Maybe we could resume this discussion and > complete the last steps. WDYT? That make sense. The current means this can be achieved is by having a jobset for each release in Cuirass, as hinted in doc/release.org from the guix-maintenance repo: --8<---------------cut here---------------start------------->8--- ** Adding a Cuirass jobset for branch =version-X.Y.Z= This jobset will have to be kept until the next release, so that substitutes remain available. The easiest way to add a new jobset is directly via the web interface of Cuirass. To be allowed to do so, you must authenticate with the Cuirass instance via a private TLS certificate imported into your browser. --8<---------------cut here---------------end--------------->8--- They should be added declaratively in the guix-maintenance repo to avoid loosing them. Would you like to give it a try? [...] >> diff --git a/doc/guix.texi b/doc/guix.texi >> index b50feed4c4..a3754b7019 100644 >> --- a/doc/guix.texi >> +++ b/doc/guix.texi >> @@ -5060,6 +5060,20 @@ Invoking guix time-machine >> large number of packages; the result is cached though and subsequent >> commands targeting the same commit are almost instantaneous. >> >> +Due to @command{guix time-machine} relying on the ``inferiors'' >> +mechanism (@pxref{Inferiors}), the oldest commit it can travel to is >> +commit @samp{6298c3ff} (``v1.0.0''), dated May 1@sup{st}, 2019, which is >> +the first release that included the inferiors mechanism. An error is >> +returned when attempting to navigate to older commits. > > There is also some issue with bootstrapping depending on your hardware. > > About time-bomb, there are also gnutls and openssl or libgit2. It was > probably transparent for you because there are substitutable, I guess. > While Python 2 had probably been removed for some reasons. > > Well, I would move the workaround to some dedicated block and move this > comment after the note about security > > @quotation Note > The history of Guix is immutable and @command{guix time-machine} > provides the exact same software as they are in a specific Guix > revision. Naturally, no security fixes are provided for old versions > of Guix or its channels. A careless use of @command{guix time-machine} > opens the door to security vulnerabilities. @xref{Invoking guix pull, > @option{--allow-downgrades}}. > @end quotation > > +Due to @command{guix time-machine} relying on the ``inferiors'' > +mechanism (@pxref{Inferiors}), the oldest commit it can travel to is > +commit @samp{6298c3ff} (``v1.0.0''), dated May 1@sup{st}, 2019, which is > +the first release that included the inferiors mechanism. An error is > +returned when attempting to navigate to older commits. > + > +@quotation Note > +Although it should technically be possible to travel to such an old > +revision, the ease to do so will largely depend on the availability of > +binary substitutes. When traveling to a distant past, some packages may > +not easily build from source anymore. One such example are old versions > +of Python 2 which had time bombs in its test suite, in the form of > +expiring SSL certificates. This particular problem can be worked around > +by setting the hardware clock to a value in the past before attempting > +the build. > +@end quotation Good suggestion, done! > The general syntax is: > > @example > > > >> new file mode 100644 >> index 0000000000..8b62ef75ea >> --- /dev/null >> +++ b/tests/guix-time-machine.sh > > [...] > >> +# Visiting a commit older than v1.0.0 fails. >> +! guix time-machine --commit=v0.15.0 > > Cool to add test. But this test needs a network access: > > $ ./pre-inst-env guix time-machine --commit=v0.15.0 -- describe > guix time-machine: error: Git error: failed to resolve address for git.savannah.gnu.org: Name or service not known > > It’s as I said elsewhere. :-) Well, I have not investigated more but I > guess a bug in some Git manipulation. From my testing, the test doesn't require networking, perhaps because it doesn't use the --commit or --branch arguments. I've tested that with this in /etc/hosts, with the first IP being bogus: --8<---------------cut here---------------start------------->8--- 192.168.254.254 git.savannah.gnu.org savannah --8<---------------cut here---------------end--------------->8--- I've investigated a bit, and it seems that reaching to the network is only done when using tags or branch names, not exact commits. That is expected I think, since tags or branch names are not immutable in Git, while a commit ID is. I've now installed this change; thanks for the review and suggestions! -- Thanks, Maxim
diff --git a/Makefile.am b/Makefile.am index 5ffcb6a12d..4228c07803 100644 --- a/Makefile.am +++ b/Makefile.am @@ -615,6 +615,7 @@ SH_TESTS = \ tests/guix-refresh.sh \ tests/guix-shell.sh \ tests/guix-shell-export-manifest.sh \ + tests/guix-time-machine.sh \ tests/guix-graph.sh \ tests/guix-describe.sh \ tests/guix-repl.sh \ diff --git a/doc/guix.texi b/doc/guix.texi index b50feed4c4..a3754b7019 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5060,6 +5060,20 @@ Invoking guix time-machine large number of packages; the result is cached though and subsequent commands targeting the same commit are almost instantaneous. +Due to @command{guix time-machine} relying on the ``inferiors'' +mechanism (@pxref{Inferiors}), the oldest commit it can travel to is +commit @samp{6298c3ff} (``v1.0.0''), dated May 1@sup{st}, 2019, which is +the first release that included the inferiors mechanism. An error is +returned when attempting to navigate to older commits. Note that +although it should technically be possible to travel to such an old +commit, the ease to do so will largely depend on the availability of +binary substitutes. When traveling to a distant past, some packages may +not easily build from source anymore. One such example are old versions +of Python 2 which had time bombs in its test suite, in the form of +expiring SSL certificates. This particular problem can be worked around +by setting the hardware clock to a value in the past before attempting +the build. + @quotation Note The history of Guix is immutable and @command{guix time-machine} provides the exact same software as they are in a specific Guix diff --git a/guix/inferior.scm b/guix/inferior.scm index 5dfd30a6c8..fca6fb4b22 100644 --- a/guix/inferior.scm +++ b/guix/inferior.scm @@ -871,11 +871,15 @@ (define* (cached-channel-instance store #:key (authenticate? #t) (cache-directory (%inferior-cache-directory)) - (ttl (* 3600 24 30))) + (ttl (* 3600 24 30)) + validate-channels) "Return a directory containing a guix filetree defined by CHANNELS, a list of channels. -The directory is a subdirectory of CACHE-DIRECTORY, where entries can be reclaimed after TTL seconds. -This procedure opens a new connection to the build daemon. AUTHENTICATE? -determines whether CHANNELS are authenticated." +The directory is a subdirectory of CACHE-DIRECTORY, where entries can be +reclaimed after TTL seconds. This procedure opens a new connection to the +build daemon. AUTHENTICATE? determines whether CHANNELS are authenticated. +VALIDATE-CHANNELS, if specified, must be a one argument procedure accepting a +list of channels that can be used to validate the channels; it should raise an +exception in case of problems." (define commits ;; Since computing the instances of CHANNELS is I/O-intensive, use a ;; cheaper way to get the commit list of CHANNELS. This limits overhead @@ -923,27 +927,30 @@ (define* (cached-channel-instance store (if (file-exists? cached) cached - (run-with-store store - (mlet* %store-monad ((instances - -> (latest-channel-instances store channels - #:authenticate? - authenticate?)) - (profile - (channel-instances->derivation instances))) - (mbegin %store-monad - ;; It's up to the caller to install a build handler to report - ;; what's going to be built. - (built-derivations (list profile)) - - ;; Cache if and only if AUTHENTICATE? is true. - (if authenticate? - (mbegin %store-monad - (symlink* (derivation->output-path profile) cached) - (add-indirect-root* cached) - (return cached)) - (mbegin %store-monad - (add-temp-root* (derivation->output-path profile)) - (return (derivation->output-path profile))))))))) + (begin + (when (procedure? validate-channels) + (validate-channels channels)) + (run-with-store store + (mlet* %store-monad ((instances + -> (latest-channel-instances store channels + #:authenticate? + authenticate?)) + (profile + (channel-instances->derivation instances))) + (mbegin %store-monad + ;; It's up to the caller to install a build handler to report + ;; what's going to be built. + (built-derivations (list profile)) + + ;; Cache if and only if AUTHENTICATE? is true. + (if authenticate? + (mbegin %store-monad + (symlink* (derivation->output-path profile) cached) + (add-indirect-root* cached) + (return cached)) + (mbegin %store-monad + (add-temp-root* (derivation->output-path profile)) + (return (derivation->output-path profile)))))))))) (define* (inferior-for-channels channels #:key diff --git a/guix/scripts/time-machine.scm b/guix/scripts/time-machine.scm index d7c71ef705..e4fe511382 100644 --- a/guix/scripts/time-machine.scm +++ b/guix/scripts/time-machine.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2019 Konrad Hinsen <konrad.hinsen@fastmail.net> ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com> +;;; Copyright © 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com> ;;; ;;; This file is part of GNU Guix. ;;; @@ -19,13 +20,15 @@ ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. (define-module (guix scripts time-machine) + #:use-module (guix channels) + #:use-module (guix diagnostics) #:use-module (guix ui) #:use-module (guix scripts) #:use-module (guix inferior) #:use-module (guix store) #:use-module (guix status) #:use-module ((guix git) - #:select (with-git-error-handling)) + #:select (update-cached-checkout with-git-error-handling)) #:use-module ((guix utils) #:select (%current-system)) #:use-module ((guix scripts pull) @@ -38,9 +41,17 @@ (define-module (guix scripts time-machine) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) #:use-module (srfi srfi-37) + #:use-module (srfi srfi-71) #:export (guix-time-machine)) +;;; The required inferiors mechanism relied on by 'guix time-machine' was +;;; firmed up in v1.0.0; it is the oldest, safest commit that can be travelled +;;; to. +(define %oldest-possible-commit + "6298c3ffd9654d3231a6f25390b056483e8f407c") ;v1.0.0 + ;;; ;;; Command-line options. @@ -81,7 +92,7 @@ (define %options (alist-delete 'repository-url result)))) (option '("commit") #t #f (lambda (opt name arg result) - (alist-cons 'ref `(commit . ,arg) result))) + (alist-cons 'ref `(tag-or-commit . ,arg) result))) (option '("branch") #t #f (lambda (opt name arg result) (alist-cons 'ref `(branch . ,arg) result))) @@ -140,8 +151,27 @@ (define-command (guix-time-machine . args) (let* ((opts (parse-args args)) (channels (channel-list opts)) (command-line (assoc-ref opts 'exec)) + (ref (assoc-ref opts 'ref)) (substitutes? (assoc-ref opts 'substitutes?)) (authenticate? (assoc-ref opts 'authenticate-channels?))) + + (define (validate-guix-channel channels) + "Finds the Guix channel among CHANNELS, and validates that REF as +captured from the closure, a git reference specification such as a commit hash +or tag associated to CHANNEL, is valid and new enough to satisfy the 'guix +time-machine' requirements. A `formatted-message' condition is raised +otherwise." + (let* ((guix-channel (find guix-channel? channels)) + (checkout commit relation (update-cached-checkout + (channel-url guix-channel) + #:ref (or ref '()) + #:starting-commit + %oldest-possible-commit))) + (unless (memq relation '(ancestor self)) + (raise (formatted-message + (G_ "cannot travel past commit `~a' from May 1st, 2019") + (string-take %oldest-possible-commit 12)))))) + (when command-line (let* ((directory (with-store store @@ -153,6 +183,8 @@ (define-command (guix-time-machine . args) #:dry-run? #f) (set-build-options-from-command-line store opts) (cached-channel-instance store channels - #:authenticate? authenticate?))))) + #:authenticate? authenticate? + #:validate-channels + validate-guix-channel))))) (executable (string-append directory "/bin/guix"))) (apply execl (cons* executable executable command-line)))))))) diff --git a/tests/guix-time-machine.sh b/tests/guix-time-machine.sh new file mode 100644 index 0000000000..8b62ef75ea --- /dev/null +++ b/tests/guix-time-machine.sh @@ -0,0 +1,28 @@ +# GNU Guix --- Functional package management for GNU +# Copyright © 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com> +# +# This file is part of GNU Guix. +# +# GNU Guix is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or (at +# your option) any later version. +# +# GNU Guix is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +# +# Test the 'guix time-machine' command-line utility. +# + +guix time-machine --version + +# Visiting a commit older than v1.0.0 fails. +! guix time-machine --commit=v0.15.0 + +exit 0
* doc/guix.texi (Invoking guix time-machine): Document limitation. * guix/inferior.scm (cached-channel-instance): New VALIDATE-CHANNELS argument. Use it to validate channels when there are no cache hit. * guix/scripts/time-machine.scm (%options): Tag the given reference with 'tag-or-commit instead of 'commit. (%oldest-possible-commit): New variable. (guix-time-machine) <validate-guix-channel>: New nested procedure. Pass it to the 'cached-channel-instance' call. * tests/guix-time-machine.sh: New test. * Makefile.am (SH_TESTS): Register it. Suggested-by: Simon Tournier <zimon.toutoune@gmail.com> Reviewed-by: Ludovic Courtès <ludo@gnu.org> --- Changes in v2: - Test it - Delay validation work to inside cached-channel-instance, when there's no cache hit - Expound doc to mention possible problems and possible workarounds Makefile.am | 1 + doc/guix.texi | 14 +++++++++ guix/inferior.scm | 57 ++++++++++++++++++++--------------- guix/scripts/time-machine.scm | 38 +++++++++++++++++++++-- tests/guix-time-machine.sh | 28 +++++++++++++++++ 5 files changed, 110 insertions(+), 28 deletions(-) create mode 100644 tests/guix-time-machine.sh