diff mbox series

[bug#74112] Fix determinism issue in guix package

Message ID ZyJ1wSX6cPIdnLpN@kernelpanicroom
State New
Headers
Series [bug#74112] Fix determinism issue in guix package |

Commit Message

Jakob Kirsch Oct. 30, 2024, 6:06 p.m. UTC 
  I've recently noticed that `guix challenge guix` returns different hashes for each substitute server and also every build with `guix build --no-grafts guix` fails.
Running `guix build --no-grafts guix --cores=1` seems to reliably produce the same hash though. As someone pointed out on XMPP, Guile seems to have issues with parallel builds so this patch disables them for the guix package.

I think this has high importance because the main guix package should be reproducible in order to trust the whole chain of packages.
From 0f3df56dd0c430c09ba2839c9e2d5b32948201ae Mon Sep 17 00:00:00 2001
Message-ID: <0f3df56dd0c430c09ba2839c9e2d5b32948201ae.1730311414.git.jakob.kirsch@web.de>
From: Jakob Kirsch <jakob.kirsch@web.de>
Date: Wed, 30 Oct 2024 19:02:15 +0100
Subject: [PATCH v1] gnu: guix: Fix determinism issue

* gnu/packages/package-management.scm (guix): Fix determinism issue by disabling parallel build.

Change-Id: Ie28e16ed1f15cbc0da0c0d70b2c461e2baa3ff0a
---
 gnu/packages/package-management.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


base-commit: d6f775c30c6f47e174f6110d1089edc6315600e4
--
2.46.0

Comments

Hilton Chain Nov. 5, 2024, 10:31 a.m. UTC | #1 
Hi Jakob,

On Tue, 05 Nov 2024 02:38:34 +0800,
Jakob Kirsch via Bug reports for GNU Guix wrote:
>
> I've recently noticed that `guix challenge guix` fails since guix isn't
> reproducible at the moment. As someone pointed out on XMPP, this is due to
> parallelism issues with Guile so running `guix build guix --no-grafts
> --cores=1 --check` reliably produces the same output.
>
> I've sent a simple patch in #74112 to address this and I think this is
> important since the main guix package should definitely be reproducible so it
> can be trusted.

I can confirm the reproducibility issue.

I have two x86_64-linux machines building guix to verify the fix, I'll apply
your patch once they produce matching outputs.

Thanks for reporting!
Hilton Chain Nov. 5, 2024, 3:25 p.m. UTC | #2 
Hello again,

On Tue, 05 Nov 2024 18:31:04 +0800,
Hilton Chain via Guix-patches via wrote:
>
> Hi Jakob,
>
> On Tue, 05 Nov 2024 02:38:34 +0800,
> Jakob Kirsch via Bug reports for GNU Guix wrote:
> >
> > I've recently noticed that `guix challenge guix` fails since guix isn't
> > reproducible at the moment. As someone pointed out on XMPP, this is due to
> > parallelism issues with Guile so running `guix build guix --no-grafts
> > --cores=1 --check` reliably produces the same output.
> >
> > I've sent a simple patch in #74112 to address this and I think this is
> > important since the main guix package should definitely be reproducible so it
> > can be trusted.
>
> I can confirm the reproducibility issue.
>
> I have two x86_64-linux machines building guix to verify the fix, I'll apply
> your patch once they produce matching outputs.

Took me quite a while to build 5 rounds. :)

--8<---------------cut here---------------start------------->8---
$ guix hash --serializer=nar /gnu/store/fs7x07jfn7igpkwv3alrs9by21q70y13-guix-1.4.0-26.5ab3c4c
0kh87wb4qn97kwzrf4igal71cjvv143j6jr2y3dwfzcy1madj1ll
--8<---------------cut here---------------end--------------->8---

Applied #74112 as 4c56d0cccdc44e12484b26332715f54768738c5f, thanks!
Simon Tournier Nov. 7, 2024, 5:54 p.m. UTC | #3 
Hi,

On Tue, 05 Nov 2024 at 23:25, Hilton Chain via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

>> I can confirm the reproducibility issue.
>>
>> I have two x86_64-linux machines building guix to verify the fix, I'll apply
>> your patch once they produce matching outputs.
>
> Took me quite a while to build 5 rounds. :)
>
> --8<---------------cut here---------------start------------->8---
> $ guix hash --serializer=nar /gnu/store/fs7x07jfn7igpkwv3alrs9by21q70y13-guix-1.4.0-26.5ab3c4c
> 0kh87wb4qn97kwzrf4igal71cjvv143j6jr2y3dwfzcy1madj1ll
> --8<---------------cut here---------------end--------------->8---
>
> Applied #74112 as 4c56d0cccdc44e12484b26332715f54768738c5f, thanks!

Maybe I am missing something.  To my knowledge, .go files produced by
Guile are not always reproducible, see bug#20272 [1].  And, from my
understanding, Guix cannot be reproducible until this bug had been
fixed.  Therefore, I am not convinced that this patch is worth under
this frame considering the build-time penalty it brings.

That’s said, maybe it’s better than nothing and the package ’guix’ is
barely built after all.  I do not know.

What people think?

Cheers,
simon

1: https://issues.guix.gnu.org/issue/20272
Maxim Cournoyer Nov. 10, 2024, 9:11 a.m. UTC | #4 
Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Tue, 05 Nov 2024 at 23:25, Hilton Chain via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:
>
>>> I can confirm the reproducibility issue.
>>>
>>> I have two x86_64-linux machines building guix to verify the fix, I'll apply
>>> your patch once they produce matching outputs.
>>
>> Took me quite a while to build 5 rounds. :)
>>
>> --8<---------------cut here---------------start------------->8---
>> $ guix hash --serializer=nar /gnu/store/fs7x07jfn7igpkwv3alrs9by21q70y13-guix-1.4.0-26.5ab3c4c
>> 0kh87wb4qn97kwzrf4igal71cjvv143j6jr2y3dwfzcy1madj1ll
>> --8<---------------cut here---------------end--------------->8---
>>
>> Applied #74112 as 4c56d0cccdc44e12484b26332715f54768738c5f, thanks!
>
> Maybe I am missing something.  To my knowledge, .go files produced by
> Guile are not always reproducible, see bug#20272 [1].  And, from my
> understanding, Guix cannot be reproducible until this bug had been
> fixed.  Therefore, I am not convinced that this patch is worth under
> this frame considering the build-time penalty it brings.
>
> That’s said, maybe it’s better than nothing and the package ’guix’ is
> barely built after all.  I do not know.
>
> What people think?

Perhaps we should set the default parallel-build? to #f in the
guile-build-system at least in the meantime, with a prominent comment as
to why and a reference to the upstream issue?  Many Guile packages use
the gnu-build-system so that wouldn't cover all of them like
'guix'... I'm not sure.

It'd be nicer to fix the underlying guile issue (again?), but I doubt
many people are up to this.
Ludovic Courtès Nov. 14, 2024, 10:01 a.m. UTC | #5 
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

>> Maybe I am missing something.  To my knowledge, .go files produced by
>> Guile are not always reproducible, see bug#20272 [1].  And, from my
>> understanding, Guix cannot be reproducible until this bug had been
>> fixed.  Therefore, I am not convinced that this patch is worth under
>> this frame considering the build-time penalty it brings.
>>
>> That’s said, maybe it’s better than nothing and the package ’guix’ is
>> barely built after all.  I do not know.
>>
>> What people think?
>
> Perhaps we should set the default parallel-build? to #f in the
> guile-build-system at least in the meantime, with a prominent comment as
> to why and a reference to the upstream issue?  Many Guile packages use
> the gnu-build-system so that wouldn't cover all of them like
> 'guix'... I'm not sure.

Sounds good to me: packages using ‘guile-build-system’ are usually
relatively small so the impact is negligible.

‘guix’ is a little different because it takes so much time to build
sequentially…

Ludo’.
diff mbox series

Patch

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 1763d2d59f..478a74385b 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -199,7 +199,8 @@  (define-public guix
                 (file-name (string-append "guix-" version "-checkout"))))
       (build-system gnu-build-system)
       (arguments
-       `(#:configure-flags (list
+       `(#:parallel-build? #false ; for reproducibility
+         #:configure-flags (list

                             ;; Provide channel metadata for 'guix describe'.
                             ;; Don't pass '--with-channel-url' and