[bug#43745,04/27] gnu: ocaml-migrate-parsetree: Update to 1.7.3.

* gnu/packages/ocaml.scm (ocaml-migrate-parsetree): Update to 1.7.3.
---
 gnu/packages/ocaml.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

On Thu, 01 Oct 2020 at 15:41, Julien Lepiller <julien@lepiller.eu> wrote:
> * gnu/packages/ocaml.scm (ocaml-migrate-parsetree): Update to 1.7.3.
> ---
>  gnu/packages/ocaml.scm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

LGTM but '--check' returns:

--8<---------------cut here---------------start------------->8---
guix build: error: derivation `/gnu/store/fgkmww5s1srlh8frn1jbzz952mkn4pvi-ocaml-migrate-parsetree-1.7.3.drv' may not be deterministic: output `/gnu/store/3pb3cz6s0p37p4737gm6cj50p1vh02q8-ocaml-migrate-parsetree-1.7.3' differs
--8<---------------cut here---------------end--------------->8---

Could you open a bug report when you will merge to master?


All the best,
simon

Le Tue, 13 Oct 2020 00:53:43 +0200,
zimoun <zimon.toutoune@gmail.com> a écrit :

> 
> On Thu, 01 Oct 2020 at 15:41, Julien Lepiller <julien@lepiller.eu>
> wrote:
> > * gnu/packages/ocaml.scm (ocaml-migrate-parsetree): Update to 1.7.3.
> > ---
> >  gnu/packages/ocaml.scm | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> LGTM but '--check' returns:
> 
> --8<---------------cut here---------------start------------->8---
> guix build: error: derivation
> `/gnu/store/fgkmww5s1srlh8frn1jbzz952mkn4pvi-ocaml-migrate-parsetree-1.7.3.drv'
> may not be deterministic: output
> `/gnu/store/3pb3cz6s0p37p4737gm6cj50p1vh02q8-ocaml-migrate-parsetree-1.7.3'
> differs --8<---------------cut
> here---------------end--------------->8---
> 
> Could you open a bug report when you will merge to master?
> 
> 
> All the best,
> simon

Thank you! I've been investigating this, but I don't really understand
what's happening.  So, I've created myself an environment in which I
could compile the sources of migrate-parsetree:

cd s1
/my/guix/repo/pre-inst-env guix environment -C ocaml-migrate-parsetree
dune build @install
exit

mv s1 s2
copy sources to s1 again (to prevent issues due to recorded build
paths), and build again.

This allows me to keep and compare all the intermediate build products.
From diffoscope and _build/log, I found the first differing file that
is a cmo file.  I note that the build is not in order, but arguments
are always sorted. This is expected, as dune runs with 4 cores.

Next, I've created this very simple file (one line), called test.ml:

module Ast_402 = Migrate_parsetree__Ast_402

(this is the first line of Ast_402.ml-gen, which is the file that
builds the first differing cmo.

Then, I build it using:

OCAMLC=ocamlc.opt -w @1..3@5..28@30..39@43@46..47@49..57@61..62-40
-strict-sequence -strict-formats-short-paths -keep-locs -w -49
-nopervasives -nostdlib -g -bin-annot -no-alias-deps -opaque -o a.cmo
-c -implem test.ml

using exactly the same options as dune, excluding a -I (include).  This
creates a.cmo, a.cmt and a.cmi. Then, using sha256sum, I can see if the
files differ.  Removing these files and trying again gives me always
the same result.  However, if I instead move them to b.cm{o,t,i}, the
next time I build a.cmo, there is a difference between a.cmo and b.cmo,
as well as the cmt files (but not the cmi files):

${OCAMLC}
sha256sum a.cmo
de1caa8b636e97e3e7c964ea84ea52bc532d80653afd13a5a106687886255861  a.cmo
rm a.cm*
${OCAMLC}
sha256sum a.cmo
de1caa8b636e97e3e7c964ea84ea52bc532d80653afd13a5a106687886255861  a.cmo
...
mv {a,b}.cmo; mv {a,b}.cmt; mv {a,b}.cmi
${OCAMLC}
sha256sum a.cmo
8cfd4b3d214d924f8229bd3697cc3eace58d4e20eb7cb27c27e7b670fa9809e6  a.cmo
rm a.cm*
${OCAMLC}
sha256sum a.cmo
8cfd4b3d214d924f8229bd3697cc3eace58d4e20eb7cb27c27e7b670fa9809e6  a.cmo
...


and with every additional files (c.cmo, d.cmo, ...) I get a different
hash, but as long as I keep the same files there, nothing changes.

The hash also changes if I create additional files, such as:

touch e.cmi (the cmi file seems to be the only one whose presence
changes the content of the cmo and cmt files).


So, my hypothesis is that dune is building files out of order, but lets
ocaml read the generated cmi files.  Since the build is not in order,
when it builds the same file in two different builds, the cmi are not
the same and the result is different.  Since dune will always use all
my cores, I used a trick I learned from LFS:

echo 0 > /sys/devices/system/cpu/cpu1/online
(and similar for every other core, except cpu0)

This way, I have a single-core machine and, hopefully, dune runs
sequentially. This time, --rounds=2 passed (after removing the existing
store item of course).

Hi Julien,

Wow!  Thank you for this detailed investigation.


On Tue, 13 Oct 2020 at 04:03, Julien Lepiller <julien@lepiller.eu> wrote:

> So, my hypothesis is that dune is building files out of order, but lets
> ocaml read the generated cmi files.  Since the build is not in order,
> when it builds the same file in two different builds, the cmi are not
> the same and the result is different.  Since dune will always use all
> my cores, I used a trick I learned from LFS:
>
> echo 0 > /sys/devices/system/cpu/cpu1/online
> (and similar for every other core, except cpu0)
>
> This way, I have a single-core machine and, hopefully, dune runs
> sequentially. This time, --rounds=2 passed (after removing the existing
> store item of course).

Where do you this “echo 0 > /sys/devices/system/cpu/cpu1/online“ to
disable all except cpu0?  Because, I have tried as root on my 4 cpu
machine and then “./pre-inst-env guix build ocaml-migrate-parsetree
–no-grafts –check“ still returns an error.


Cheers,
simon

Your existing build was built unreproducibly. You need to remove it with guix gc first. Then you can run --rounds=2.

Alternatively add "-j1" as an argument to dune (as #:build-flags). That will change the output, ensuring you don't compare with an unreproducible build.

Le 13 octobre 2020 08:09:55 GMT-04:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>Hi Julien,
>
>Wow!  Thank you for this detailed investigation.
>
>
>On Tue, 13 Oct 2020 at 04:03, Julien Lepiller <julien@lepiller.eu>
>wrote:
>
>> So, my hypothesis is that dune is building files out of order, but
>lets
>> ocaml read the generated cmi files.  Since the build is not in order,
>> when it builds the same file in two different builds, the cmi are not
>> the same and the result is different.  Since dune will always use all
>> my cores, I used a trick I learned from LFS:
>>
>> echo 0 > /sys/devices/system/cpu/cpu1/online
>> (and similar for every other core, except cpu0)
>>
>> This way, I have a single-core machine and, hopefully, dune runs
>> sequentially. This time, --rounds=2 passed (after removing the
>existing
>> store item of course).
>
>Where do you this “echo 0 > /sys/devices/system/cpu/cpu1/online“ to
>disable all except cpu0?  Because, I have tried as root on my 4 cpu
>machine and then “./pre-inst-env guix build ocaml-migrate-parsetree
>–no-grafts –check“ still returns an error.
>
>
>Cheers,
>simon

On Tue, 13 Oct 2020 at 17:24, Julien Lepiller <julien@lepiller.eu> wrote:
>
> Your existing build was built unreproducibly. You need to remove it with guix gc first. Then you can run --rounds=2.
>
> Alternatively add "-j1" as an argument to dune (as #:build-flags). That will change the output, ensuring you don't compare with an unreproducible build.

Yeah, for sure!  Sorry.
It works with "echo 0>/sys/devices/system/cpu/cpu1/online" etc.  Nice
trick, thanks.

Cheers,
simon