[bug#38546,00/11] Update Julia, Fix precompilation, add HTTP.jl

Hi Simon,

Sorry, I forgot to send the dsfmt patch.  Also, julia's
SOURCE_DATE_EPOCH patch was named differently. I've fixed this in theattached patches.  You need to apply Add-dsfmt.patch, Update-to-1.3.1
and then julia-use-SOURCE_DATE_EPOCH.

About reproducibility: if I'm not wrong, sys.so contains Base library
precompiled ([1]).  Precompilation is still non deterministic (here's
[2] an issue on github). Something I did to check precompilation:

Since I'm tired of this, I've started investigating better.

Loaded a guix container
#+begin_src bash
mkdir -p /tmp/jlprecompiled/
./pre-inst-env guix environment -u nixbld --no-cwd --container --network --expose=$(realpath determinism) --share=/tmp/jlprecompiled/ --ad-hoc julia coreutils nss-certs curl tar gzip libfaketime
#+end_src

from this container, run some time this:

#+begin_src bash
export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"
export SSL_CERT_FILE="$SSL_CERT_DIR/ca-certificates.crt"
export GIT_SSL_CAINFO="$SSL_CERT_FILE"

SOURCE_DATE_EPOCH=1 julia --startup-file=no -q -e 'using Pkg; Pkg.instantiate()'
SOURCE_DATE_EPOCH=1 faketime '2008-12-24 08:15:42' julia --startup-file=no -q -e 'using Random; using Pkg; Pkg.add("Compat"); Random.seed!(0); using Compat'

cp -r ~/.julia/compiled/v1.3/Compat/ /tmp/jlprecompiled/$RANDOM
sha256sum ~/.julia/compiled/v1.3/Compat/*
# 6c10d65f6cd9cdd5b98e4d5e2b82b7bd00508f0daccef0967dec92cb6468052e  /home/nixbld/.julia/compiled/v1.3/Compat/GSFWK_Pua4I.ji
ls -lah ~/.julia/compiled/v1.3/Compat/*
# -rw------- 1 nixbld users 102K Jan 21 07:39 /home/nixbld/.julia/compiled/v1.3/Compat/GSFWK_Pua4I.ji
#+end_src

I could not get the same results twice (also, size differs).  I'll work
on this on some spare time (for example, there are other places where
SOURCE_DATE_EPOCH can be used, but this [3] is a problem I need to
solve first).

Maybe 1.3.1 (when reviewed) can be merged, since we have
the same problem with julia 1.1, but we can wait for the
source-date-epoch and julia-xyz patches until we solve this.

What do you think?

Thanks, Nicolò

[1] https://docs.julialang.org/en/v1/devdocs/sysimg/
[2] https://github.com/JuliaLang/julia/issues/25900
[3] https://github.com/JuliaLang/julia/issues/34115
zimoun <zimon.toutoune@gmail.com> writes:

> Hi Nicolò,
>
> Thank you for working on this.
> Julia rocks! ;-)
>
>
> However, I am lost with your patch set.
>
> I have tried only the first patch you attached. I hit this error:
>
> --8<---------------cut here---------------start------------->8---
> error: dsfmt: unbound variable
> --8<---------------cut here---------------end--------------->8---
>
> So something is missing. :-)
>
> Then I notice that you change the list of 'patches' and you "forgot"
> to send them too. :-)
>
>
> Almost related, does this update fix the reproducibility bug [1]? For example,
>
> --8<---------------cut here---------------start------------->8---
> ./pre-inst-env guix build julia
> ./pre-inst-env guix build julia --check --no-grafts -K
> --8<---------------cut here---------------end--------------->8---
>
>
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22304
>
>
> All the best,
> simon

Hi Nicolò,

Cool that you figured out a source of non-reproducibility.


On Tue, 21 Jan 2020 at 14:45, Nicolò Balzarotti <anothersms@gmail.com> wrote:

> Sorry, I forgot to send the dsfmt patch.  Also, julia's
> SOURCE_DATE_EPOCH patch was named differently. I've fixed this in theattached patches.  You need to apply Add-dsfmt.patch, Update-to-1.3.1
> and then julia-use-SOURCE_DATE_EPOCH.

This patch 'julia-SOURCE_DATE_EPOCH-mtime.patch' is the one you
mentioned here [#], right?

Could you send it as an upstream PR?

[#] https://github.com/JuliaLang/julia/issues/34115#issuecomment-568171025


> About reproducibility: if I'm not wrong, sys.so contains Base library
> precompiled ([1]).  Precompilation is still non deterministic (here's
> [2] an issue on github). Something I did to check precompilation:

I am not sure to well understand the source of non-determinism.

Does the patch about SOURCE_DATE_EPOCH fix the issue of [1] and [2]?
Or is it something else?


> I could not get the same results twice (also, size differs).  I'll work
> on this on some spare time (for example, there are other places where
> SOURCE_DATE_EPOCH can be used, but this [3] is a problem I need to
> solve first).

Is the problem [3] not solved by 'julia-SOURCE_DATE_EPOCH-mtime.patch'?



> Maybe 1.3.1 (when reviewed) can be merged, since we have
> the same problem with julia 1.1, but we can wait for the
> source-date-epoch and julia-xyz patches until we solve this.

My opinion is: if a patch is floating around to fix the
source-date-epoch issue, let try to push it upstream. If it is
rejected, let talk later if Guix will include it or not. And in the
meantime, I will try to review the 1.3.1 because yes I agree that it
should be included even if we know it is not reproducible -- the
package Guitarix [@] is updated and not reproducible neither.

[@] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21803


Thank you for working on this.


All the best,
simon


> [1] https://docs.julialang.org/en/v1/devdocs/sysimg/
> [2] https://github.com/JuliaLang/julia/issues/25900
> [3] https://github.com/JuliaLang/julia/issues/34115

Hi Simon,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Nicolò,
>
> Cool that you figured out a source of non-reproducibility.
>
>
> On Tue, 21 Jan 2020 at 14:45, Nicolò Balzarotti <anothersms@gmail.com> wrote:
>
>> Sorry, I forgot to send the dsfmt patch.  Also, julia's
>> SOURCE_DATE_EPOCH patch was named differently. I've fixed this in theattached patches.  You need to apply Add-dsfmt.patch, Update-to-1.3.1
>> and then julia-use-SOURCE_DATE_EPOCH.
>
> This patch 'julia-SOURCE_DATE_EPOCH-mtime.patch' is the one you
> mentioned here [#], right?
>
> Could you send it as an upstream PR?
>
> [#] https://github.com/JuliaLang/julia/issues/34115#issuecomment-568171025
>
>
>> About reproducibility: if I'm not wrong, sys.so contains Base library
>> precompiled ([1]).  Precompilation is still non deterministic (here's
>> [2] an issue on github). Something I did to check precompilation:
>
> I am not sure to well understand the source of non-determinism.
>
> Does the patch about SOURCE_DATE_EPOCH fix the issue of [1] and [2]?
> Or is it something else?
>

The first patch (the one I mention in [#]) fixes _one source of_
non-determinism (the stored mtime inside the precompile cache).  But
others are present (precompiling the same file 2 times lead to
differences even in file size, something I reported on [2]).  What I'm
working on is trying to find out what other sources are and patch them,
too.  I patched some c code (src/support/timefuncs.c) so that it follows
SOURCE_DATE_EPOCH too.  This finally lead to same-size files.  I'm now
working on parsing binary (.ji) files directly with julia built in
function to have a textual representation of the differences.

I can't send SOURCE_DATE_EPOCH patches upstream until [3] is merged, as
patching timefuncts leads to an endless test suite (as they were using
time() to check for passed time. If time() always returns 1, tests
sleeps forever).  Yesterday evening I completed that patch (and has been
approved).  My plans are on finding the hopefully last source of
non-determinism, patch it and send everything upstream (hoping they
care).

>
>> I could not get the same results twice (also, size differs).  I'll work
>> on this on some spare time (for example, there are other places where
>> SOURCE_DATE_EPOCH can be used, but this [3] is a problem I need to
>> solve first).
>
> Is the problem [3] not solved by 'julia-SOURCE_DATE_EPOCH-mtime.patch'?
>
>
>
>> Maybe 1.3.1 (when reviewed) can be merged, since we have
>> the same problem with julia 1.1, but we can wait for the
>> source-date-epoch and julia-xyz patches until we solve this.
>
> My opinion is: if a patch is floating around to fix the
> source-date-epoch issue, let try to push it upstream. If it is
> rejected, let talk later if Guix will include it or not. And in the
> meantime, I will try to review the 1.3.1 because yes I agree that it
> should be included even if we know it is not reproducible -- the
> package Guitarix [@] is updated and not reproducible neither.
>

Thanks! I hope in a 1.4 release where everything is fixed ;)

Nicolò

> [@] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21803
>
>
> Thank you for working on this.
>
>
> All the best,
> simon
>
>
>> [1] https://docs.julialang.org/en/v1/devdocs/sysimg/
>> [2] https://github.com/JuliaLang/julia/issues/25900
>> [3] https://github.com/JuliaLang/julia/issues/34115

Hi Nicolò,

On Wed, 22 Jan 2020 at 10:59, Nicolò Balzarotti <anothersms@gmail.com> wrote:

> The first patch (the one I mention in [#]) fixes _one source of_
> non-determinism (the stored mtime inside the precompile cache).  But
> others are present (precompiling the same file 2 times lead to
> differences even in file size, something I reported on [2]).  What I'm
> working on is trying to find out what other sources are and patch them,
> too.  I patched some c code (src/support/timefuncs.c) so that it follows
> SOURCE_DATE_EPOCH too.  This finally lead to same-size files.  I'm now
> working on parsing binary (.ji) files directly with julia built in
> function to have a textual representation of the differences.

Thank you for the explanations. Cool!


> I can't send SOURCE_DATE_EPOCH patches upstream until [3] is merged, as
> patching timefuncts leads to an endless test suite (as they were using
> time() to check for passed time. If time() always returns 1, tests
> sleeps forever).  Yesterday evening I completed that patch (and has been
> approved).  My plans are on finding the hopefully last source of
> non-determinism, patch it and send everything upstream (hoping they
> care).

Awesome if upstream are responsive.


> Thanks! I hope in a 1.4 release where everything is fixed ;)

I have seen that 1.4-rc1 is just released. Hope that everything about
reproducibility will be fixed. ;-)


I have not yet tried your patches. Sorry bit busy at work.
And I will not have any slot next week, so I will be back on the topic
after the Guix Days. Except if someone beats me. :-)
(see you there :-)


Thank you for improving the Julia ecosystem in Guix because Julia is
becoming quite popular.

Cheers,
simon

Hi Nicoló,

On Fri, 24 Jan 2020 at 18:26, zimoun <zimon.toutoune@gmail.com> wrote:

> I have not yet tried your patches. Sorry bit busy at work.
> And I will not have any slot next week, so I will be back on the topic
> after the Guix Days. Except if someone beats me. :-)
> (see you there :-)

I have tried them. Ouch! The tests eat so much...

Do you have a special motivation to rename 'llvm-patch' to 'julia-patch'?
(I agree, it is better :-))

Well, only 2 inputs are missing. Do you think it is affordable to pack
them and so remove the commentary?

--8<---------------cut here---------------start------------->8---
;; FIXME: The following inputs are downloaded from upstream to allow us
;; to use the lightweight Julia release tarball.  Ideally, these inputs
;; would eventually be replaced with proper Guix packages.
;; Find dependencies versions here:
;; https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/Versions.make
--8<---------------cut here---------------end--------------->8---


Otherwise, there is some issues about indentation -- I think you use
Emacs, so check that '.dir-locals.el' is correctly setup -- double
space for sentences in description,
'patch/julia-SOURCE_DATE_EPOCH-mtime.patch' should be added with the
same commit that "Update", etc.
I have half-corrected so I can send you these 2 v2-patches if you want to. :-)

And I have not tried yet to build with '--check'.


All the best,
simon

Hi Nicoló,


On Tue, 4 Feb 2020 at 19:03, zimoun <zimon.toutoune@gmail.com> wrote:

> I have tried them. Ouch! The tests eat so much...

The tests fail on my desktop machine with 8GB of RAM.

--8<---------------cut here---------------start------------->8---
running testset Distributed...
ERROR: LoadError: TaskFailedException:
Timed out waiting to read host:port string from worker.
Stacktrace:
 [1] worker_from_id(::Distributed.ProcessGroup, ::Int64) at
/tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:1059
 [2] worker_from_id at
/tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:1056
[inlined]
 [3] #remote_do#156 at
/tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:482
[inlined]
 [4] remote_do at
/tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:482
[inlined]
 [5] kill at /tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/managers.jl:534
[inlined]
 [6] create_worker(::Distributed.LocalManager, ::WorkerConfig) at
/tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:581
 [7] setup_launched_worker(::Distributed.LocalManager, ::WorkerConfig,
::Array{Int64,1}) at
/tmp/guix-build-julia-1.3.1.drv-0/julia-1.3.1/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:523
 [8] (::Distributed.var"#43#46"{Distributed.LocalManager,Array{Int64,1},WorkerConfig})()
at ./task.jl:333

[...more failed...]
--8<---------------cut here---------------end--------------->8---


Do they pass on your machine?


> And I have not tried yet to build with '--check'.

Still not reproducible as expected [1]. :-)

[1] http://issues.guix.gnu.org/issue/22304


Cheers,
simon

ps:
I think you are busy by your thesis. Cross the fingers. ;-)

Hi Nicoló,

I have seen your message on IRC.
Cool for the release 1.4.


Could we try to include first this 1.3 one?  Then push the 1.4 if it
is already ready.

Because then, this release will be usable "forever" (or almost ;-)
even if it is not perfect and issues are fixed with the 1.4.
I mean, once a package is inside Guix, then it becomes "easily" usable
with 'guix time-machine' and saved on Software Heritage.
Therefore, considering the big picture of Scientific Reproducibility,
it matters -- for the future to be able to travel back -- to also have
1.3. ;-)

I think we are almost done. From my point of view, we just need to
address (or comment) these 3 points:

1.
> Do you have a special motivation to rename 'llvm-patch' to 'julia-patch'?
> (I agree, it is better :-))


2.
> Well, only 2 inputs are missing. Do you think it is affordable to pack
> them and so remove the commentary?
>
> --8<---------------cut here---------------start------------->8---
> ;; FIXME: The following inputs are downloaded from upstream to allow us
> ;; to use the lightweight Julia release tarball.  Ideally, these inputs
> ;; would eventually be replaced with proper Guix packages.
> ;; Find dependencies versions here:
> ;; https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/Versions.make
> --8<---------------cut here---------------end--------------->8---


3.
> Otherwise, there is some issues about indentation -- I think you use
> Emacs, so check that '.dir-locals.el' is correctly setup -- double
> space for sentences in description,
> 'patch/julia-SOURCE_DATE_EPOCH-mtime.patch' should be added with the
> same commit that "Update", etc.
> I have half-corrected so I can send you these 2 v2-patches if you want to. :-)



What do you think?


Cheers,
simon

zimoun <zimon.toutoune@gmail.com> writes:

Hi,
> Hi Nicoló,
>
> I have seen your message on IRC.
> Cool for the release 1.4.
>
>
> Could we try to include first this 1.3 one?  Then push the 1.4 if it
> is already ready.
>
> Because then, this release will be usable "forever" (or almost ;-)
> even if it is not perfect and issues are fixed with the 1.4.
> I mean, once a package is inside Guix, then it becomes "easily" usable
> with 'guix time-machine' and saved on Software Heritage.
> Therefore, considering the big picture of Scientific Reproducibility,
> it matters -- for the future to be able to travel back -- to also have
> 1.3. ;-)
>

Ok, let's do both
> I think we are almost done. From my point of view, we just need to
> address (or comment) these 3 points:
>
> 1.
>> Do you have a special motivation to rename 'llvm-patch' to 'julia-patch'?
>> (I agree, it is better :-))
>
Yes, because now it's not only used by llvm-julia but also by the new
libunwind-julia.  So calling llvm-patch a function that takes a patch
from the julia repo and that is applied to a "julia module" is measleading.

>
> 2.
>> Well, only 2 inputs are missing. Do you think it is affordable to pack
>> them and so remove the commentary?
>>
>> --8<---------------cut here---------------start------------->8---
>> ;; FIXME: The following inputs are downloaded from upstream to allow us
>> ;; to use the lightweight Julia release tarball.  Ideally, these inputs
>> ;; would eventually be replaced with proper Guix packages.
>> ;; Find dependencies versions here:
>> ;; https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/Versions.make
>> --8<---------------cut here---------------end--------------->8---
>

I indeed tried.  The main problem is that in the Makefile there's no
USE_SYSTEM_{OBJCONV/LIBWHICH}, and I didn't want to mess with the it.
But yes, I think I could at least try again.

>
> 3.
>> Otherwise, there is some issues about indentation -- I think you use
>> Emacs, so check that '.dir-locals.el' is correctly setup -- double
>> space for sentences in description,
>> 'patch/julia-SOURCE_DATE_EPOCH-mtime.patch' should be added with the
>> same commit that "Update", etc.
>> I have half-corrected so I can send you these 2 v2-patches if you want to. :-)
>
>
If you already fixed them, yes please send a v2 patch and I'll continue
working from there.

>
> What do you think?
>
Fine for me, do we need also to debug a broken test right?
>
> Cheers,
> simon


Nicolò

On Tue, 11 Feb 2020 at 16:10, Nicolò Balzarotti <anothersms@gmail.com> wrote:

> > 1.
> >> Do you have a special motivation to rename 'llvm-patch' to 'julia-patch'?
> >> (I agree, it is better :-))
> >
> Yes, because now it's not only used by llvm-julia but also by the new
> libunwind-julia.  So calling llvm-patch a function that takes a patch
> from the julia repo and that is applied to a "julia module" is measleading.

Ok.


> > 2.
> >> Well, only 2 inputs are missing. Do you think it is affordable to pack
> >> them and so remove the commentary?
> >>
> >> --8<---------------cut here---------------start------------->8---
> >> ;; FIXME: The following inputs are downloaded from upstream to allow us
> >> ;; to use the lightweight Julia release tarball.  Ideally, these inputs
> >> ;; would eventually be replaced with proper Guix packages.
> >> ;; Find dependencies versions here:
> >> ;; https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/Versions.make
> >> --8<---------------cut here---------------end--------------->8---
>
> I indeed tried.  The main problem is that in the Makefile there's no
> USE_SYSTEM_{OBJCONV/LIBWHICH}, and I didn't want to mess with the it.
> But yes, I think I could at least try again.

Let me check how it will be crawled by SWH to be sure that Guix will
be able to still build Julia 1.3 even if GitHub does not exist
anymore.
And I am in favour to let these 2 inputs.


> > 3.
> >> Otherwise, there is some issues about indentation -- I think you use
> >> Emacs, so check that '.dir-locals.el' is correctly setup -- double
> >> space for sentences in description,
> >> 'patch/julia-SOURCE_DATE_EPOCH-mtime.patch' should be added with the
> >> same commit that "Update", etc.
> >> I have half-corrected so I can send you these 2 v2-patches if you want to. :-)
> >
> If you already fixed them, yes please send a v2 patch and I'll continue
> working from there.

Stay tuned. :-)
I try to send it today...


> Fine for me, do we need also to debug a broken test right?

I do not know if the test are broken. I am not able to run them on my
desktop machine. Not enough memory.


Cheers,
simon

zimoun <zimon.toutoune@gmail.com> writes:

Hi

> On Tue, 11 Feb 2020 at 16:10, Nicolò Balzarotti <anothersms@gmail.com> wrote:
>
>> > 1.
>> >> Do you have a special motivation to rename 'llvm-patch' to 'julia-patch'?
>> >> (I agree, it is better :-))
>> >
>> Yes, because now it's not only used by llvm-julia but also by the new
>> libunwind-julia.  So calling llvm-patch a function that takes a patch
>> from the julia repo and that is applied to a "julia module" is measleading.
>
> Ok.
>
>
>> > 2.
>> >> Well, only 2 inputs are missing. Do you think it is affordable to pack
>> >> them and so remove the commentary?
>> >>
>> >> --8<---------------cut here---------------start------------->8---
>> >> ;; FIXME: The following inputs are downloaded from upstream to allow us
>> >> ;; to use the lightweight Julia release tarball.  Ideally, these inputs
>> >> ;; would eventually be replaced with proper Guix packages.
>> >> ;; Find dependencies versions here:
>> >> ;; https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/Versions.make
>> >> --8<---------------cut here---------------end--------------->8---
>>
>> I indeed tried.  The main problem is that in the Makefile there's no
>> USE_SYSTEM_{OBJCONV/LIBWHICH}, and I didn't want to mess with the it.
>> But yes, I think I could at least try again.
>
> Let me check how it will be crawled by SWH to be sure that Guix will
> be able to still build Julia 1.3 even if GitHub does not exist
> anymore.
> And I am in favour to let these 2 inputs.
>
>
>> > 3.
>> >> Otherwise, there is some issues about indentation -- I think you use
>> >> Emacs, so check that '.dir-locals.el' is correctly setup -- double
>> >> space for sentences in description,
>> >> 'patch/julia-SOURCE_DATE_EPOCH-mtime.patch' should be added with the
>> >> same commit that "Update", etc.
>> >> I have half-corrected so I can send you these 2 v2-patches if you want to. :-)
>> >
>> If you already fixed them, yes please send a v2 patch and I'll continue
>> working from there.
>
> Stay tuned. :-)
> I try to send it today...
>
Don't worry, I'm still working on my PhD thesis so my spare time until
the end of the week is low :D
>
>> Fine for me, do we need also to debug a broken test right?
>
> I do not know if the test are broken. I am not able to run them on my
> desktop machine. Not enough memory.
>
Ok, so I'll run tests a few times on my server then
>
> Cheers,
> simon

Thanks, Nicolò