| Message ID | 20210323194211.16398-1-zimon.toutoune@gmail.com |
|---|---|
| Headers | show |
| Series | Add Zygote (piece for Machine Learning with Julia) | expand |
zimoun <zimon.toutoune@gmail.com> writes: > Hi, Hi! > > This adds all the packages required to build Zygote, the source-to-source > automatic differentiation in Julia. Wonderful, nice to see the first contribution to julia packages :) > > Last, well, I feel motivated to implement an (recursive) importer, even > inaccurate, it will greatly help. > I found yesterday julihub, which might help in that regard a lot. I'll share here my notes (still no implementation), if I'm not missing something obvious, this importer should be super easy. Let me know what you think about those: 1. refresh the package list (json) curl https://juliahub.com/app/packages/info 2. search the required package (.name field) 3. get info on the package (.metadata.docslink) / pkg.json eg. curl 'https://juliahub.com/docs/Convex/Sr0e5/0.14.5/pkg.json' 4. extract: - version - homepage - license - repository - description - direct dependencies (used in propagated-inputs field) store them into a list - indirect dependencies add them to the dependency list - binary (jll) dependencies tell the user those must be fixed - remove duplicates/packages already in guix - repeat 4 for each dependency Thanks, Nicolò
Hi, On Tue, 23 Mar 2021 at 23:48, Nicolò Balzarotti <anothersms@gmail.com> wrote: > zimoun <zimon.toutoune@gmail.com> writes: >> This adds all the packages required to build Zygote, the source-to-source >> automatic differentiation in Julia. > Wonderful, nice to see the first contribution to julia packages :) If you want to review, then it would help. :-) You do not need to be committer because it is checking and commenting, or by simply saying LGTM. For example, I am not committer and I reviewed ocaml packages or camlboot. It would help to reduce the load on the committer’s shoulders. ;-) Well, I would like to have these stuffs in the next release 1.2.1. >> Last, well, I feel motivated to implement an (recursive) importer, even >> inaccurate, it will greatly help. >> > I found yesterday julihub, which might help in that regard a lot. > I'll Cool! I am moving the discussion about that on guix-devel and I will comment once I have read a bit this juliahub thing. :-) Thanks, simon
zimoun <zimon.toutoune@gmail.com> writes: > If you want to review, then it would help. :-) Here you go: Patch applies and everything builds fine. I also checked the licenses and they seems fine. I run some example on the Zygote tutorial and it worked Some notes: julia-macrotools: > +that let you do deep transformations of code..") two dots julia-colors: Instead of removing the tests, calling julia with --depwarn=yes should do the trick. I changed the (check) function in julia-build-system adding replacing invoke "julia" with invoke "julia" "--depwarn=yes", removed your phase from julia-colors and everything went well (built zygote and HTTP, which should cover all packages we have). Would you mind fixing the build system directly? As I guess other packages will fail the same way julia-compilersupportlibraries-jll: julia-openspecfun-jll: I'd change package description, as it's the default and shared between all jll packages julia-constructionbase: typo: lightwight -> lightweight julia-irtool: typo: simpe -> simple julia-macrotools: typo: Jlia -> Julia julia-specialfuntions: > Most of these functions were formerly part of Base in early versions of Julia. I'd remove this from the description julia-forwarddiff: synopsis seems wrong julia-zygoterules: extra space at the end of synopsis Thanks, Nicolò
Hi Nicoló, On Wed, 24 Mar 2021 at 09:15, Nicolò Balzarotti <anothersms@gmail.com> wrote: > zimoun <zimon.toutoune@gmail.com> writes: > >> If you want to review, then it would help. :-) > > Here you go: Thanks for the prompt review! > julia-colors: > Instead of removing the tests, calling julia with --depwarn=yes > should do the trick. > I changed the (check) function in julia-build-system adding > replacing invoke "julia" with invoke "julia" "--depwarn=yes", removed > your phase from julia-colors and everything went well (built zygote and > HTTP, which should cover all packages we have). Would you mind > fixing the build system directly? As I guess other packages will fail > the same way Ok, I will double-check if we change the option of the build system or simply locally tweak the arguments case per case. Thanks again! Cheers, simon
Hi Nicoló, On Wed, 24 Mar 2021 at 10:35, zimoun <zimon.toutoune@gmail.com> wrote: > On Wed, 24 Mar 2021 at 09:15, Nicolò Balzarotti <anothersms@gmail.com> wrote: >> I changed the (check) function in julia-build-system adding >> replacing invoke "julia" with invoke "julia" "--depwarn=yes", removed >> your phase from julia-colors and everything went well (built zygote and >> HTTP, which should cover all packages we have). Would you mind >> fixing the build system directly? As I guess other packages will fail >> the same way > > Ok, I will double-check if we change the option of the build system or > simply locally tweak the arguments case per case. Hum, I get these kind of messages: --8<---------------cut here---------------start------------->8--- Internal error: encountered unexpected error in runtime: ErrorException("type UnionAll has no field size") jl_errorf at /tmp/guix-build-julia-1.5.3.drv-0/julia-1.5.3/src/rtutils.c:77 --8<---------------cut here---------------end--------------->8--- for julia-datastructures for instance. Even if the build is fine and my small experiments seems correct. Well, I have set this deprecation warning as a first commit in the v2. Thanks, simon
zimoun <zimon.toutoune@gmail.com> writes: > Hi Nicoló, > Hum, I get these kind of messages: > > --8<---------------cut here---------------start------------->8--- > Internal error: encountered unexpected error in runtime: > ErrorException("type UnionAll has no field size") > jl_errorf at /tmp/guix-build-julia-1.5.3.drv-0/julia-1.5.3/src/rtutils.c:77 > --8<---------------cut here---------------end--------------->8--- > > for julia-datastructures for instance. Even if the build is fine and > my small experiments seems correct. > I'd say it's a julia bug [fn:1]. can be reproduced with: ```julia primitive type Foo{T} 32 end foo(T, x) = (Core.bitcast(Foo{T}, x); x) code_typed(foo, Tuple{Any, Int32}) ``` We need to test this with julia 1.5.4 or 1.6.0 (the latter has been released yesterday :D. I had a build recipe for some version in between 1.5.3 and 1.6.0, I'll check if it still works). Before the guix 1.2.1 release, we might want to upgrade to 1.5.4 at least. > Well, I have set this deprecation warning as a first commit in the v2. Thanks! I should have some time to re-check those later today Nicolò [fn:1] https://github.com/Juliaang/julia/issues/37532
Hi, On Thu, 25 Mar 2021 at 10:47, Nicolò Balzarotti <anothersms@gmail.com> wrote: > zimoun <zimon.toutoune@gmail.com> writes: > I'd say it's a julia bug [fn:1]. ok > We need to test this with julia 1.5.4 or 1.6.0 (the latter has been > released yesterday :D. I had a build recipe for some version in > between 1.5.3 and 1.6.0, I'll check if it still works). Before the guix > 1.2.1 release, we might want to upgrade to 1.5.4 at least. Please go ahead. If 1.6.0 is not included for 1.2.1, it will for the next 1.3. ;-) IMHO, we should try to address: the reproducibility issue and the precompiled twice, <http://issues.guix.gnu.org/issue/22304 <http://issues.guix.gnu.org/issue/47354> <https://yhetil.org/guix/86h7l1ic7e.fsf@gmail.com> Thanks, simon
Nicolò Balzarotti <anothersms@gmail.com> writes: > zimoun <zimon.toutoune@gmail.com> writes: > I should have some time to re-check those later today > Ok the everything still builds (I had a few problems applying the patches, but it might depend on my setup). Lint gives warnings: julia-zygote@0.6.4: permanent redirect from https://fluxml.ai/Zygote.jl to https://fluxml.ai/Zygote.jl/ julia-zygote@0.6.4: can be upgraded to 0.6.6 julia-chainrules@0.7.54: can be upgraded to 0.7.55 julia-openspecfun-jll@0.5.3+4: sentences in description should be followed by two spaces; possible infraction at 129 julia-compilersupportlibraries-jll@0.4.0+1: sentences in description should be followed by two spaces; possible infraction at 134 julia-compilersupportlibraries-jll@0.4.0+1: line 344 is way too long (91 characters) julia-chainrulescore@0.9.29: can be upgraded to 0.9.33 julia-macrotools@0.5.6: permanent redirect from https://fluxml.ai/MacroTools.jl to https://fluxml.ai/MacroTools.jl/ Other than that, LGTM (I opened an issue [fn:1] to constructionbase.jl to have latest version tagged, if they respond promptly we might use the tagged version) [fn:1] https://github.com/JuliaObjects/ConstructionBase.jl/issues/42 Thanks, Nicolò
Hi Nicoló, On Sat, 27 Mar 2021 at 22:01, Nicolò Balzarotti <anothersms@gmail.com> wrote: > Nicolò Balzarotti <anothersms@gmail.com> writes: > >> zimoun <zimon.toutoune@gmail.com> writes: > >> I should have some time to re-check those later today >> > Ok the everything still builds (I had a few problems applying the > patches, but it might depend on my setup). Ah. > Lint gives warnings: > julia-zygote@0.6.4: permanent redirect from https://fluxml.ai/Zygote.jl to > https://fluxml.ai/Zygote.jl/ > julia-macrotools@0.5.6: permanent redirect from > https://fluxml.ai/MacroTools.jl to https://fluxml.ai/MacroTools.jl/ I find weird to have to have trailing ’/’ for these when all the others does not have. > julia-zygote@0.6.4: can be upgraded to 0.6.6 > julia-chainrules@0.7.54: can be upgraded to 0.7.55 > julia-chainrulescore@0.9.29: can be upgraded to 0.9.33 Well, let apply this series and update them later. :-) > julia-openspecfun-jll@0.5.3+4: sentences in description should be followed by > two spaces; possible infraction at 129 > julia-compilersupportlibraries-jll@0.4.0+1: sentences in description should be > followed by two spaces; possible infraction at 134 Indeed. Ludo, could you fix it before pushing? > julia-compilersupportlibraries-jll@0.4.0+1: line 344 is way too long (91 > characters) Hum, but life is too short. ;-) > Other than that, LGTM Thanks! > (I opened an issue [fn:1] to constructionbase.jl to have latest > version tagged, Cool! > if they respond promptly we might use the tagged version) Well, let merge this series. Once in, it becomes much easier to update them. And less scary for committer. :-) Pushing ~30 new Julia packages is not the same than tweaking a version+hash or a description. ;-) Thanks again! Cheers, simon PS: Teaser: I have started something about an importer. :-) I have checked JuliaHub but I am not able to find the dependencies list, so I am going to use General.
zimoun <zimon.toutoune@gmail.com> writes: > Well, let apply this series and update them later. :-) > Well, let merge this series. Once in, it becomes much easier to update > them. And less scary for committer. :-) Pushing ~30 new Julia > packages is not the same than tweaking a version+hash or a > description. ;-) Ok > PS: > Teaser: I have started something about an importer. :-) > I have checked JuliaHub but I am not able to find the dependencies list, > so I am going to use General. Great! If you can use the general registry directly is fine. Dependency list is under .deps Example: curl 'https://juliahub.com/docs/Flux/QdkVy/0.11.6/pkg.json' | jq .deps those with direct: false are depdencencies of dependencies
Hi, On Sat, 27 Mar 2021 at 22:41, Nicolò Balzarotti <anothersms@gmail.com> wrote: > Great! If you can use the general registry directly is fine. The plan is to start with Pkg.TOML has you did. And see after if a parser could be implemented with PEG or other. I am not enough comfortable with PEG to go with it (even if I run the tutorial etc.). For instance, the 'cpan' importer checks if some Perl packages are there. Well, let start by not reinventing the wheel and see later. > Dependency list is under .deps > > Example: > curl 'https://juliahub.com/docs/Flux/QdkVy/0.11.6/pkg.json' | jq .deps Yeah, but Project.toml distinguishes between "deps" and "extra", which gives 'inputs' and 'propgated-inputs', roughly. But, yeah JuliaHub is nicer because it provides partial synopses and licenses. Maybe I will change my mind. :-) Cheers, simon
Hi there! I’ve pushed v2, comrades! Thanks for the great work! Simon, I noticed the trap you’d set up: patches 10 and 20 were missing from v2. :-) I took patches 9 and 19 from v1 instead, and everything builds fine, but lemme know if anything’s amiss! Ludo’.
Hi, On Mon, 29 Mar 2021 at 16:50, Ludovic Courtès <ludo@gnu.org> wrote: > I’ve pushed v2, comrades! Thanks for the great work! \o/ Thanks Nicoló for the review. > Simon, I noticed the trap you’d set up: patches 10 and 20 were missing > from v2. :-) I took patches 9 and 19 from v1 instead, and everything > builds fine, but lemme know if anything’s amiss! Oh, sorry! Indeed, I have missed the shell rule and forgot the '0'. For the record, these 2 patches are the same for v1 and v2. Thanks, simon --8<---------------cut here---------------start------------->8--- $ diff v2-0010-gnu-Add-julia-colortypes.patch 0009-gnu-Add-julia-colortypes.patch 1c1 < From 6597c28170e52c435dea0e29f7783260781574b0 Mon Sep 17 00:00:00 2001 --- > From 2ef94ee9e0f516d77449e8f179b66dc3135bff24 Mon Sep 17 00:00:00 2001 4c4 < Subject: [PATCH v2 10/30] gnu: Add julia-colortypes. --- > Subject: [PATCH 09/29] gnu: Add julia-colortypes. 12c12 < index 4abeaa765f..8d28aab7f1 100644 --- > index d417a996f8..3213efd715 100644 --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ diff v2-0020-gnu-Add-julia-calculus.patch 0019-gnu-Add-julia-calculus.patch 1c1 < From 3d96c37625d6841a6b378ea0d57b53e479207b78 Mon Sep 17 00:00:00 2001 --- > From 5cf3bb5ab498d53bf419e18d1391afb5473b54e8 Mon Sep 17 00:00:00 2001 4c4 < Subject: [PATCH v2 20/30] gnu: Add julia-calculus. --- > Subject: [PATCH 19/29] gnu: Add julia-calculus. 12c12 < index c5c04d6579..75b46a1c7c 100644 --- > index d14fa66fe5..dfab3c20af 100644 --8<---------------cut here---------------end--------------->8---
Hi, This adds all the packages required to build Zygote, the source-to-source automatic differentiation in Julia. <https://github.com/FluxML/Zygote.jl> It is the first brick to have Flux, the Julia machine learning framework. <https://github.com/FluxML/Flux.jl> Noting that (at least 3) large pieces are still missing: Juno, Documenter and CUDA. Speaking about the Julia package CUDA, it is required by the test suite of Zygote, that's why the #:tests? is #f. However, we manually check the package by running the small examples, e.g., --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix environment -C --ad-hoc julia julia-zygote -- julia julia> using Zygote [ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] julia> f(x) = 5x + 3 f (generic function with 1 method) julia> f(10), f'(10) (53, 5) julia> @code_llvm f'(10) ; @ /gnu/store/5m6676w77bbz44prbfxhxmk88srgxy81-profile/share/julia/packages/Zygote/src/compiler/interface.jl:62 within `#43' define i64 @"julia_#43_2036"(i64) { top: ret i64 5 } julia> W, b = rand(2, 3), rand(2); julia> predict(x) = W*x .+ b; julia> g = gradient(Params([W, b])) do sum(predict([1,2,3])) end Grads(...) julia> g[W], g[b] ([1.0 2.0 3.0; 1.0 2.0 3.0], 2-element Fill{Float64}: entries equal to 1.0) --8<---------------cut here---------------end--------------->8--- All the packages are not archived because of the rate limit. They should be soon. --8<---------------cut here---------------start------------->8--- for n in $(git log -30 --format="%s" --reverse | cut -d'-' -f2 | sed 's/\.//'); do ./pre-inst-env guix lint -c archival julia-${n} ; done gnu/packages/julia-xyz.scm:397:7: julia-constructionbase@1.1.0: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:817:5: julia-macrotools@0.5.6: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:1040:5: julia-reexport@1.0.0: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:723:5: julia-irtools@0.4.2: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:283:5: julia-colortypes@0.10.12: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:1059:5: julia-requires@1.1.3: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:1085:5: julia-richardson@1.4.0: scheduled Software Heritage archival gnu/packages/julia-xyz.scm:552:5: julia-finitedifferences@0.12.2: Software Heritage rate limit reached; try again later gnu/packages/julia-xyz.scm:211:5: julia-chainrulestestutils@0.6.4: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:528:5: julia-fillarrays@0.11.7: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:309:5: julia-commonsubexpressions@0.3.0: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:131:5: julia-calculus@0.5.1: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:441:5: julia-diffresults@1.0.3: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:488:5: julia-difftests@0.1.1: archival rate limit exceeded; try again later guix lint: error: julia-compilersupportlibraries: unknown package guix lint: error: julia-openspecfun: unknown package gnu/packages/julia-xyz.scm:1112:5: julia-specialfunctions@1.3.0: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:155:5: julia-chainrules@0.7.54: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:613:5: julia-forwarddiff@0.10.17: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:1221:5: julia-zygoterules@0.2.1: archival rate limit exceeded; try again later gnu/packages/julia-xyz.scm:1243:5: julia-zygote@0.6.4: archival rate limit exceeded; try again later --8<---------------cut here---------------end--------------->8--- Last, well, I feel motivated to implement an (recursive) importer, even inaccurate, it will greatly help. All the best, simon zimoun (29): gnu: Add julia-constructionbase. gnu: Add julia-macrotools. gnu: Add julia-nanmath. gnu: Add julia-reexport. gnu: Add julia-irtools. gnu: Add julia-staticarrays. gnu: Add julia-unitful. gnu: Add julia-abstractffts. gnu: Add julia-colortypes. gnu: Add julia-colors. gnu: Add julia-example. gnu: Add julia-requires. gnu: Add julia-chainrulescore. gnu: Add julia-richardson. gnu: Add julia-finitedifferences. gnu: Add julia-chainrulestestutils. gnu: Add julia-fillarrays. gnu: Add julia-commonsubexpressions. gnu: Add julia-calculus. gnu: Add julia-diffresults. gnu: Add julia-difftests. gnu: Add julia-compilersupportlibraries-jll. gnu: Add julia-openspecfun-jll. gnu: Add julia-specialfunctions. gnu: Add julia-diffrules. gnu: Add julia-chainrules. gnu: Add julia-forwarddiff. gnu: Add julia-zygoterules. gnu: Add julia-zygote. gnu/packages/julia-xyz.scm | 777 +++++++++++++++++++++++++++++++++++++ 1 file changed, 777 insertions(+) base-commit: a2f3a456bec357394fe550d927c26bbbdaa7ba6e