Message ID | 87lfcfcmf3.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me |
---|---|
State | Accepted |
Headers | show |
Series | [bug#45972] Add julia-json with dependencies | expand |
Context | Check | Description |
---|---|---|
cbaines/applying patch | fail | View Laminar job |
cbaines/issue | success | View issue |
Hi, Nicolò Balzarotti <anothersms@gmail.com> skribis: > Almost all of Julia packages are under MIT (expat), I double checked and > they are fine. Also, 3 packages (out of 8) have been updated since my > submission, so I updated them now and guix lint does not complain anymore. Great. > This is the first batch, I finally decided to submit in small batches > the 100+ packages I have; you have been warned :D Oh, fun! :-) So, actually, we have a problem: > +(define-public julia-orderedcollections > + (package > + (name "julia-orderedcollections") > + (version "1.3.2") > + (source > + (origin > + (method git-fetch) > + (uri (git-reference > + (url "https://github.com/JuliaCollections/OrderedCollections.jl") > + (commit (string-append "v" version)))) > + (file-name "OrderedCollections") I was surprised that ‘guix lint’ doesn’t complain about this file name, I thought it required the file name to match the package name and at least that’s the spirit (I’ll take a look). Anyway, I went ahead and replaced all these by (git-file-name name version) as is done elsewhere. But now the tests would fail like so: --8<---------------cut here---------------start------------->8--- starting phase `check' ERROR: LoadError: ArgumentError: Package Adapt not found in current path: - Run `import Pkg; Pkg.add("Adapt")` to install the Adapt package. --8<---------------cut here---------------end--------------->8--- My understanding is that the first patch expects the source file name to match the Julia package name. IMO, that shouldn’t be the case. Can we either extract the Julia package name from metadata that’s in the package itself (?), or otherwise pass it to all the phases via ‘julia-build-system’? Sorry for not noticing earlier! Thanks, Ludo’.
> My understanding is that the first patch expects the source file name to > match the Julia package name. That's correct, we use it in the build system. > IMO, that shouldn’t be the case. At first I wasn't sure it was ok, but it went on through the revision process when I submitted the first package (Compat) so I tought it was fine. > Can we either extract the Julia package name from metadata that’s in > the package itself (?), or otherwise pass it to all the phases via > ‘julia-build-system’? Sure, I'd just read it from Package.toml (nowadays almost all the packages have this file, and for when it's missing we also have the julia-create-package-toml procedure that creates it). The file is toml, but I don't see a toml parser in guix. So, I'd use a function like the following: #+begin_src scheme (define (package.toml->name file) (call-with-input-file file (lambda (in) (let loop ((line (read-line in 'concat))) (if (eof-object? line) #f ;What to do? (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line))) (if m (match:substring m 1) (loop (read-line in 'concat))))))))) #+end_src As you can see this is very minimal/naive (takes the first name = "" occurrence, does not even consider comments, for which I'd add a negative lookahead "^(?!#)" which I can't get to work with ice-9 regex), but tested on a few packages it working. I don't know what to do when the match is not found (since it's something that might happen only during development, the #f fallback should not be too bad, btw, as the build will fail). The other way I think this is easily solvable is by asking julia directly, by reading the output of: (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println") doing something like cargo's manifest-target procedure does. But it'd go the other way if it's ok. Let me know! Once decided, I'll submit the updated patches > > Sorry for not noticing earlier! > np and thanks again!
Buon giorno! Nicolò Balzarotti <anothersms@gmail.com> skribis: >> My understanding is that the first patch expects the source file name to >> match the Julia package name. > > That's correct, we use it in the build system. > >> IMO, that shouldn’t be the case. > > At first I wasn't sure it was ok, but it went on through the revision > process when I submitted the first package (Compat) so I tought it was fine. If it was me it was probably an oversight, I’m sorry about that. >> Can we either extract the Julia package name from metadata that’s in >> the package itself (?), or otherwise pass it to all the phases via >> ‘julia-build-system’? > > Sure, I'd just read it from Package.toml (nowadays almost all the > packages have this file, and for when it's missing we also have the > julia-create-package-toml procedure that creates it). > > The file is toml, but I don't see a toml parser in guix. So, I'd use a > function like the following: > > #+begin_src scheme > (define (package.toml->name file) > (call-with-input-file file > (lambda (in) > (let loop ((line (read-line in 'concat))) > (if (eof-object? line) > #f ;What to do? > (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line))) > (if m (match:substring m 1) > (loop (read-line in 'concat))))))))) > #+end_src Sounds reasonable to me. If I understand the toml format correctly, “name=.*” is guaranteed to be on a line on its own, so that looks safe. > As you can see this is very minimal/naive (takes the first name = "" > occurrence, does not even consider comments, for which I'd add a > negative lookahead "^(?!#)" which I can't get to work with ice-9 regex), > but tested on a few packages it working. I don't know what to do when > the match is not found (since it's something that might happen only > during development, the #f fallback should not be too bad, btw, as the > build will fail). Yes, sounds good. To be on the safe side, we can add a #:julia-package-name parameter to the build system; it would default to #f, in which case the name is extracted from the toml file. > The other way I think this is easily solvable is by asking julia > directly, by reading the output of: > > (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println") > > doing something like cargo's manifest-target procedure does. But it'd > go the other way if it's ok. Yeah, the basic “parser” seems to be good enough. Thanks! Ludo’.
Hi, On Thu, 28 Jan 2021 at 01:30, Nicolò Balzarotti <anothersms@gmail.com> wrote: > Sure, I'd just read it from Package.toml (nowadays almost all the > packages have this file, and for when it's missing we also have the > julia-create-package-toml procedure that creates it). > > The file is toml, but I don't see a toml parser in guix. So, I'd use a > function like the following: > > #+begin_src scheme > (define (package.toml->name file) > (call-with-input-file file > (lambda (in) > (let loop ((line (read-line in 'concat))) > (if (eof-object? line) > #f ;What to do? > (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line))) > (if m (match:substring m 1) > (loop (read-line in 'concat))))))))) > #+end_src [...] > The other way I think this is easily solvable is by asking julia > directly, by reading the output of: > > (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println") With a bit more glue, could this be transformed into something like “julia->guix-package”? And so have a Julia package importer, even if it fails for some cases. WDYT? All the best, simon
zimoun <zimon.toutoune@gmail.com> writes: > Hi, Hi Simon! >> >> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println") > > With a bit more glue, could this be transformed into something like > “julia->guix-package”? And so have a Julia package importer, even if it > fails for some cases. Well, if you mean "Can we use Pkg.jl to generate package definitions for us?" the answer is "probably yes, but I never investigated this". That line uses just Julia Base TOML.jl, which for some reason is defined inside module Pkg.jl (and it's not top-level). If you instead meant "Can we have a Julia importer?" some time ago I wrote one in Julia, I've not used it in the last year, but it did work quite well back then. I attach it here for reference. But before digging into Pkg3, I need to push a new set of packages which contains julia-jllwrappers, which is needed for Julia packages which require binary distributions to work. After that, it will be possible to create an importer that work "ok" even with packages depending on .so libraries. using Pkg using Pkg.TOML using LibGit2 const base = ["Base64", "CRC32c", "Dates", "DelimitedFiles", "Distributed", "FileWatching", "Future", "InteractiveUtils", "Libdl", "LibGit2", "LinearAlgebra", "Logging", "Markdown", "Mmap", "Pkg", "Printf", "Profile", "Random", "REPL", "Serialization", "SHA", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "SuiteSparse", "Test", "Unicode", "UUIDs"] const disabled = ["WinRPM", "Homebrew", "CMake", "Docile", "Color", "HTTPClient", "ICU", "Calendar", "LegacyStrings", "Nulls"] registrypath = expanduser("~/.julia-old/registries/General/") registry = joinpath(registrypath, "Registry.toml") const register = TOML.parse(join(readlines(registry), "\n")) jlpkgname(info) = "julia-$(lowercase(info.name))" function getrev(path) versions = TOML.parse(join(readlines(path), "\n")) versionv = findmax(map(x -> VersionNumber(x), keys(versions) |> collect)) (rev = versions[string(versionv[1])]["git-tree-sha1"], ver = versionv[1]) end function getpackagebyuuid(uuid) uuid in keys(register["packages"]) || return nothing path = register["packages"][uuid]["path"] getpath(x) = joinpath(registrypath, joinpath(path, x)) package = TOML.parse(join(readlines(getpath("Package.toml")), "\n")) deppath = getpath("Deps.toml") isfile(deppath) || return nothing deps = TOML.parse(join(readlines(deppath), "\n")) (name = package["name"], uuid = package["uuid"], repo = package["repo"], deps = deps, vers = getrev(getpath("Versions.toml"))) end function getpackage(wanted) uuid = findfirst(p -> lowercase(p["name"]) == lowercase(wanted), register["packages"]) uuid == nothing && return nothing return getpackagebyuuid(uuid) end function getdeps!(deps, vers, recursive, out) flat(arr::Array) = mapreduce(x -> isa(x, Array) ? flat(x) : x, append!, arr, init=[]) v = map(p -> VersionNumber.(split(p, '-')), keys(deps) |> collect) valid = findall(x -> length(x) == 1 || (x[2] == v"0" && x[1] <= vers) || x[1] <= vers <= x[2], v) f = flat(map(x -> values(x), values(collect(values(deps))[valid]))) push!.(Ref(out), f) # if recursive # push! # end nothing end function have(info) file = "/home/nixo/git/guix/gnu/packages/julia-xyz.scm" return "(name \"" * jlpkgname(info) * "\")" in strip.(readlines(file)) end function gethash(info) wd = mktempdir() if info.name in base || info.name in disabled || have(info) return "" end println(stderr, "Cloning $(info.name) in $wd") repo = LibGit2.clone(info.repo, wd) hash = cd(wd) do out = Pipe() try LibGit2.checkout!(repo, string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit, LibGit2.GitTag(repo, "v" * string(info.vers.ver)))))) catch e try LibGit2.checkout!(repo, string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit, LibGit2.GitCommit(repo, "v" * string(info.vers.ver)))))) catch e # FIXME: if this happens, return the commit too and use it in the package println(stderr, "Failed to checkout $(e), continuing") end end run(pipeline(`guix hash -rx .`, stdout=out)) readline(out) end rm(wd, recursive = true) hash end function makepackage(info; done = []) if info === nothing @warn "Could not find package (have you cloned the registry?)" return elseif info in done return "" end push!(done, info) deps = String[] getdeps!(info.deps, info.vers.ver, true, deps) # TODO: remove deps that are in base deps = filter(x -> x !== nothing, getpackagebyuuid.(deps)) deplist = join(map(name -> "(\"$name\" ,$name)", jlpkgname.(deps)), '\n') packagedeps = join(makepackage.(deps, done = done), "") hash = gethash(info) hash == "" && return "" """ $packagedeps (define-public $(jlpkgname(info)) (package (name "$(jlpkgname(info))") (version "$(info.vers.ver)") (source (origin (method git-fetch) (uri (git-reference (url "$(info.repo)") (commit (string-append "v" version)))) (file-name "$(info.name)") (sha256 (base32 "$hash")))) (propagated-inputs `($(deplist))) (build-system julia-build-system) (home-page "$(info.repo)") (synopsis "") (description "") (license license:expat))) """ end println.(makepackage.(getpackage.(ARGS)))
As a self reminder, I found this repo https://github.com/cdluminate/DistroHelper.jl which might contain something useful for our purposes
Hi, On Sat, 30 Jan 2021 at 21:13, Nicolò Balzarotti <anothersms@gmail.com> wrote: > zimoun <zimon.toutoune@gmail.com> writes: >>> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println") >> >> With a bit more glue, could this be transformed into something like >> “julia->guix-package”? And so have a Julia package importer, even if it >> fails for some cases. > > Well, if you mean "Can we use Pkg.jl to generate package definitions for > us?" the answer is "probably yes, but I never investigated this". That > line uses just Julia Base TOML.jl, which for some reason is defined > inside module Pkg.jl (and it's not top-level). I mean the correct way is to write a TOML parser, probably using PEG. As it is done with the other importers. It is a piece of work. Not especially hard but it could take some time. So instead, the question is, using a bit of Julia glue and Guile glue, is it possible to extract the necessary information to have a Guix package? Well, maybe these glue code is the same as writing a TOML parser. :-) > If you instead meant "Can we have a Julia importer?" some time ago I > wrote one in Julia, I've not used it in the last year, but it did work > quite well back then. I attach it here for reference. Fun! One thing is how to deal with ’Manifest.toml’. And pre-compiled substitutes do not make too much sense in the Julia world. Well, I do not know how these Manifest.toml overlap with how Guix works. Cheers, simon
> I mean the correct way is to write a TOML parser, probably using PEG. > As it is done with the other importers. > It is a piece of work. Not especially hard but it could take some time. That wouldn't be too hard, but also it would not be enough. As you can see in [fn:1], there's just the list of dependencies, but no info on how to get them, so parsing the Julia General Registry [fn:2] is still required. > So instead, the question is, using a bit of Julia glue and Guile glue, > is it possible to extract the necessary information to have a Guix > package? > > Well, maybe these glue code is the same as writing a TOML parser. :-) This might be feasible, but as I said above, unfortunately I don't think that a TOML parser would be enough. But asking "Pkg3" to resolve dependencies for us, yes, this makes sense. > One thing is how to deal with ’Manifest.toml’. And pre-compiled > substitutes do not make too much sense in the Julia world. Well, I do > not know how these Manifest.toml overlap with how Guix works. WDYM? Julia uses .ji files (which are the analogoues of .go for guile and .pyc for python), if this is what you are referring to. I'm just ignoring Manifest.toml, which should be the same as venv in the python world. [fn:1] https://github.com/JuliaLang/TOML.jl/blob/master/Project.toml [fn:2] https://github.com/JuliaRegistries/General
Hi, On Sun, 31 Jan 2021 at 21:00, Nicolò Balzarotti <anothersms@gmail.com> wrote: > That wouldn't be too hard, but also it would not be enough. As you can > see in [fn:1], there's just the list of dependencies, but no info on how > to get them, so parsing the Julia General Registry [fn:2] is still required. [...] > I'm just ignoring Manifest.toml, which should be the same as venv in the > python world. Sorry, you have right. I have misremembered an old presentation about Pkg3 by Stefan Karpinski. Well, thanks for the pointers. And I gave a look at the Pkg doc. ;-) Once a PEG parser for TOML files is written, all the information is in the Registry (by default this General repository), recursively. However, the “resolve” part is not straightforward, IMHO. It is what confused me in my previous email and I thought was “Manifest.toml”. Anyway. Instead of reinventing the wheel and reimplement Pkg.jl in Guix, maybe it is worth to have Julia code as you did that extracts the relevant information and then generates the relevant Guix packages. However, it should mean that “guix import julia” requires to have the package julia implicitly used. Why not. :-) Cheers, simon
From aa1231bb89d1a7271181d437ebe36952ec8ccfc4 Mon Sep 17 00:00:00 2001 From: nixo <nicolo@nixo.xyz> Date: Tue, 19 Jan 2021 00:06:34 +0100 Subject: [PATCH v2 9/9] gnu: Add julia-json. * gnu/packages/julia-xyz.scm (julia-json): New variable. --- gnu/packages/julia-xyz.scm | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/gnu/packages/julia-xyz.scm b/gnu/packages/julia-xyz.scm index 70d7f1d718..a79b8d07b6 100644 --- a/gnu/packages/julia-xyz.scm +++ b/gnu/packages/julia-xyz.scm @@ -125,6 +125,31 @@ scaled by a constant factor. Consequently, they have a fixed number of digits (bits) after the decimal (radix) point.") (license license:expat))) +(define-public julia-json + (package + (name "julia-json") + (version "0.21.1") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/JuliaIO/JSON.jl") + (commit (string-append "v" version)))) + (file-name "JSON") + (sha256 + (base32 "1f9k613kbknmp4fgjxvjaw4d5sfbx8a5hmcszmp1w9rqfqngjx9m")))) + (build-system julia-build-system) + (propagated-inputs + `(("julia-datastructures" ,julia-datastructures) + ("julia-fixedpointnumbers" ,julia-fixedpointnumbers) + ("julia-parsers" ,julia-parsers) + ("julia-offsetarrays" ,julia-offsetarrays))) + (home-page "https://github.com/JuliaIO/JSON.jl") + (synopsis "JSON parsing and printing library for Julia") + (description "@code{JSON.jl} is a pure Julia module which supports parsing +and printing JSON documents.") + (license license:expat))) + (define-public julia-orderedcollections (package (name "julia-orderedcollections") -- 2.30.0