[bug#45972] Add julia-json with dependencies

I applied all your suggestions :)

> If you’re confident with the licenses and ‘guix lint’ is happy,
> please push!

Almost all of Julia packages are under MIT (expat), I double checked and
they are fine.  Also, 3 packages (out of 8) have been updated since my
submission, so I updated them now and guix lint does not complain anymore.

> Nice to see more Julia packages.

This is the first batch, I finally decided to submit in small batches
the 100+ packages I have; you have been warned :D

Here's the updated patchset (if somebody can push).

Thanks Ludo for the review

Hi,

Nicolò Balzarotti <anothersms@gmail.com> skribis:

> Almost all of Julia packages are under MIT (expat), I double checked and
> they are fine.  Also, 3 packages (out of 8) have been updated since my
> submission, so I updated them now and guix lint does not complain anymore.

Great.

> This is the first batch, I finally decided to submit in small batches
> the 100+ packages I have; you have been warned :D

Oh, fun! :-)

So, actually, we have a problem:

> +(define-public julia-orderedcollections
> +  (package
> +    (name "julia-orderedcollections")
> +    (version "1.3.2")
> +    (source
> +     (origin
> +       (method git-fetch)
> +       (uri (git-reference
> +             (url "https://github.com/JuliaCollections/OrderedCollections.jl")
> +             (commit (string-append "v" version))))
> +       (file-name "OrderedCollections")

I was surprised that ‘guix lint’ doesn’t complain about this file name,
I thought it required the file name to match the package name and at
least that’s the spirit (I’ll take a look).

Anyway, I went ahead and replaced all these by (git-file-name name
version) as is done elsewhere.  But now the tests would fail like so:

--8<---------------cut here---------------start------------->8---
starting phase `check'
ERROR: LoadError: ArgumentError: Package Adapt not found in current path:
- Run `import Pkg; Pkg.add("Adapt")` to install the Adapt package.
--8<---------------cut here---------------end--------------->8---

My understanding is that the first patch expects the source file name to
match the Julia package name.  IMO, that shouldn’t be the case.  Can we
either extract the Julia package name from metadata that’s in the
package itself (?), or otherwise pass it to all the phases via
‘julia-build-system’?

Sorry for not noticing earlier!

Thanks,
Ludo’.

> My understanding is that the first patch expects the source file name to
> match the Julia package name.

That's correct, we use it in the build system.

> IMO, that shouldn’t be the case.

At first I wasn't sure it was ok, but it went on through the revision
process when I submitted the first package (Compat) so I tought it was fine.

> Can we either extract the Julia package name from metadata that’s in
> the package itself (?), or otherwise pass it to all the phases via
> ‘julia-build-system’?

Sure, I'd just read it from Package.toml (nowadays almost all the
packages have this file, and for when it's missing we also have the
julia-create-package-toml procedure that creates it).

The file is toml, but I don't see a toml parser in guix.  So, I'd use a
function like the following:

#+begin_src scheme
(define (package.toml->name file)
  (call-with-input-file file
    (lambda (in)
      (let loop ((line (read-line in 'concat)))
        (if (eof-object? line)
            #f                          ;What to do?
            (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line)))
              (if m (match:substring m 1)
                  (loop (read-line in 'concat)))))))))
#+end_src

As you can see this is very minimal/naive (takes the first name = ""
occurrence, does not even consider comments, for which I'd add a
negative lookahead "^(?!#)" which I can't get to work with ice-9 regex),
but tested on a few packages it working.  I don't know what to do when
the match is not found (since it's something that might happen only
during development, the #f fallback should not be too bad, btw, as the
build will fail).

The other way I think this is easily solvable is by asking julia
directly, by reading the output of:

(invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")

doing something like cargo's manifest-target procedure does.  But it'd
go the other way if it's ok.

Let me know!  Once decided, I'll submit the updated patches

>
> Sorry for not noticing earlier!
>
np and thanks again!

Buon giorno!

Nicolò Balzarotti <anothersms@gmail.com> skribis:

>> My understanding is that the first patch expects the source file name to
>> match the Julia package name.
>
> That's correct, we use it in the build system.
>
>> IMO, that shouldn’t be the case.
>
> At first I wasn't sure it was ok, but it went on through the revision
> process when I submitted the first package (Compat) so I tought it was fine.

If it was me it was probably an oversight, I’m sorry about that.

>> Can we either extract the Julia package name from metadata that’s in
>> the package itself (?), or otherwise pass it to all the phases via
>> ‘julia-build-system’?
>
> Sure, I'd just read it from Package.toml (nowadays almost all the
> packages have this file, and for when it's missing we also have the
> julia-create-package-toml procedure that creates it).
>
> The file is toml, but I don't see a toml parser in guix.  So, I'd use a
> function like the following:
>
> #+begin_src scheme
> (define (package.toml->name file)
>   (call-with-input-file file
>     (lambda (in)
>       (let loop ((line (read-line in 'concat)))
>         (if (eof-object? line)
>             #f                          ;What to do?
>             (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line)))
>               (if m (match:substring m 1)
>                   (loop (read-line in 'concat)))))))))
> #+end_src

Sounds reasonable to me.  If I understand the toml format correctly,
“name=.*” is guaranteed to be on a line on its own, so that looks safe.

> As you can see this is very minimal/naive (takes the first name = ""
> occurrence, does not even consider comments, for which I'd add a
> negative lookahead "^(?!#)" which I can't get to work with ice-9 regex),
> but tested on a few packages it working.  I don't know what to do when
> the match is not found (since it's something that might happen only
> during development, the #f fallback should not be too bad, btw, as the
> build will fail).

Yes, sounds good.

To be on the safe side, we can add a #:julia-package-name parameter to
the build system; it would default to #f, in which case the name is
extracted from the toml file.

> The other way I think this is easily solvable is by asking julia
> directly, by reading the output of:
>
> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")
>
> doing something like cargo's manifest-target procedure does.  But it'd
> go the other way if it's ok.

Yeah, the basic “parser” seems to be good enough.

Thanks!

Ludo’.

Hi,

On Thu, 28 Jan 2021 at 01:30, Nicolò Balzarotti <anothersms@gmail.com> wrote:

> Sure, I'd just read it from Package.toml (nowadays almost all the
> packages have this file, and for when it's missing we also have the
> julia-create-package-toml procedure that creates it).
>
> The file is toml, but I don't see a toml parser in guix.  So, I'd use a
> function like the following:
>
> #+begin_src scheme
> (define (package.toml->name file)
>   (call-with-input-file file
>     (lambda (in)
>       (let loop ((line (read-line in 'concat)))
>         (if (eof-object? line)
>             #f                          ;What to do?
>             (let ((m (string-match "name\\s*=\\s*\"(.*)\"" line)))
>               (if m (match:substring m 1)
>                   (loop (read-line in 'concat)))))))))
> #+end_src

[...]

> The other way I think this is easily solvable is by asking julia
> directly, by reading the output of:
>
> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")

With a bit more glue, could this be transformed into something like
“julia->guix-package”?  And so have a Julia package importer, even if it
fails for some cases.

WDYT?


All the best,
simon

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,

Hi Simon!

>>
>> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")
>
> With a bit more glue, could this be transformed into something like
> “julia->guix-package”?  And so have a Julia package importer, even if it
> fails for some cases.

Well, if you mean "Can we use Pkg.jl to generate package definitions for
us?" the answer is "probably yes, but I never investigated this".  That
line uses just Julia Base TOML.jl, which for some reason is defined
inside module Pkg.jl (and it's not top-level).

If you instead meant "Can we have a Julia importer?" some time ago I
wrote one in Julia, I've not used it in the last year, but it did work
quite well back then.  I attach it here for reference.

But before digging into Pkg3, I need to push a new set of packages which
contains julia-jllwrappers, which is needed for Julia packages which
require binary distributions to work.  After that, it will be possible
to create an importer that work "ok" even with packages depending on
.so libraries.
using Pkg
using Pkg.TOML
using LibGit2

const base = ["Base64",
              "CRC32c",
              "Dates",
              "DelimitedFiles",
              "Distributed",
              "FileWatching",
              "Future",
              "InteractiveUtils",
              "Libdl",
              "LibGit2",
              "LinearAlgebra",
              "Logging",
              "Markdown",
              "Mmap",
              "Pkg",
              "Printf",
              "Profile",
              "Random",
              "REPL",
              "Serialization",
              "SHA",
              "SharedArrays",
              "Sockets",
              "SparseArrays",
              "Statistics",
              "SuiteSparse",
              "Test",
              "Unicode",
              "UUIDs"]
const disabled = ["WinRPM", "Homebrew", "CMake",
                  "Docile", "Color", 
                  "HTTPClient", "ICU", "Calendar", "LegacyStrings",
                  "Nulls"]

registrypath = expanduser("~/.julia-old/registries/General/")
registry = joinpath(registrypath, "Registry.toml")
const register = TOML.parse(join(readlines(registry), "\n"))

jlpkgname(info) = "julia-$(lowercase(info.name))"

function getrev(path)
    versions = TOML.parse(join(readlines(path), "\n"))
    versionv = findmax(map(x -> VersionNumber(x), keys(versions) |> collect))
    (rev = versions[string(versionv[1])]["git-tree-sha1"],
     ver = versionv[1])
end

function getpackagebyuuid(uuid)
    uuid in keys(register["packages"]) || return nothing
    path = register["packages"][uuid]["path"]
    getpath(x) =  joinpath(registrypath, joinpath(path, x))
    package = TOML.parse(join(readlines(getpath("Package.toml")), "\n"))
    deppath = getpath("Deps.toml")
    isfile(deppath) || return nothing
    deps = TOML.parse(join(readlines(deppath), "\n"))
    (name = package["name"],
     uuid = package["uuid"],
     repo = package["repo"],
     deps = deps,
     vers = getrev(getpath("Versions.toml")))
end

function getpackage(wanted)
    uuid = findfirst(p -> lowercase(p["name"]) == lowercase(wanted),
                     register["packages"])
    uuid == nothing && return nothing
    return getpackagebyuuid(uuid)
end

function getdeps!(deps, vers, recursive, out)
    flat(arr::Array) = mapreduce(x -> isa(x, Array) ? flat(x) : x, append!,
                                 arr, init=[])
    v = map(p -> VersionNumber.(split(p, '-')), keys(deps) |> collect)
    valid = findall(x -> length(x) == 1 ||
                    (x[2] == v"0" && x[1] <= vers) ||
                    x[1] <= vers <= x[2], v)
    f = flat(map(x -> values(x), values(collect(values(deps))[valid])))
    push!.(Ref(out), f)
    # if recursive
    #     push!
    # end
    nothing
end

function have(info)
    file = "/home/nixo/git/guix/gnu/packages/julia-xyz.scm"
    return "(name \"" * jlpkgname(info) * "\")" in strip.(readlines(file))
end

function gethash(info)
    wd = mktempdir()
    if info.name in base ||
        info.name in disabled ||
        have(info)
        return ""
    end
    println(stderr, "Cloning $(info.name) in $wd")
    repo = LibGit2.clone(info.repo, wd)
    hash = cd(wd) do
        out = Pipe()
        try
            LibGit2.checkout!(repo,
                              string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit,
                                                                  LibGit2.GitTag(repo, "v" * string(info.vers.ver))))))
        catch e
            try
                LibGit2.checkout!(repo,
                                  string(LibGit2.GitHash(LibGit2.peel(LibGit2.GitCommit,
                                                                      LibGit2.GitCommit(repo, "v" * string(info.vers.ver))))))
            catch e
                # FIXME: if this happens, return the commit too and use it in the package
                println(stderr, "Failed to checkout $(e), continuing")
            end
        end
        run(pipeline(`guix hash -rx .`, stdout=out))
        readline(out)
    end
    rm(wd, recursive = true)
    hash
end

function makepackage(info; done = [])
    if info === nothing
        @warn "Could not find package (have you cloned the registry?)"
        return
    elseif info in done
        return ""
    end
    push!(done, info)
    deps = String[]
    getdeps!(info.deps, info.vers.ver, true, deps)
    # TODO: remove deps that are in base
    deps = filter(x -> x !== nothing, getpackagebyuuid.(deps))
    deplist = join(map(name -> "(\"$name\" ,$name)",
                       jlpkgname.(deps)), '\n')
    packagedeps = join(makepackage.(deps, done = done), "")
    hash = gethash(info)
    hash == "" && return ""
    """
    $packagedeps
    (define-public $(jlpkgname(info))
      (package
       (name "$(jlpkgname(info))")
       (version "$(info.vers.ver)")
       (source
          (origin
            (method git-fetch)
            (uri (git-reference
              (url "$(info.repo)")
              (commit (string-append "v" version))))
            (file-name "$(info.name)")
            (sha256
                 (base32 "$hash"))))
       (propagated-inputs `($(deplist)))
       (build-system julia-build-system)
       (home-page "$(info.repo)")
       (synopsis "")
       (description "")
       (license license:expat)))
    """
end

println.(makepackage.(getpackage.(ARGS)))

As a self reminder, I found this repo
https://github.com/cdluminate/DistroHelper.jl
which might contain something useful for our purposes

Hi,

On Sat, 30 Jan 2021 at 21:13, Nicolò Balzarotti <anothersms@gmail.com> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>>> (invoke-julia "using Pkg; Pkg.TOML.parsefile("Project.toml")["name"] |> println")
>>
>> With a bit more glue, could this be transformed into something like
>> “julia->guix-package”?  And so have a Julia package importer, even if it
>> fails for some cases.
>
> Well, if you mean "Can we use Pkg.jl to generate package definitions for
> us?" the answer is "probably yes, but I never investigated this".  That
> line uses just Julia Base TOML.jl, which for some reason is defined
> inside module Pkg.jl (and it's not top-level).

I mean the correct way is to write a TOML parser, probably using PEG.
As it is done with the other importers.

It is a piece of work.  Not especially hard but it could take some time.

So instead, the question is, using a bit of Julia glue and Guile glue,
is it possible to extract the necessary information to have a Guix
package?

Well, maybe these glue code is the same as writing a TOML parser. :-)


> If you instead meant "Can we have a Julia importer?" some time ago I
> wrote one in Julia, I've not used it in the last year, but it did work
> quite well back then.  I attach it here for reference.

Fun!

One thing is how to deal with ’Manifest.toml’.  And pre-compiled
substitutes do not make too much sense in the Julia world.  Well, I do
not know how these Manifest.toml overlap with how Guix works.


Cheers,
simon

> I mean the correct way is to write a TOML parser, probably using PEG.
> As it is done with the other importers.
> It is a piece of work.  Not especially hard but it could take some time.

That wouldn't be too hard, but also it would not be enough.  As you can
see in [fn:1], there's just the list of dependencies, but no info on how
to get them, so parsing the Julia General Registry [fn:2] is still required.

> So instead, the question is, using a bit of Julia glue and Guile glue,
> is it possible to extract the necessary information to have a Guix
> package?
>
> Well, maybe these glue code is the same as writing a TOML parser. :-)

This might be feasible, but as I said above, unfortunately I don't think
that a TOML parser would be enough.  But asking "Pkg3" to resolve
dependencies for us, yes, this makes sense.

> One thing is how to deal with ’Manifest.toml’.  And pre-compiled
> substitutes do not make too much sense in the Julia world.  Well, I do
> not know how these Manifest.toml overlap with how Guix works.
WDYM? Julia uses .ji files (which are the analogoues of .go for guile
and .pyc for python), if this is what you are referring to.

I'm just ignoring Manifest.toml, which should be the same as venv in the
python world.

[fn:1] https://github.com/JuliaLang/TOML.jl/blob/master/Project.toml
[fn:2] https://github.com/JuliaRegistries/General

Hi,

On Sun, 31 Jan 2021 at 21:00, Nicolò Balzarotti <anothersms@gmail.com> wrote:

> That wouldn't be too hard, but also it would not be enough.  As you can
> see in [fn:1], there's just the list of dependencies, but no info on how
> to get them, so parsing the Julia General Registry [fn:2] is still required.

[...]

> I'm just ignoring Manifest.toml, which should be the same as venv in the
> python world.

Sorry, you have right.  I have misremembered an old presentation about
Pkg3 by Stefan Karpinski.

Well, thanks for the pointers.  And I gave a look at the Pkg doc. ;-)

Once a PEG parser for TOML files is written, all the information is in
the Registry (by default this General repository), recursively.

However, the “resolve” part is not straightforward, IMHO.  It is what
confused me in my previous email and I thought was “Manifest.toml”.
Anyway.

Instead of reinventing the wheel and reimplement Pkg.jl in Guix, maybe
it is worth to have Julia code as you did that extracts the relevant
information and then generates the relevant Guix packages.  However, it
should mean that “guix import julia” requires to have the package julia
implicitly used.  Why not. :-)


Cheers,
simon