| Message ID | 3LOAUDT0FLL4U.2SOD925YP915T@wilsonb.com |
|---|---|
| State | New |
| Headers | show |
| Series | [bug#48463] gnu: Add j. | expand |
| Context | Check | Description |
|---|---|---|
| cbaines/comparison | success | View comparision |
| cbaines/git branch | success | View Git branch |
| cbaines/applying patch | fail | View Laminar job |
| cbaines/issue | success | View issue |
Hi elaexuotee, Am Sonntag, den 16.05.2021, 19:54 +0900 schrieb elaexuotee@wilsonb.com: > Hey Guix, > > This packages up the J programming lanuage. The build scripts that > upstream provides are pretty hostile to package maintainers, so we > have to jump through quite a few hoops to get the build working. Yup, that much is visible from the package description. > I have had this in my personal, private channel for a while, tweaking > it over time, so I believe it works fairly well in practice. Here are > the high-level design decisions I have made: > > 1) Provide multiple J versions; > > It is extremely common for J users to have multiple versions > installed in parallel since there are often major, breaking changes > between (non-patch-level) releases. > > In addition, a very large fraction of J users keep the beta installed > as well, so I have elected to provide this too. That should be fine, we provide multiple versions for other packages where applicable as well. Do note, that we should try to keep the number limited, though. You may perhaps want to export make-jlib, so that the user can build their own. > 2) Bundle together all variants of the interpreter along with a > wrapper script that selects the most featureful one at runtime; > > There are 3 versions of the interpreter, supporting different CPU > feature-sets: vanilla, AVX, AVX2. This package elects to build all > these variants and provides a wrapper script that launches the "most > AVX-y" version possible at runtime. > > Essentially, this is trying to make up for upstream's lack of a fat > binary. Are fat binaries the accepted HPC way? I'm not up-to-date to this. > 3) Try to mirror the FHS standard for installation locations; > > For the most part, J uses idiosyncratic defaults for where it looks > for library files and the like. This package elects to configure > things so files sit in more standard locations---e.g. runtime scripts > sit under .../share/j; configuration files sit under .../etc/j; etc. > > 4) Point the J package manager to $HOME/.j/<version>. > > J maintains its own mini package repository for "addons", which are > simply officially-supported J scripts. This package sets things up so > that these packages get installed to .j/<version>/addons. In > addition, other related directories are gathered under .j/<version>. Those should be fine. > Items on the TODO list: > > 1) Write an importer and create packages for J addons; > > This is currently in progress. > > 2) Create a something like a jsoftware-union procedure to bundle up J > with user-selectable addons; > > This is necessary since the interpreter looks for addons at a > specified path. We probably want to mimick something like texlive and > provide both an "all batteries included" package along with one where > users can select which addons they want. Both sound like good ideas, although I'm a little less sure about the texlive one. Would a meta-package like gnome also be acceptable? > 3) Support more older versions of J. > > J is open source starting with j801, so it'd be nice to have packages > for the entire j80x series too. Unfortunately, those use an entirely > different set of build scripts from the current j90x series, so it > will require munging-work similar to what we already have. As above, I'd prefer if it was one procedure and one package pointing to the latest j80x, assuming all of our added patches can also be applied to earlier versions. > +(define-record-type* <jlib-build-configuration> > + jlib-build-configuration make-jlib-build-configuration > + jlib-build-configuration? > + (builder jlib-build-configuration-builder (default > "guix.gnu.org")) > + (version jlib-build-configuration-version (default #f)) > + (revision jlib-build-configuration-revision (default #f)) > + (hash jlib-build-configuration-hash (default #f)) > + (type jlib-build-configuration-type (default 'release)) > + (patches jlib-build-configuration-patches (default '())) > + (extra-inputs jlib-build-configuration-extra-inputs (default '())) > + (extra-envars jlib-build-configuration-extra-envars (default > '()))) > + > +(define make-jlib > + (match-lambda > + (($ <jlib-build-configuration> > + builder jversion revision hash type patches extra-inputs > extra-envars) Please try to use keyword arguments. > + (properties `((jversion . ,jversion) > + (jrevision . ,revision) > + (jtype . ,type))) Are those used anywhere? Can jversion and jrevision not be parsed from (package-version jlib)? Regards, Leo
Thanks for taking a look! Leo Prikler <leo.prikler@student.tugraz.at> wrote: > That should be fine, we provide multiple versions for other packages > where applicable as well. Do note, that we should try to keep the > number limited, though. You may perhaps want to export make-jlib, so > that the user can build their own. Cool. I like the idea of exporting the build procedures. > Are fat binaries the accepted HPC way? I'm not up-to-date to this. I believe so? Don't quote me on that though. For this particular package the overhoad is just ~8MB, so I don't think it's much of an issue either way. It would probably be more of a hassle to split j into a package for each variant. > Both sound like good ideas, although I'm a little less sure about the > texlive one. Would a meta-package like gnome also be acceptable? A metapackage is certainly practical. The entire set of J addons currently weighs in at a whopping 50MB. hehe. However, there are about 125 addons in total, so it simply feels "more correct" to let the user also choose just the addons they want, which I think would require something like jsoftware-union. That said, either way, an "all batteries include" package would be good to have, and since this is way easier than packaging the addons separately, I'll definitely work on the metapackage first. > As above, I'd prefer if it was one procedure and one package pointing > to the latest j80x, assuming all of our added patches can also be > applied to earlier versions. Yeah, that would definitely be ideal. Unfortunately, it's not so straigthforward. The the "major versions" are the xxx part of jxxx-y, with large changes between each, e.g. j902 introduced and non-compatible API change over j901. So, from a pure J user perspective, having all jxxx versions available is ideal. However, each version seems to require its own set of build flags and sometimes version-specific patches (like in the j901 case). We probably don't want to push those settings out into user manifest specs. On the other hand, there are already 10 versions from j801 to j903. > > +(define make-jlib > > + (match-lambda > > + (($ <jlib-build-configuration> > > + builder jversion revision hash type patches extra-inputs > > extra-envars) > Please try to use keyword arguments. Actually, maybe these "build configuration" records could solve the above "too many versions" problem. We could offer only the latest J (and J beta?) packages in the repo, but let (gnu packages jsoftware) export a set of "default configurations" for old versions to be used with the make-j procedure. Any particular reason to avoid using records though? Currently, its letting us share configuration options between j902 and j903 that don't work in j901. Personally, I thought this felt like a nice declarative, scheme-y way to go, but my Scheme is still very amateurish. Definitely wanting to rectify any strange ideas I may have. > > + (properties `((jversion . ,jversion) > > + (jrevision . ,revision) > > + (jtype . ,type))) > Are those used anywhere? Can jversion and jrevision not be parsed from > (package-version jlib)? The make-j procedure uses them. We'd have to parse out these from both the version string and package name with a what's essentially the inverse of the jname procedure. I found it a lot more readable and less hassle to just propagate this data directly. However, if there's a particular reason why using properties is problematic, I'll try to see what I can do.
Hi, Am Montag, den 24.05.2021, 22:37 +0900 schrieb elaexuotee@wilsonb.com: > > Are fat binaries the accepted HPC way? I'm not up-to-date to this. > > I believe so? Don't quote me on that though. For this particular > package the > overhoad is just ~8MB, so I don't think it's much of an issue either > way. It > would probably be more of a hassle to split j into a package for each > variant. In that case it will probably be fine. > > Both sound like good ideas, although I'm a little less sure about > > the > > texlive one. Would a meta-package like gnome also be acceptable? > > A metapackage is certainly practical. The entire set of J addons > currently weighs in at a whopping 50MB. hehe. However, there are > about 125 addons in total, so it simply feels "more correct" to let > the user also choose just the addons they want, which I think would > require something like jsoftware-union. > > That said, either way, an "all batteries include" package would be > good to have, and since this is way easier than packaging the addons > separately, I'll definitely work on the metapackage first. stuff-union usually implies, that stuff can't just be put into a profile and expected to work (like texlive), whereas in other cases (e.g. the gnome metapackage) people can use its parts on their own provided that they have the right combination (which typically implies having icons etc.) > > As above, I'd prefer if it was one procedure and one package > > pointing to the latest j80x, assuming all of our added patches can > > also be applied to earlier versions. > > Yeah, that would definitely be ideal. > > Unfortunately, it's not so straigthforward. The the "major versions" > are the xxx part of jxxx-y, with large changes between each, e.g. > j902 introduced and non-compatible API change over j901. So, from a > pure J user perspective, having all jxxx versions available is ideal. > > However, each version seems to require its own set of build flags and > sometimes version-specific patches (like in the j901 case). We > probably don't want to push those settings out into user manifest > specs. > > On the other hand, there are already 10 versions from j801 to j903. That's less than the number of Rust versions we need just for bootstrap, but still a few too many in my opinion. Are all of those still used in production or would it be wiser to to put some of them into Guix-Past [1]? > > > +(define make-jlib > > > + (match-lambda > > > + (($ <jlib-build-configuration> > > > + builder jversion revision hash type patches extra-inputs > > > extra-envars) > > Please try to use keyword arguments. > > Actually, maybe these "build configuration" records could solve the > above "too many versions" problem. We could offer only the latest J > (and J beta?) packages in the repo, but let (gnu packages jsoftware) > export a set of "default configurations" for old versions to be used > with the make-j procedure. I don't think there's much to be gained if we provide packages without packages. > Any particular reason to avoid using records though? Currently, its > letting us share configuration options between j902 and j903 that > don't work in j901. > Personally, I thought this felt like a nice declarative, scheme-y way > to go, but my Scheme is still very amateurish. Definitely wanting to > rectify any strange ideas I may have. Having keyword lists is also nice and declarative, but more importantly, it lets you call the function as a normal function without having to worry about constructing some record in a particular way. The configuration you're using doesn't have a specific setting? Just override it with your own. It's as simple as calling (append old- config extra-config). > > > + (properties `((jversion . ,jversion) > > > + (jrevision . ,revision) > > > + (jtype . ,type))) > > Are those used anywhere? Can jversion and jrevision not be parsed > > from (package-version jlib)? > > The make-j procedure uses them. We'd have to parse out these from > both the version string and package name with a what's essentially > the inverse of the jname procedure. I found it a lot more readable > and less hassle to just propagate this data directly. > > However, if there's a particular reason why using properties is > problematic, I'll try to see what I can do. I find this way of mistreating the version field very weird. You're not adding anything new by doing this and you're not making anything more secure (I'd argue, that it's less secure, because you could update the version and forget about the property or vice versa). First of all, do you even need all this info? `j' is an objectively bad name for a package. `j-beta' is not better in any way, it only avoids the user installing a potentially unstable "J" package. (Note, that we typically use "-next" for that, however). However, this info is meaningless when hardcoded into a procedure. You can just inherit from the package that's generated and override the name as needed. Next, should ijconsole not simply be a package like jlib (both properly prefixed with jsoftware- of course)? Then you can take whatever version bits you need from the package version itself. Regards, Leo [1] https://gitlab.inria.fr/guix-hpc/guix-past
As always, thanks for the quick turnaround. Leo Prikler <leo.prikler@student.tugraz.at> wrote: > stuff-union usually implies, that stuff can't just be put into a > profile and expected to work (like texlive), whereas in other cases > (e.g. the gnome metapackage) people can use its parts on their own > provided that they have the right combination (which typically implies > having icons etc.) Oh, okay. For some reason, I was just imagining "metapackage" to mean "all batteries included", but your description makes a lot more sense. In this case, having addons without the J interpreter is pretty useless, so IIUC a jsoftware-union makes more sense than a metapackage in this case? Just to be clear, J looks for addons at a path burned into $store/...-jlib-$version/etc/j/profilex.ijs. Currently, that points to $HOME/.j/$jversion/addons, but we'll need to change that to a single store output path that contains the union of all desired addons, hence why I was thinking the *-union approach is needed. For now, instead of messing with the union, would it make sense to just create a (non-public?) jsoftware-addons package that contains all addons and point jlib at those? > That's less than the number of Rust versions we need just for > bootstrap, but still a few too many in my opinion. Are all of those > still used in production or would it be wiser to to put some of them > into Guix-Past [1]? Oh cool. Guix Past might be a better solution! Thanks for sharing. I don't have any solid data, since "in production" J is mostly dominated by the financial sector from what I hear, but just going by my impression from lurking on the J mailing list, the j80x series (and even earlier non-free versions) does still have a user base. Personally, I just want to make these older versions available to the community but am agnostic about the *how* so will defer to whatever you (and others) think is best. > Having keyword lists is also nice and declarative, but more > importantly, it lets you call the function as a normal function without > having to worry about constructing some record in a particular way. > The configuration you're using doesn't have a specific setting? Just > override it with your own. It's as simple as calling (append old- > config extra-config). Oh! (append old-config extra-config) is exactly the kind of thing I was trying to achieve; however, I'm not sure I quite grok the usage you are describing. Something like this? (define* (proc #:key foo bar) ...) (define old-config '(#:foo 0)) (define extra-config '(#:bar 1)) (apply proc (append old-config extra-config)) > I find this way of mistreating the version field very weird. You're > not adding anything new by doing this and you're not making anything > more secure (I'd argue, that it's less secure, because you could update > the version and forget about the property or vice versa). I agree the version stuff is a bit crappy. It's something that got munged badly over time and deserves more thought. Here are the issues I'm trying to solve: 1) The source URLs look like ../j<version>-<type>-<patch> or ../j<version>-<type>, where <type> is either "beta" or "release", and depending on whether a patch-level exists; 2) The type string ("release" or "beta") is a build setting; and 3) ijconsole installs to something like ../bin/ijconsole-902, without including the patch-level, like ../bin/ijconsole-902-b. 4) Users should probably be able to easily install the latest J release as well as J beta concurrently without caring about exactly which version these are. Those imply that <version>, <type>, and <patch> are indeed handled separately, no matter concrete method utilized. My thinking was that 4 means we want a separate *-beta package, meaning that the version field should probably just look like "<version>-<patch>". Additionally, relating back to the above discussion about multiple versions 5) It would be nice to be able to install the latest j901 without having to know that this corresponds to patch level f, i.e. currently j@901-f. > First of all, do you even need all this info? `j' is an objectively bad name > for a package. `j-beta' is not better in any way, it only avoids the user > installing a potentially unstable "J" package. Yeah, "j" is pretty short but it does mirror "r". I'm agnostic about package name, though. Suggestions welcome. > (Note, that we typically use "-next" for that, however). Thanks for pointing out this convention. > However, this info is meaningless when hardcoded into a procedure. You can > just inherit from the package that's generated and override the name as > needed. I see what you mean in a general sense but not quite how to apply it in this case. I am all ears if you have a simpler, cleaner solution that addresses the constraints listed above! > Next, should ijconsole not simply be a package like jlib (both properly > prefixed with jsoftware- of course)? This is effectively what make-j does, no? ijconsole needs to know the path to jlib, so make-j points it at the correct one and wraps everything up into a package. Anyway, thanks again for taking a look!
Hi, Am Dienstag, den 25.05.2021, 12:57 +0900 schrieb elaexuotee@wilsonb.com: > As always, thanks for the quick turnaround. > > Leo Prikler <leo.prikler@student.tugraz.at> wrote: > > stuff-union usually implies, that stuff can't just be put into a > > profile and expected to work (like texlive), whereas in other cases > > (e.g. the gnome metapackage) people can use its parts on their own > > provided that they have the right combination (which typically > > implies having icons etc.) > > Oh, okay. For some reason, I was just imagining "metapackage" to > mean "all batteries included", but your description makes a lot more > sense. In this case, having addons without the J interpreter is > pretty useless, so IIUC a jsoftware-union makes more sense than a > metapackage in this case? > > Just to be clear, J looks for addons at a path burned into > $store/...-jlib-$version/etc/j/profilex.ijs. Currently, that points > to $HOME/.j/$jversion/addons, but we'll need to change that to a > single store output path that contains the union of all desired > addons, hence why I was thinking the *-union approach is needed. > > For now, instead of messing with the union, would it make sense to > just create a (non-public?) jsoftware-addons package that contains > all addons and point jlib at those? Are those addons data packages or can they be compiled without needing jlib? If they're pure data, then you can pack them up and refer them as you wanted to, if not, you first have to create jlib-minimal without them, then package up everything and finally do jlib. Oh, and one thing that was previously ignored is that using XDG standards for things we can't simply put in the store (like configuration if it needs that) is preferable to having a ~/.j directory. > > That's less than the number of Rust versions we need just for > > bootstrap, but still a few too many in my opinion. Are all of > > those still used in production or would it be wiser to to put some > > of them into Guix-Past [1]? > > Oh cool. Guix Past might be a better solution! Thanks for sharing. > > I don't have any solid data, since "in production" J is mostly > dominated by the financial sector from what I hear, but just going by > my impression from lurking on the J mailing list, the j80x series > (and even earlier non-free versions) does still have a user base. > > Personally, I just want to make these older versions available to the > community but am agnostic about the *how* so will defer to whatever > you (and others) think is best. People are also still running ancient versions of Debian. Python 2 has officially reached end of life, yet it is used as well. At some point we do have to declare software that people are still using as old :) > > Having keyword lists is also nice and declarative, but more > > importantly, it lets you call the function as a normal function > > without having to worry about constructing some record in a > > particular way. The configuration you're using doesn't have a > > specific setting? Just override it with your own. It's as simple > > as calling (append old-config extra-config). > > Oh! (append old-config extra-config) is exactly the kind of thing I > was trying > to achieve; however, I'm not sure I quite grok the usage you are > describing. > Something like this? > > (define* (proc #:key foo bar) ...) > (define old-config '(#:foo 0)) > (define extra-config '(#:bar 1)) > (apply proc (append old-config extra-config)) Yup. > > I find this way of mistreating the version field very > > weird. You're not adding anything new by doing this and you're not > > making anything more secure (I'd argue, that it's less secure, > > because you could update the version and forget about the property > > or vice versa). > > I agree the version stuff is a bit crappy. It's something that got > munged badly over time and deserves more thought. Here are the > issues I'm trying to solve: > > 1) The source URLs look like ../j<version>-<type>-<patch> or > ../j<version>-<type>, where <type> is either "beta" or > "release", and > depending on whether a patch-level exists; > 2) The type string ("release" or "beta") is a build setting; and > 3) ijconsole installs to something like ../bin/ijconsole-902, > without > including the patch-level, like ../bin/ijconsole-902-b. > 4) Users should probably be able to easily install the latest J > release as > well as J beta concurrently without caring about exactly which > version > these are. > > Those imply that <version>, <type>, and <patch> are indeed handled > separately, no matter concrete method utilized. Those four issues do call for separate keyword arguments, but they need not necessarily be reflected in the version string. You can have two packages with the same version, but different hashes in Guix. > My thinking was that 4 means we want a separate *-beta package, > meaning that the version field should probably just look like > "<version>-<patch>". > Additionally, relating back to the above discussion about multiple > versions Sounds good, though <version>.<patch> would be equally acceptable. Your call. > 5) It would be nice to be able to install the latest j901 without > having to > know that this corresponds to patch level f, i.e. currently > j@901-f. I think `guix build j@901' ought to resolve this automatically. > > First of all, do you even need all this info? `j' is an > > objectively bad name > > for a package. `j-beta' is not better in any way, it only avoids > > the user installing a potentially unstable "J" package. > > Yeah, "j" is pretty short but it does mirror "r". I'm agnostic about > package name, though. Suggestions welcome. Fair enough, though this remains a problem for single letter programs in which others would likely want to claim the letter as well. R has the benefit of being old and well-known enough, it would probably be rlang otherwise (not to be confused with erlang). > > (Note, that we typically use "-next" for that, however). > > Thanks for pointing out this convention. > > > However, this info is meaningless when hardcoded into a > > procedure. You can just inherit from the package that's generated > > and override the name as needed. > > I see what you mean in a general sense but not quite how to apply it > in this case. I am all ears if you have a simpler, cleaner solution > that addresses the constraints listed above! I think the cleaner solution here is to not store those properties in the package, but still having them as parameters to the make-j function call. > > Next, should ijconsole not simply be a package like jlib (both > > properly prefixed with jsoftware- of course)? > > This is effectively what make-j does, no? ijconsole needs to know the > path to jlib, so make-j points it at the correct one and wraps > everything up into a package. I don't really understand why those needs to be done in two steps however. ijconsole could already contain the fully functioning j program. Or partially functioning if we account for addons. Cheers!
Hi, There is now a mechanism for tuning packages for particular subarchitectures without resorting to irreproducible and unsubstitutable packages, see <https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/>. Might be useful here. Greetings, Maxime.
From c231924ab9a72979001640d866db1988f8fd3bde Mon Sep 17 00:00:00 2001 From: "B. Wilson" <elaexuotee@wilsonb.com> Date: Sun, 16 May 2021 19:02:29 +0900 Subject: [PATCH] gnu: Add j. To: guix-patches@gnu.org * gnu/packages/jsoftware.scm: New file. * gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file. * gnu/local.mk: (GNU_SYSTEM_MODULES, dist_patch_DATA): Add them. --- gnu/local.mk | 2 + gnu/packages/jsoftware.scm | 432 ++++++++++++++++++ .../patches/jsoftware-j901-f-fixes.patch | 80 ++++ 3 files changed, 514 insertions(+) create mode 100644 gnu/packages/jsoftware.scm create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch diff --git a/gnu/local.mk b/gnu/local.mk index c3b0274945..b706ec4e86 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -305,6 +305,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/javascript.scm \ %D%/packages/jemalloc.scm \ %D%/packages/jrnl.scm \ + %D%/packages/jsoftware.scm \ %D%/packages/jose.scm \ %D%/packages/julia.scm \ %D%/packages/julia-xyz.scm \ @@ -1270,6 +1271,7 @@ dist_patch_DATA = \ %D%/packages/patches/jfsutils-add-sysmacros.patch \ %D%/packages/patches/jfsutils-include-systypes.patch \ %D%/packages/patches/jsoncpp-fix-inverted-case.patch \ + %D%/packages/patches/jsoftware-j901-f-fixes.patch \ %D%/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch \ %D%/packages/patches/kdbusaddons-kinit-file-name.patch \ %D%/packages/patches/libffi-3.3-powerpc-fixes.patch \ diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm new file mode 100644 index 0000000000..f6f6099e42 --- /dev/null +++ b/gnu/packages/jsoftware.scm @@ -0,0 +1,432 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 B. Wilson <elaexuotee@wilsonb.com> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (gnu packages jsoftware) + #:use-module (guix records) + #:use-module (guix packages) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix git-download) + #:use-module (guix build-system gnu) + #:use-module (guix build-system trivial) + #:use-module (guix build utils) + #:use-module (guix utils) + #:use-module (gnu packages) + #:use-module (gnu packages libedit) + #:use-module (gnu packages llvm) + #:use-module (gnu packages maths) + #:use-module (guix gexp) + #:use-module (ice-9 match)) + + +(define (jname prefix jtype) + "Return a package name for the given prefix and jtype, e.g. `jlib', + `jlib-beta', `j', and `j-beta'." + (let ((postfix (if (eq? jtype 'release) "" + (string-append "-" (symbol->string jtype))))) + (string-append prefix postfix))) + +(define (jversion->string version revision) + "Return a string representation of a J version and (optional) revision pair." + (let ((postfix (if (not revision) "" + (string-append "-" revision)))) + (string-append version postfix))) + +(define (jinfo->git-tag jversion jtype jrevision) + "Given version parameters, construct a git tag for upstream releases." + (let ((postfix (if (not jrevision) "" + (string-append "-" jrevision)))) + (string-append "j" jversion "-" (symbol->string jtype) postfix))) + +(define-record-type* <jlib-build-configuration> + jlib-build-configuration make-jlib-build-configuration + jlib-build-configuration? + (builder jlib-build-configuration-builder (default "guix.gnu.org")) + (version jlib-build-configuration-version (default #f)) + (revision jlib-build-configuration-revision (default #f)) + (hash jlib-build-configuration-hash (default #f)) + (type jlib-build-configuration-type (default 'release)) + (patches jlib-build-configuration-patches (default '())) + (extra-inputs jlib-build-configuration-extra-inputs (default '())) + (extra-envars jlib-build-configuration-extra-envars (default '()))) + +(define make-jlib + (match-lambda + (($ <jlib-build-configuration> + builder jversion revision hash type patches extra-inputs extra-envars) + (package + (name (jname "jlib" type)) + (version (jversion->string jversion revision)) + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/jsoftware/jsource") + (commit (jinfo->git-tag jversion type revision)))) + (sha256 (base32 hash)) + (file-name (git-file-name name version)) + (patches patches))) + (properties `((jversion . ,jversion) + (jrevision . ,revision) + (jtype . ,type))) + (build-system gnu-build-system) + (native-inputs `(("clang-toolchain" ,clang-toolchain))) + (inputs (cons* `("libedit" ,libedit) + `("libomp" ,libomp) + extra-inputs)) + (arguments + `(#:modules (((ice-9 ftw) #:select (scandir)) + ((ice-9 popen) #:select (open-pipe* close-pipe)) + ((ice-9 regex) #:select (match:substring string-match)) + ((ice-9 threads) #:select (parallel par-for-each)) + ((srfi srfi-26) #:select (cut)) + ((srfi srfi-1) #:select (fold)) + ,@%gnu-build-system-modules) + #:phases + ;; Upstream's build system consists of ad-hoc scripts that build + ;; build up (very complicated) environment variables to pass to make. + ;; The basic build process looks like this: + ;; + ;; 1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values; + ;; 2) Set jplatform and j64x environment variables; + ;; 3) Run make2/build_jconsole.sh and make2/build_libj.sh; + ;; + ;; However, upstream expects users to run J directly from the source + ;; directory; they do not supply a make `install' target. Thus it + ;; takes some massaging to install files in FHS-style directories. + (modify-phases %standard-phases + ;; In particular, we have to set up + ;; + ;; 1) jsrc/jversion.h as in a typical build; + ;; 2) jlibrary/bin/profilex.ijs to point to writable directories; + ;; 3) make2/build_*.sh to respect standard build conventions; + ;; 4) jsrc/jconsole.c to fix libedit dlopen; and + ;; 5) Hard coded references to addons directory. + (replace 'configure + (lambda* (#:key target inputs outputs #:allow-other-keys) + (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain")) + (clang (string-append clang-toolchain "/bin/clang")) + (libedit (assoc-ref inputs "libedit")) + (out (assoc-ref outputs "out")) + (share (string-append out "/share/j"))) + ;; Set up build constants + (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h") + (substitute* "jsrc/jversion.h" + (("^#define jversion.*$") + (format #f "#define jversion ~s\n" ,version)) + (("^#define jtype.*$") + (format #f "#define jtype ~s\n" (symbol->string ',type))) + (("^#define jbuilder.*$") + (format #f "#define jbuilder ~s\n" ,builder))) + ;; Create profilex.ijs overrides to point to the correct + ;; store items. Note that we set ~install and ~addons + ;; directories to reside under ~user to allow installing + ;; and loading addons. TODO: Guix-ify J addons as well. + (call-with-output-file "jlibrary/bin/profilex.ijs" + (lambda (port) + (display + (string-join + (list + (format #f "system=. '~a/system'" share) + (format #f "tools=. '~a/tools'" share) + ;; Upstream defaults to spamming $HOME with + ;; unhidden userdata directories. Set this to be + ;; $HOME/.j/<jtype>/<jversion> instead + "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''" + "jversion=. ({.~ i.&'-') jversion" + "jsuffix=. >@{&('';'-beta') jtype -: 'beta'" + "user=. home,'/.j/',jversion,jsuffix" + "addons=. user,'/addons'" + "break=. user,'/break'" + "config=. user,'/config'" + "install=. user,'/install'" + "snap=. user,'/snap'" + "temp=. user,'/temp'" + "\n") + "\n") + port))) + ;; Munge the build scripts into reason: + ;; 1. Short-circuit the fragile compiler detection; + ;; 2. Make sure to include our CFLAGS and LFLAGS; and + ;; 3. Propagate script errors to top level. + (for-each + (lambda (file) + (with-directory-excursion "make2" + (substitute* file + ;; The `compiler' variable doesn't point to the actual + ;; compiler. It is just a switch to tell the build + ;; scripts whether to use gcc- or clang-specific + ;; flags. + (("^compiler=.*$") "compiler=clang\n") + (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS ")) + (("^(common=\")(\\$USETHREAD.*)$" _ def rest) + (string-append def "$CFLAGS " rest)) + (("^#!.*" shebang) + (string-append shebang "set -o errexit\n"))))) + '("build_jconsole.sh" "build_libj.sh")) + ;; The jconsole manually loads libedit with dlopen. The path + ;; must be absolute to correctly point to our input. + (substitute* "jsrc/jconsole.c" + (("libedit\\.so\\.[0-9]" so-file) + (format #f "~a/lib/~a" libedit so-file))) + ;; The ~addons/dev directory supplies tentative J-script + ;; definitions of new J engine functionality. Since we point + ;; ~addons under the ~user directory, we move it under + ;; ~system instead, which sits as-is in the output. + (with-directory-excursion "jsrc" + (for-each + (lambda (file) + (substitute* file (("~addons/dev") "~system/dev"))) + (scandir "." + (lambda (f) (eq? (stat:type (stat f)) 'regular))))) + ;; Implementation of 9!:14 records build time which breaks + ;; build reproducibility. Note that upstream code depends on + ;; the exact format of these strings, so we need to mimic the + ;; standard. + (substitute* "jsrc/j.c" + (("__DATE__") "\"Jan 01 1970\"") + (("__TIME__") "\"00:00:00\"")) + ;; Upstream recommends using clang, with GCC support being + ;; second-class, often resulting in build failures. + (setenv "CC" clang)))) + ;; The build output depends primarily on the values of the + ;; `jplatform' and `j64x' environment variables. If the target is + ;; ARM, then `jplatform' is "raspberry", otherwise it is `linux'. + ;; In addition to 32- and 64- bit versions, `j64x' controlls + ;; whether AVX or AVX2 variants of libj are built. + ;; + ;; However, build targets are not fine-grained enough to + ;; distinguish between CPU features. Thus we build and install all + ;; variants of libj, expecting jconsole to be called with a wrapper + ;; script that detects AVX features and loads the appropriate libj + ;; at runtime. + (replace 'build + (lambda _ + (setenv "USE_OPENMP" "1") + (setenv "USE_THREAD" "1") + (for-each (lambda (var-val) (apply setenv var-val)) + (quote ,extra-envars)) + ;; The build scripts assume that PWD is make2. + (with-directory-excursion "make2" + (let* ((platform ,(if (target-arm?) "raspberry" "linux")) + (jplat (string-append "jplatform=" platform)) + (target-bit ,(if (target-64bit?) "64" "32")) + (jbit (string-append "j64x=" "j" target-bit)) + (jbit-avx (string-append jbit "avx")) + (jbit-avx2 (string-append jbit "avx2"))) + (parallel + ;; Since jconsole doesn't depend on AVX features, we just + ;; build it once. + (invoke "env" jplat jbit "./build_jconsole.sh") + (invoke "env" jplat jbit "./build_libj.sh") + (if ,(target-64bit?) + (parallel + (invoke "env" jplat jbit-avx "./build_libj.sh") + (invoke "env" jplat jbit-avx2 + "./build_libj.sh")))))))) + ;; The test suite is expected to be run as follows for each variant + ;; of libj that we build: + ;; + ;; $ echo 'RUN ddall' | jconsole test/tsu.ijs + ;; + ;; This requires a working jconsole with accessible jlibrary files. + ;; We simply place these all under test/bin. + (replace 'check + (lambda _ + (let ((jplatform ,(if (target-arm?) "raspberry" "linux"))) + (mkdir-p "test/bin") + (for-each + (lambda (dir) + (let ((source (string-append "jlibrary/" dir)) + (dest (string-append "test/bin/" dir))) + (begin + (mkdir-p dest) + (copy-recursively source dest)))) + '("system" "tools" "addons")) + ;; The jlibrary/dev directory only sometimes exists, but + ;; needs to be copied into the ~system directory when it + ;; does. + (if (access? "jlibrary/dev" R_OK) + (copy-recursively "jlibrary/dev" "test/bin/system/dev")) + (par-for-each + (lambda (dir) + (let* ((jbin (string-append "bin/" jplatform)) + (jbit ,(if (target-64bit?) "j64" "j32")) + (jconsole (string-append jbin "/" jbit + "/jconsole")) + (source (string-append jbin "/" dir)) + (dest (string-append "test/bin/" dir))) + (begin + (mkdir-p dest) + (copy-recursively source dest) + (install-file "jlibrary/bin/profile.ijs" dest) + (install-file jconsole dest) + (let* ((jc (string-append dest "/jconsole")) + (tests "test/tsu.ijs") + (port (open-pipe* OPEN_WRITE jc tests))) + (display "RUN ddall\n" port) + (when (not (zero? (status:exit-val + (close-pipe port)))) + (error "Some J build tests failed.")))))) + (scandir (string-append "bin/" jplatform) + (negate (cut member <> '("." ".."))))) + #t))) + ;; Now that everything is built, installation is fairly + ;; straightforward, following FHS conventions. The only quirk is + ;; that we install jconsole under /libexec to make room for the + ;; wrapper replacement under /bin. + (replace 'install + (lambda* (#:key outputs inputs #:allow-other-keys) + (let* ((jplat ,(if (target-arm?) "raspberry" "linux")) + (jbit ,(if (target-64bit?) "j64" "j32")) + (jconsole (string-join + `("bin" ,jplat ,jbit "jconsole") "/")) + (out (assoc-ref outputs "out")) + (etc (string-append out "/etc/j")) + (lib (string-append out "/lib/j")) + (libexec (string-append out "/libexec")) + (share (string-append out "/share/j")) + (system (string-append share "/system")) + (dev (string-append system "/dev"))) + (mkdir-p lib) + (for-each + (lambda (jarch) + (let* ((jbin (string-join `("bin" ,jplat ,jarch) "/")) + (javx-match (string-match "avx.*" jarch)) + (javx (if (not javx-match) "" + (match:substring javx-match))) + (sep (if javx-match "-" "")) + (source (string-append jbin "/libj.so")) + (dest (format #f "~a/libj~a~a.so" lib sep javx))) + (copy-file source dest))) + (scandir (string-append "bin/" jplat) + (negate (cut member <> '("." ".."))))) + (install-file jconsole libexec) + (copy-recursively "jlibrary/system" system) + (if (access? "jlibrary/dev" R_OK) + (copy-recursively "jlibrary/dev" dev)) + (install-file "jlibrary/bin/profile.ijs" etc) + (install-file "jlibrary/bin/profilex.ijs" etc))))))) + (home-page "https://www.jsoftware.com/") + (synopsis "Ascii-only, array programming language in the APL family") + (description + "J is a high-level, general-purpose programming language that is +particularly suited to the mathematical, statistical, and logical analysis of +data. It is a powerful tool for developing algorithms and exploring problems +that are not already well understood.") + (license license:gpl3+))))) + + +(define jlib-901 + (make-jlib + (jlib-build-configuration + (version "901") + (revision "f") + (hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p") + (patches (search-patches "jsoftware-j901-f-fixes.patch"))))) + + +(define jlib-build-configuration-with-sleef + (jlib-build-configuration + (extra-inputs `(("sleef" ,sleef))) + (extra-envars `(("USE_SLEEF_SRC" "0") + ("LDFLAGS" "-lsleef"))))) + +(define jlib-902 + (make-jlib + (jlib-build-configuration + (inherit jlib-build-configuration-with-sleef) + (version "902") + (revision "b") + (hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0")))) + +(define jlib-903-beta + (make-jlib + (jlib-build-configuration + (inherit jlib-build-configuration-with-sleef) + (version "903") + (revision "h") + (type 'beta) + (hash "13sw2ffgx6pm699qfi50drlp8y1vmzj4swzx82g6vhyjpjci1w4h")))) + + +(define (make-ijconsole jlib) + "Generate a G-exp wrapper script that detects AVX/AVX2 support at runtime and + executes jconsole with the appropriate libj.so and profile.ijs." + (program-file (string-append "ijconsole-" (package-version jlib)) + #~(begin + (use-modules ((ice-9 rdelim) #:select (read-line)) + ((ice-9 regex) #:select (regexp-match? string-match))) + + (define (cpu-feature-line? string) + (string-prefix? "flags" string)) + + (define (contains-word? word string) + (regexp-match? + (string-match (string-join `("\\<" ,word "\\>") "") + string))) + + (define (has-cpu-feature? feature) + (with-input-from-file "/proc/cpuinfo" + (lambda () + (catch 'found + (lambda () + (let loop ((line (read-line))) + (cond ((eof-object? line) #f) + ((and (cpu-feature-line? line) + (contains-word? feature line)) + (throw 'found)) + (else (loop (read-line)))))) + (const #t))))) + + (let* ((jconsole (string-append #$jlib "/libexec/jconsole")) + (libj (format #f "~a/lib/j/libj-~a.so" #$jlib + (cond ((has-cpu-feature? "avx2") "avx2") + ((has-cpu-feature? "avx") "avx") + (else "")))) + (jprofile (string-append #$jlib "/etc/j/profile.ijs"))) + (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile + (cdr (command-line))))))) + +(define (make-j jlib) + "Generate a J package containing /bin/ijconsole-<jversion> linked against the + given jlib library package. Note that the ijconsole executable is versioned + since it is typical for J users to have multiple versions installed on the + same system." + (let ((jversion (assq-ref (package-properties jlib) 'jversion)) + (jtype (assq-ref (package-properties jlib) 'jtype))) + (package (inherit jlib) + (name (jname "j" jtype)) + (source #f) + (inputs `(("jlib" ,jlib) + ("ijconsole" ,(make-ijconsole jlib)))) + (build-system trivial-build-system) + (arguments + `(#:modules ((guix build utils)) + #:builder + (begin + (use-modules (guix build utils)) + (let* ((ijconsole (assoc-ref %build-inputs "ijconsole")) + (out (assoc-ref %outputs "out")) + (bin (string-append out "/bin"))) + (mkdir-p bin) + (symlink ijconsole + (string-append bin "/ijconsole-" ,jversion))))))))) + +(define-public j-901 (make-j jlib-901)) +(define-public j-902 (make-j jlib-902)) +(define-public j-903-beta (make-j jlib-903-beta)) diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch new file mode 100644 index 0000000000..0ac7e94de4 --- /dev/null +++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch @@ -0,0 +1,80 @@ +This patch fixes two separate issues with ustream sources: + +* Normalize import paths in jsrc/cip.c + +Upstream claims to have some build requirements that force them to use strange +import paths. However, these paths do not exist inside our build chroot. + +* Fix unititialized variable warning + +Clang 9 issues some warnings which cause the build to fail since upstream +compiles with -Werror. + + +diff --git a/jsrc/cip.c b/jsrc/cip.c +index 61da4088..fb3c03b6 100644 +--- a/jsrc/cip.c ++++ b/jsrc/cip.c +@@ -3,9 +3,9 @@ + /* */ + /* Conjunctions: Inner Product */ + +-#include "../../jsource/jsrc/j.h" +-#include "../../jsource/jsrc/vasm.h" +-#include "../../jsource/jsrc/gemm.h" ++#include "j.h" ++#include "vasm.h" ++#include "gemm.h" + + #define MAXAROWS 384 // max rows of a that we can process to stay in L2 cache a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH this is wired to 128*3 - check if you chage + +@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0 + switch(c){ + case CPLUSDOT: + #define F |= +-#include "../../jsource/jsrc/cip_t.h" ++#include "cip_t.h" + break; + case CSTARDOT: + #define F &= +-#include "../../jsource/jsrc/cip_t.h" ++#include "cip_t.h" + break; + case CNE: + #define F ^= +-#include "../../jsource/jsrc/cip_t.h" ++#include "cip_t.h" + break; + } + R z; +diff --git a/jsrc/gemm.c b/jsrc/gemm.c +index 51fe306e..b105dfc1 100644 +--- a/jsrc/gemm.c ++++ b/jsrc/gemm.c +@@ -318,7 +318,7 @@ dgemm_nn (I m, + _B); + + // loop 3 +- I i; ++ I i=0; + #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c) + for (i=0; i<mb; ++i) { + I mc; +@@ -501,7 +501,7 @@ igemm_nn (I m, + _B); + + // loop 3 +- I i; ++ I i=0; + #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c) + for (i=0; i<mb; ++i) { + I mc; +@@ -831,7 +831,7 @@ zgemm_nn (I m, + _B); + + // loop 3 +- I i; ++ I i=0; + #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c) + for (i=0; i<mb; ++i) { + I mc; -- 2.31.1