| Message ID | 3f45eba5ce4b94bcf046b1f9368336ad236a74e3.1724369334.git.arunisaac@systemreboot.net |
|---|---|
| State | New |
| Headers | show |
| Series | slurm: Enable REST API. | expand |
Hi Arun, Arun Isaac <arunisaac@systemreboot.net> skribis: > * gnu/packages/parallel.scm (slurm)[inputs]: Add dbus, http-parser, libjwt, > librdkafka, libtaml, linux-libre-headers and (mariadb "dev"). > [arguments]: Add --enable-slurmrestd, --with-bpf, --with-http-parser and > --with-rdkafka to configure flags. > > Change-Id: Icddaa8f883754129d975e82fccd732a47a467517 I’m curious what the REST interface has to offer compared to DRMAA. > + (inputs (list dbus > + freeipmi > + http-parser > `(,hwloc-2 "lib") > json-c > + libjwt > + librdkafka > + libyaml > + linux-libre-headers > linux-pam > + `(,mariadb "dev") That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is not necessary; it’s included by default). It’s a bit of Inheritance Game in this file (ah ha!) so we should make sure the other slurm variants still work fine. qa.guix hasn’t tested it yet. Thanks, Ludo’.
Hi Ludo, > I’m curious what the REST interface has to offer compared to DRMAA. I don't know, to be honest. I've started working on ravanan, a new implementation of the Common Workflow Language that takes inspiration from Guix and integrates with Guix for strong reproducibility. https://git.systemreboot.net/ravanan/about/ ravanan uses the slurm REST API to talk to slurm, and that's why I need this patch. I chose the slurm REST API over DRMAA rather arbitrarily. In the future, I could support both in ravanan. So, it's still good to have both. >> + (inputs (list dbus >> + freeipmi >> + http-parser >> `(,hwloc-2 "lib") >> json-c >> + libjwt >> + librdkafka >> + libyaml >> + linux-libre-headers >> linux-pam >> + `(,mariadb "dev") > > That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is > not necessary; it’s included by default). Good catch! I'll remove it before pushing. > It’s a bit of Inheritance Game in this file (ah ha!) so we should make > sure the other slurm variants still work fine. qa.guix hasn’t tested it > yet. I have built the other slurm variants, and they seem to build fine. I also used these packages to migrate a cluster from ancient slurm 18.08 to modern slurm 23.02 in increments of two versions, and tested functionality at each step. They all seem to work. So, I am relatively confident these packages work. Regards, Arun
>> That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is >> not necessary; it’s included by default). > > Good catch! I'll remove it before pushing. But, how do I refer to an implicit input (linux-libre-headers, in this case) using this-package-input? Is there some other way to do this? Thanks!
Hey Arun, Arun Isaac <arunisaac@systemreboot.net> skribis: >>> That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is >>> not necessary; it’s included by default). >> >> Good catch! I'll remove it before pushing. > > But, how do I refer to an implicit input (linux-libre-headers, in this > case) using this-package-input? Is there some other way to do this? It depends on the details, but perhaps you can use ‘search-input-file’? ‘this-package-input’ doesn’t let you access implicit inputs; it just accesses the regular inputs fields. HTH, Ludo’.
>> But, how do I refer to an implicit input (linux-libre-headers, in this >> case) using this-package-input? Is there some other way to do this? > > It depends on the details, but perhaps you can use ‘search-input-file’? > ‘this-package-input’ doesn’t let you access implicit inputs; it just > accesses the regular inputs fields. Fixed with search-input-directory. Patchset v2 follows. Thanks!
Hey, Arun Isaac <arunisaac@systemreboot.net> skribis: >> I’m curious what the REST interface has to offer compared to DRMAA. > > I don't know, to be honest. > > I've started working on ravanan, a new implementation of the Common > Workflow Language that takes inspiration from Guix and integrates with > Guix for strong reproducibility. > > https://git.systemreboot.net/ravanan/about/ Nice! > ravanan uses the slurm REST API to talk to slurm, and that's why I need > this patch. I chose the slurm REST API over DRMAA rather arbitrarily. In > the future, I could support both in ravanan. So, it's still good to have > both. I’m mentioning DRMAA because Ricardo wrote guile-drmaa for similar purposes, and I think DRMAA is implemented by several batch schedulers. But I don’t know whether it would be a good fit for what you need. Ludo’.
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm index e75e902a68..21c945382f 100644 --- a/gnu/packages/parallel.scm +++ b/gnu/packages/parallel.scm @@ -15,6 +15,7 @@ ;;; Copyright © 2024 Zheng Junjie <873216071@qq.com> ;;; Copyright © 2024 David Elsing <david.elsing@posteo.net> ;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr> +;;; Copyright © 2024 Arun Isaac <arunisaac@systemreboot.net> ;;; ;;; This file is part of GNU Guix. ;;; @@ -49,13 +50,16 @@ (define-module (gnu packages parallel) #:use-module (gnu packages base) #:use-module (gnu packages bash) #:use-module (gnu packages check) + #:use-module (gnu packages databases) #:use-module (gnu packages documentation) #:use-module (gnu packages flex) #:use-module (gnu packages freeipmi) + #:use-module (gnu packages glib) #:use-module (gnu packages libevent) #:use-module (gnu packages linux) #:use-module (gnu packages maths) #:use-module (gnu packages mpi) + #:use-module (gnu packages networking) #:use-module (gnu packages perl) #:use-module (gnu packages pkg-config) #:use-module (gnu packages python) @@ -63,6 +67,7 @@ (define-module (gnu packages parallel) #:use-module (gnu packages python-science) #:use-module (gnu packages python-xyz) #:use-module (gnu packages readline) + #:use-module (gnu packages serialization) #:use-module (gnu packages tcl) #:use-module (gnu packages tls) #:use-module (gnu packages web)) @@ -223,12 +228,18 @@ (define-public slurm (mkdir "contribs") (rename-file "tmp-pmi" "contribs/pmi") (rename-file "tmp-pmi2" "contribs/pmi2"))))) - ;; FIXME: More optional inputs could be added, - ;; in particular mysql and gtk+. - (inputs (list freeipmi + ;; FIXME: More optional inputs (for example, gtk+) could be added. + (inputs (list dbus + freeipmi + http-parser `(,hwloc-2 "lib") json-c + libjwt + librdkafka + libyaml + linux-libre-headers linux-pam + `(,mariadb "dev") openpmix munge numactl @@ -238,13 +249,20 @@ (define-public slurm (build-system gnu-build-system) (arguments (list #:configure-flags - #~(list "--enable-pam" "--sysconfdir=/etc/slurm" + #~(list "--enable-pam" + "--enable-slurmrestd" + "--sysconfdir=/etc/slurm" "--disable-static" + (string-append "--with-bpf=" + #$(this-package-input "linux-libre-headers")) (string-append "--with-freeipmi=" #$(this-package-input "freeipmi")) + (string-append "--with-http-parser=" + #$(this-package-input "http-parser")) (string-append "--with-hwloc=" (ungexp (this-package-input "hwloc") "lib")) (string-append "--with-json=" #$(this-package-input "json-c")) (string-append "--with-munge=" #$(this-package-input "munge")) + (string-append "--with-rdkafka=" #$(this-package-input "librdkafka")) ;; Use PMIx bundled with Open MPI (this is required for Open MPI 5.x). ;; Note: Older versions that inherit from this package lack the