[bug#72766,2/2] gnu: slurm: Enable REST API.
Commit Message
* gnu/packages/parallel.scm (slurm)[inputs]: Add dbus, http-parser, libjwt,
librdkafka, libtaml, linux-libre-headers and (mariadb "dev").
[arguments]: Add --enable-slurmrestd, --with-bpf, --with-http-parser and
--with-rdkafka to configure flags.
Change-Id: Icddaa8f883754129d975e82fccd732a47a467517
---
gnu/packages/parallel.scm | 26 ++++++++++++++++++++++----
1 file changed, 22 insertions(+), 4 deletions(-)
Comments
Hi Arun,
Arun Isaac <arunisaac@systemreboot.net> skribis:
> * gnu/packages/parallel.scm (slurm)[inputs]: Add dbus, http-parser, libjwt,
> librdkafka, libtaml, linux-libre-headers and (mariadb "dev").
> [arguments]: Add --enable-slurmrestd, --with-bpf, --with-http-parser and
> --with-rdkafka to configure flags.
>
> Change-Id: Icddaa8f883754129d975e82fccd732a47a467517
I’m curious what the REST interface has to offer compared to DRMAA.
> + (inputs (list dbus
> + freeipmi
> + http-parser
> `(,hwloc-2 "lib")
> json-c
> + libjwt
> + librdkafka
> + libyaml
> + linux-libre-headers
> linux-pam
> + `(,mariadb "dev")
That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is
not necessary; it’s included by default).
It’s a bit of Inheritance Game in this file (ah ha!) so we should make
sure the other slurm variants still work fine. qa.guix hasn’t tested it
yet.
Thanks,
Ludo’.
Hi Ludo,
> I’m curious what the REST interface has to offer compared to DRMAA.
I don't know, to be honest.
I've started working on ravanan, a new implementation of the Common
Workflow Language that takes inspiration from Guix and integrates with
Guix for strong reproducibility.
https://git.systemreboot.net/ravanan/about/
ravanan uses the slurm REST API to talk to slurm, and that's why I need
this patch. I chose the slurm REST API over DRMAA rather arbitrarily. In
the future, I could support both in ravanan. So, it's still good to have
both.
>> + (inputs (list dbus
>> + freeipmi
>> + http-parser
>> `(,hwloc-2 "lib")
>> json-c
>> + libjwt
>> + librdkafka
>> + libyaml
>> + linux-libre-headers
>> linux-pam
>> + `(,mariadb "dev")
>
> That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is
> not necessary; it’s included by default).
Good catch! I'll remove it before pushing.
> It’s a bit of Inheritance Game in this file (ah ha!) so we should make
> sure the other slurm variants still work fine. qa.guix hasn’t tested it
> yet.
I have built the other slurm variants, and they seem to build fine. I
also used these packages to migrate a cluster from ancient slurm 18.08
to modern slurm 23.02 in increments of two versions, and tested
functionality at each step. They all seem to work. So, I am relatively
confident these packages work.
Regards,
Arun
>> That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is
>> not necessary; it’s included by default).
>
> Good catch! I'll remove it before pushing.
But, how do I refer to an implicit input (linux-libre-headers, in this
case) using this-package-input? Is there some other way to do this?
Thanks!
Hey Arun,
Arun Isaac <arunisaac@systemreboot.net> skribis:
>>> That’s quite a few heavyweight dependencies (‘linux-libre-headers’ is
>>> not necessary; it’s included by default).
>>
>> Good catch! I'll remove it before pushing.
>
> But, how do I refer to an implicit input (linux-libre-headers, in this
> case) using this-package-input? Is there some other way to do this?
It depends on the details, but perhaps you can use ‘search-input-file’?
‘this-package-input’ doesn’t let you access implicit inputs; it just
accesses the regular inputs fields.
HTH,
Ludo’.
>> But, how do I refer to an implicit input (linux-libre-headers, in this
>> case) using this-package-input? Is there some other way to do this?
>
> It depends on the details, but perhaps you can use ‘search-input-file’?
> ‘this-package-input’ doesn’t let you access implicit inputs; it just
> accesses the regular inputs fields.
Fixed with search-input-directory. Patchset v2 follows.
Thanks!
Hey,
Arun Isaac <arunisaac@systemreboot.net> skribis:
>> I’m curious what the REST interface has to offer compared to DRMAA.
>
> I don't know, to be honest.
>
> I've started working on ravanan, a new implementation of the Common
> Workflow Language that takes inspiration from Guix and integrates with
> Guix for strong reproducibility.
>
> https://git.systemreboot.net/ravanan/about/
Nice!
> ravanan uses the slurm REST API to talk to slurm, and that's why I need
> this patch. I chose the slurm REST API over DRMAA rather arbitrarily. In
> the future, I could support both in ravanan. So, it's still good to have
> both.
I’m mentioning DRMAA because Ricardo wrote guile-drmaa for similar
purposes, and I think DRMAA is implemented by several batch schedulers.
But I don’t know whether it would be a good fit for what you need.
Ludo’.
@@ -15,6 +15,7 @@
;;; Copyright © 2024 Zheng Junjie <873216071@qq.com>
;;; Copyright © 2024 David Elsing <david.elsing@posteo.net>
;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
+;;; Copyright © 2024 Arun Isaac <arunisaac@systemreboot.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -49,13 +50,16 @@ (define-module (gnu packages parallel)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages check)
+ #:use-module (gnu packages databases)
#:use-module (gnu packages documentation)
#:use-module (gnu packages flex)
#:use-module (gnu packages freeipmi)
+ #:use-module (gnu packages glib)
#:use-module (gnu packages libevent)
#:use-module (gnu packages linux)
#:use-module (gnu packages maths)
#:use-module (gnu packages mpi)
+ #:use-module (gnu packages networking)
#:use-module (gnu packages perl)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python)
@@ -63,6 +67,7 @@ (define-module (gnu packages parallel)
#:use-module (gnu packages python-science)
#:use-module (gnu packages python-xyz)
#:use-module (gnu packages readline)
+ #:use-module (gnu packages serialization)
#:use-module (gnu packages tcl)
#:use-module (gnu packages tls)
#:use-module (gnu packages web))
@@ -223,12 +228,18 @@ (define-public slurm
(mkdir "contribs")
(rename-file "tmp-pmi" "contribs/pmi")
(rename-file "tmp-pmi2" "contribs/pmi2")))))
- ;; FIXME: More optional inputs could be added,
- ;; in particular mysql and gtk+.
- (inputs (list freeipmi
+ ;; FIXME: More optional inputs (for example, gtk+) could be added.
+ (inputs (list dbus
+ freeipmi
+ http-parser
`(,hwloc-2 "lib")
json-c
+ libjwt
+ librdkafka
+ libyaml
+ linux-libre-headers
linux-pam
+ `(,mariadb "dev")
openpmix
munge
numactl
@@ -238,13 +249,20 @@ (define-public slurm
(build-system gnu-build-system)
(arguments
(list #:configure-flags
- #~(list "--enable-pam" "--sysconfdir=/etc/slurm"
+ #~(list "--enable-pam"
+ "--enable-slurmrestd"
+ "--sysconfdir=/etc/slurm"
"--disable-static"
+ (string-append "--with-bpf="
+ #$(this-package-input "linux-libre-headers"))
(string-append "--with-freeipmi=" #$(this-package-input "freeipmi"))
+ (string-append "--with-http-parser="
+ #$(this-package-input "http-parser"))
(string-append "--with-hwloc="
(ungexp (this-package-input "hwloc") "lib"))
(string-append "--with-json=" #$(this-package-input "json-c"))
(string-append "--with-munge=" #$(this-package-input "munge"))
+ (string-append "--with-rdkafka=" #$(this-package-input "librdkafka"))
;; Use PMIx bundled with Open MPI (this is required for Open MPI 5.x).
;; Note: Older versions that inherit from this package lack the