| Message ID | 20210228133314.22935-1-gheber@hdfgroup.org |
|---|---|
| State | New |
| Headers | show |
| Series | [bug#46830,Added,hdf5-1.12-parallel-openmpi] * gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on HDF5 1.12.0 | expand |
| Context | Check | Description |
|---|---|---|
| cbaines/comparison | success | View comparision |
| cbaines/git branch | success | View Git branch |
| cbaines/applying patch | success | View Laminar job |
| cbaines/issue | success | View issue |
This needs to be a separate package since (as with all HDF5 major releases) it's not 100% API compatible with HDF5 1.10. It would break other packages, e.g., `petsc-lite`
Hi, On Sun, 28 Feb 2021 at 07:33, Gerd Heber <gerd.heber@gmail.com> wrote: > +(define-public hdf5-1.12-parallel-openmpi > + (package/inherit hdf5-1.12 > + (name "hdf5-1.12-parallel-openmpi") > + (inputs > + `(("mpi" ,openmpi) > + ,@(package-inputs hdf5))) > + (arguments > + (substitute-keyword-arguments (package-arguments hdf5) > + ((#:configure-flags flags) > + ``("--enable-parallel" > + ,@(delete "--enable-cxx" > + (delete "--enable-threadsafe" ,flags)))) > + ((#:phases phases) > + `(modify-phases ,phases > + (add-after 'build 'mpi-setup > + ,%openmpi-setup) > + (add-before 'check 'patch-tests > + (lambda _ > + ;; OpenMPI's mpirun will exit with non-zero status if it > + ;; detects an "abnormal termination", i.e. any process not > + ;; calling MPI_Finalize(). Since the test is explicitly > + ;; avoiding MPI_Finalize so as not to have at_exit and thus > + ;; H5C_flush_cache from being called, mpirun will always > + ;; complain, so turn this test off. > + (substitute* "testpar/Makefile" > + (("(^TEST_PROG_PARA.*)t_pflush1(.*)" front back) > + (string-append front back "\n"))) > + (substitute* "tools/test/h5diff/testph5diff.sh" > + (("/bin/sh") (which "sh"))) > + #t)))))) > + (synopsis "Management suite for data with parallel IO support"))) > + Why duplicates? Something like: --8<---------------cut here---------------start------------->8--- (define-public hdf5-1.12-parallel-openmpi (package/inherit hdf5-parallel-openmpi (version (package-version hdf5-1.12)) (source (package-source hdf5-1.12))) --8<---------------cut here---------------end--------------->8--- seems simpler. All the best, simon
Hi, (Please CC the bug number.) On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber@gmail.com> wrote: > On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune@gmail.com> wrote: > > --8<---------------cut here---------------start------------->8--- > > (define-public hdf5-1.12-parallel-openmpi > > (package/inherit hdf5-parallel-openmpi > > (version (package-version hdf5-1.12)) > > (source (package-source hdf5-1.12))) > > --8<---------------cut here---------------end--------------->8--- > Hey, thanks, very clever! Should I submit a new patch or is there a > way to patch the patch? The usual way is to send a v2 which is the patch to apply to the Git tree (and *not* a patch to apply to a patch to apply to the tree). However, personally I am not in favor to add to Guix proper too much versions of variant packages; but that's just my humble opinion. Instead, it could be submitted to specific channels, for instance <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in Guix-HPC. Cheers, simon
zimoun <zimon.toutoune@gmail.com> skribis: > On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber@gmail.com> wrote: >> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune@gmail.com> wrote: > >> > --8<---------------cut here---------------start------------->8--- >> > (define-public hdf5-1.12-parallel-openmpi >> > (package/inherit hdf5-parallel-openmpi >> > (version (package-version hdf5-1.12)) >> > (source (package-source hdf5-1.12))) >> > --8<---------------cut here---------------end--------------->8--- > >> Hey, thanks, very clever! Should I submit a new patch or is there a >> way to patch the patch? > > The usual way is to send a v2 which is the patch to apply to the Git > tree (and *not* a patch to apply to a patch to apply to the tree). > > However, personally I am not in favor to add to Guix proper too much > versions of variant packages; but that's just my humble opinion. > Instead, it could be submitted to specific channels, for instance > <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in > Guix-HPC. In this particular case, I think the proposed variant is fine; it’s consistent with what we’re already doing with hdf5. Ludo’.
Ludovic, how are you? Thanks for taking the time to reply. This conversation makes me wonder what the Guix model for packages such as HDF5 might be. On the one hand, there should be defaults for typical, i.e., non-HPC users, and they probably belong into gnu/packages/maths.scm. Once you add MPI to the mix, things get a little more complicated, and channels, such as Guix-HPC might be the better place. (?) Should we run our own channel? In HDF5's case, we also use versions in a peculiar way, where the expectation is that 1.8.x, 1.10.x, and 1.12.x, etc. version are not compatible for end-user applications, mostly because of API changes. What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12 belong into maths.scm, plus the thread-safe builds, but not hdf5-parallel-openmpi? I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI) atomicity tests fail with the OpenMPI version used in the current hdf5-parallel-openmpi. And then there is MPICH... I would also like to see HDFView as a Guix package. We have a Spack package, but the Java story on Guix (I don't blame you ;-) is a little confusing. I'm sold on the merits of Guix and you are doing fantastic work. What's your recommendation, and what can we (The HDF Group) do to help? Best, G. On Tue, Mar 2, 2021 at 1:39 PM Ludovic Courtès <ludo@gnu.org> wrote: > > zimoun <zimon.toutoune@gmail.com> skribis: > > > On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber@gmail.com> wrote: > >> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune@gmail.com> wrote: > > > >> > --8<---------------cut here---------------start------------->8--- > >> > (define-public hdf5-1.12-parallel-openmpi > >> > (package/inherit hdf5-parallel-openmpi > >> > (version (package-version hdf5-1.12)) > >> > (source (package-source hdf5-1.12))) > >> > --8<---------------cut here---------------end--------------->8--- > > > >> Hey, thanks, very clever! Should I submit a new patch or is there a > >> way to patch the patch? > > > > The usual way is to send a v2 which is the patch to apply to the Git > > tree (and *not* a patch to apply to a patch to apply to the tree). > > > > However, personally I am not in favor to add to Guix proper too much > > versions of variant packages; but that's just my humble opinion. > > Instead, it could be submitted to specific channels, for instance > > <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in > > Guix-HPC. > > In this particular case, I think the proposed variant is fine; it’s > consistent with what we’re already doing with hdf5. > > Ludo’.
Hi Gerd, On Wed, 03 Mar 2021 at 08:10, Gerd Heber <gerd.heber@gmail.com> wrote: > What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12 > belong into maths.scm, plus the thread-safe builds, but not > hdf5-parallel-openmpi? From my understanding, hdf5 at versions 1.8 and 1.10 are used by other packages. When those very packages will not use at one of these versions, I guess the very version will be removed. hdf5-parallel-openmpi is used by petsc-openmpi for instance. And this hdf5 variant is built with version 1.10. Since there is no package that requires hdf5-parallel-openmpi at another version than 1.10, I do not see the point to include it in Guix proper. Especially when the custom variant is straightforward to locally create and buildable on a reasonable amount of time. However, these words are not totally acceptable. :-) If I take the shoes of a regular scientist, then they only wants the package at hand and not necessary RTFM how to do package transformations. > I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI) > atomicity tests fail > with the OpenMPI version used in the current hdf5-parallel-openmpi. Yes, and we could imagine different versions of openmpi. And then compiled with different compiler versions, etc… > And then there is MPICH... …and the matrix of combinations is exploding. ;-) One issue with the channel is to provide substitutes for that channel. For example, the channel guix-science uses TravisCI to build the package from GitHub. <https://github.com/guix-science/guix-science/blob/master/.github/workflows/build.yml> That’s said, the cons about channels–and so the pros about include the hdf5 variant in Guix proper–is to keep consistency and detect breakage: Guix proper updates a package that becomes incompatible with one the variant living a channel. All in Guix proper, then Guix CI will detect it. Some dependencies in Guix proper and hdf variant in a channel, then the channel CI probably not since nothing changed (from the channel side) and so nothing triggered a build. I do not know. Well, let stay pragmatic. :-) > I would also like to see HDFView as a Guix package. We have a Spack > package, but It would be really cool! > I'm sold on the merits of Guix and you are doing fantastic work. > What's your recommendation, and what can we (The HDF Group) do to help? Thanks the HDF Group for their interest on Guix. Where the package definition ends (channel vs Guix proper) is one thing, maybe a start should to have these hdf5 variant definitions. Then from a pragmatic point of view, depending on these definitions (number, length, etc.), they could ends in (gnu packages maths) or maybe its own module (gnu packages hdf) or maybe in a channel. WDYT? All the best, simon
Hi Gerd, Gerd Heber <gerd.heber@gmail.com> skribis: > Ludovic, how are you? Thanks for taking the time to reply. This > conversation makes > me wonder what the Guix model for packages such as HDF5 might be. > On the one hand, there should be defaults for typical, i.e., non-HPC users, and > they probably belong into gnu/packages/maths.scm. Once you add MPI to the > mix, things get a little more complicated, and channels, such as > Guix-HPC might be > the better place. (?) Should we run our own channel? As you write, the goal for packages in Guix proper should be to have “good defaults”, and possibly useful variants, as is currently the case with HDF5. We won’t keep all versions and variants in Guix proper because that’d be too much of a maintenance burden. For “unusual” variants, I recommend maintaining your own channel. (This is something we did at Inria for example with an Open MPI variant that includes the MPI1 compatibility layer.) Additionally, you can use things like ‘--with-input=openmpi=mpich’. We may eventually get support for “parameterized packages”, which will allow users to choose between HDF5 variants from the command line: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html > What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12 > belong into maths.scm, plus the thread-safe builds, but not > hdf5-parallel-openmpi? For Guix proper, I think we should keep the number of HDF5 variants stable. We should remove old versions once they’re no longer needed by any in-tree package. > I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI) > atomicity tests fail > with the OpenMPI version used in the current hdf5-parallel-openmpi. > And then there is MPICH... > > I would also like to see HDFView as a Guix package. We have a Spack package, but > the Java story on Guix (I don't blame you ;-) is a little confusing. That’s a different story :-) but please don’t hesitate to share your experience, frustration, and questions regarding Java on the mailing lists, I’m sure you’ll get some guidance. > I'm sold on the merits of Guix and you are doing fantastic work. > What's your recommendation, and what can we (The HDF Group) do to help? It’s great to get feedback from upstream; you can certainly help us make wise(r) decisions regarding HDF5 packaging, in particular in deciding which variants and versions make sense, what defaults are reasonable, and how well they work in an HPC and non-HPC setting. WDYT? What changes would you like to see? Thanks for your interest! Ludo’.
Hello, thanks for your patch, that I only see just now (teams were not yet a thing in 2021...). This version of hdf5 has been retired, so I am closing this issue. It might be useful to add a parallel package to hdf5@1.14 instead. Andreas
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm index 58433d4b0c..df101635ab 100644 --- a/gnu/packages/maths.scm +++ b/gnu/packages/maths.scm @@ -1451,7 +1451,7 @@ Swath).") (license (license:non-copyleft home-page)))) (define-public hdf5-parallel-openmpi - (package/inherit hdf5-1.10 ;use the latest + (package/inherit hdf5-1.10 (name "hdf5-parallel-openmpi") (inputs `(("mpi" ,openmpi) @@ -1482,6 +1482,38 @@ Swath).") #t)))))) (synopsis "Management suite for data with parallel IO support"))) +(define-public hdf5-1.12-parallel-openmpi + (package/inherit hdf5-1.12 + (name "hdf5-1.12-parallel-openmpi") + (inputs + `(("mpi" ,openmpi) + ,@(package-inputs hdf5))) + (arguments + (substitute-keyword-arguments (package-arguments hdf5) + ((#:configure-flags flags) + ``("--enable-parallel" + ,@(delete "--enable-cxx" + (delete "--enable-threadsafe" ,flags)))) + ((#:phases phases) + `(modify-phases ,phases + (add-after 'build 'mpi-setup + ,%openmpi-setup) + (add-before 'check 'patch-tests + (lambda _ + ;; OpenMPI's mpirun will exit with non-zero status if it + ;; detects an "abnormal termination", i.e. any process not + ;; calling MPI_Finalize(). Since the test is explicitly + ;; avoiding MPI_Finalize so as not to have at_exit and thus + ;; H5C_flush_cache from being called, mpirun will always + ;; complain, so turn this test off. + (substitute* "testpar/Makefile" + (("(^TEST_PROG_PARA.*)t_pflush1(.*)" front back) + (string-append front back "\n"))) + (substitute* "tools/test/h5diff/testph5diff.sh" + (("/bin/sh") (which "sh"))) + #t)))))) + (synopsis "Management suite for data with parallel IO support"))) + (define-public hdf5-blosc (package (name "hdf5-blosc")