[bug#64448,0/2] Fix hdf5-1.14 wrappers and remove generated source files from all hdf5 versions.

When using hdf5 with CMake, I noticed that it does not work for the 1.14
version, because the h5cc and h5c++ wrappers contain the flag
"-I/tmp/guix-build-hdf5-1.14.0.drv-0/hdf5-1.14.0/src/H5FDsubfiling". It
is required in the 'configure phase, but can be removed afterwards.

Additionally, I noticed that the hdf5 source tarballs contain many
autogenerated files, so I also removed them. Somehow, the 1.8.23 release
is missing two m4 scripts (which are included in later releases, but
different from this one), so they are copied from the 1.8.21 tarball
(the latest earlier release which includes them).
The genparser script is missing as well, which is copied from the 1.10.9
release.

David Elsing (2):
  gnu: hdf5-1.8: Remove generated files from source
  gnu: hdf5-1.14: Remove spurious include directories.

 gnu/packages/maths.scm                      | 234 +++++++++++++++-----
 gnu/packages/patches/hdf5-config-date.patch |  14 +-
 2 files changed, 188 insertions(+), 60 deletions(-)


base-commit: 76e39909e886bb51cf9b2fa03a1523545faca880
  

I saw that the Git repository is also available, with the releases as
tags: https://github.com/HDFGroup/hdf5. Would it be preferred to use
them instead and make the snippet shorter? The missing scripts are also
included for the 1.8.23 release.
  

Hello,

adding Gerd Heber in cc, who seems to be the last person having worked on
these packages.

I am not competent at all about them, so commenting more as an outsider.
Using "guix package -A hdf":
hdf4                	4.2.14          	out        	gnu/packages/maths.scm:1289:2
hdf4-alt            	4.2.14          	out        	gnu/packages/maths.scm:1372:2
hdf5                	1.8.23          	out,fortran	gnu/packages/maths.scm:1382:2
hdf5                	1.14.0          	out,fortran	gnu/packages/maths.scm:1543:2
hdf5                	1.12.2          	out,fortran	gnu/packages/maths.scm:1523:2
hdf5                	1.10.9          	out,fortran	gnu/packages/maths.scm:1503:2
hdf5-blosc          	1.0.0           	out        	gnu/packages/maths.scm:1807:2
hdf5-parallel-openmpi	1.10.9          	out,fortran	gnu/packages/maths.scm:1775:2

and a lot of packages in other languages, for instance:
cl-hdf5-cffi        	1.8.18-1.5b5c88f	out        	gnu/packages/lisp-xyz.scm:12238:4
ecl-hdf5-cffi       	1.8.18-1.5b5c88f	out        	gnu/packages/lisp-xyz.scm:12238:4
(apparently based on 1.8.18? but it has hdf5-1.10 as input).

So there are lots of versions, and in the middle of them,
(define-public hdf5
  ;; Default version of HDF5.
  hdf5-1.10)

And
hdf5-parallel-openmpi	1.10.9          	out,fortran	gnu/packages/maths.scm:1775:2
which inherits like this:
  (package/inherit hdf5-1.10                      ;use the latest
but takes inputs like this:
    (inputs
     `(("mpi" ,openmpi)
       ,@(package-inputs hdf5)))
which, I suppose, works by chance since currently hdf5-1.10 and hdf5 are
the same (but not "the latest").

Are all of these needed? If only the latest version 1.14.0 could be kept,
this would make the patch for 1.8.23 obsolete, for instance.

Andreas
  

Am Mon, Jul 24, 2023 at 03:49:49PM +0200 schrieb Andreas Enge:
> Are all of these needed?

Partial answer at
   https://portal.hdfgroup.org/display/support/HDF5+1.8.23 :
"Future of HDF5-1.8
Please be aware that this will be the last release for HDF5 1.8.
We encourage users to move to HDF5 1.10 or HDF5 1.14. Please refer to
the release schedule for future releases of these versions."

The schedule at
   https://github.com/HDFGroup/hdf5#release-schedule
shows that 1.10 and 1.12 should see their last release this autumn,
so probably we should really move to 1.14.

$ guix refresh -l hdf5@1.8
Building the following 9 packages would ensure 16 dependent packages are rebuilt: pigx-sars-cov-2@0.0.9 pigx@0.0.3 nip2@8.7.1 python-pyvips@2.2.1 gnss-sdr@0.0.17 hdf-eos5@1.15 scilab@5.5.0 hdf-java@3.3.2 h5check@2.0.1

$ guix refresh -l hdf5@1.10
Building the following 210 packages would ensure 411 dependent packages are rebuilt:
...

$ guix refresh -l hdf5@1.12
No dependents other than itself: hdf5@1.12.2
So this could probably be dropped immediately.

$ guix refresh -l hdf5@1.14
Building the following 2 packages would ensure 2 dependent packages are rebuilt: ecl-hdf5-cffi@1.8.18-1.5b5c88f cl-hdf5-cffi@1.8.18-1.5b5c88f

I suppose that the many dependencies on 1.10 come from the definition of
the variable hd5 as hdf5-10; it is quite possible we could simply upgrade
this to hdf5-14. Someone™ should give it a try...

Andreas
  

Only HDF5 1.14.x (currently, 1.14.2) should be kept. However, hdf5 and
hdf5-parallel should be kept separate. Aside from the MPI dependence,
the parallel build is NOT a superset of the sequential version. There could
be even two versions of hdf5: a non-threadsafe build (hdf5) and a
threadsafe build (hdf5-ts? hdf5-mt?).

For HDF4, 4.2.16-2 is the latest HDF4 release. We are maintaining that
indefinitely (for NASA).

Best, G.

On Mon, Jul 24, 2023 at 8:49 AM Andreas Enge <andreas@enge.fr> wrote:

> Hello,
>
> adding Gerd Heber in cc, who seems to be the last person having worked on
> these packages.
>
> I am not competent at all about them, so commenting more as an outsider.
> Using "guix package -A hdf":
> hdf4                    4.2.14                  out
>  gnu/packages/maths.scm:1289:2
> hdf4-alt                4.2.14                  out
>  gnu/packages/maths.scm:1372:2
> hdf5                    1.8.23                  out,fortran
>  gnu/packages/maths.scm:1382:2
> hdf5                    1.14.0                  out,fortran
>  gnu/packages/maths.scm:1543:2
> hdf5                    1.12.2                  out,fortran
>  gnu/packages/maths.scm:1523:2
> hdf5                    1.10.9                  out,fortran
>  gnu/packages/maths.scm:1503:2
> hdf5-blosc              1.0.0                   out
>  gnu/packages/maths.scm:1807:2
> hdf5-parallel-openmpi   1.10.9                  out,fortran
>  gnu/packages/maths.scm:1775:2
>
> and a lot of packages in other languages, for instance:
> cl-hdf5-cffi            1.8.18-1.5b5c88f        out
>  gnu/packages/lisp-xyz.scm:12238:4
> ecl-hdf5-cffi           1.8.18-1.5b5c88f        out
>  gnu/packages/lisp-xyz.scm:12238:4
> (apparently based on 1.8.18? but it has hdf5-1.10 as input).
>
> So there are lots of versions, and in the middle of them,
> (define-public hdf5
>   ;; Default version of HDF5.
>   hdf5-1.10)
>
> And
> hdf5-parallel-openmpi   1.10.9                  out,fortran
>  gnu/packages/maths.scm:1775:2
> which inherits like this:
>   (package/inherit hdf5-1.10                      ;use the latest
> but takes inputs like this:
>     (inputs
>      `(("mpi" ,openmpi)
>        ,@(package-inputs hdf5)))
> which, I suppose, works by chance since currently hdf5-1.10 and hdf5 are
> the same (but not "the latest").
>
> Are all of these needed? If only the latest version 1.14.0 could be kept,
> this would make the patch for 1.8.23 obsolete, for instance.
>
> Andreas
>
>
  

Am Sun, Aug 20, 2023 at 07:36:50PM -0500 schrieb Gerd Heber:
> Only HDF5 1.14.x (currently, 1.14.2) should be kept. However, hdf5 and
> hdf5-parallel should be kept separate. Aside from the MPI dependence,
> the parallel build is NOT a superset of the sequential version. There could be
> even two versions of hdf5: a non-threadsafe build (hdf5) and a threadsafe build
> (hdf5-ts? hdf5-mt?).
> 
> For HDF4, 4.2.16-2 is the latest HDF4 release. We are maintaining that
> indefinitely (for NASA).

Thanks for the info, which helps us set a target!

I am trying to update hdf4, see
   https://issues.guix.gnu.org/65443

Andreas
  

And I have removed hdf5@1.12, which had no dependent packages.

Andreas
  

Hello,

> $ guix refresh -l hdf5@1.8
> Building the following 9 packages would ensure 16 dependent packages are rebuilt: pigx-sars-cov-2@0.0.9 pigx@0.0.3 nip2@8.7.1 python-pyvips@2.2.1 gnss-sdr@0.0.17 hdf-eos5@1.15 scilab@5.5.0 hdf-java@3.3.2 h5check@2.0.1

Currently, 5 packages depend directly on hdf5@1.8 (from guix graph -t reverse-package):
-scilab, for which the latest version can be built with hdf5@1.14:
 https://issues.guix.gnu.org/66201
-hdf5-eos, for which the latest version can also be built with
 hdf5@1.14: https://issues.guix.gnu.org/66204
-java-cisd-jhdf5, where newer versions seem to be built only with
 Gradle: https://sissource.ethz.ch/sispub/jhdf5. It has fastqc and pigx(-*)
 as dependencies.
-hdf-java, where the latest version is from 2017. Maybe it can be built
 with a newer HDF5 version? No other packages depend on it.
-h5check, where the latest version is from 2014. When I tried to compile
 it with hdf5@1.14, some tests failed because the file format is not
 accepted anymore. No other packages depend on it.

Cheers,
David
  

Hello,

I have submitted two new issues with the goal of getting closer to
removing hdf5@1.8 and hdf5@1.10, see
   https://issues.guix.gnu.org/77285
   https://issues.guix.gnu.org/77287

Ricardo, since you seem to be interested in this package, your input
there would be very welcome. And here also; after the proposed removal
of h5check (which, I just see, does not even build any more), the only
remaining direct dependency of hdf5@1.8 is
   java-cisd-jhdf5
and "guix refresh -l hdf5@1.8" lists
   pigx-sars-cov-2@0.0.9 trinityrnaseq@2.13.2 pigx@0.0.3
as leaf packages.

Andreas
  

Hi Andreas,

> I have submitted two new issues with the goal of getting closer 
> to
> removing hdf5@1.8 and hdf5@1.10, see
>    https://issues.guix.gnu.org/77285
>    https://issues.guix.gnu.org/77287
>
> Ricardo, since you seem to be interested in this package, your 
> input
> there would be very welcome. And here also; after the proposed 
> removal
> of h5check (which, I just see, does not even build any more), 
> the only
> remaining direct dependency of hdf5@1.8 is
>    java-cisd-jhdf5
> and "guix refresh -l hdf5@1.8" lists
>    pigx-sars-cov-2@0.0.9 trinityrnaseq@2.13.2 pigx@0.0.3
> as leaf packages.

On the python-team branch the only user of java-cisd-jhdf5 is 
fastqc, which should be updated to a more recent version.
  

Hello Ricardo,

Am Wed, Mar 26, 2025 at 06:29:39PM +0100 schrieb Ricardo Wurmus:
> On the python-team branch the only user of java-cisd-jhdf5 is fastqc, which
> should be updated to a more recent version.

looking at its git repository, fastqc still requires java-cisd-jhdf5
(there is an eight year old jar file), so it does not look as if
updating fastqc will help us get rid of java-cisd-jhdf5 and then
hdf5@1.8.

Andreas
  

Hello David,

with a lot of work on hdf5 unrelated to your original issue, this bug
report has become a bit unwieldy (sorry for that, but I think we made
good progress around this package complex!).

So I wonder whether you would be okay closing this issue and opening
new ones with updated patches?

Useful remaining work would be updating hdf5@1.14 to its latest version;
working towards getting rid of hdf5@1.8 and hdf5@1.10;
and your original issue about generated files, which maybe could be
combined with an update to hdf5@1.14 (I think there is no point in
spending much work on the outdated versions that we would prefer to
remove).
For testing updates, it might make sense to wait until the non-building
dependencies have been removed in about a month.

What do you think?

Andreas
  

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> So I wonder whether you would be okay closing this issue and opening
> new ones with updated patches?

Sure, I'm closing it now.

> Useful remaining work would be updating hdf5@1.14 to its latest version;
> working towards getting rid of hdf5@1.8 and hdf5@1.10;
> and your original issue about generated files, which maybe could be
> combined with an update to hdf5@1.14 (I think there is no point in
> spending much work on the outdated versions that we would prefer to
> remove).

Yes, that's a good suggestion, I made a new patch series which only
changes hdf5@1.14 and hdf-java: https://issues.guix.gnu.org/77590

Thanks,
David