[bug#69591,00/31] Unbundle and update python-pytorch

This patch series unbundles all dependencies of python-pytorch (updated to
version 2.2.1) and removes all* autogenerated files, completing the work by
Ludo' [1]. I also applied the unbundling to python-pytorch-for-r-torch
(version 2.0.1) and made sure that r-torch still builds.

Originally, I wanted to package PyTorch with ROCm, but decided that it would
be better to first update the package and complete the unbundling.  This was
more work than I initially thought though, so this patch series is quite long.

Caffe2 is heavily integrated in the PyTorch code, so unbundling it does
not make sense.

For QNNPACK, the situation is strange: There is an old version [2] and an
internal fork in aten/src/ATen/native/quantized/cpu/qnnpack, which PyTorch
both depends on (as well as XNNPACK). The internal fork does not depend on
other parts of the code currently, so I unbundled it as well as a package
variant of qnnpack.

ideep [3] has a separate branch for PyTorch (the master branch does not
seem maintained), which only contains header files without build
instructions. Therefore, I made a simple package with the header files.

miniz also requires a package variant for PyTorch.

Several scripts for autogenerated files depend on the compiled PyTorch
library. However, it was possible to patch the scripts with empty values
and later regenerate the files with the initial PyTorch version.
Therefore, I introduced the 'codegen2, 'build2 and 'install2 build
phases running after the initial library is installed. In 'codegen2, the
installed library is used to regenerate the files properly, which leads
to a recompilation of only 12 files in the 'build2 phase.

The core tests can be run as well, but I disabled them as they take
a long time after an already long build.

I also wanted to try and build python-pytorch on i686-linux, as it doesn't
require emulation on an x86_64 CPU, but it depends on Rust through
python-cryptography, for which there are no substitutes available currently.

Cheers,
David

* with the exception of test/cpp/api/init_baseline.h and
test/cpp/api/optim_baseline.h, which are just lists of previously
calculated numbers; or any other generated file I might have missed.

[1] https://hpc.guix.info/blog/2021/09/whats-in-a-package/
[2] https://github.com/pytorch/QNNPACK
[3] https://github.com/intel/ideep/tree/pytorch

David Elsing (31):
  gnu: asmjit: Update to commit 3ca5c18.
  gnu: Add python-typing-extensions-4.10.
  gnu: Add python-optree.
  gnu: Add python-pytest-flakefinder.
  gnu: Add python-pytest-shard.
  gnu: Add python-expecttest.
  gnu: Add python-pytest-rerunfailures-13.
  gnu: Add miniz.
  gnu: Add miniz-for-pytorch.
  gnu: Add libnop.
  gnu: Remove flatbuffers-next-shared.
  gnu: python-flatbuffers-next: Update to 23.5.26.
  gnu: pthreadpool: Update to commit 178e3e0.
  gnu: cpuinfo: Update to commit aa4b216.
  gnu: clog: Add "-DUSE_SYSTEM_LIBS=ON" configure flag.
  gnu: nnpack: Update to commit 70a77f4.
  gnu: oneapi-dnnl: Update to 3.3.5.
  gnu: Add tensorpipe.
  gnu: Add fbgemm.
  gnu: Add qnnpack.
  gnu: Add foxi.
  gnu: Add ideep-pytorch.
  gnu: xnnpack: Update to commit 51a9875.
  gnu: Remove xnnpack-for-torch2.
  gnu: Add qnnpack-pytorch.
  gnu: python-pytorch: Update to 2.2.1 and unbundle dependencies.
  gnu: python-torchvision: Update to 0.17.1.
  gnu: Add ideep-pytorch-for-r-torch.
  gnu: Add oneapi-dnnl-for-r-torch.
  gnu: Add qnnpack-pytorch-for-r-torch.
  gnu: python-pytorch-for-r-torch: Adjust to new python-pytorch.

 gnu/local.mk                                  |    9 +-
 gnu/packages/compression.scm                  |   49 +
 gnu/packages/cpp.scm                          |    6 +-
 gnu/packages/machine-learning.scm             |  874 +++-
 gnu/packages/parallel.scm                     |   30 +-
 .../patches/clog-fix-shared-build.patch       |   39 +-
 .../patches/cpuinfo-system-libraries.patch    |   50 -
 .../patches/fbgemm-use-system-libraries.patch |   62 +
 gnu/packages/patches/foxi-fix-build.patch     |   55 +
 gnu/packages/patches/miniz-for-pytorch.patch  |   47 +
 .../patches/python-optree-fix-32-bit.patch    |  122 +
 ...ython-pytorch-1.9.0-system-libraries.patch |  139 -
 .../patches/python-pytorch-fix-codegen.patch  |  167 +
 ...thon-pytorch-for-r-torch-fix-codegen.patch |  167 +
 ...pytorch-for-r-torch-system-libraries.patch |  390 ++
 .../patches/python-pytorch-runpath.patch      |   19 +-
 .../python-pytorch-system-libraries.patch     |  324 +-
 .../python-pytorch-without-kineto.patch       |   60 +
 .../python-pytorch2-system-libraries.patch    |  156 -
 .../xnnpack-for-torch2-system-libraries.patch | 2660 -----------
 .../patches/xnnpack-remove-broken-tests.patch |  335 ++
 .../patches/xnnpack-system-libraries.patch    | 4133 ++++++++++-------
 gnu/packages/python-build.scm                 |   12 +
 gnu/packages/python-check.scm                 |   88 +
 gnu/packages/python-xyz.scm                   |   29 +
 gnu/packages/serialization.scm                |   53 +-
 26 files changed, 5111 insertions(+), 4964 deletions(-)
 delete mode 100644 gnu/packages/patches/cpuinfo-system-libraries.patch
 create mode 100644 gnu/packages/patches/fbgemm-use-system-libraries.patch
 create mode 100644 gnu/packages/patches/foxi-fix-build.patch
 create mode 100644 gnu/packages/patches/miniz-for-pytorch.patch
 create mode 100644 gnu/packages/patches/python-optree-fix-32-bit.patch
 delete mode 100644 gnu/packages/patches/python-pytorch-1.9.0-system-libraries.patch
 create mode 100644 gnu/packages/patches/python-pytorch-fix-codegen.patch
 create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-fix-codegen.patch
 create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-system-libraries.patch
 create mode 100644 gnu/packages/patches/python-pytorch-without-kineto.patch
 delete mode 100644 gnu/packages/patches/python-pytorch2-system-libraries.patch
 delete mode 100644 gnu/packages/patches/xnnpack-for-torch2-system-libraries.patch
 create mode 100644 gnu/packages/patches/xnnpack-remove-broken-tests.patch