From patchwork Wed Mar 6 19:37:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Elsing X-Patchwork-Id: 2340 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id D6BB827BBE2; Wed, 6 Mar 2024 19:38:55 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 65F6727BBE9 for ; Wed, 6 Mar 2024 19:38:50 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rhx60-0001FY-Si; Wed, 06 Mar 2024 14:38:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhx5z-0001FB-3m for guix-patches@gnu.org; Wed, 06 Mar 2024 14:38:31 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rhx5y-0005WD-SG for guix-patches@gnu.org; Wed, 06 Mar 2024 14:38:30 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rhx6T-00084e-Pv for guix-patches@gnu.org; Wed, 06 Mar 2024 14:39:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69591] [PATCH 00/31] Unbundle and update python-pytorch Resent-From: David Elsing Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 06 Mar 2024 19:39:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 69591 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69591@debbugs.gnu.org Cc: David Elsing X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.170975390430977 (code B ref -1); Wed, 06 Mar 2024 19:39:01 +0000 Received: (at submit) by debbugs.gnu.org; 6 Mar 2024 19:38:24 +0000 Received: from localhost ([127.0.0.1]:51488 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rhx5o-00083T-RI for submit@debbugs.gnu.org; Wed, 06 Mar 2024 14:38:24 -0500 Received: from lists.gnu.org ([209.51.188.17]:43452) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rhx5k-00083G-3S for submit@debbugs.gnu.org; Wed, 06 Mar 2024 14:38:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhx5E-00019s-RP for guix-patches@gnu.org; Wed, 06 Mar 2024 14:37:44 -0500 Received: from mout01.posteo.de ([185.67.36.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhx5C-0005PS-9w for guix-patches@gnu.org; Wed, 06 Mar 2024 14:37:44 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 5FBB8240028 for ; Wed, 6 Mar 2024 20:37:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1709753858; bh=UtszHbtRuqDAXRr8BuGVCJHzEvT/t0o/9sn52hTMd8g=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=pVaAe2JMVl7VEVJxt81N8EWSxypIiccje8ONOZDYFVrLbzMjQgRVsIr/GAWNL3BZY wDEKrWN0GMdU85dmIkwkGcuTwjseV1VgM3/ELvL1PlWIYE6xI2+YxZis6YeuM2PCIb k/bnxaNT1YmhTolHaPAe2i1gV3JWsjDm7Ozxs29OKUiAjFwoFfMiqmGUBjqhlnagry 6Sl5HsVhtZPpiiMFGtl7EweVjGHvpbgn1NXMZnamClGtdTiBR+pHj6BTgAuzehULtL MuZNxzso6bJklsMxSYwARmLQJouzvBr+Tp/14o++lujZKoPQWozYRByg5XEWPu5Myu NJ9StNUvfWN1w== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4TqjPs6b2dz9rxQ; Wed, 6 Mar 2024 20:37:37 +0100 (CET) From: David Elsing Date: Wed, 6 Mar 2024 19:37:23 +0000 Message-ID: <20240306193725.17463-1-david.elsing@posteo.net> MIME-Version: 1.0 Received-SPF: pass client-ip=185.67.36.65; envelope-from=david.elsing@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches This patch series unbundles all dependencies of python-pytorch (updated to version 2.2.1) and removes all* autogenerated files, completing the work by Ludo' [1]. I also applied the unbundling to python-pytorch-for-r-torch (version 2.0.1) and made sure that r-torch still builds. Originally, I wanted to package PyTorch with ROCm, but decided that it would be better to first update the package and complete the unbundling. This was more work than I initially thought though, so this patch series is quite long. Caffe2 is heavily integrated in the PyTorch code, so unbundling it does not make sense. For QNNPACK, the situation is strange: There is an old version [2] and an internal fork in aten/src/ATen/native/quantized/cpu/qnnpack, which PyTorch both depends on (as well as XNNPACK). The internal fork does not depend on other parts of the code currently, so I unbundled it as well as a package variant of qnnpack. ideep [3] has a separate branch for PyTorch (the master branch does not seem maintained), which only contains header files without build instructions. Therefore, I made a simple package with the header files. miniz also requires a package variant for PyTorch. Several scripts for autogenerated files depend on the compiled PyTorch library. However, it was possible to patch the scripts with empty values and later regenerate the files with the initial PyTorch version. Therefore, I introduced the 'codegen2, 'build2 and 'install2 build phases running after the initial library is installed. In 'codegen2, the installed library is used to regenerate the files properly, which leads to a recompilation of only 12 files in the 'build2 phase. The core tests can be run as well, but I disabled them as they take a long time after an already long build. I also wanted to try and build python-pytorch on i686-linux, as it doesn't require emulation on an x86_64 CPU, but it depends on Rust through python-cryptography, for which there are no substitutes available currently. Cheers, David * with the exception of test/cpp/api/init_baseline.h and test/cpp/api/optim_baseline.h, which are just lists of previously calculated numbers; or any other generated file I might have missed. [1] https://hpc.guix.info/blog/2021/09/whats-in-a-package/ [2] https://github.com/pytorch/QNNPACK [3] https://github.com/intel/ideep/tree/pytorch David Elsing (31): gnu: asmjit: Update to commit 3ca5c18. gnu: Add python-typing-extensions-4.10. gnu: Add python-optree. gnu: Add python-pytest-flakefinder. gnu: Add python-pytest-shard. gnu: Add python-expecttest. gnu: Add python-pytest-rerunfailures-13. gnu: Add miniz. gnu: Add miniz-for-pytorch. gnu: Add libnop. gnu: Remove flatbuffers-next-shared. gnu: python-flatbuffers-next: Update to 23.5.26. gnu: pthreadpool: Update to commit 178e3e0. gnu: cpuinfo: Update to commit aa4b216. gnu: clog: Add "-DUSE_SYSTEM_LIBS=ON" configure flag. gnu: nnpack: Update to commit 70a77f4. gnu: oneapi-dnnl: Update to 3.3.5. gnu: Add tensorpipe. gnu: Add fbgemm. gnu: Add qnnpack. gnu: Add foxi. gnu: Add ideep-pytorch. gnu: xnnpack: Update to commit 51a9875. gnu: Remove xnnpack-for-torch2. gnu: Add qnnpack-pytorch. gnu: python-pytorch: Update to 2.2.1 and unbundle dependencies. gnu: python-torchvision: Update to 0.17.1. gnu: Add ideep-pytorch-for-r-torch. gnu: Add oneapi-dnnl-for-r-torch. gnu: Add qnnpack-pytorch-for-r-torch. gnu: python-pytorch-for-r-torch: Adjust to new python-pytorch. gnu/local.mk | 9 +- gnu/packages/compression.scm | 49 + gnu/packages/cpp.scm | 6 +- gnu/packages/machine-learning.scm | 874 +++- gnu/packages/parallel.scm | 30 +- .../patches/clog-fix-shared-build.patch | 39 +- .../patches/cpuinfo-system-libraries.patch | 50 - .../patches/fbgemm-use-system-libraries.patch | 62 + gnu/packages/patches/foxi-fix-build.patch | 55 + gnu/packages/patches/miniz-for-pytorch.patch | 47 + .../patches/python-optree-fix-32-bit.patch | 122 + ...ython-pytorch-1.9.0-system-libraries.patch | 139 - .../patches/python-pytorch-fix-codegen.patch | 167 + ...thon-pytorch-for-r-torch-fix-codegen.patch | 167 + ...pytorch-for-r-torch-system-libraries.patch | 390 ++ .../patches/python-pytorch-runpath.patch | 19 +- .../python-pytorch-system-libraries.patch | 324 +- .../python-pytorch-without-kineto.patch | 60 + .../python-pytorch2-system-libraries.patch | 156 - .../xnnpack-for-torch2-system-libraries.patch | 2660 ----------- .../patches/xnnpack-remove-broken-tests.patch | 335 ++ .../patches/xnnpack-system-libraries.patch | 4133 ++++++++++------- gnu/packages/python-build.scm | 12 + gnu/packages/python-check.scm | 88 + gnu/packages/python-xyz.scm | 29 + gnu/packages/serialization.scm | 53 +- 26 files changed, 5111 insertions(+), 4964 deletions(-) delete mode 100644 gnu/packages/patches/cpuinfo-system-libraries.patch create mode 100644 gnu/packages/patches/fbgemm-use-system-libraries.patch create mode 100644 gnu/packages/patches/foxi-fix-build.patch create mode 100644 gnu/packages/patches/miniz-for-pytorch.patch create mode 100644 gnu/packages/patches/python-optree-fix-32-bit.patch delete mode 100644 gnu/packages/patches/python-pytorch-1.9.0-system-libraries.patch create mode 100644 gnu/packages/patches/python-pytorch-fix-codegen.patch create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-fix-codegen.patch create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-system-libraries.patch create mode 100644 gnu/packages/patches/python-pytorch-without-kineto.patch delete mode 100644 gnu/packages/patches/python-pytorch2-system-libraries.patch delete mode 100644 gnu/packages/patches/xnnpack-for-torch2-system-libraries.patch create mode 100644 gnu/packages/patches/xnnpack-remove-broken-tests.patch