From patchwork Sat Sep 7 10:08:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Graves X-Patchwork-Id: 67712 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 9BAAB27BBEA; Sat, 7 Sep 2024 11:10:42 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_VALIDITY_CERTIFIED, RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE,SPF_HELO_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 06CC327BBE2 for ; Sat, 7 Sep 2024 11:10:42 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1smsOV-0001Ep-AZ; Sat, 07 Sep 2024 06:10:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1smsOM-00019z-0b for guix-patches@gnu.org; Sat, 07 Sep 2024 06:10:06 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1smsOL-0007rR-Gt for guix-patches@gnu.org; Sat, 07 Sep 2024 06:10:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=zzC1fOM+UTR4uoVY8RHcBxdEspGh4ZeosllLJWD0+Ec=; b=UG/xXNbJqYF2VKulti02YNJREf7kJ9ctkMhlqUOdVKUUgt5+lVr5ZfK+kqbqtYVwyed6147lXVJVH29yLvofG2mofdky8mJoEC4NG9w03o61Mj2sWFJIuz5c4uEW4nLYUjiDgvC9nxfLo4Sg5x/6OTyNodD3v76/OEUHifq7Z2bpGJBGzJ6d+Y/ycRLCoFspnyQ35t3KxsKViANyplCydBQ3ZBRhuA58JUe/zm5pytu1jrSUOUdNm7WeC9lYIoIu2WpNtBpPJ9kpVY6xJicdEtvyLN5aIaBnfEbJh5nmpORvA76XC60yzmkZq4T81qoVeqkVPZUPKxCuYqBbYaVlLg==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1smsOM-0001Sl-36 for guix-patches@gnu.org; Sat, 07 Sep 2024 06:10:06 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#73094] [PATCH 10/10] gnu: Add python-safetensors. Resent-From: Nicolas Graves Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 07 Sep 2024 10:10:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73094 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 73094@debbugs.gnu.org Cc: ngraves@ngraves.fr Received: via spool by 73094-submit@debbugs.gnu.org id=B73094.17257037815327 (code B ref 73094); Sat, 07 Sep 2024 10:10:06 +0000 Received: (at 73094) by debbugs.gnu.org; 7 Sep 2024 10:09:41 +0000 Received: from localhost ([127.0.0.1]:54972 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1smsNx-0001Nr-4i for submit@debbugs.gnu.org; Sat, 07 Sep 2024 06:09:41 -0400 Received: from 3.mo561.mail-out.ovh.net ([46.105.44.175]:46007) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1smsNu-0001NU-DL for 73094@debbugs.gnu.org; Sat, 07 Sep 2024 06:09:39 -0400 Received: from director4.ghost.mail-out.ovh.net (unknown [10.108.9.73]) by mo561.mail-out.ovh.net (Postfix) with ESMTP id 4X18344M0lz1Pb4 for <73094@debbugs.gnu.org>; Sat, 7 Sep 2024 10:09:36 +0000 (UTC) Received: from ghost-submission-55b549bf7b-ncpc4 (unknown [10.111.174.38]) by director4.ghost.mail-out.ovh.net (Postfix) with ESMTPS id 4D1B41FD79; Sat, 7 Sep 2024 10:09:35 +0000 (UTC) Received: from ngraves.fr ([37.59.142.98]) by ghost-submission-55b549bf7b-ncpc4 with ESMTPSA id RjfKKl8m3Ga4yQAAUyFbHg (envelope-from ); Sat, 07 Sep 2024 10:09:35 +0000 Authentication-Results: garm.ovh; auth=pass (GARM-98R002f7ce8435-8bef-4d8f-ab8e-b48646af4fee, 9FD0014F98746B3795650381BBEC0664E0950E21) smtp.auth=ngraves@ngraves.fr X-OVh-ClientIp: 81.67.146.208 Date: Sat, 7 Sep 2024 12:08:52 +0200 Message-ID: <20240907100908.25197-10-ngraves@ngraves.fr> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240907100908.25197-1-ngraves@ngraves.fr> References: <20240907100908.25197-1-ngraves@ngraves.fr> MIME-Version: 1.0 X-Ovh-Tracer-Id: 2765210171848909538 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeeftddrudeifedgvdeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefpihgtohhlrghsucfirhgrvhgvshcuoehnghhrrghvvghssehnghhrrghvvghsrdhfrheqnecuggftrfgrthhtvghrnhepkeegjedtieegieetkeejueejgedtkeekvdegheevfeeivdeiuedvhfdvuddtgfetnecuffhomhgrihhnpehhuhhgghhinhhgfhgrtggvrdgtohenucfkphepuddvjedrtddrtddruddpkedurdeijedrudegiedrvddtkedpfeejrdehledrudegvddrleeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpehnghhrrghvvghssehnghhrrghvvghsrdhfrhdpnhgspghrtghpthhtohepuddprhgtphhtthhopeejfedtleegseguvggssghughhsrdhgnhhurdhorhhgpdfovfetjfhoshhtpehmohehiedupdhmohguvgepshhmthhpohhuth DKIM-Signature: a=rsa-sha256; bh=zzC1fOM+UTR4uoVY8RHcBxdEspGh4ZeosllLJWD0+Ec=; c=relaxed/relaxed; d=ngraves.fr; h=From; s=ovhmo4487190-selector1; t=1725703776; v=1; b=YZ1giW+UfB7zcF7TSzKsEHIAYEVPtY5Rd+W8vliwyu1NPHfiyzzE/3szfmi0zdWWredKbC7u JI5Cjs5djj8ANtfG8+oJ5h/q18Lx8sT06FB35JDH/YsEnlS0nbskNJjA8fVqtBOVlMPLkMSwClT OQz9bHez+vVKWm31UFx2NA2fWdFaA2uCURmmz2h2uT7cGyOorR9QSH2KgGAUAMkilZm2S6Rwmpd AnWFrrlqBSBzWGgeXohGY1z9flK4jzNukq0UHju5yfFqvFrsSezOFJSznwNNAUAoxf49Nd5b+O4 LupFNeLHv83BRbAtmMWs4DuQ/6hM07yiqKp1O2WTdjD0w== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Nicolas Graves X-ACL-Warn: , Nicolas Graves via Guix-patches X-Patchwork-Original-From: Nicolas Graves via Guix-patches via From: Nicolas Graves Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * gnu/packages/machine-learning.scm (python-safetensors): New variable. Change-Id: I90a1684d06756ce87ca0862d745a75be5919f0b2 --- gnu/packages/machine-learning.scm | 97 +++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm index a4aeb97be7..12be1d7bf6 100644 --- a/gnu/packages/machine-learning.scm +++ b/gnu/packages/machine-learning.scm @@ -1120,6 +1120,103 @@ (define-public rust-safetensors @code{PyTorch} counterparts.") (license license:asl2.0))) +(define-public python-safetensors + (package + (name "python-safetensors") + (version "0.4.3") + (source + (origin + (method url-fetch) + (uri (pypi-uri "safetensors" version)) + (sha256 + (base32 "1hhiwy67jarm70l0k26fs1cjhzkgzrh79q14bklj2yp0qi8gr19g")) + (modules '((guix build utils) + (ice-9 ftw))) + (snippet + #~(begin ;; Only keeping bindings. + (for-each (lambda (file) + (unless (member file '("." ".." "bindings" "PKG-INFO")) + (delete-file-recursively file))) + (scandir ".")) + (for-each (lambda (file) + (unless (member file '("." "..")) + (rename-file (string-append "bindings/python/" file) file))) + (scandir "bindings/python")))))) + (build-system cargo-build-system) + (arguments + (list + #:imported-modules `(,@%cargo-build-system-modules + ,@%pyproject-build-system-modules) + #:modules '((guix build cargo-build-system) + ((guix build pyproject-build-system) #:prefix py:) + (guix build utils) + (ice-9 regex) + (ice-9 textual-ports)) + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack-rust-crates 'inject-safetensors + (lambda _ + (substitute* "Cargo.toml" + (("\\[dependencies\\]") + (format #f "[dependencies]~%safetensors = ~s" + #$(package-version rust-safetensors)))) + (let ((file-path "Cargo.toml")) + (call-with-input-file file-path + (lambda (port) + (let* ((content (get-string-all port)) + (top-match (string-match + "\\[dependencies.safetensors" content))) + (call-with-output-file file-path + (lambda (out) + (format out "~a" (match:prefix top-match)))))))))) + (replace 'check + (lambda _ + (copy-file "target/release/libsafetensors_rust.so" + "py_src/safetensors/_safetensors_rust.so") + (invoke "python3" + "-c" (format #f + "import sys; sys.path.append(\"~a/py_src\")" + (getcwd)) + "-m" "pytest" + "-n" "auto" + "--dist=loadfile" + "-s" "-v" "./tests/" + "--ignore=./tests/test_flax_comparison.py"))) + (add-after 'install 'install-python + (lambda _ + (let* ((pversion #$(version-major+minor (package-version python))) + (lib (string-append #$output "/lib/python" pversion + "/site-packages/")) + (info (string-append lib "safetensors-" + #$(package-version this-package) + ".dist-info"))) + (mkdir-p info) + (copy-file "PKG-INFO" (string-append info "/METADATA")) + (copy-recursively + "py_src/safetensors" + (string-append lib "safetensors")))))) + #:cargo-inputs + `(("rust-pyo3" ,rust-pyo3-0.21) + ("rust-memmap2" ,rust-memmap2-0.9) + ("rust-safetensors" ,rust-safetensors) + ("rust-serde-json" ,rust-serde-json-1)))) + (inputs + (list rust-safetensors)) + (native-inputs + (list python-h5py + python-minimal + python-numpy + python-pytest + python-pytest-xdist + python-pytorch + tensorflow)) + (home-page "https://huggingface.co/docs/safetensors") + (synopsis "Simple and safe way to store and distribute tensors") + (description "This package provides a fast (zero-copy) and safe +(dedicated) format for storing tensors safely. This package builds upon +@code{rust-safetensors} and provides Python bindings.") + (license license:asl2.0))) + (define-public python-sentencepiece (package (name "python-sentencepiece")