From patchwork Sun Jun 4 23:28:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Tai X-Patchwork-Id: 50664 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id CA8F727BBE2; Mon, 5 Jun 2023 00:29:33 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 24FAE27BBE9 for ; Mon, 5 Jun 2023 00:29:31 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q5x9k-0004LW-D4; Sun, 04 Jun 2023 19:29:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q5x9i-0004LC-Jh for guix-patches@gnu.org; Sun, 04 Jun 2023 19:29:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q5x9i-0006Rw-BL for guix-patches@gnu.org; Sun, 04 Jun 2023 19:29:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q5x9i-00022g-6f for guix-patches@gnu.org; Sun, 04 Jun 2023 19:29:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63897] [PATCH] gnu: tensorflow_lite: Update to 2.12.0 Resent-From: Andy Tai Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 04 Jun 2023 23:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 63897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 63897@debbugs.gnu.org Cc: Andy Tai X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16859213117807 (code B ref -1); Sun, 04 Jun 2023 23:29:02 +0000 Received: (at submit) by debbugs.gnu.org; 4 Jun 2023 23:28:31 +0000 Received: from localhost ([127.0.0.1]:47562 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5x9C-00021q-1M for submit@debbugs.gnu.org; Sun, 04 Jun 2023 19:28:30 -0400 Received: from lists.gnu.org ([209.51.188.17]:32822) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5x99-00021g-Oz for submit@debbugs.gnu.org; Sun, 04 Jun 2023 19:28:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q5x99-0004Ip-IN for guix-patches@gnu.org; Sun, 04 Jun 2023 19:28:27 -0400 Received: from mail-oo1-xc2b.google.com ([2607:f8b0:4864:20::c2b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q5x96-0006PR-Sa for guix-patches@gnu.org; Sun, 04 Jun 2023 19:28:27 -0400 Received: by mail-oo1-xc2b.google.com with SMTP id 006d021491bc7-5585f2f070bso3076947eaf.0 for ; Sun, 04 Jun 2023 16:28:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685921303; x=1688513303; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:sender:from:to:cc:subject:date:message-id:reply-to; bh=Ci1kZdGW4TGNATteMQhOyS2wu5DfwZ+MaxA/Kj5zC2A=; b=OnlnhbVlILBbDZUAdch9/Vi0URsYwMYzqlSyAxFol+N+ENoXP98cN1STA/6wUH23W3 8M66bJeAFc+VHZSKpeYrLc/LJbFyPLJ7IoYneR2aXAxz8uG0MeM4DafzwfAryoNjSuHi NVwc3tkt5bMPlLIzqZszgwC52f14jQepIygSZFUOXJz8xjCVEdOmNrterrPrgTmWbNkZ u7MO2cMcbDqrLpmxVAWnPtleNXeqJd6UnRUdjJXeywVYald6M1DpMJTd9LTKNeUpd2yz eqQW3xvtCCdkS7Q3zuIBC3gzf7bV7A9PdehOFvd3hoikV3hLgDTvyluRJSipK3H0mkHP J34Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685921303; x=1688513303; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:sender:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Ci1kZdGW4TGNATteMQhOyS2wu5DfwZ+MaxA/Kj5zC2A=; b=OO5qNCVLb3PVbjyx603IkhFNlPcpUOo0MBqPiK+FJ299+EiES1lEB7Rp72ohgyUpjh FH19+9/ny+5w5GaCTlFvqSAhFSeq+bii0QWdl1IZORJkzP3lcqIoNHBgd4BxsNKNhlZo bo/dNLjA7kcPDLeLrIyfzfCeam+rKWN14YUTcVqSVSuh+y5weQZuXqfjnVAq88nTT5b+ l0G2gz73osVvPcGvBj97a3J4L1LJ6oePy2yTqCZ+0fltGDwIemgZSraf2BxygNUIhsDh ZFTy7cJVHbrn8o/Fj6sEW+FLwfk092cqDSslzyZVhuWIUgsYjsPx6vy/eLS90q6V+uok GWIA== X-Gm-Message-State: AC+VfDwe+RliCARRlzlSNO+CSZc4TSTFrPLGUj0MyUaaNP0NVwNkkHHu 875Oft6Dx/JgLnCdY+GedQ87i0Af4XU= X-Google-Smtp-Source: ACHHUZ45dlSMxFShxRuFs1sUsDehHb+VKrD8LxPSfQeQwc/e/FsYaikC1SyTOHMny95lhdJt48Vm0A== X-Received: by 2002:a05:6359:67a6:b0:129:ca90:3b6c with SMTP id sq38-20020a05635967a600b00129ca903b6cmr309705rwb.17.1685921302920; Sun, 04 Jun 2023 16:28:22 -0700 (PDT) Received: from localhost.localdomain ([98.37.226.98]) by smtp.googlemail.com with ESMTPSA id i7-20020a17090adc0700b0025043a8185dsm4654974pjv.23.2023.06.04.16.28.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 04 Jun 2023 16:28:22 -0700 (PDT) From: Andy Tai Date: Sun, 4 Jun 2023 16:28:18 -0700 Message-Id: X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::c2b; envelope-from=lichengtai@gmail.com; helo=mail-oo1-xc2b.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * gnu/packages/algebra.scm (eigen-for-tensorflow-lite): Remove variable * gnu/packages/machine-learning.scm (tensorflow-lite): Update to 2.12.0 [arguments] (configure-flags): Turn on GPU and RUY use; Use cmake to find packages already in Guix as dependencies for absl-cpp, eigen, flatbuffer, neon2ssl, cpuinfo, ruy. [arguments] (phases): Remove operations setting up absl, eigen, ruy, and neon2ssl as native inouts as now we use Guix's packages of these as dependencies [arguments] (inputs): Add cpuinfo, eigen, fp16, mesa-header, opencl, pthreadpool, ruy, vulkan and xnnpack as explicit inputs [arguments] (native-inputs): Remove local setup of neon2ssl and ruy as we now use Guix's packages of these as explicit dependencies * gnu/packages/maths.scm: Remove extra statement * gnu/packages/serialization.scm (flatbuffers-next-shared): New variable, flatbuffers-next built with -fPIC as needed by tensorflow-lite --- gnu/packages/algebra.scm | 32 -------- gnu/packages/machine-learning.scm | 118 ++++++++++++++++-------------- gnu/packages/maths.scm | 1 - gnu/packages/serialization.scm | 12 +++ 4 files changed, 75 insertions(+), 88 deletions(-) base-commit: eed55a6544d5bda2245ec853e5fa4b28e1865bea prerequisite-patch-id: 5e9e341f5896c5eee0f64a7a031cc57491e76eb9 prerequisite-patch-id: b17d63ac48fd66475dda11cd37f74bb63aca5cfb diff --git a/gnu/packages/algebra.scm b/gnu/packages/algebra.scm index 3babf90b3c..81df0c97ac 100644 --- a/gnu/packages/algebra.scm +++ b/gnu/packages/algebra.scm @@ -1155,38 +1155,6 @@ (define-public eigen-for-tensorflow (substitute* "test/CMakeLists.txt" (("ei_add_test\\(stddeque") "#"))))))))))) -(define-public eigen-for-tensorflow-lite - ;; This commit was taken from - ;; tensorflow/lite/tools/cmake/modules/eigen.cmake - (let ((commit "d10b27fe37736d2944630ecd7557cefa95cf87c9") - (revision "1")) - (package (inherit eigen) - (name "eigen-for-tensorflow-lite") - (version (git-version "3.3.7" revision commit)) - (source (origin - (method git-fetch) - (uri (git-reference - (url "https://gitlab.com/libeigen/eigen") - (commit commit))) - (sha256 - (base32 - "0v8a20cwvwmp3hw4275b37frw33v92z0mr8f4dn6y8k0rz92hrrf")) - (file-name (git-file-name name version)) - (modules '((guix build utils))) - (snippet - ;; Ther are test failures in the "unsupported" directory, but - ;; maintainers say it's unsupported anyway, so just skip - ;; them. - '(begin - (substitute* "unsupported/CMakeLists.txt" - (("add_subdirectory\\(test.*") - "# Do not build the tests for unsupported features.\n")))))) - (arguments - (substitute-keyword-arguments (package-arguments eigen) - ((#:phases phases) - `(modify-phases ,phases - (delete 'disable-some-tests)))))))) - (define-public xtensor (package (name "xtensor") diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm index c9bc5e8b68..15fe4be995 100644 --- a/gnu/packages/machine-learning.scm +++ b/gnu/packages/machine-learning.scm @@ -54,6 +54,7 @@ (define-module (gnu packages machine-learning) #:use-module (gnu packages) #:use-module (gnu packages adns) #:use-module (gnu packages algebra) + #:use-module (gnu packages assembly) #:use-module (gnu packages audio) #:use-module (gnu packages autotools) #:use-module (gnu packages base) @@ -88,6 +89,7 @@ (define-module (gnu packages machine-learning) #:use-module (gnu packages ninja) #:use-module (gnu packages ocaml) #:use-module (gnu packages onc-rpc) + #:use-module (gnu packages opencl) #:use-module (gnu packages parallel) #:use-module (gnu packages perl) #:use-module (gnu packages pkg-config) @@ -107,6 +109,7 @@ (define-module (gnu packages machine-learning) #:use-module (gnu packages swig) #:use-module (gnu packages time) #:use-module (gnu packages tls) + #:use-module (gnu packages vulkan) #:use-module (gnu packages video) #:use-module (gnu packages web) #:use-module (gnu packages xml) @@ -2642,7 +2645,7 @@ (define-public tensorflow (define-public tensorflow-lite (package (name "tensorflow-lite") - (version "2.5.0") + (version "2.12.0") (source (origin (method git-fetch) @@ -2652,35 +2655,53 @@ (define-public tensorflow-lite (file-name (git-file-name name version)) (sha256 (base32 - "1jdw2i1rq06zqd6aabh7bbm0avsg4pygnfmd7gviv0blhih9054l")))) + "051knmq6ysg4936d9baaqsx6j3rfsp4ffi2pwrxx4llan682rvpi")))) (build-system cmake-build-system) (arguments `(#:tests? #false ; no "check" target #:build-type "Release" #:configure-flags (list - "-DTFLITE_ENABLE_GPU=OFF" - "-DTFLITE_ENABLE_RUY=OFF" + ;"--trace" + ;; Use Guix's own versions of library dependencies. + "-DTFLITE_ENABLE_INSTALL=ON" ;; so cmake can be used to find this from other packages + "-DCMAKE_FIND_PACKAGE_PREFER_CONFIG=ON" + "-DTFLITE_ENABLE_GPU=ON" + "-DTFLITE_ENABLE_RUY=ON" + + ;; TO DO: turn on Farmhash + ;"-DSYSTEM_FARMHASH=ON" + + + (string-append "-Dabsl_DIR=" (assoc-ref %build-inputs "abseil-cpp") + "/lib/cmake/absl") + (string-append "-DEigen3_DIR=" (assoc-ref %build-inputs "eigen") + "/share/eigen3/cmake") + (string-append "-DFlatBuffers_DIR=" (assoc-ref %build-inputs "flatbuffers") + "/lib/cmake/flatbuffers") + (string-append "-DNEON_2_SSE_DIR=" (assoc-ref %build-inputs "neon2sse") + "/lib/cmake/NEON_2_SSE") + (string-append "-Dcpuinfo_DIR=" (assoc-ref %build-inputs "cpuinfo") + "/share/cpuinfo") + (string-append "-Druy_DIR=" (assoc-ref %build-inputs "ruy") + "/lib/cmake/ruy") ;; TODO: The build system attempts to build xnnpack from source. We ;; would like to use our xnnpack package here, but this requires more ;; work. "-DTFLITE_ENABLE_XNNPACK=OFF" - ;; Pretend we've already fetched abseil. We won't actually build it - ;; but use the existing package. - "-Dabseil-cpp_POPULATED=TRUE" - - ;; Don't fetch the sources. We have already built flatbuffers. - "-Dflatbuffers_POPULATED=TRUE" + ;;; Don't fetch the sources. We have these already + "-Degl_headers_POPULATED=TRUE" + "-Dfp16_headers_POPULATED=TRUE" + "-Dopencl_headers_POPULATED=TRUE" + "-Dopengl_headers_POPULATED=TRUE" + "-Dvulkan_headers_POPULATED=TRUE" "-DFFT2D_SOURCE_DIR=/tmp/fft2d" - "-Dneon2sse_SOURCE_DIR=/tmp/neon2sse" - "-Dneon2sse_BINARY_DIR=/tmp/neon2sse-bin" "-DFARMHASH_SOURCE_DIR=/tmp/farmhash" "-Dgemmlowp_SOURCE_DIR=/tmp/gemmlowp" - (string-append "-DRUY_SOURCE_DIR=" - (assoc-ref %build-inputs "ruy-src"))) + ) #:phases (modify-phases %standard-phases (add-after 'unpack 'chdir @@ -2689,20 +2710,19 @@ (define-public tensorflow-lite (lambda* (#:key inputs #:allow-other-keys) ;; Use external cmake finders instead of these stubs that won't ;; find anything but the bundled sources. - (delete-file "tools/cmake/modules/Findabsl.cmake") - (delete-file "tools/cmake/modules/Findeigen.cmake") + ;; TO DO: properly use Guix's pthreaqdpool + ;; roght now not using pthreadpool because + ;; we are not enabling xnnpack + (substitute* "CMakeLists.txt" + (("if\\(NOT DEFINED PTHREADPOOL_SOURCE_DIR\\)") + "if(false)")) (substitute* "CMakeLists.txt" - (("find_package\\(eigen REQUIRED") - "find_package(eigen REQUIRED NAMES Eigen3")) - (substitute* "tools/cmake/modules/Findflatbuffers.cmake" - (("get_target_property.*") - (format #false "set(FLATBUFFERS_INCLUDE_DIRS ~a/include)\n" - (assoc-ref inputs "flatbuffers")))) + (("if\\(NOT TARGET pthreadpool\\)") + "if(false)")) ;; Don't fetch source code; we already have everything we need. (substitute* '("tools/cmake/modules/fft2d.cmake" - "tools/cmake/modules/ruy.cmake" "tools/cmake/modules/farmhash.cmake" "tools/cmake/modules/neon2sse.cmake" "tools/cmake/modules/gemmlowp.cmake") @@ -2718,10 +2738,9 @@ (define-public tensorflow-lite (invoke "tar" "--strip-components=1" "-xf" (assoc-ref inputs "fft2d-src"))) - (copy-recursively (assoc-ref inputs "neon2sse-src") - "/tmp/neon2sse/") (copy-recursively (assoc-ref inputs "gemmlowp-src") - "/tmp/gemmlowp/"))) + "/tmp/gemmlowp/") + )) (add-after 'copy-sources 'prepare-shared-library-build (lambda _ (chdir "c"))) (replace 'install @@ -2738,9 +2757,21 @@ (define-public tensorflow-lite (find-files "." "\\.h$"))))))))) (inputs `(("abseil-cpp" ,abseil-cpp-20200923.3) - ("eigen" ,eigen-for-tensorflow-lite) - ("flatbuffers" ,flatbuffers) - ("python" ,python))) + ("cpuinfo" ,cpuinfo) + ("eigen" ,eigen) + ("fp16" ,fp16) + ("flatbuffers" ,flatbuffers-next-shared) + ;("gemmlowp" ,gemmlowp) ;; TO DO + ("mesa-headers" ,mesa-headers) + ("neon2sse" ,neon2sse) + ("opencl-clhpp" ,opencl-clhpp) + ("opencl-headers" ,opencl-headers) + ("opencl-icd-loader" ,opencl-icd-loader) + ("pthreadpool" ,pthreadpool) + ("python" ,python) + ("ruy" ,ruy) + ;("xnnpack" ,xnnpack) ;; TO DO: properly use Guix's copy of xnnpack + ("vulkan-headers" , vulkan-headers))) (native-inputs `(("pkg-config" ,pkg-config) ("gemmlowp-src" @@ -2756,17 +2787,6 @@ (define-public tensorflow-lite (sha256 (base32 "1sbp8kmr2azwlvfbzryy1frxi99jhsh1nc93bdbxdf8zdgpv0kxl"))))) - ("neon2sse-src" - ,(let ((commit "a1652fd5253afbf3e39357b012974f93511f6108")) - (origin - (method git-fetch) - (uri (git-reference - (url "https://github.com/intel/ARM_NEON_2_x86_SSE") - (commit commit))) - (file-name (git-file-name "neon2sse" (string-take commit 8))) - (sha256 - (base32 - "1q8gkxag9wlnwdwad2pclsrkwzrdjy94hyrkayrsvxyj7szb5y8i"))))) ("farmhash-src" ,(let ((commit "816a4ae622e964763ca0862d9dbd19324a1eaf45")) (origin @@ -2782,24 +2802,12 @@ (define-public tensorflow-lite ,(origin (method url-fetch) (uri (string-append "https://storage.googleapis.com/" - "mirror.tensorflow.org/" - "www.kurims.kyoto-u.ac.jp/~ooura/fft2d.tgz")) + "mirror.tensorflow.org/github.com/petewarden/" + "OouraFFT/archive/v1.0.tar.gz")) (file-name "fft2d.tar.gz") (sha256 (base32 - "1jfflzi74fag9z4qmgwvp90aif4dpbr1657izmxlgvf4hy8fk9xd")))) - ("ruy-src" - ,(let ((commit "9c56af3fce210a8a103eda19bd6f47c08a9e3d90")) - (origin - (method git-fetch) - (uri (git-reference - (url "https://github.com/google/ruy") - (commit commit) - (recursive? #true))) - (file-name (git-file-name "ruy" (string-take commit 8))) - (sha256 - (base32 - "1cfd5gk6kaj8kbl3h98gx1ap8czd59y6p8qq8nr28fklpyzf5cis"))))))) + "1jfflzi74fag9z4qmgwvp90aif4dpbr1657izmxlgvf4hy8fk9xd")))))) (home-page "https://tensorflow.org") (synopsis "Machine learning framework") (description diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm index c6fabf1973..25a7b61156 100644 --- a/gnu/packages/maths.scm +++ b/gnu/packages/maths.scm @@ -85,7 +85,6 @@ (define-module (gnu packages maths) #:use-module (guix packages) #:use-module (guix download) #:use-module (guix git-download) - #:use-module (guix gexp) #:use-module (guix utils) #:use-module ((guix build utils) #:select (alist-replace)) #:use-module (guix build-system ant) diff --git a/gnu/packages/serialization.scm b/gnu/packages/serialization.scm index 39954d6764..268a5d3d9b 100644 --- a/gnu/packages/serialization.scm +++ b/gnu/packages/serialization.scm @@ -748,6 +748,18 @@ (define-public flatbuffers-next (base32 "1z3a6l8g2y53i5xzraswfs2i0i3kk52zv7nzc2q3fgisbyiri3pz")))))) +(define-public flatbuffers-next-shared + (package + (inherit flatbuffers-next) + (name "flatbuffers-shared") + (version "23.1.21") + (arguments + (substitute-keyword-arguments (package-arguments flatbuffers-next) + ((#:configure-flags flags) + ;; make C++ fiels compiled with -fPIC, needed for shared lib". + #~(cons "-DFLATBUFFERS_CXX_FLAGS=-fPIC" #$flags)))))) + + (define-public python-feather-format (package (name "python-feather-format")