From patchwork Thu Oct 21 13:09:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jean-Baptiste Volatier X-Patchwork-Id: 34000 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 5E77127BBE3; Thu, 21 Oct 2021 14:14:16 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,T_DKIM_INVALID, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 4CC7B27BBE1 for ; Thu, 21 Oct 2021 14:14:15 +0100 (BST) Received: from localhost ([::1]:55830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mdXte-0000KS-Dw for patchwork@mira.cbaines.net; Thu, 21 Oct 2021 09:14:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39640) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mdXqZ-000580-Mt for guix-patches@gnu.org; Thu, 21 Oct 2021 09:11:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44963) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mdXqZ-0007s2-C9 for guix-patches@gnu.org; Thu, 21 Oct 2021 09:11:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mdXqY-0008Oa-Vq for guix-patches@gnu.org; Thu, 21 Oct 2021 09:11:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#51319] [PATCH 1/2] build-system/julia: Enable Julia Pkg to find installed packages Resent-From: Jean-Baptiste Volatier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Oct 2021 13:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51319 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 51319@debbugs.gnu.org Cc: Jean-Baptiste Volatier Received: via spool by 51319-submit@debbugs.gnu.org id=B51319.163482182032193 (code B ref 51319); Thu, 21 Oct 2021 13:11:02 +0000 Received: (at 51319) by debbugs.gnu.org; 21 Oct 2021 13:10:20 +0000 Received: from localhost ([127.0.0.1]:56507 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mdXpr-0008NB-Qv for submit@debbugs.gnu.org; Thu, 21 Oct 2021 09:10:20 -0400 Received: from mail-40133.protonmail.ch ([185.70.40.133]:18355) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mdXpn-0008MV-3U for 51319@debbugs.gnu.org; Thu, 21 Oct 2021 09:10:16 -0400 Date: Thu, 21 Oct 2021 13:09:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1634821804; bh=cEdR2dCvWdAvcCyYVXz5VXAYrVBsAQxdqJl8j64rjac=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=pbz02UYIMykj7eEOIQzUbz+DbjbnkIAGXmXD3yNq3DoFxmPFw4OJ0ju4RWWenUxtL eNOz+UdADI+2imGB3jMG6Kwjo3RARkiibRjBck0/924MN/uK90S1WlFOzkp69i/+Ga ztfxbef9t8TsjFKiTfv4ybv6CO0SAiOoCMzlCNvW0o1EAIPT+7D42wTICmnnCbm5LV l8gB117HAyhxZnjLXQFdfMpevO7CLBppgwLM7oXcmDcb2/EdSIxFwz/Dwy7MHVEXT7 BVhqeqvXQUHj34JHle/DQukzO8FChI2YMF7vMp24mOasoLeE8YSP6SnHk1GGkNlRKZ QCfdsDGaDjLgQ== Message-ID: <20211021130952.14696-2-jbv@pm.me> In-Reply-To: <20211021130952.14696-1-jbv@pm.me> References: <20211021130952.14696-1-jbv@pm.me> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" Reply-to: Jean-Baptiste Volatier X-ACL-Warn: , Jean-Baptiste Volatier via Guix-patches X-Patchwork-Original-From: Jean-Baptiste Volatier via Guix-patches via From: Jean-Baptiste Volatier X-getmail-retrieved-from-mailbox: Patches * guix/build/julia-build-system.scm (link-depot): New phase. Julia built-in package manager (Pkg) looks for packages in JULIA_DEPOT_PATH/packages/PACKAGENAME/XXXX where XXXX is a string encoding package UUID and SHA1 of files. The link-depot phase creates a link at the correct location to allow Pkg to find packages that were already installed by Guix. (%package-path): modified package path from packages/ to loadpath/ (project.toml->uuid): retrive package uuid from TOML file * gnu/packages/julia-xyz.scm: correct paths to reflect change of %package-path * gnu/packages/julia.scm: correct paths to reflect change of %package-path --- gnu/packages/julia-xyz.scm | 2 +- gnu/packages/julia.scm | 2 +- guix/build/julia-build-system.scm | 46 ++++++++++++++++++++++++++++--- 3 files changed, 44 insertions(+), 6 deletions(-) -- 2.33.1 diff --git a/gnu/packages/julia-xyz.scm b/gnu/packages/julia-xyz.scm index c1ebc0fba1..1e407a2277 100644 --- a/gnu/packages/julia-xyz.scm +++ b/gnu/packages/julia-xyz.scm @@ -4610,7 +4610,7 @@ (define-public julia-uris (lambda* (#:key source outputs #:allow-other-keys) (let ((out (assoc-ref outputs "out"))) (chdir - (string-append out "/share/julia/packages/URIs/test"))) + (string-append out "/share/julia/loadpath/URIs/test"))) #t))))) ;; required for tests (inputs `(("julia-json" ,julia-json))) diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm index 411f2e2e10..5b9bf91ed6 100644 --- a/gnu/packages/julia.scm +++ b/gnu/packages/julia.scm @@ -673,7 +673,7 @@ (define-public julia (native-search-paths (list (search-path-specification (variable "JULIA_LOAD_PATH") - (files (list "share/julia/packages/"))) + (files (list "share/julia/loadpath/"))) (search-path-specification (variable "JULIA_DEPOT_PATH") (files (list "share/julia/"))))) diff --git a/guix/build/julia-build-system.scm b/guix/build/julia-build-system.scm index d74acf2a05..c7f2119974 100644 --- a/guix/build/julia-build-system.scm +++ b/guix/build/julia-build-system.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019, 2020 Nicolò Balzarotti +;;; Copyright © 2021 Jean-Baptiste Volatier ;;; ;;; This file is part of GNU Guix. ;;; @@ -20,10 +21,13 @@ (define-module (guix build julia-build-system) #:use-module ((guix build gnu-build-system) #:prefix gnu:) #:use-module (guix build utils) + #:use-module (rnrs io ports) #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (ice-9 rdelim) + #:use-module (ice-9 popen) #:export (%standard-phases + %package-path julia-create-package-toml julia-build)) @@ -37,7 +41,7 @@ (define (invoke-julia code) (invoke "julia" "-e" code)) ;; subpath where we store the package content -(define %package-path "/share/julia/packages/") +(define %package-path "/share/julia/loadpath/") (define (project.toml->name file) "Look for Julia package name in the TOML file FILE (usually named @@ -51,6 +55,18 @@ (define (project.toml->name file) (if m (match:substring m 1) (loop (read-line in 'concat))))))))) +(define (project.toml->uuid file) + "Look for Julia package uuid in the TOML file FILE (usually named +Project.toml)." + (call-with-input-file file + (lambda (in) + (let loop ((line (read-line in 'concat))) + (if (eof-object? line) + #f + (let ((m (string-match "uuid\\s*=\\s*\"(.*)\"" line))) + (if m (match:substring m 1) + (loop (read-line in 'concat))))))))) + (define* (install #:key source inputs outputs julia-package-name #:allow-other-keys) (let* ((out (assoc-ref outputs "out")) @@ -73,7 +89,7 @@ (define* (precompile #:key source inputs outputs julia-package-name (setenv "JULIA_DEPOT_PATH" builddir) ;; Add new package dir to the load path. (setenv "JULIA_LOAD_PATH" - (string-append builddir "packages/" ":" + (string-append builddir "loadpath/" ":" (or (getenv "JULIA_LOAD_PATH") ""))) ;; Actual precompilation: @@ -97,15 +113,36 @@ (define* (check #:key tests? source inputs outputs julia-package-name (setenv "SOURCE_DATE_EPOCH" "1") (setenv "JULIA_DEPOT_PATH" builddir) (setenv "JULIA_LOAD_PATH" - (string-append builddir "packages/" ":" + (string-append builddir "loadpath/" ":" (or (getenv "JULIA_LOAD_PATH") ""))) (setenv "HOME" "/tmp") (invoke "julia" "--depwarn=yes" - (string-append builddir "packages/" + (string-append builddir "loadpath/" package "/test/runtests.jl")))) #t) +(define* (link-depot #:key source inputs outputs julia-package-name + #:allow-other-keys) + (let* ((out (assoc-ref outputs "out")) + (package-name (or + julia-package-name + (project.toml->name "Project.toml"))) + (package-dir (string-append out %package-path package-name)) + (uuid (project.toml->uuid "Project.toml")) + (pipe (open-pipe* OPEN_READ "julia" "-e" + (format #f "using Pkg; +println(Base.version_slug(Base.UUID(\"~a\"), + Base.SHA1(Pkg.GitTools.tree_hash(\".\"))))" uuid package-dir))) + (slug (string-trim-right (get-string-all pipe)))) + ;; When installing a package, julia looks first at in the JULIA_DEPOT_PATH + ;; for a path like packages/PACKAGE/XXXX + ;; Where XXXX is a slug encoding the package UUID and SHA1 of the files + ;; Here we create a link with the correct path to enable julia to find the package + (mkdir-p (string-append out "/share/julia/packages/" package-name)) + (symlink package-dir (string-append out "/share/julia/packages/" package-name "/" slug))) + #t) + (define (julia-create-package-toml outputs source name uuid version deps) @@ -138,6 +175,7 @@ (define %standard-phases (delete 'check) ; tests must be run after installation (replace 'install install) (add-after 'install 'precompile precompile) + (add-after 'unpack 'link-depot link-depot) (add-after 'install 'check check) ;; TODO: In the future we could add a "system-image-generation" phase ;; where we use PackageCompiler.jl to speed up package loading times