Message ID | 20210209105843.13230-1-efraim@flashner.co.il |
---|---|
State | Accepted |
Headers | show |
Series | [bug#46399] build-system/cargo: Propagate crates across builds. | expand |
Context | Check | Description |
---|---|---|
cbaines/submitting builds | success | |
cbaines/comparison | success | View comparision |
cbaines/git branch | success | View Git branch |
cbaines/applying patch | success | View Laminar job |
cbaines/issue | success | View issue |
Pushed with minor revisions. Also patched a few packages which needed #:install-source #? in order to continue to build.
Hello Efraim! Thank you for improving Rust development with Guix. I have a question about this change to doc/guix.texi when translating. Sorry for noticing so late just before the string freeze. On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote: > diff --git a/doc/guix.texi b/doc/guix.texi > index 7d18703283..0d153a1470 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > […] > @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the > It adds @code{rustc} and @code{cargo} to the set of inputs. > A different Rust package can be specified with the @code{#:rust} parameter. > > -Regular cargo dependencies should be added to the package definition via the > -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the > +Regular cargo dependencies should be added to the package definition similarly > +to other packages; those needed only at build time to native-inputs, others to > +inputs. If you need to add source-only crates then you should add them to via > +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the > spec can be a package or a source definition. Note that the spec must > evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml} > file at its root, or it will be ignored. Similarly, cargo dev-dependencies I have no experience in Rust. Do I understand correctly that source-only crates among the inputs should be added both to inputs and to #:cargo-inputs? Is the > +inputs. If you need to add source-only crates then you should add them to via a typo which should read “add them too” instead of “add them to”? I look at the subsequent change f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons > diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm > index d67e9dc02f..7dde00ffdb 100644 > --- a/gnu/packages/crates-graphics.scm > +++ b/gnu/packages/crates-graphics.scm > @@ -2415,8 +2415,11 @@ the wayland protocol, client side.") > (arguments > - `(#:skip-build? #t > - #:cargo-inputs > + `(#:cargo-inputs > (("rust-nix" ,rust-nix-0.18) > ("rust-once-cell" ,rust-once-cell-1) > - ("rust-smallvec" ,rust-smallvec-1) > - ("rust-wayland-sys" ,rust-wayland-sys-0.28)))) > + ("rust-smallvec" ,rust-smallvec-1)))) > + (inputs > + `(("rust-nix" ,rust-nix-0.18) > + ("rust-once-cell" ,rust-once-cell-1) > + ("rust-smallvec" ,rust-smallvec-1) > + ("rust-wayland-sys" ,rust-wayland-sys-0.28))) > (home-page "https://github.com/smithay/wayland-rs") and do not understand why rust-wayland-sys is only part of inputs but rust-once-cell is also among the cargo-inputs. Also the inputs are empty for older versions of rust-wayland-commons. Could you clarify what is written in the doc/guix.texi? Regards, Florian
On Sun, Apr 11, 2021 at 05:11:21PM +0200, pelzflorian (Florian Pelz) wrote: > Hello Efraim! > > Thank you for improving Rust development with Guix. I have a question > about this change to doc/guix.texi when translating. Sorry for > noticing so late just before the string freeze. > > On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote: > > diff --git a/doc/guix.texi b/doc/guix.texi > > index 7d18703283..0d153a1470 100644 > > --- a/doc/guix.texi > > +++ b/doc/guix.texi > > […] > > @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the > > It adds @code{rustc} and @code{cargo} to the set of inputs. > > A different Rust package can be specified with the @code{#:rust} parameter. > > > > -Regular cargo dependencies should be added to the package definition via the > > -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the > > +Regular cargo dependencies should be added to the package definition similarly > > +to other packages; those needed only at build time to native-inputs, others to > > +inputs. If you need to add source-only crates then you should add them to via > > +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the > > spec can be a package or a source definition. Note that the spec must > > evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml} > > file at its root, or it will be ignored. Similarly, cargo dev-dependencies > > I have no experience in Rust. Do I understand correctly that > source-only crates among the inputs should be added both to inputs and > to #:cargo-inputs? Is the > > > +inputs. If you need to add source-only crates then you should add them to via > > a typo which should read “add them too” instead of “add them to”? I went back and refreshed myself with what I wrote there. source-only crates should be added to both regular inputs and to #:cargo-inputs. Actually converting the ~1000 rust packages to actually make good use of the changes will allow us to drop nearly all the #:cargo-inputs and just use regular inputs, but until then we still need the propagated source magic that comes from #:cargo-inputs. Yes, definitely a typo, it should be "add them too". > I look at the subsequent change > f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons > > > diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm > > index d67e9dc02f..7dde00ffdb 100644 > > --- a/gnu/packages/crates-graphics.scm > > +++ b/gnu/packages/crates-graphics.scm > > @@ -2415,8 +2415,11 @@ the wayland protocol, client side.") > > (arguments > > - `(#:skip-build? #t > > - #:cargo-inputs > > + `(#:cargo-inputs > > (("rust-nix" ,rust-nix-0.18) > > ("rust-once-cell" ,rust-once-cell-1) > > - ("rust-smallvec" ,rust-smallvec-1) > > - ("rust-wayland-sys" ,rust-wayland-sys-0.28)))) > > + ("rust-smallvec" ,rust-smallvec-1)))) > > + (inputs > > + `(("rust-nix" ,rust-nix-0.18) > > + ("rust-once-cell" ,rust-once-cell-1) > > + ("rust-smallvec" ,rust-smallvec-1) > > + ("rust-wayland-sys" ,rust-wayland-sys-0.28))) > > (home-page "https://github.com/smithay/wayland-rs") > > and do not understand why rust-wayland-sys is only part of inputs but > rust-once-cell is also among the cargo-inputs. Also the inputs are > empty for older versions of rust-wayland-commons. #:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs to future builds. The changes to the cargo-build-system mimic this so we can use regular inputs and not #:cargo-inputs. rust-wayland-sys-0.28 has enough of its dependencies converted that it can be used as just an input but rust-once-cell-1 hasn't yet so it still needs to be in #:cargo-inputs. As far as the other versions of rust-wayland-commons, I started by focusing on alacritty so we could remove some of the build phases and move them to the crates which actually needed patching. > Could you clarify what is written in the doc/guix.texi? > > Regards, > Florian I think for now the best option is to add the rust inputs of all types to both inputs and #:cargo-inputs (or native-inputs and #:cargo-development-inputs) and then remove them from #:cargo-inputs when it's possible to do so. Ultimately Someone™ should revive the wip-rust branch and convert all the crates over in one go and adjust the importer and manual.
On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote: > I went back and refreshed myself with what I wrote there. source-only > crates should be added to both regular inputs and to #:cargo-inputs. Thank you for clarifying. I will translate the manual accordingly. > #:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs > to future builds. I had not known about this difference. > I think for now the best option is to add the rust inputs of all types > to both inputs and #:cargo-inputs (or native-inputs and > #:cargo-development-inputs) and then remove them from #:cargo-inputs > when it's possible to do so. Ultimately Someone™ should revive the > wip-rust branch and convert all the crates over in one go and adjust the > importer and manual. I suppose this paragraph of your reply should be added to the manual. Regards, Florian
On Tue, Apr 13, 2021 at 02:31:39PM +0200, pelzflorian (Florian Pelz) wrote: > On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote: > > I went back and refreshed myself with what I wrote there. source-only > > crates should be added to both regular inputs and to #:cargo-inputs. > > Thank you for clarifying. I will translate the manual accordingly. We are in "string freeze" for the upcoming release, so please delay your changes to the manual until the release is completed.
diff --git a/doc/guix.texi b/doc/guix.texi index 7d18703283..0d153a1470 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -32,7 +32,7 @@ Copyright @copyright{} 2015, 2016, 2017, 2019, 2020 Leo Famulari@* Copyright @copyright{} 2015, 2016, 2017, 2018, 2019, 2020 Ricardo Wurmus@* Copyright @copyright{} 2016 Ben Woodcroft@* Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@* -Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@* +Copyright @copyright{} 2016, 2017, 2018, 2019, 2020, 2021 Efraim Flashner@* Copyright @copyright{} 2016 John Darrington@* Copyright @copyright{} 2016, 2017 Nikita Gillmann@* Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@* @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the It adds @code{rustc} and @code{cargo} to the set of inputs. A different Rust package can be specified with the @code{#:rust} parameter. -Regular cargo dependencies should be added to the package definition via the -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the +Regular cargo dependencies should be added to the package definition similarly +to other packages; those needed only at build time to native-inputs, others to +inputs. If you need to add source-only crates then you should add them to via +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the spec can be a package or a source definition. Note that the spec must evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml} file at its root, or it will be ignored. Similarly, cargo dev-dependencies @@ -7449,8 +7451,11 @@ In its @code{configure} phase, this build system will make any source inputs specified in the @code{#:cargo-inputs} and @code{#:cargo-development-inputs} parameters available to cargo. It will also remove an included @code{Cargo.lock} file to be recreated by @code{cargo} during the -@code{build} phase. The @code{install} phase installs the binaries -defined by the crate. +@code{build} phase. The @code{package} phase will run @code{cargo package} +to create a source crate for future use. The @code{install} phase installs +the binaries defined by the crate. Unless @code{install-source? #f} is +defined it will also install a source crate repository of itself and unpacked +sources, to ease in future hacking on rust packages. @end defvr @defvr {Scheme Variable} chicken-build-system diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm index 6c8edf6bac..01667ef045 100644 --- a/guix/build-system/cargo.scm +++ b/guix/build-system/cargo.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> +;;; Copyright © 2021 Efraim Flashner <efraim@flashner.co.il> ;;; ;;; This file is part of GNU Guix. ;;; @@ -77,8 +78,10 @@ to NAME and VERSION." (vendor-dir "guix-vendor") (cargo-build-flags ''("--release")) (cargo-test-flags ''("--release")) + (cargo-package-flags ''("--no-metadata" "--no-verify")) (features ''()) (skip-build? #f) + (install-source? #t) (phases '(@ (guix build cargo-build-system) %standard-phases)) (outputs '("out")) @@ -106,6 +109,7 @@ to NAME and VERSION." #:vendor-dir ,vendor-dir #:cargo-build-flags ,cargo-build-flags #:cargo-test-flags ,cargo-test-flags + #:cargo-package-flags ,cargo-package-flags #:features ,features #:skip-build? ,skip-build? #:tests? ,(and tests? (not skip-build?)) diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm index 1d21b33895..0328ede4cb 100644 --- a/guix/build/cargo-build-system.scm +++ b/guix/build/cargo-build-system.scm @@ -2,7 +2,7 @@ ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com> -;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il> +;;; Copyright © 2019, 2020, 2021 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2020 Marius Bakke <marius@gnu.org> ;;; @@ -73,6 +73,38 @@ Cargo.toml file present at its root." " | cut -d/ -f2" " | grep -q '^Cargo.toml$'"))))) +(define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys) + (define (inputs->rust-inputs inputs) + "Filter using the label part from INPUTS." + (filter (lambda (input) + (match input + ((name . _) (rust-package? name)))) + inputs)) + (define (inputs->directories inputs) + "Extract the directory part from INPUTS." + (match inputs + (((names . directories) ...) + directories))) + + (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs)))) + (unless (null? rust-inputs) + (mkdir-p "target/package") + (mkdir-p vendor-dir) + ;; TODO: copy only regular inputs to target/package, not native-inputs. + (for-each (lambda (input-crate) + (copy-recursively (string-append input-crate + "/share/cargo/registry") + "target/package")) + (delete-duplicates rust-inputs)) + + (for-each (lambda (crate) + (invoke "tar" "xzf" crate "-C" vendor-dir)) + (find-files "target/package" "\\.crate$")))) + #t) + +(define (rust-package? name) + (string-prefix? "rust-" name)) + (define* (configure #:key inputs (vendor-dir "guix-vendor") #:allow-other-keys) @@ -170,9 +202,23 @@ directory = '" port) (apply invoke "cargo" "test" cargo-test-flags) #t)) -(define* (install #:key inputs outputs skip-build? features #:allow-other-keys) +(define* (package #:key + (cargo-package-flags '("--no-metadata" "--no-verify")) + #:allow-other-keys) + "Run 'cargo-package' for a given Cargo package." + (apply invoke `("cargo" "package" ,@cargo-package-flags))) + +(define* (install #:key + inputs + outputs + skip-build? + (install-source? #t) + features + #:allow-other-keys) "Install a given Cargo package." - (let* ((out (assoc-ref outputs "out"))) + (let* ((out (assoc-ref outputs "out")) + (registry (string-append out "/share/cargo/registry")) + (sources (string-append out "/share/cargo/src"))) (mkdir-p out) ;; Make cargo reuse all the artifacts we just built instead @@ -186,6 +232,18 @@ directory = '" port) (invoke "cargo" "install" "--no-track" "--path" "." "--root" out "--features" (string-join features))) + (when install-source? + ;; Install crate tarballs and unpacked sources for later use. + ;; TODO: Is there a better format/directory for these files? + (mkdir-p sources) + (for-each (lambda (crate) + (install-file crate registry)) + (find-files "target/package" "\\.crate$")) + + (for-each (lambda (crate) + (invoke "tar" "xzf" crate "-C" sources)) + (find-files registry "\\.crate$"))) + #t)) (define %standard-phases @@ -195,6 +253,8 @@ directory = '" port) (replace 'build build) (replace 'check check) (replace 'install install) + (add-after 'build 'package package) + (add-after 'unpack 'unpack-rust-crates unpack-rust-crates) (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums))) (define* (cargo-build #:key inputs (phases %standard-phases)