[bug#64804] gnu: rust: Update to Rust 1.71.0
Commit Message
this one was quite a bit of effort but i seem like i actually got it
working!
the major changes that i've done is for Rust 1.70, i made a patch that
enables the cc feature flag so it would compile the outline asm files
instead of using included binary .a files which were removed by guix.
i also ignored a lot more tests for Rust 1.71, mainly gitoxide tests as
they require the network and the git binary and we don't have that in
the build.
also rustfmt requires rustc's shared libraries so i set the RUSTFLAGS
environment variable to add the "out" outputs lib folder to the RUNPATH
so the rustfmt binary works!
---
.../patches/rust-1.70-fix-rustix-build.patch | 20 +++
gnu/packages/rust.scm | 137 ++++++++++++++++--
2 files changed, 141 insertions(+), 16 deletions(-)
create mode 100644 gnu/packages/patches/rust-1.70-fix-rustix-build.patch
base-commit: 00ed2901f5171e4f9435641a91678217cae38030
Comments
Hi Fries,
While I'm not a Rust packager (and therefore may miss some stuff during
review), there are a few glaring issues which stand out for me. Also
forgive the formatting; not having the message proper to reply to
complicates things.
First and foremost, Rust has 11052 reverse dependencies and this patch
would rebuild 25834 packages. This should be submitted to the
core-updates branch. I'm not sure whether this patch should be
resubmitted for it or if others can just keep this in mind when
considering merging it.
The next major thing is that this should be split into multiple
patches. Make each discrete new Rust in its own patch, for example.
Ideally, each patch would be atomic so that only a subset could be
applied and packages would still build fine. If I were doing it, I
would commit each intermediate version of Rust and bump the default
Rust version with that commit. You may also be able to get away with
adding each intermediate Rust and only bumping the version of the
default rust for the last one; others will have to weigh in. See
Contributing > Submitting Patches > Sending a Patch Series > Multiple
Patches in the manual for how to submit such a patch series if you need
it.
Make sure your commit messages match the proper style; see
https://www.gnu.org/prep/standards/html_node/Change-Logs.html#Change-Logs
and the commit history.
There's one point where you add three new, empty lines near the
beginning of the patch. Prune those.
Where you write new modify-phases forms and new snippets, use
g-expressions.
> + [target."cfg(windows)".dependencies.windows-sys]
> + version = "0.45.0"
Shouldn't we be able to remove this altogether somehow since we're not
the target operating system?
> + ;; Rust 1.70 uses the rustix library which on Linux, it defaults to
> + ;; using outline ASM which without the cc cargo feature enabled, it
> + ;; will expect a precompiled binary library. This patch will
enable the cargo
> + ;; cc feature flag inside the fd-lock vendored Cargo.toml file,
which is the
> + ;; crate that uses rustix.
This comment is difficult to parse. Could it be reworded more clearly?
Perhaps, "Rust 1.70 adds the rustix library which uses outline ASM. The
vendored fd-lock crate uses rustix. It expects a precompiled binary
library without the "cc" Cargo feature enabled. This patch enables the
"cc" feature flag inside the vendored fd-lock Cargo.toml file."
That said, fd-lock is already packaged in Guix; the vendored version
should be avoided.
> + (("(checksum = )\".*\"" all name)
> + (string-append name "\"" ,%cargo-reference-hash "\"")))
Clever.
> + ;;; Function to make creating a list to ignore tests a bit easier.
This should probably be a docstring instead of a comment, and it should
describe what the function does - "Accept a list of strings containing
test names, and return a list of forms for skipping those tests" maybe.
> + (define (make-ignore-test-list strs)
> + (map (lambda (str)
> + (let ((ignore-string (format #f "#[ignore]\n~a" str)))
> + `((,str) ,ignore-string)))
> + strs))
A few things here. Firstly, I'm not sure that this procedure should
exist. Ideally we'd like to fix rather than ignore as many tests as
possible - although I see you're using this to rewrite existing stuff
more cleanly. Either way, break this whole function and its usage out
into a separate patch. Move it to be near other helper functions, not
in the middle of package definitions. Also perhaps make it a bit
cleaner like:
> (define (ignore-rust-tests strs)
> (map (lambda (str)
> `((,str) ,(string-append "#[ignore]\n" str))
> strs))
It's worth noting this is essentially a non-hygienic macro; maybe look
into rewriting it as a macro instead. See
https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility
for an introduction to the ideas behind macros, and
https://www.gnu.org/software/guile/manual/html_node/Macros.html -
especially
https://www.gnu.org/software/guile/manual/html_node/Syntax-Rules.html.
I've also found the (incorrectly) linked and very excellent
http://www.phyast.pitt.edu/~micheles/syntax-rules.pdf (WARNING! IS A
PDF! also available from
https://gist.github.com/jgarte/beb03e000943b7426f00b3d04ed01262
(WARNING! IS GITHUB!)) to be incredibly helpful. Ideally instead of
using `(substitutes* <file> ...)` you'd simply have some thing like
`(ignore-rust-tests <file> '(<str> <strs> ...))` which produces the
right code. Note that if you rewrite this as a macro, you won't be able
to use docstrings.
> + ;; Gitoxide tests seem to require the internet to run
> + ;; and Guix build containers don't have the internet.
You can just say, "Gitoxide tests require the network" or similar.
> - (substitute*
> -
"src/tools/cargo/tests/testsuite/init/simple_hg_ignore_exists/mod.rs"
> - (("fn simple_hg_ignore_exists")
> - "#[ignore]\nfn simple_hg_ignore_exists"))
> (substitute*
>
"src/tools/cargo/tests/testsuite/init/mercurial_autodetect/mod.rs"
> - (("fn mercurial_autodetect")
> - "#[ignore]\nfn mercurial_autodetect"))))
> + ,@(make-ignore-test-list '("fn case")))
> + (substitute*
> + "src/tools/cargo/tests/testsuite/init/simple_hg/mod.rs"
> + ,@(make-ignore-test-list '("fn case")))
> + (substitute*
> +
"src/tools/cargo/tests/testsuite/init/simple_hg_ignore_exists/mod.rs"
> + ,@(make-ignore-test-list '("fn case")))
In this code, you move down the substitute* call on
simple_hg_ignore_exists/mod.rs and add code above it. This produces a
larger diff than necessary. Avoid changing code order like this without
a good reason.
> + ;; Append the default output's lib folder to the RUSTFLAGS
> + ;; environment variable. this lets programs that depend on
> + ;; rustc's shared libraries like rustfmt work.
Minor writing stuff. Capitalize the 't' in "this;" move "like rustfmt"
to after "programs."
> + (setenv "RUSTFLAGS"
> + (format #f "-C link-arg=-Wl,-rpath,~a/lib"
> + (assoc-ref outputs "out")))
I'm not sure that this is the best way to do this. Setting build flags
in an environment variable just feels wrong. I'll let someone with more
knowledge of the domain weigh in, but I do want to mention it.
I don't think any of the rest of this review is blocking, but you may
want to consider it anyway.
I don't see "format" used in package definitions very often; I don't
know if there's some reason to oppose it, but it's worth considering
just using "string-append" instead.
Make sure this builds on architectures besides x86_64 if you can. I see
in the commit history for Rust there seem to be some issues around
aarch64 and riscv64 support in particular, so testing for those
platforms may reveal more issues.
Make sure to run "guix style" and "guix lint".
You may want to have a separate patch, either before or after your
changes, to port all of the default rust package's modify-phases to
g-expressions; "guix style -S arguments" should help there.
You'll need to wait for review from experts on the subject, and
obviously someone with commit, but this is very impressive for a first
patch. Rust has been languishing for a while and it's clearly not just
because it's not used. Well done.
- Juli
Hello,
it would be wonderful if this could be merged. I am trying to package
broot (command line file manager) and one of it's dependency requires
rust 1.70.0.
Pretty please.
Also bump,
We shouldn't be 4 versions behind upstream as a lot of newer rust
apps/programs require on what used to be unstable features in
older/current versions of rust. This is currently a stopgap for me when
packaging swww, a wayland compositor wallpaper changer.
Thanks all.
Hi,
such rust related changes are targeted at the rust team branch. IIRC
the rust team is a one-man army, so it absolutely possible that things
take a while :/.
But reading #64804, Efraim already bumped rust to 1.70 on the rust-team
branch, so it will be available after the next rust-team merge. In the
meantime you'd have to use the rust-team branch if you really want to
use rust-1.70.
mfg²
Milan Svoboda <milan.svoboda@centrum.cz> writes:
> Hello,
>
> it would be wonderful if this could be merged. I am trying to package broot
> (command line file manager) and one of it's dependency requires rust 1.70.0.
>
> Pretty please.
Closing, as Rust is already at a higher version.
new file mode 100644
@@ -0,0 +1,20 @@
+--- a/vendor/fd-lock/Cargo.toml 2023-05-31 14:44:48.000000000 -0700
++++ b/vendor/fd-lock/Cargo.toml 2023-07-14 21:19:34.637702319 -0700
+@@ -45,7 +45,7 @@
+
+ [target."cfg(unix)".dependencies.rustix]
+ version = "0.37.0"
+-features = ["fs"]
++features = ["fs", "cc"]
+
+ [target."cfg(windows)".dependencies.windows-sys]
+ version = "0.45.0"
+--- a/src/bootstrap/Cargo.lock 2023-07-11 20:32:40.000000000 -0700
++++ b/src/bootstrap/Cargo.lock 2023-07-14 22:41:53.269284713 -0700
+@@ -618,6 +618,7 @@
+ dependencies = [
+ "bitflags",
++ "cc",
+ "errno",
+ "io-lifetimes",
+ "libc",
@@ -16,6 +16,7 @@
;;; Copyright © 2022 Zheng Junjie <873216071@qq.com>
;;; Copyright © 2022 Jim Newsome <jnewsome@torproject.org>
;;; Copyright © 2022 Mark H Weaver <mhw@netris.org>
+;;; Copyright © 2023 Fries <fries1234@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -122,6 +123,9 @@ (define* (rust-bootstrapped-package base-rust version checksum)
(alist-replace "rustc-bootstrap" (list base-rust)
(package-native-inputs base-rust))))))
+
+
+
;;; Note: mrustc's only purpose is to be able to bootstap Rust; it's designed
;;; to be used in source form.
(define %mrustc-commit "597593aba86fa2edbea80c6e09f0b1b2a480722d")
@@ -705,6 +709,81 @@ (define rust-1.68
(rust-bootstrapped-package
rust-1.67 "1.68.2" "15ifyd5jj8rd979dkakp887hgmhndr68pqaqvd2hqkfdywirqcwk"))
+(define rust-1.69
+ (let ((base-rust
+ (rust-bootstrapped-package
+ rust-1.68 "1.69.0" "03zn7kx5bi5mdfsqfccj4h8gd6abm7spj0kjsfxwlv5dcwc9f1gv")))
+ (package
+ (inherit base-rust)
+ (source
+ (origin
+ (inherit (package-source base-rust))
+ (snippet
+ '(begin
+ (for-each delete-file-recursively
+ '("src/llvm-project"
+ "vendor/tikv-jemalloc-sys/jemalloc"))
+ ;; Also remove the bundled (mostly Windows) libraries.
+ (for-each delete-file
+ (find-files "vendor" ".*\\.(a|dll|exe|lib)$")))))))))
+
+(define rust-1.70
+ (let ((base-rust
+ (rust-bootstrapped-package
+ rust-1.69 "1.70.0" "0z6j7d0ni0rmfznv0w3mrf882m11kyh51g2bxkj40l3s1c0axgxj")))
+ (package
+ (inherit base-rust)
+ (source
+ (origin
+ (inherit (package-source base-rust))
+ ;; Rust 1.70 uses the rustix library which on Linux, it defaults to
+ ;; using outline ASM which without the cc cargo feature enabled, it
+ ;; will expect a precompiled binary library. This patch will enable the cargo
+ ;; cc feature flag inside the fd-lock vendored Cargo.toml file, which is the
+ ;; crate that uses rustix.
+ (patches (search-patches "rust-1.70-fix-rustix-build.patch"))
+ (patch-flags '("-p1"))))
+ (arguments
+ (substitute-keyword-arguments (package-arguments base-rust)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (replace 'build
+ (lambda* (#:key parallel-build? #:allow-other-keys)
+ (let ((job-spec (string-append
+ "-j" (if parallel-build?
+ (number->string (parallel-job-count))
+ "1"))))
+ (invoke "./x.py" job-spec "build" "--stage=1"
+ "library/std"
+ "src/tools/cargo")))))))))))
+
+(define rust-1.71
+ (let ((base-rust
+ (rust-bootstrapped-package
+ rust-1.70 "1.71.0" "15jc0d13cmrh2xvpkyyvsbwgn3w4klqiwf2wlgzfp22mvjmy8rx6")))
+ (package
+ (inherit base-rust)
+ (arguments
+ (substitute-keyword-arguments (package-arguments base-rust)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (replace 'patch-cargo-checksums
+ (lambda _
+ (substitute* '("Cargo.lock"
+ "src/bootstrap/Cargo.lock"
+ "src/tools/rust-analyzer/Cargo.lock"
+ "src/tools/cargo/Cargo.lock")
+ (("(checksum = )\".*\"" all name)
+ (string-append name "\"" ,%cargo-reference-hash "\"")))
+ (generate-all-checksums "vendor"))))))))))
+
+;;; Function to make creating a list to ignore tests a bit easier.
+(define (make-ignore-test-list strs)
+ (map (lambda (str)
+ (let ((ignore-string (format #f "#[ignore]\n~a" str)))
+ `((,str) ,ignore-string)))
+ strs))
+
;;; Note: Only the latest versions of Rust are supported and tested. The
;;; intermediate rusts are built for bootstrapping purposes and should not
;;; be relied upon. This is to ease maintenance and reduce the time
@@ -713,7 +792,7 @@ (define rust-1.68
;;; Here we take the latest included Rust, make it public, and re-enable tests
;;; and extra components such as rustfmt.
(define-public rust
- (let ((base-rust rust-1.67))
+ (let ((base-rust rust-1.71))
(package
(inherit base-rust)
(outputs (cons "rustfmt" (package-outputs base-rust)))
@@ -748,23 +827,43 @@ (define-public rust
(which "env")))))
(add-after 'unpack 'disable-tests-requiring-git
(lambda _
- (substitute* "src/tools/cargo/tests/testsuite/new.rs"
- (("fn author_prefers_cargo")
- "#[ignore]\nfn author_prefers_cargo")
- (("fn finds_author_git")
- "#[ignore]\nfn finds_author_git")
- (("fn finds_local_author_git")
- "#[ignore]\nfn finds_local_author_git"))))
+ (substitute* "src/tools/cargo/tests/testsuite/git.rs"
+ ,@(make-ignore-test-list
+ '("fn fetch_downloads_with_git2_first_then_with_gitoxide_and_vice_versa"
+ "fn git_fetch_cli_env_clean"
+ "fn git_with_cli_force"
+ "fn use_the_cli")))
+ ;; Gitoxide tests seem to require the internet to run
+ ;; and Guix build containers don't have the internet.
+ (substitute* "src/tools/cargo/tests/testsuite/git_shallow.rs"
+ ,@(make-ignore-test-list
+ '("fn gitoxide_clones_git_dependency_with_shallow_protocol_and_git2_is_used_for_followup_fetches"
+ "fn gitoxide_clones_registry_with_shallow_protocol_and_aborts_and_updates_again"
+ "fn gitoxide_clones_registry_with_shallow_protocol_and_follow_up_fetch_maintains_shallowness"
+ "fn gitoxide_clones_registry_with_shallow_protocol_and_follow_up_with_git2_fetch"
+ "fn gitoxide_clones_registry_without_shallow_protocol_and_follow_up_fetch_uses_shallowness"
+ "fn gitoxide_clones_shallow_two_revs_same_deps"
+ "fn gitoxide_git_dependencies_switch_from_branch_to_rev"
+ "fn gitoxide_shallow_clone_followed_by_non_shallow_update"
+ "fn shallow_deps_work_with_revisions_and_branches_mixed_on_same_dependency")))
+ (substitute* "src/tools/cargo/tests/testsuite/offline.rs"
+ ,@(make-ignore-test-list '("fn gitoxide_cargo_compile_offline_with_cached_git_dep_shallow_dep")))
+ (substitute* "src/tools/cargo/tests/testsuite/patch.rs"
+ ,@(make-ignore-test-list '("fn gitoxide_clones_shallow_old_git_patch")))))
(add-after 'unpack 'disable-tests-requiring-mercurial
(lambda _
- (substitute*
- "src/tools/cargo/tests/testsuite/init/simple_hg_ignore_exists/mod.rs"
- (("fn simple_hg_ignore_exists")
- "#[ignore]\nfn simple_hg_ignore_exists"))
(substitute*
"src/tools/cargo/tests/testsuite/init/mercurial_autodetect/mod.rs"
- (("fn mercurial_autodetect")
- "#[ignore]\nfn mercurial_autodetect"))))
+ ,@(make-ignore-test-list '("fn case")))
+ (substitute*
+ "src/tools/cargo/tests/testsuite/init/simple_hg/mod.rs"
+ ,@(make-ignore-test-list '("fn case")))
+ (substitute*
+ "src/tools/cargo/tests/testsuite/init/simple_hg_ignore_exists/mod.rs"
+ ,@(make-ignore-test-list '("fn case")))
+ (substitute*
+ "src/tools/cargo/tests/testsuite/new.rs"
+ ,@(make-ignore-test-list '("fn simple_hg")))))
(add-after 'unpack 'disable-tests-broken-on-aarch64
(lambda _
(with-directory-excursion "src/tools/cargo/tests/testsuite/"
@@ -807,7 +906,7 @@ (define-public rust
;; We skip the test since it's drastically unlikely Guix's
;; packaging will introduce a bug here.
(lambda _
- (delete-file "src/test/ui/parser/shebang/sneaky-attrib.rs")))
+ (delete-file "tests/ui/parser/shebang/sneaky-attrib.rs")))
(add-after 'unpack 'patch-process-tests
(lambda* (#:key inputs #:allow-other-keys)
(let ((bash (assoc-ref inputs "bash")))
@@ -840,11 +939,17 @@ (define-public rust
"gdb = \"" gdb "/bin/gdb\"\n"))))))
(replace 'build
;; Phase overridden to also build rustfmt.
- (lambda* (#:key parallel-build? #:allow-other-keys)
+ (lambda* (#:key parallel-build? outputs #:allow-other-keys)
(let ((job-spec (string-append
"-j" (if parallel-build?
(number->string (parallel-job-count))
"1"))))
+ ;; Append the default output's lib folder to the RUSTFLAGS
+ ;; environment variable. this lets programs that depend on
+ ;; rustc's shared libraries like rustfmt work.
+ (setenv "RUSTFLAGS"
+ (format #f "-C link-arg=-Wl,-rpath,~a/lib"
+ (assoc-ref outputs "out")))
(invoke "./x.py" job-spec "build"
"library/std" ;rustc
"src/tools/cargo"