[bug#78280,rust-team] doc: Document lockfile importer based Rust packaging workflow.

Message ID 77c65c3299edd79017a5f732ac2f98ec773d2d96.1746535085.git.hako@ultrarare.space
State New
Headers
Series [bug#78280,rust-team] doc: Document lockfile importer based Rust packaging workflow. |

Commit Message

Hilton Chain May 6, 2025, 12:47 p.m. UTC
  * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update
documentation.
* doc/guix-cookbook.texi (Packaging)[Packaging Workflow]: New section.

Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc
---
 doc/contributing.texi  |  97 +++++++----
 doc/guix-cookbook.texi | 368 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 437 insertions(+), 28 deletions(-)

--
2.49.0
  

Comments

Murilo May 6, 2025, 4:30 p.m. UTC | #1
Hi, a few suggestions.

On Tue May 6, 2025 at 9:47 AM -03, Hilton Chain wrote:
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi

> +In the Rust community it is common for multiple incompatible versions of a
s/community/ecosystem/
I think ecosystem is a better word here

> +specifically for some Rust applications, and can't simply identity them by
s/identity/identify/
Typo

> +version.  In this case we can use a @code{for-@var{program}} suffix, for
s/program/application/
Just to keep it consistent with the previous definitions (binary crate)

> --- a/doc/guix-cookbook.texi
> +++ b/doc/guix-cookbook.texi

> +The following sections are real life examples on working with specific build
s/are real life/provide real-life/
More cohesive and real-life as an adjective for the examples noun

> +Since @code{cargo-audit} is available on crates.io, we can generate a template
> +via the crates.io importer (@pxref{Invoking guix import,,, guix, GNU Guix
How about adding the used command to import here?
...
> +Reference Manual}).  After manual editing, we'll have the following definiton:

> +The identifier used to invoke @code{cargo-inputs}, @code{'cargo-audit} here,
The identifier used to invoke @code{cargo-inputs}, in this case @code{'cargo-audit},

> +# Or use short options, in this case the shell processes pathes before passing
> +# them to Guix, allowing expansion of @code{~}, for example.
> +$ guix import -i gnu/packages/rust-crates.scm \
$ guix import -i ~/my-local-guix/gnu/packages/rust-crates.scm \
Could use an example with expanding '~', for extra clarification

> +At this stage, package @code{cargo-audit} is buildable.
s/package/the package/
more cohesive

> +Finally we'll unbundle vendored dependencies.  The lockfile importer inserts
s/unbundle/unbundle the/
more cohesive

> +To facility various tasks in the common workflow, several scripts are provided
s/facility/facilitate/
typo

> +@code{niri} has Cargo workspace dependencies.  When packaging a Cargo workspace,
> +parameter @code{#:cargo-package-crates} is required.
I think here is a good place to mention how to indentify if an application has
dependencies that are workspaces.

My suggestion is something along these lines:

```
We can see in the @code{niri} dependencies, inside @code{Cargo.toml}, that
@code{pipewire} is being fetched from an external source, and by looking at
@code{pipewire-rs}' @code{Cargo.toml}, we can see that @code{pipewire-rs} is a
workspace. Thus, @code{niri} has Cargo workspace dependencies.  When packaging a
Cargo workspace, the parameter @code{#:cargo-package-crates} is required.
```
Also, is it really required for all Cargo workspaces? Or only required for
"Cargo workspace dependencies" (the ones with #:skip-build? #t)?
I think you mean `s/Cargo workspace,/Cargo workspace dependency,/`, but I'm not
sure.

> +To use our packaged development snapshots, it's also necessary to modify
> +@file{Cargo.toml} in a build phase, with a package-specific substitution
> +pattern.
Could it be in the source snippet too? or is it more desirable to have it in
a build phase?

Thank you for all the work and effort!
  

Patch

diff --git a/doc/contributing.texi b/doc/contributing.texi
index ab4f30d54b..148b4b0f71 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -1600,34 +1600,75 @@  Rust Crates
 @subsection Rust Crates

 @cindex rust
-Rust programs standing for themselves are named as any other package, using the
-lowercase upstream name.
-
-To prevent namespace collisions we prefix all other Rust packages with the
-@code{rust-} prefix.  The name should be changed to lowercase as appropriate and
-dashes should remain in place.
-
-In the rust ecosystem it is common for multiple incompatible versions of a
-package to be used at any given time, so all package definitions should have a
-versioned suffix.  The versioned suffix is the left-most non-zero digit (and
-any leading zeros, of course).  This follows the ``caret'' version scheme
-intended by Cargo.  Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
-
-Because of the difficulty in reusing rust packages as pre-compiled inputs for
-other packages the Cargo build system (@pxref{Build Systems,
-@code{cargo-build-system}}) presents the @code{#:cargo-inputs} and
-@code{cargo-development-inputs} keywords as build system arguments.  It would be
-helpful to think of these as similar to @code{propagated-inputs} and
-@code{native-inputs}.  Rust @code{dependencies} and @code{build-dependencies}
-should go in @code{#:cargo-inputs}, and @code{dev-dependencies} should go in
-@code{#:cargo-development-inputs}.  If a Rust package links to other libraries
-then the standard placement in @code{inputs} and the like should be used.
-
-Care should be taken to ensure the correct version of dependencies are used; to
-this end we try to refrain from skipping the tests or using @code{#:skip-build?}
-when possible.  Of course this is not always possible, as the package may be
-developed for a different Operating System, depend on features from the Nightly
-Rust compiler, or the test suite may have atrophied since it was released.
+Rust applications (binary crates) and libraries (library crates) are packaged
+separately.  We put our main efforts into applications and only package
+libraries as sources, utilizing automation with a manual focus on unbundling
+vendored dependencies.
+
+Rust applications are treated like any other package and named using the
+lowercase upstream name.  When using the Cargo build system (@pxref{Build
+Systems, @code{cargo-build-system}}), Rust applications should have the
+@code{#:install-source?} parameter set to @code{#f}, as this parameter only
+makes sense for libraries.  When the package source is a Cargo workspace,
+@code{#:cargo-install-paths} must be set to enable relevant support.
+
+Rust libraries are hidden from the user interface and managed in two modules:
+
+@enumerate
+@item
+@code{(gnu packages rust-crates)}, storing source definitions imported from Rust
+packages' @file{Cargo.lock} via the lockfile importer (@pxref{Invoking guix
+import, @code{crate}, @option{--lockfile=@var{file}}}).
+
+Imported definitions must be checked and have vendored dependencies unbundled
+before being contributed to Guix.  This module is managed by the Rust team
+(@pxref{Teams}).
+
+@item
+@code{(gnu packages rust-sources)}, storing more complex definitions that need
+to be full packages.  This includes Rust libraries requiring external inputs to
+unbundle and Cargo workspaces.
+
+These libraries should have the @code{#:skip-build?} parameter set to @code{#t}.
+For Cargo workspaces, @code{#:cargo-package-crates} must be set.
+
+Since they are added manually, the following naming convention applies:
+
+To prevent namespace collisions they are named with @code{rust-} prefix.  The
+name should be changed to lowercase as appropriate and dashes should remain in
+place.
+
+In the Rust community it is common for multiple incompatible versions of a
+package to be used at any given time, so all libraries should have a versioned
+suffix.  The versioned suffix is the left-most non-zero digit (and any leading
+zeros, of course).  This follows the ``caret'' version scheme intended by Cargo.
+Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
+
+In practice we usually package development snapshots of Rust libraries
+specifically for some Rust applications, and can't simply identity them by
+version.  In this case we can use a @code{for-@var{program}} suffix, for
+example, @code{rust-pipewire-for-niri} and @code{rust-pubgrub-for-uv}.
+@end enumerate
+
+Rust libraries are not referenced directly.  @code{(guix build-sytem cargo)}
+provides a @code{cargo-inputs} procedure to lookup input lists generated by the
+lockfile importer.
+
+@cindex cargo inputs
+@findex cargo-inputs
+@deffn {Procedure} cargo-inputs name [#:module '(gnu packages rust-crates)]
+Lookup Cargo inputs for @var{name} in @var{module}, return an empty list if
+unavailable.
+
+@var{name} must be consistent with the one used in lockfile importer invocation.
+
+@var{module} must provide a public interface @code{lookup-cargo-inputs}, a
+template is available in the @file{etc/teams/rust} directory of Guix source
+tree.
+@end deffn
+
+@xref{Packaging Rust Crates,,, guix-cookbook, GNU Guix Cookbook}, for a more
+detailed packaging workflow.


 @node Elm Packages
diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi
index fb58866d40..19379bf36f 100644
--- a/doc/guix-cookbook.texi
+++ b/doc/guix-cookbook.texi
@@ -105,6 +105,7 @@  Top
 Packaging

 * Packaging Tutorial::          A tutorial on how to add packages to Guix.
+* Packaging Workflows::         Real life examples on working with specific build systems.

 Packaging Tutorial

@@ -129,6 +130,16 @@  Top
 * Automatic update::
 * Inheritance::

+Packaging Workflows
+
+* Packaging Rust Crates::
+
+Packaging Rust Crates
+
+* Common Workflow for Rust Packaging::
+* Cargo Workspaces and Development Snapshots::
+* Using Rust Libraries in Other Build Systems::
+
 System Configuration

 * Auto-Login to a Specific TTY::  Automatically Login a User to a Specific TTY
@@ -518,6 +529,7 @@  Packaging

 @menu
 * Packaging Tutorial::          A tutorial on how to add packages to Guix.
+* Packaging Workflows::         Real life examples on working with specific build systems.
 @end menu

 @node Packaging Tutorial
@@ -1598,7 +1610,363 @@  References
 @uref{https://www.gnu.org/software/guix/guix-ghm-andreas-20130823.pdf, ``GNU Guix: Package without a scheme!''}, by Andreas Enge
 @end itemize

+@node Packaging Workflows
+@section Packaging Workflows
+
+The following sections are real life examples on working with specific build
+systems, serving as extensions to the concise packaging guidelines
+(@pxref{Packaging Guidelines,,, guix, GNU Guix Reference Manual}).
+
+@menu
+* Packaging Rust Crates::
+@end menu
+
+@node Packaging Rust Crates
+@subsection Packaging Rust Crates
+
+In preparation, add the following packages to our environment:
+
+@example
+$ guix shell rust rust:cargo cargo-audit cargo-license
+@end example
+
+@menu
+* Common Workflow for Rust Packaging::
+* Cargo Workspaces and Development Snapshots::
+* Using Rust Libraries in Other Build Systems::
+@end menu
+
+@node Common Workflow for Rust Packaging
+@subsubsection Common Workflow for Rust Packaging
+
+In this example, we'll package @code{cargo-audit}, which is published on the
+@uref{https://crates.io, crates.io} Rust package repository.  All its
+dependencies are on crates.io as well.
+
+@enumerate
+@item
+Since @code{cargo-audit} is available on crates.io, we can generate a template
+via the crates.io importer (@pxref{Invoking guix import,,, guix, GNU Guix
+Reference Manual}).  After manual editing, we'll have the following definiton:
+
+@lisp
+(define-public cargo-audit
+  (package
+    (name "cargo-audit")
+    (version "0.21.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "cargo-audit" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n"))))
+    (build-system cargo-build-system)
+    (arguments (list #:install-source? #f))
+    (inputs (cargo-inputs 'cargo-audit))
+    (home-page "https://rustsec.org/")
+    (synopsis "Audit Cargo.lock for crates with security vulnerabilities")
+    (description
+     "This package provides a Cargo subcommand, @@command@{cargo audit@}, to
+audit @@file@{Cargo.lock@} for crates with security vulnerabilities.")
+    (license (list license:asl2.0 license:expat))))
+@end lisp
+
+The identifier used to invoke @code{cargo-inputs}, @code{'cargo-audit} here,
+must be unique, usually matching the variable name of the package.
+
+@item
+Unpack package source and navigate to the unpacked directory, then run the
+following commands:
+
+@example
+$ cargo generate-lockfile
+$ cargo audit
+$ cargo license
+@end example
+
+@command{cargo generate-lockfile} updates dependencies to compatible versions.
+Applying it to all Rust applications helps reduce a great number of Rust
+libraries we need to check later.  Although sometimes libraries may fail to
+follow the @uref{https://semver.org/, semantic versioning} scheme, it's still
+acceptable.
+
+@command{cargo audit} checks known vulnerabilities and @command{cargo license}
+checks licenses, for all the dependencies.  We must have an acceptable output of
+@command{cargo audit} and ensure all dependencies are licensed with our
+supported licenses (@pxref{Defining Packages,,, guix, GNU Guix Reference
+Manual}).
+
+@item
+Import dependencies from the generated lockfile:
+
+@example
+$ guix import --insert=gnu/packages/rust-crates.scm \
+      crate --lockfile=/path/to/Cargo.lock cargo-audit
+
+# Or use short options, in this case the shell processes pathes before passing
+# them to Guix, allowing expansion of @code{~}, for example.
+$ guix import -i gnu/packages/rust-crates.scm \
+      crate -f /path/to/Cargo.lock cargo-audit
+@end example
+
+@code{cargo-audit} here must be consistent with the identifier used for
+@code{cargo-inputs} invocation in the package definition.
+
+At this stage, package @code{cargo-audit} is buildable.
+
+@item
+Finally we'll unbundle vendored dependencies.  The lockfile importer inserts
+@code{TODO:} comments for libraries with high probability of bundled
+dependencies.  @code{cargo-build-system} also performs additional check for
+binary files, which usually indicate bundling, in its
+@code{check-for-pregenerated-files} phase:
+
+@example
+$ ./pre-inst-env guix build cargo-audit
+@dots{}
+starting phase `check-for-pregenerated-files'
+Searching for binary files...
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst
+./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_aclui.a
+./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_activeds.a
+./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_asycfilt.a
+./guix-vendor/rust-winapi-i686-pc-windows-gnu-0.4.0.tar.gz/lib/libwinapi_amsi.a
+@dots{}
+@end example
+
+Although Rust libraries are not publicly exported, we can still select them via
+the Guix command-line interface through expression:
+
+@example
+$ guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)'
+@end example
+
+To unbundle most dependencies, a snippet is sufficient:
+
+@lisp
+(define rust-curl-sys-0.4.80+curl-8.12.1
+  (crate-source "curl-sys" "0.4.80+curl-8.12.1"
+                "0d7ppx4kq77hc5nyff6jydmfabpgd0i3ppjvn8x0q833mhpdzxsm"
+                #:snippet '(delete-file-recursively "curl")))
+@end lisp
+
+@lisp
+(define rust-bzip2-sys-0.1.13+1.0.8
+  (crate-source "bzip2-sys" "0.1.13+1.0.8"
+                "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2"
+                #:snippet
+                '(begin
+                   (delete-file-recursively "bzip2-1.0.8")
+                   (delete-file "build.rs")
+                   (with-output-to-file "build.rs"
+                     (lambda _
+                       (format #t "fn main() @{~@@
+                        println!(\"cargo:rustc-link-lib=bz2\");~@@
+                        @}~%"))))))
+@end lisp
+
+In a more complex case, where unbundling one dependency requires a build process
+that involves other packages, we should make a full package in @code{(gnu
+packages rust-sources)} first and reference it in the imported definition.
+
+For example, we have defined a @code{rust-ring-0.17} in @code{(gnu packages
+rust-sources)}, then the imported definition in @code{(gnu packages
+rust-crates)} should be modified to reference it.
+
+@lisp
+(define rust-ring-0.17.14 rust-ring-0.17)
+@end lisp
+
+When one dependency can be safely removed, modify it to @code{#f}.
+
+@lisp
+(define rust-openssl-src-300.4.2+3.4.1 #f)
+@end lisp
+
+To facility various tasks in the common workflow, several scripts are provided
+in the @file{etc/teams/rust} directory of Guix source tree.
+@end enumerate
+
+@node Cargo Workspaces and Development Snapshots
+@subsubsection Cargo Workspaces and Development Snapshots
+
+In this example, we'll package @code{niri}, which depends on development
+snapshots (also Cargo workspaces here).
+
+As we can't ensure compatibility of a development snapshot, before executing
+@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it
+to a known working revision.
+
+To use our packaged development snapshots, it's also necessary to modify
+@file{Cargo.toml} in a build phase, with a package-specific substitution
+pattern.
+
+@lisp
+(define-public niri
+  (package
+   (name "niri")
+   (version "25.02")
+   (source (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/YaLTeR/niri")
+                   (commit (string-append "v" version))))
+             (file-name (git-file-name name version))
+             (sha256
+              (base32
+               "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r"))))
+   (build-system cargo-build-system)
+   (arguments
+    (list #:install-source? #f
+          #:phases
+          #~(modify-phases %standard-phases
+              (add-after 'unpack 'use-guix-vendored-dependencies
+                (lambda _
+                  (substitute* "Cargo.toml"
+                    (("# version =.*")
+                     "version = \"*\"")
+                    (("git.*optional")
+                     "version = \"*\", optional")
+                    (("^git = .*")
+                     "")))))))
+   (native-inputs
+    (list pkg-config))
+   (inputs
+    (cons* clang
+           libdisplay-info
+           libinput-minimal
+           libseat
+           libxkbcommon
+           mesa
+           pango
+           pipewire
+           wayland
+           (cargo-inputs 'niri)))
+   (home-page "https://github.com/YaLTeR/niri")
+   (synopsis "Scrollable-tiling Wayland compositor")
+   (description
+    "Niri is a scrollable-tiling Wayland compositor which arranges windows in a
+scrollable format.  It is considered stable for daily use and performs most
+functions expected of a Wayland compositor.")
+   (license license:gpl3+)))
+@end lisp
+
+@code{niri} has Cargo workspace dependencies.  When packaging a Cargo workspace,
+parameter @code{#:cargo-package-crates} is required.
+
+@lisp
+(define-public rust-pipewire-for-niri
+  (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9")
+        (revision "0"))
+    (package
+      (name "rust-pipewire")
+      (version (git-version "0.8.0" revision commit))
+      (source
+       (origin
+         (method git-fetch)
+         (uri (git-reference
+               (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git")
+               (commit commit)))
+         (file-name (git-file-name name version))
+         (sha256
+          (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v"))))
+      (build-system cargo-build-system)
+      (arguments
+       (list #:skip-build? #t
+             #:cargo-package-crates
+             ''("libspa-sys" "libspa" "pipewire-sys" "pipewire")))
+      (inputs (cargo-inputs 'rust-pipewire-for-niri))
+      (home-page "https://pipewire.org/")
+      (synopsis "Rust bindings for PipeWire")
+      (description "This package provides Rust bindings for PipeWire.")
+      (license license:expat))))
+@end lisp
+
+Don't forget to modify all workspace members in @code{(gnu packages
+rust-crates)}:
+
+@lisp
+(define rust-pipewire-0.8.0.fd3d8f7 rust-pipewire-for-niri)
+(define rust-pipewire-sys-0.8.0.fd3d8f7 rust-pipewire-for-niri)
+@dots{}
+(define rust-libspa-0.8.0.fd3d8f7 rust-pipewire-for-niri)
+(define rust-libspa-sys-0.8.0.fd3d8f7 rust-pipewire-for-niri)
+@end lisp
+
+@node Using Rust Libraries in Other Build Systems
+@subsubsection Using Rust Libraries in Other Build Systems
+
+In this example, we'll package @code{libchewing}, which combines two build
+systems.
+
+When building Rust packages in other build systems, we need to add @code{rust},
+and @code{rust:cargo} to @code{native-inputs}, import and use modules from both
+build systems, and apply necessary build phases from @code{cargo-build-system}.
+
+For cross-compilation support, we'll also add @code{rust-sysroot} created with
+@code{(make-rust-sysroot (%current-target-system))} to @code{native-inputs}, and
+set @code{#:cargo-target} for @code{cargo-build-system}'s build phases.
+
+@lisp
+(define-public libchewing
+  (package
+    (name "libchewing")
+    (version "0.9.1")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/chewing/libchewing")
+             (commit (string-append "v" version))))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32 "0gh64wvrk5pn0fhmpvj1j99d5g7f7697rk96zbkc8l72yjr819z5"))))
+    (build-system cmake-build-system)
+    (arguments
+     (list #:imported-modules
+           (append %cmake-build-system-modules
+                   %cargo-build-system-modules)
+           #:modules
+           '(((guix build cargo-build-system) #:prefix cargo:)
+             (guix build cmake-build-system)
+             (guix build utils))
+           #:phases
+           #~(modify-phases %standard-phases
+               (add-after 'unpack 'prepare-cargo-build-system
+                 (lambda args
+                   (for-each
+                    (lambda (phase)
+                      (format #t "Running cargo phase: ~a~%" phase)
+                      (apply (assoc-ref cargo:%standard-phases phase)
+                             ;; For cross-compilation.
+                             #:cargo-target #$(cargo-triplet)
+                             args))
+                    '(unpack-rust-crates
+                      configure
+                      check-for-pregenerated-files
+                      patch-cargo-checksums)))))))
+    (native-inputs
+     (append
+      (list rust `(,rust "cargo") )
+      ;; For cross-compilation.
+      (or (and=> (%current-target-system)
+                 (compose list make-rust-sysroot))
+          '())))
+    (inputs
+     (cons* corrosion ncurses sqlite (cargo-inputs 'libchewing)))
+    (synopsis "Chinese phonetic input method")
+    (description "Chewing is an intelligent phonetic (Zhuyin/Bopomofo) input
+method, one of the most popular choices for Traditional Chinese users.")
+    (home-page "https://chewing.im/")
+    (license license:lgpl2.1+)))
+@end lisp
+
+
 @c *********************************************************************
 @node System Configuration
 @chapter System Configuration