[bug#77093,rust-team,18/18] doc: Document lockfile importer based Rust packaging workflow.

  * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update
documentation.

Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc
---
 doc/contributing.texi | 415 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 390 insertions(+), 25 deletions(-)
  

On Tue, Mar 18, 2025 at 03:24:31PM +0800, Hilton Chain wrote:
> * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update
> documentation.
> 
> Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc
> ---
>  doc/contributing.texi | 415 +++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 390 insertions(+), 25 deletions(-)
> 
> diff --git a/doc/contributing.texi b/doc/contributing.texi
> index ab4f30d54b..e7c5116a3d 100644
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi
> @@ -1600,34 +1600,399 @@ Rust Crates
>  @subsection Rust Crates
>  
>  @cindex rust
> -Rust programs standing for themselves are named as any other package, using the
> -lowercase upstream name.
> +As currently there's no achievable way to reuse Rust packages as pre-compiled
> +inputs for other packages at a distribution scale, and our previous approach on
> +making all Rust dependencies as full packages doesn't cope well with Rust
> +ecosystem thus causing issues on both contribution and maintenance sides, we
> +have switched to a different packaging model.

I'm not sure the first paragraph is necessary.

> +
> +Rust programs (binary crates) and dependencies (library crates) are treated
> +separately.  We put main efforts into programs and only package Rust
> +dependencies as sources, utilizing automation with manual focus on unbundling
> +vendored dependencies.  The following paragraphs will explain them and give
> +several examples.

Rust programs (binary crates) and dependencies (library crates) are treated
separately.  We put our main efforts into programs and only package Rust
dependencies as sources, utilizing automation with a manual focus on unbundling
vendored dependencies.  The following paragraphs will explain them and give
several examples.

> +
> +Rust programs are treated like any other package and named using the lowercase
> +upstream name.  When using the Cargo build system (@pxref{Build Systems,
> +@code{cargo-build-system}}), Rust programs should have @code{#:install-source?}
> +argument set to @code{#f}, as this argument only makes sense to dependencies.

argument set to @code{#f}, as this argument only makes sense for dependencies.

> +When the package source is a Cargo workspace, @code{#:cargo-install-paths} must
> +be set to enable relevant support.
> +
> +Rust dependencies are managed in two modules:
>  
> -To prevent namespace collisions we prefix all other Rust packages with the
> -@code{rust-} prefix.  The name should be changed to lowercase as appropriate and
> -dashes should remain in place.
> +@enumerate
> +@item
> +@code{(gnu packages rust-crates)}, storing source definitions imported from Rust
> +programs' @file{Cargo.lock} via the lockfile importer (@pxref{Invoking guix
> +import, crate, @code{--lockfile=@var{file}}}).
> +
> +Imported definitions must be checked and have vendored sources unbundled before
> +contributing to Guix.  Naturally, this module serves as a store for both sources
> +and unbundling strategies.
> +
> +This module is managed by Rust team (@pxref{Teams}) to ensure there's always one

This module is managed by the Rust team (@pxref{Teams}) to ensure there's always one

> +version containing all changes from other branches, so that the maintained
> +version can be used directly in case of merge conflicts, thus coordination is
> +required for other committers to modify it.
> +
> +Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup
> +script @file{etc/teams/rust/cleanup-crates.sh} for this moudle.

script @file{etc/teams/rust/cleanup-crates.sh} for this module.

> +
> +@item
> +@code{(gnu packages rust-sources)}, storing more complex definitions that need
> +to be full packages.  This includes Rust dependencies requiring external inputs
> +to unbundle and Cargo workspaces.
> +
> +These dependencies should have @code{#:skip-build?} argument set to @code{#t}.

These dependencies should have the @code{#:skip-build?} argument set to @code{#t}.

> +For Cargo workspaces, @code{#:cargo-package-crates} must be set.
> +
> +Since they are added manually, the following naming convention applies:
> +
> +To prevent namespace collisions they are named with @code{rust-} prefix.  The
> +name should be changed to lowercase as appropriate and dashes should remain in
> +place.
>  
>  In the rust ecosystem it is common for multiple incompatible versions of a
> -package to be used at any given time, so all package definitions should have a
> -versioned suffix.  The versioned suffix is the left-most non-zero digit (and
> -any leading zeros, of course).  This follows the ``caret'' version scheme
> -intended by Cargo.  Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
> -
> -Because of the difficulty in reusing rust packages as pre-compiled inputs for
> -other packages the Cargo build system (@pxref{Build Systems,
> -@code{cargo-build-system}}) presents the @code{#:cargo-inputs} and
> -@code{cargo-development-inputs} keywords as build system arguments.  It would be
> -helpful to think of these as similar to @code{propagated-inputs} and
> -@code{native-inputs}.  Rust @code{dependencies} and @code{build-dependencies}
> -should go in @code{#:cargo-inputs}, and @code{dev-dependencies} should go in
> -@code{#:cargo-development-inputs}.  If a Rust package links to other libraries
> -then the standard placement in @code{inputs} and the like should be used.
> -
> -Care should be taken to ensure the correct version of dependencies are used; to
> -this end we try to refrain from skipping the tests or using @code{#:skip-build?}
> -when possible.  Of course this is not always possible, as the package may be
> -developed for a different Operating System, depend on features from the Nightly
> -Rust compiler, or the test suite may have atrophied since it was released.
> +package to be used at any given time, so all dependencies should have a
> +versioned suffix.  The versioned suffix is the left-most non-zero digit (and any
> +leading zeros, of course).  This follows the ``caret'' version scheme intended
> +by Cargo.  Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
> +
> +In practice we are usually packaging development snapshots of Rust dependencies
> +specifically for some Rust programs, and can't simply identity them by version.
> +In this case we can use a @code{for-@var{program}} suffix.  Examples@:
> +@code{rust-pipewire-for-niri}, @code{rust-pubgrub-for-uv}.
> +@end enumerate
> +
> +Let's demonstrate the packaging workflow by examples, note that package-specific
> +issues are not involved here.
> +
> +In preparation, we'll adding the following packages to our environment:
> +
> +@example
> +guix shell rust rust:cargo cargo-audit cargo-license
> +@end example
> +
> +Example 1: @code{cargo-audit}, which is published on the crates.io Rust package
> +repository @uref{https://crates.io, crates.io}.
> +
> +@enumerate
> +@item
> +We can generate a draft definition via the crates.io importer (@pxref{Invoking
> +guix import, crate}).  In the end we'll have the following definiton:
> +
> +@lisp
> +(define-public cargo-audit
> +  (package
> +    (name "cargo-audit")
> +    (version "0.21.2")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (crate-uri "cargo-audit" version))
> +       (file-name (string-append name "-" version ".tar.gz"))
> +       (sha256
> +        (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n"))))
> +    (build-system cargo-build-system)
> +    (arguments (list #:install-source? #f))
> +    (inputs (cargo-inputs 'cargo-audit))
> +    (home-page "https://rustsec.org/")
> +    (synopsis "Audit Cargo.lock for crates with security vulnerabilities")
> +    (description
> +     "This package provides a Cargo subcommand, @@command@{cargo audit@}, to
> +audit @@file@{Cargo.lock@} for crates with security vulnerabilities.")
> +    (license (list license:asl2.0 license:expat))))
> +@end lisp
> +
> +@code{cargo-inputs} is a procedure defined in @code{guix build-system cargo},
> +facilitating dependency modification.  @code{'cargo-audit} used here must be a
> +unique identifier, usually same to the program's variable name.
> +
> +@item
> +Unpack package source and navigate to the unpacked directory, then execute the
> +following commands:
> +
> +@example
> +cargo generate-lockfile
> +cargo audit
> +cargo license
> +@end example
> +
> +@command{cargo generate-lockfile} updates dependencies to compatible versions,
> +@command{cargo audit} checks known vulnerabilities and @command{cargo license}
> +checks licenses of all dependencies.
> +
> +We must have an acceptable output of @command{cargo audit} and ensure all
> +dependencies are licensed with our supported licenses (@pxref{Defining Packages,
> +@code{license}}).
> +
> +@item
> +Import dependencies from previously generated lockfile:
> +
> +@example
> +guix import --insert=gnu/packages/rust-crates.scm \
> +    crate --lockfile=/path/to/Cargo.lock cargo-audit
> +@end example
> +
> +@code{cargo-audit} used here must be consistent with the identifier used for
> +@code{cargo-inputs}.
> +
> +At this stage, @code{cargo-audit} is buildable.
> +
> +@item
> +Finally we'll unbundle vendored sources.  The lockfile importer inserts
> +@code{TODO:} comments to dependencies with high probability of bundled sources.
> +@code{cargo-build-system} also performs additional check in its
> +@code{check-for-pregenerated-files} phase:
> +
> +@example
> +$ guix build cargo-audit
> +@dots{}
> +starting phase `check-for-pregenerated-files'
> +Searching for binary files...
> +./guix-vendor/rust-addr2line-0.21.0.tar.gz/rustfmt.toml
> +./guix-vendor/rust-arc-swap-1.7.1.tar.gz/rustfmt.toml
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst
> +@dots{}
> +@end example
> +
> +Although dependencies in @code{(gnu packages rust-crates)} are not exported, we
> +can still select them in Guix command-line interface through expression:
> +
> +@example
> +guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)'
> +@end example
> +
> +For most dependencies, a snippet is sufficient:
> +
> +@lisp
> +(define rust-bzip2-sys-0.1.13+1.0.8
> +  (crate-source "bzip2-sys" "0.1.13+1.0.8"
> +                "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2"
> +                #:snippet
> +                '(begin
> +                   (delete-file-recursively "bzip2-1.0.8")
> +                   (delete-file "build.rs")
> +                   ;; Inspired by Debian's patch.
> +                   (with-output-to-file "build.rs"
> +                     (lambda _
> +                       (format #t "fn main() @{~@@
> +                        println!(\"cargo:rustc-link-lib=bz2\");~@@
> +                        @}~%"))))))
> +@end lisp
> +
> +In a more complex case, where unbundling one dependency requires other packages,
> +we should package the dependency in @code{(gnu packages rust-sources)} first and
> +point the imported definition to a symbol with the same name.
> +
> +For example we have defined a @code{rust-ring-0.17} in @code{(gnu packages
> +rust-sources)}, then the imported definition in @code{(gnu packages
> +rust-crates)} should be modified to point to it:
> +
> +@lisp
> +(define rust-ring-0.17.14 'rust-ring-0.17)
> +@end lisp
> +
> +When one dependency can be safely removed, modify it to @code{#f}.
> +
> +@lisp
> +(define rust-openssl-src-300.4.2+3.4.1 #f)
> +@end lisp
> +
> +Such modifications are processed by @code{cargo-inputs} procedure.
> +
> +@code{cargo-inputs} can also be configured by keywoard arguments
> +@code{#:crates-module} (default: @code{(gnu packages rust-crates)}, module to
> +resolve imported definitions) and @code{#:sources-module} (default: @code{(gnu
> +packages rust-sources)}, module to resolve modified symbols), which are useful
> +to maintain an independent channel.
> +@end enumerate
> +
> +Example 2: @code{niri}, which is not available on crates.io and depends on
> +development snapshots (also Cargo workspaces in this example).
> +
> +@enumerate
> +@item
> +As we can't ensure compatibility of a development snapshot, before executing
> +@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it
> +in a known working revision.
> +
> +To use our packaged development snapshots, it's also necessary to modify
> +@file{Cargo.toml} in build environment, the substitution pattern is
> +package-specific.
> +
> +@code{cargo-inputs} returns a list, all list operations apply to it too.
> +
> +@lisp
> +(define-public niri
> +  (package
> +   (name "niri")
> +   (version "25.02")
> +   (source (origin
> +             (method git-fetch)
> +             (uri (git-reference
> +                   (url "https://github.com/YaLTeR/niri")
> +                   (commit (string-append "v" version))))
> +             (file-name (git-file-name name version))
> +             (sha256
> +              (base32
> +               "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r"))))
> +   (build-system cargo-build-system)
> +   (arguments
> +    (list #:install-source? #f
> +          #:phases
> +          #~(modify-phases %standard-phases
> +              (add-after 'unpack 'use-guix-vendored-dependencies
> +                (lambda _
> +                  (substitute* "Cargo.toml"
> +                    (("# version =.*")
> +                     "version = \"*\"")
> +                    (("git.*optional")
> +                     "version = \"*\", optional")
> +                    (("^git = .*")
> +                     "")))))))
> +   (native-inputs
> +    (list pkg-config))
> +   (inputs
> +    (cons* clang
> +           libdisplay-info
> +           libinput-minimal
> +           libseat
> +           libxkbcommon
> +           mesa
> +           pango
> +           pipewire
> +           wayland
> +           (cargo-inputs 'niri)))
> +   (home-page "https://github.com/YaLTeR/niri")
> +   (synopsis "Scrollable-tiling Wayland compositor")
> +   (description
> +    "Niri is a scrollable-tiling Wayland compositor which arranges windows in a
> +scrollable format.  It is considered stable for daily use and performs most
> +functions expected of a Wayland compositor.")
> +   (license license:gpl3)))
> +@end lisp
> +
> +@item
> +@code{niri} also has Cargo workspace dependencies.  When packaging a Cargo
> +workspace, build argument @code{#:cargo-package-crates} is required.
> +
> +@lisp
> +(define-public rust-pipewire-for-niri
> +  (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9")
> +        (revision "0"))
> +    (package
> +      (name "rust-pipewire")
> +      (version (git-version "0.8.0" revision commit))
> +      (source
> +       (origin
> +         (method git-fetch)
> +         (uri (git-reference
> +               (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git")
> +               (commit commit)))
> +         (file-name (git-file-name name version))
> +         (sha256
> +          (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v"))))
> +      (build-system cargo-build-system)
> +      (arguments
> +       (list #:skip-build? #t
> +             #:cargo-package-crates
> +             ''("libspa-sys" "libspa" "pipewire-sys" "pipewire")))
> +      (inputs (cargo-inputs 'rust-pipewire-for-niri))
> +      (home-page "https://pipewire.org/")
> +      (synopsis "Rust bindings for PipeWire")
> +      (description "This package provides Rust bindings for PipeWire.")
> +      (license license:expat))))
> +@end lisp
> +
> +Don't forget to modify all workspace members in @code{(gnu packages
> +rust-crates)}:
> +
> +@lisp
> +(define rust-pipewire-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +(define rust-pipewire-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +@dots{}
> +(define rust-libspa-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +(define rust-libspa-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +@end lisp
> +@end enumerate
> +
> +Example 3: @code{fish}, which combines two build systems.
> +
> +@enumerate
> +@item
> +When building Rust packages in other build systems, we need to add @code{rust}
> +and @code{rust:cargo} to @code{native-inputs}, import and use modules from both
> +build systems, and apply necessary build phases from @code{cargo-build-system}.
> +
> +@lisp
> +(define-public fish
> +  (package
> +    (name "fish")
> +    (version "4.0.0")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (string-append "https://github.com/fish-shell/fish-shell/"
> +                           "releases/download/" version "/"
> +                           "fish-" version ".tar.xz"))
> +       (sha256
> +        (base32 "1wv9kjwg6ax8m2f85i58l9f9cndshn1f15n8skc68w1mf3cmpnig"))))
> +    (build-system cmake-build-system)
> +    (inputs
> +     (cons* fish-foreign-env ncurses pcre2
> +            python                    ;for fish_config and manpage completions
> +            (cargo-inputs 'fish)))
> +    (native-inputs
> +     (list doxygen groff                ;for 'fish --help'
> +           pkg-config
> +           procps                       ;for the test suite
> +           rust
> +           `(,rust "cargo")))
> +    (arguments
> +     (list #:imported-modules
> +           (append %cargo-build-system-modules
> +                   %cmake-build-system-modules)
> +           #:modules
> +           (((guix build cargo-build-system) #:prefix cargo:)
> +            (guix build cmake-build-system)
> +            (guix build utils))
> +           #:phases
> +           #~(modify-phases %standard-phases
> +               (add-after 'unpack 'use-guix-vendored-dependencies
> +                 (lambda _
> +                   (substitute* "Cargo.toml"
> +                     (("git.*tag.*,")
> +                      "version = \"*\","))))
> +               (add-after 'unpack 'prepare-cargo-build-system
> +                 (lambda args
> +                   (for-each
> +                    (lambda (phase)
> +                      (format #t "Running cargo phase: ~a~%" phase)
> +                      (apply (assoc-ref cargo:%standard-phases phase)
> +                             args))
> +                    '(unpack-rust-crates
> +                      configure
> +                      check-for-pregenerated-files
> +                      patch-cargo-checksums)))))))
> +    (synopsis "The friendly interactive shell")
> +    (description
> +     "Fish (friendly interactive shell) is a shell focused on interactive use,
> +discoverability, and friendliness.  Fish has very user-friendly and powerful
> +tab-completion, including descriptions of every completion, completion of
> +strings with wildcards, and many completions for specific commands.  It also
> +has extensive and discoverable help.  A special @@command@{help@} command gives
> +access to all the fish documentation in your web browser.  Other features
> +include smart terminal handling based on terminfo, an easy to search history,
> +and syntax highlighting.")
> +    (home-page "https://fishshell.com/")
> +    (license license:gpl2)))
> +@end lisp
> +@end enumerate
>  
>  
>  @node Elm Packages
> -- 
> 2.48.1
> 
> 
> 
>
  

On Tue, 18 Mar 2025 19:49:54 +0800,
Efraim Flashner wrote:
>
> [1  <text/plain; utf-8 (quoted-printable)>]
> On Tue, Mar 18, 2025 at 03:24:31PM +0800, Hilton Chain wrote:
> > * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update
> > documentation.
> >
> > Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc
> > ---
> >  doc/contributing.texi | 415 +++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 390 insertions(+), 25 deletions(-)
> >
> > diff --git a/doc/contributing.texi b/doc/contributing.texi
> > index ab4f30d54b..e7c5116a3d 100644
> > --- a/doc/contributing.texi
> > +++ b/doc/contributing.texi
> > @@ -1600,34 +1600,399 @@ Rust Crates
> >  @subsection Rust Crates
> >
> >  @cindex rust
> > -Rust programs standing for themselves are named as any other package, using the
> > -lowercase upstream name.
> > +As currently there's no achievable way to reuse Rust packages as pre-compiled
> > +inputs for other packages at a distribution scale, and our previous approach on
> > +making all Rust dependencies as full packages doesn't cope well with Rust
> > +ecosystem thus causing issues on both contribution and maintenance sides, we
> > +have switched to a different packaging model.
>
> I'm not sure the first paragraph is necessary.

I think we need to clarify why do we only do this for Rust, since similar
workflow can be applied to other ecosystems with a package manager as well.


Thanks for the review, I'll prepare V2 soon.