From patchwork Tue Mar 18 14:27:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hilton Chain X-Patchwork-Id: 40341 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B0EAE27BBEA; Tue, 18 Mar 2025 14:31:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-5.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,FROM_SUSPICIOUS_NTLD,MAILING_LIST_MULTI, PDS_OTHER_BAD_TLD,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_VALIDITY_CERTIFIED, RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE,SPF_HELO_PASS, URIBL_BLOCKED,URIBL_SBL_A autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 292DA27BBE2 for ; Tue, 18 Mar 2025 14:31:17 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tuXyN-0007Hh-5n; Tue, 18 Mar 2025 10:31:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tuXxb-0006Q9-Mf for guix-patches@gnu.org; Tue, 18 Mar 2025 10:30:30 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tuXxY-0001y4-FZ; Tue, 18 Mar 2025 10:30:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=fYh+Bj/HtW5J1+1CPK1CvQMLgPhMjNp3CvRauv6IPNA=; b=Oh8Oqe9cGgRbrOYDu8YJcmo8UTOAFC+kt5CgwdgY1BwZJ3PkmIk8B97XnocpU33c1KJJ0dm3mXV3/gi7fYQ88RcxVZPpYHgT4MHBD2/Cza6LoKxzK4kiHIwyn9Mg+J1x9vsBOd2AO57kACH6PZqmlYciWNOmddUj5GNobA0kUyVvTQi9NbsYaBcZtg1ehV9+mZkzKBVUUsoohn1+EX7kjnBq2sPxkoaj3LqMfeme3kt6qi9NhZeqaf/0VPkdMARRLNNf2z2hWY0V4jy32L2gW5emjqh6IkNtp2wX8Mny4JjnkjOKxjXbUjF7oXupnK28QiO9aZYNg3IDV/hYRRKzWA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tuXxY-0004yj-8e; Tue, 18 Mar 2025 10:30:24 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#77093] [PATCH rust-team v2 17/17] doc: Document lockfile importer based Rust packaging workflow. Resent-From: Hilton Chain Original-Sender: "Debbugs-submit" Resent-CC: ludo@gnu.org, maxim.cournoyer@gmail.com, guix-patches@gnu.org Resent-Date: Tue, 18 Mar 2025 14:30:24 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 77093 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 77093@debbugs.gnu.org Cc: Hilton Chain , Efraim Flashner , Ludovic =?utf-8?q?Court=C3=A8s?= , Maxim Cournoyer X-Debbugs-Original-Xcc: Ludovic =?utf-8?q?Court=C3=A8s?= , Maxim Cournoyer Received: via spool by 77093-submit@debbugs.gnu.org id=B77093.174230818518470 (code B ref 77093); Tue, 18 Mar 2025 14:30:24 +0000 Received: (at 77093) by debbugs.gnu.org; 18 Mar 2025 14:29:45 +0000 Received: from localhost ([127.0.0.1]:41750 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tuXws-0004nh-7W for submit@debbugs.gnu.org; Tue, 18 Mar 2025 10:29:45 -0400 Received: from mx.boiledscript.com ([88.99.243.112]:48336) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tuXvO-0004Yb-8M for 77093@debbugs.gnu.org; Tue, 18 Mar 2025 10:28:17 -0400 From: Hilton Chain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ultrarare.space; s=mail; t=1742308089; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fYh+Bj/HtW5J1+1CPK1CvQMLgPhMjNp3CvRauv6IPNA=; b=NI4Ms1eYTbjlSB5yOYWzh9VxwxTW/+s12q7SiFJ1+JOuZDx5QaXoliXQtb9TuABv7PJXIi B8sNdq1hCptdmARUPl/VppuEoz8cs1L9c0gKpTBh4nkYuFBDwO+ACtFkcS7FCi/AAlUxnX 6+E0UZGOthHkwrGXhxyZpm0nfCoRL8qgbFc6p8Rskp+TCnMqoLwGBPS9J2VtdVgTjX5xjR /h9wGVK73iut+1Ydoz6hyuI6y4rYkj7KrSWGiyPPkJq/e5yf1tZWDMy15ShDDIz8YSTbcr Eyr6gaa6NmaDF84mBK4PbKxYUa7u2v5yJi+nej5NSLqYRZ9Pc4nKOHlzd+BE9w== Date: Tue, 18 Mar 2025 22:27:00 +0800 Message-ID: <613319a41cd8d7823cb1d0aa6cc710cf24239a6b.1742306960.git.hako@ultrarare.space> In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update documentation. Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc --- doc/contributing.texi | 415 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 390 insertions(+), 25 deletions(-) diff --git a/doc/contributing.texi b/doc/contributing.texi index ab4f30d54b..837074dead 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -1600,34 +1600,399 @@ Rust Crates @subsection Rust Crates @cindex rust -Rust programs standing for themselves are named as any other package, using the -lowercase upstream name. +As currently there's no achievable way to reuse Rust packages as pre-compiled +inputs for other packages at a distribution scale, and our previous approach on +making all Rust dependencies as full packages doesn't cope well with Rust +ecosystem thus causing issues on both contribution and maintenance sides, we +have switched to a different packaging model. + +Rust programs (binary crates) and dependencies (library crates) are treated +separately. We put our main efforts into programs and only package Rust +dependencies as sources, utilizing automation with a manual focus on unbundling +vendored dependencies. The following paragraphs will explain them and give +several examples. + +Rust programs are treated like any other package and named using the lowercase +upstream name. When using the Cargo build system (@pxref{Build Systems, +@code{cargo-build-system}}), Rust programs should have @code{#:install-source?} +argument set to @code{#f}, as this argument only makes sense for dependencies. +When the package source is a Cargo workspace, @code{#:cargo-install-paths} must +be set to enable relevant support. + +Rust dependencies are managed in two modules: -To prevent namespace collisions we prefix all other Rust packages with the -@code{rust-} prefix. The name should be changed to lowercase as appropriate and -dashes should remain in place. +@enumerate +@item +@code{(gnu packages rust-crates)}, storing source definitions imported from Rust +programs' @file{Cargo.lock} via the lockfile importer (@pxref{Invoking guix +import, crate, @code{--lockfile=@var{file}}}). + +Imported definitions must be checked and have vendored sources unbundled before +contributing to Guix. Naturally, this module serves as a store for both sources +and unbundling strategies. + +This module is managed by the Rust team (@pxref{Teams}) to ensure there's always +one version containing all changes from other branches, so that the maintained +version can be used directly in case of merge conflicts, thus coordination is +required for other committers to modify it. + +Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup +script @file{etc/teams/rust/cleanup-crates.sh} for this module. + +@item +@code{(gnu packages rust-sources)}, storing more complex definitions that need +to be full packages. This includes Rust dependencies requiring external inputs +to unbundle and Cargo workspaces. + +These dependencies should have the @code{#:skip-build?} argument set to +@code{#t}. For Cargo workspaces, @code{#:cargo-package-crates} must be set. + +Since they are added manually, the following naming convention applies: + +To prevent namespace collisions they are named with @code{rust-} prefix. The +name should be changed to lowercase as appropriate and dashes should remain in +place. In the rust ecosystem it is common for multiple incompatible versions of a -package to be used at any given time, so all package definitions should have a -versioned suffix. The versioned suffix is the left-most non-zero digit (and -any leading zeros, of course). This follows the ``caret'' version scheme -intended by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}. - -Because of the difficulty in reusing rust packages as pre-compiled inputs for -other packages the Cargo build system (@pxref{Build Systems, -@code{cargo-build-system}}) presents the @code{#:cargo-inputs} and -@code{cargo-development-inputs} keywords as build system arguments. It would be -helpful to think of these as similar to @code{propagated-inputs} and -@code{native-inputs}. Rust @code{dependencies} and @code{build-dependencies} -should go in @code{#:cargo-inputs}, and @code{dev-dependencies} should go in -@code{#:cargo-development-inputs}. If a Rust package links to other libraries -then the standard placement in @code{inputs} and the like should be used. - -Care should be taken to ensure the correct version of dependencies are used; to -this end we try to refrain from skipping the tests or using @code{#:skip-build?} -when possible. Of course this is not always possible, as the package may be -developed for a different Operating System, depend on features from the Nightly -Rust compiler, or the test suite may have atrophied since it was released. +package to be used at any given time, so all dependencies should have a +versioned suffix. The versioned suffix is the left-most non-zero digit (and any +leading zeros, of course). This follows the ``caret'' version scheme intended +by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}. + +In practice we are usually packaging development snapshots of Rust dependencies +specifically for some Rust programs, and can't simply identity them by version. +In this case we can use a @code{for-@var{program}} suffix. Examples@: +@code{rust-pipewire-for-niri}, @code{rust-pubgrub-for-uv}. +@end enumerate + +Let's demonstrate the packaging workflow by examples, note that package-specific +issues are not involved here. + +In preparation, we'll add the following packages to our environment: + +@example +guix shell rust rust:cargo cargo-audit cargo-license +@end example + +Example 1: @code{cargo-audit}, which is published on the crates.io Rust package +repository @uref{https://crates.io, crates.io}. + +@enumerate +@item +We can generate a draft definition via the crates.io importer (@pxref{Invoking +guix import, crate}). In the end we'll have the following definiton: + +@lisp +(define-public cargo-audit + (package + (name "cargo-audit") + (version "0.21.2") + (source + (origin + (method url-fetch) + (uri (crate-uri "cargo-audit" version)) + (file-name (string-append name "-" version ".tar.gz")) + (sha256 + (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n")))) + (build-system cargo-build-system) + (arguments (list #:install-source? #f)) + (inputs (cargo-inputs 'cargo-audit)) + (home-page "https://rustsec.org/") + (synopsis "Audit Cargo.lock for crates with security vulnerabilities") + (description + "This package provides a Cargo subcommand, @@command@{cargo audit@}, to +audit @@file@{Cargo.lock@} for crates with security vulnerabilities.") + (license (list license:asl2.0 license:expat)))) +@end lisp + +@code{cargo-inputs} is a procedure defined in @code{guix build-system cargo}, +facilitating dependency modification. @code{'cargo-audit} used here must be a +unique identifier, usually same to the program's variable name. + +@item +Unpack package source and navigate to the unpacked directory, then execute the +following commands: + +@example +cargo generate-lockfile +cargo audit +cargo license +@end example + +@command{cargo generate-lockfile} updates dependencies to compatible versions, +@command{cargo audit} checks known vulnerabilities and @command{cargo license} +checks licenses of all dependencies. + +We must have an acceptable output of @command{cargo audit} and ensure all +dependencies are licensed with our supported licenses (@pxref{Defining Packages, +@code{license}}). + +@item +Import dependencies from previously generated lockfile: + +@example +guix import --insert=gnu/packages/rust-crates.scm \ + crate --lockfile=/path/to/Cargo.lock cargo-audit +@end example + +@code{cargo-audit} used here must be consistent with the identifier used for +@code{cargo-inputs}. + +At this stage, @code{cargo-audit} is buildable. + +@item +Finally we'll unbundle vendored sources. The lockfile importer inserts +@code{TODO:} comments to dependencies with high probability of bundled sources. +@code{cargo-build-system} also performs additional check in its +@code{check-for-pregenerated-files} phase: + +@example +$ ./pre-inst-env guix build cargo-audit +@dots{} +starting phase `check-for-pregenerated-files' +Searching for binary files... +./guix-vendor/rust-addr2line-0.21.0.tar.gz/rustfmt.toml +./guix-vendor/rust-arc-swap-1.7.1.tar.gz/rustfmt.toml +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst +@dots{} +@end example + +Although dependencies in @code{(gnu packages rust-crates)} are not exported, we +can still select them in Guix command-line interface through expression: + +@example +guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)' +@end example + +For most dependencies, a snippet is sufficient: + +@lisp +(define rust-bzip2-sys-0.1.13+1.0.8 + (crate-source "bzip2-sys" "0.1.13+1.0.8" + "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2" + #:snippet + '(begin + (delete-file-recursively "bzip2-1.0.8") + (delete-file "build.rs") + ;; Inspired by Debian's patch. + (with-output-to-file "build.rs" + (lambda _ + (format #t "fn main() @{~@@ + println!(\"cargo:rustc-link-lib=bz2\");~@@ + @}~%")))))) +@end lisp + +In a more complex case, where unbundling one dependency requires other packages, +we should package the dependency in @code{(gnu packages rust-sources)} first and +point the imported definition to a symbol with the same name. + +For example we have defined a @code{rust-ring-0.17} in @code{(gnu packages +rust-sources)}, then the imported definition in @code{(gnu packages +rust-crates)} should be modified to point to it: + +@lisp +(define rust-ring-0.17.14 'rust-ring-0.17) +@end lisp + +When one dependency can be safely removed, modify it to @code{#f}. + +@lisp +(define rust-openssl-src-300.4.2+3.4.1 #f) +@end lisp + +Such modifications are processed by @code{cargo-inputs} procedure. + +@code{cargo-inputs} can also be configured by keywoard arguments +@code{#:crates-module} (default: @code{(gnu packages rust-crates)}, module to +resolve imported definitions) and @code{#:sources-module} (default: @code{(gnu +packages rust-sources)}, module to resolve modified symbols), which are useful +to maintain an independent channel. +@end enumerate + +Example 2: @code{niri}, which is not available on crates.io and depends on +development snapshots (also Cargo workspaces in this example). + +@enumerate +@item +As we can't ensure compatibility of a development snapshot, before executing +@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it +in a known working revision. + +To use our packaged development snapshots, it's also necessary to modify +@file{Cargo.toml} in build environment, the substitution pattern is +package-specific. + +@code{cargo-inputs} returns a list, all list operations apply to it too. + +@lisp +(define-public niri + (package + (name "niri") + (version "25.02") + (source (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/YaLTeR/niri") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r")))) + (build-system cargo-build-system) + (arguments + (list #:install-source? #f + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack 'use-guix-vendored-dependencies + (lambda _ + (substitute* "Cargo.toml" + (("# version =.*") + "version = \"*\"") + (("git.*optional") + "version = \"*\", optional") + (("^git = .*") + ""))))))) + (native-inputs + (list pkg-config)) + (inputs + (cons* clang + libdisplay-info + libinput-minimal + libseat + libxkbcommon + mesa + pango + pipewire + wayland + (cargo-inputs 'niri))) + (home-page "https://github.com/YaLTeR/niri") + (synopsis "Scrollable-tiling Wayland compositor") + (description + "Niri is a scrollable-tiling Wayland compositor which arranges windows in a +scrollable format. It is considered stable for daily use and performs most +functions expected of a Wayland compositor.") + (license license:gpl3))) +@end lisp + +@item +@code{niri} also has Cargo workspace dependencies. When packaging a Cargo +workspace, build argument @code{#:cargo-package-crates} is required. + +@lisp +(define-public rust-pipewire-for-niri + (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9") + (revision "0")) + (package + (name "rust-pipewire") + (version (git-version "0.8.0" revision commit)) + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git") + (commit commit))) + (file-name (git-file-name name version)) + (sha256 + (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v")))) + (build-system cargo-build-system) + (arguments + (list #:skip-build? #t + #:cargo-package-crates + ''("libspa-sys" "libspa" "pipewire-sys" "pipewire"))) + (inputs (cargo-inputs 'rust-pipewire-for-niri)) + (home-page "https://pipewire.org/") + (synopsis "Rust bindings for PipeWire") + (description "This package provides Rust bindings for PipeWire.") + (license license:expat)))) +@end lisp + +Don't forget to modify all workspace members in @code{(gnu packages +rust-crates)}: + +@lisp +(define rust-pipewire-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +(define rust-pipewire-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +@dots{} +(define rust-libspa-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +(define rust-libspa-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +@end lisp +@end enumerate + +Example 3: @code{fish}, which combines two build systems. + +@enumerate +@item +When building Rust packages in other build systems, we need to add @code{rust} +and @code{rust:cargo} to @code{native-inputs}, import and use modules from both +build systems, and apply necessary build phases from @code{cargo-build-system}. + +@lisp +(define-public fish + (package + (name "fish") + (version "4.0.0") + (source + (origin + (method url-fetch) + (uri (string-append "https://github.com/fish-shell/fish-shell/" + "releases/download/" version "/" + "fish-" version ".tar.xz")) + (sha256 + (base32 "1wv9kjwg6ax8m2f85i58l9f9cndshn1f15n8skc68w1mf3cmpnig")))) + (build-system cmake-build-system) + (inputs + (cons* fish-foreign-env ncurses pcre2 + python ;for fish_config and manpage completions + (cargo-inputs 'fish))) + (native-inputs + (list doxygen groff ;for 'fish --help' + pkg-config + procps ;for the test suite + rust + `(,rust "cargo"))) + (arguments + (list #:imported-modules + (append %cargo-build-system-modules + %cmake-build-system-modules) + #:modules + (((guix build cargo-build-system) #:prefix cargo:) + (guix build cmake-build-system) + (guix build utils)) + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack 'use-guix-vendored-dependencies + (lambda _ + (substitute* "Cargo.toml" + (("git.*tag.*,") + "version = \"*\",")))) + (add-after 'unpack 'prepare-cargo-build-system + (lambda args + (for-each + (lambda (phase) + (format #t "Running cargo phase: ~a~%" phase) + (apply (assoc-ref cargo:%standard-phases phase) + args)) + '(unpack-rust-crates + configure + check-for-pregenerated-files + patch-cargo-checksums))))))) + (synopsis "The friendly interactive shell") + (description + "Fish (friendly interactive shell) is a shell focused on interactive use, +discoverability, and friendliness. Fish has very user-friendly and powerful +tab-completion, including descriptions of every completion, completion of +strings with wildcards, and many completions for specific commands. It also +has extensive and discoverable help. A special @@command@{help@} command gives +access to all the fish documentation in your web browser. Other features +include smart terminal handling based on terminfo, an easy to search history, +and syntax highlighting.") + (home-page "https://fishshell.com/") + (license license:gpl2))) +@end lisp +@end enumerate @node Elm Packages