From patchwork Sun Mar 23 07:23:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hilton Chain X-Patchwork-Id: 40701 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3C39E27BBEA; Sun, 23 Mar 2025 07:26:39 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-5.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,FROM_SUSPICIOUS_NTLD,MAILING_LIST_MULTI, PDS_OTHER_BAD_TLD,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_VALIDITY_CERTIFIED, RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE,SPF_HELO_PASS, URIBL_BLOCKED,URIBL_SBL_A autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id F2A3727BBE2 for ; Sun, 23 Mar 2025 07:26:36 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1twFil-00006I-2v; Sun, 23 Mar 2025 03:26:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1twFig-0008Uc-HD for guix-patches@gnu.org; Sun, 23 Mar 2025 03:26:06 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1twFif-0007Qe-QG; Sun, 23 Mar 2025 03:26:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:From:To:In-Reply-To:References:Subject; bh=gFpklLjfUq41WPTpcxU/7hnv7wKJEuTxt5Ot13FjGMA=; b=JKeQOlKPsgHz2CcSb/0l8aRvMo1RmF092ieaFO0tqw768KqUm9IO4XTLfS7UOwFkDLl5pfhJ/vLbDGF36ebqBVeTAJVcaoImoI2TgIE5eSWWgopknor+ZHH9Yb6WJN4AoMVm/2JEaYfwoC64zMlFT3IeLRV+eclv9Y0fdjaEv+KqIoVyrmvP7XErvODR7nFIHRMG+X3t5d2MXq6VyZhgyzYW4w4mJjQ2v0BM0yUvizET3vATLHJMJ3Eqo1CkW9X3GSDSLry8QWaYEF8B5qxlu71uA103eRqP/sRKJ6Stpf9VBi1lrjm7XDXe+/58Xahp5UCjjAN0fZUx+ZFea5DiTQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1twFic-0003eV-PV; Sun, 23 Mar 2025 03:26:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#77093] [PATCH rust-team v3 00/17] New Rust packaging workflow based on lockfile importer. References: In-Reply-To: Resent-From: Hilton Chain Original-Sender: "Debbugs-submit" Resent-CC: hako@ultrarare.space, efraim@flashner.co.il, maxim.cournoyer@gmail.com, ludo@gnu.org, guix-patches@gnu.org Resent-Date: Sun, 23 Mar 2025 07:26:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 77093 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 77093@debbugs.gnu.org Cc: Hilton Chain , Hilton Chain , Efraim Flashner , Maxim Cournoyer , Ludovic =?utf-8?q?Court=C3=A8s?= X-Debbugs-Original-Xcc: Hilton Chain , Efraim Flashner , Maxim Cournoyer , Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 77093-submit@debbugs.gnu.org id=B77093.174271473313721 (code B ref 77093); Sun, 23 Mar 2025 07:26:02 +0000 Received: (at 77093) by debbugs.gnu.org; 23 Mar 2025 07:25:33 +0000 Received: from localhost ([127.0.0.1]:47265 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1twFi0-0003Ys-6E for submit@debbugs.gnu.org; Sun, 23 Mar 2025 03:25:32 -0400 Received: from mx.boiledscript.com ([2a01:4f8:10b:392::42]:39424) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1twFhg-0003QF-On for 77093@debbugs.gnu.org; Sun, 23 Mar 2025 03:25:14 -0400 From: Hilton Chain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ultrarare.space; s=mail; t=1742714702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=uykQ3f9snLl0Jk8YahIOIV0raQ2+niRJWypqAqTgSSI=; b=d9cB/Nd8tRtZD/H/Wj+3e/vPoIpD5rtQqgRijSzUfjAf3hKZl+YNvSzfxxc2wklS9XNTuA lS4aFHSULImSpkvqlQC/6as10sESUIACYUhvX8LfyDXtbpSoGJ5Ng/iMdJIP71AFrlvMu2 GDq4ZEtSopGDoQqZ0i0NCotpTTghqkbkoCnz7O6bJphJveTyVW1uK353VF2+DpY0mXpH7N cdEBZkU57zXPxjxzVtds4U4PC3n87fWyQXjNeJjrsdrfCZ6f8xCRsVCYyZ8RynWHqmYfHx 89+D4j3zlfqYVxB2SmBnL3IB6EHEUxl2yYjqmWHW2Ygn/ASIhMXDn19QbUZ4sw== Date: Sun, 23 Mar 2025 15:23:04 +0800 Message-ID: MIME-Version: 1.0 X-MS-Reactions: disallow X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches V2 -> V3: * Add Efraim's etc/teams/rust/audit-rust-crates script. * Add (guix import crate cargo-lock) to Rust team's scope. * ‘definer’ -> ‘define-prefix’ * Adjust documentation and move examples to Guix Cookbook. * Document ‘cargo-inputs’ and remove mentioning of files under etc/teams/rust. Diff inserted at end. Efraim Flashner (1): build/cargo: Don't try to unpack sanity-check.py. Hilton Chain (16): build/cargo: Pass ‘--offline’ to cargo. build/cargo: Print out all non-empty binary files. build-system: cargo: Support packaging Cargo workspace. build-system: cargo: Support installing Cargo workspace. build/cargo: Set default value of arguments for build phases. build/cargo: Support non-workspace directory source inputs. scripts: import: Document argument for ‘--insert’ option in help message. scripts: import: Add two newlines for ‘--insert’ option. scripts: import: Support expressions defined by 'define. scripts: import: Pass "--insert" to importers. scripts: import: Skip existing definition for ‘--insert’ option. import: crate: crate-name->package-name: Move to (guix build-system cargo). build-system: cargo: Add ‘cargo-inputs’. import: crate: Add Cargo.lock parser. import: crate: Add ‘--lockfile’ option. doc: Document lockfile importer based Rust packaging workflow. Makefile.am | 1 + doc/contributing.texi | 91 +++++--- doc/guix-cookbook.texi | 352 ++++++++++++++++++++++++++++++ doc/guix.texi | 24 ++ etc/teams.scm | 1 + etc/teams/rust/audit-rust-crates | 70 ++++++ etc/teams/rust/cleanup-crates.sh | 37 ++++ etc/teams/rust/rust-crates.tmpl | 42 ++++ gnu/local.mk | 2 + gnu/packages/rust-crates.scm | 42 ++++ gnu/packages/rust-sources.scm | 29 +++ guix/build-system/cargo.scm | 59 ++++- guix/build/cargo-build-system.scm | 150 +++++++++---- guix/import/crate.scm | 81 ++++++- guix/import/crate/cargo-lock.scm | 105 +++++++++ guix/scripts/import.scm | 54 +++-- guix/scripts/import/crate.scm | 58 ++++- guix/utils.scm | 29 ++- tests/crate.scm | 88 ++++++++ 19 files changed, 1209 insertions(+), 106 deletions(-) create mode 100755 etc/teams/rust/audit-rust-crates create mode 100755 etc/teams/rust/cleanup-crates.sh create mode 100644 etc/teams/rust/rust-crates.tmpl create mode 100644 gnu/packages/rust-crates.scm create mode 100644 gnu/packages/rust-sources.scm create mode 100644 guix/import/crate/cargo-lock.scm base-commit: 127e1e89a83785f4ad8362bdc76b0d850279cf2f --- 2.49.0 diff --git a/doc/contributing.texi b/doc/contributing.texi index 837074dead..17f22390d5 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -1600,12 +1600,6 @@ Rust Crates @subsection Rust Crates @cindex rust -As currently there's no achievable way to reuse Rust packages as pre-compiled -inputs for other packages at a distribution scale, and our previous approach on -making all Rust dependencies as full packages doesn't cope well with Rust -ecosystem thus causing issues on both contribution and maintenance sides, we -have switched to a different packaging model. - Rust programs (binary crates) and dependencies (library crates) are treated separately. We put our main efforts into programs and only package Rust dependencies as sources, utilizing automation with a manual focus on unbundling @@ -1625,19 +1619,11 @@ Rust Crates @item @code{(gnu packages rust-crates)}, storing source definitions imported from Rust programs' @file{Cargo.lock} via the lockfile importer (@pxref{Invoking guix -import, crate, @code{--lockfile=@var{file}}}). +import, @code{crate}, @code{--lockfile=@var{file}}}). Imported definitions must be checked and have vendored sources unbundled before -contributing to Guix. Naturally, this module serves as a store for both sources -and unbundling strategies. - -This module is managed by the Rust team (@pxref{Teams}) to ensure there's always -one version containing all changes from other branches, so that the maintained -version can be used directly in case of merge conflicts, thus coordination is -required for other committers to modify it. - -Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup -script @file{etc/teams/rust/cleanup-crates.sh} for this module. +being contributed to Guix. This module is managed by the Rust team +(@pxref{Teams}). @item @code{(gnu packages rust-sources)}, storing more complex definitions that need @@ -1661,338 +1647,28 @@ Rust Crates In practice we are usually packaging development snapshots of Rust dependencies specifically for some Rust programs, and can't simply identity them by version. -In this case we can use a @code{for-@var{program}} suffix. Examples@: -@code{rust-pipewire-for-niri}, @code{rust-pubgrub-for-uv}. -@end enumerate - -Let's demonstrate the packaging workflow by examples, note that package-specific -issues are not involved here. - -In preparation, we'll add the following packages to our environment: - -@example -guix shell rust rust:cargo cargo-audit cargo-license -@end example - -Example 1: @code{cargo-audit}, which is published on the crates.io Rust package -repository @uref{https://crates.io, crates.io}. - -@enumerate -@item -We can generate a draft definition via the crates.io importer (@pxref{Invoking -guix import, crate}). In the end we'll have the following definiton: - -@lisp -(define-public cargo-audit - (package - (name "cargo-audit") - (version "0.21.2") - (source - (origin - (method url-fetch) - (uri (crate-uri "cargo-audit" version)) - (file-name (string-append name "-" version ".tar.gz")) - (sha256 - (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n")))) - (build-system cargo-build-system) - (arguments (list #:install-source? #f)) - (inputs (cargo-inputs 'cargo-audit)) - (home-page "https://rustsec.org/") - (synopsis "Audit Cargo.lock for crates with security vulnerabilities") - (description - "This package provides a Cargo subcommand, @@command@{cargo audit@}, to -audit @@file@{Cargo.lock@} for crates with security vulnerabilities.") - (license (list license:asl2.0 license:expat)))) -@end lisp - -@code{cargo-inputs} is a procedure defined in @code{guix build-system cargo}, -facilitating dependency modification. @code{'cargo-audit} used here must be a -unique identifier, usually same to the program's variable name. - -@item -Unpack package source and navigate to the unpacked directory, then execute the -following commands: - -@example -cargo generate-lockfile -cargo audit -cargo license -@end example - -@command{cargo generate-lockfile} updates dependencies to compatible versions, -@command{cargo audit} checks known vulnerabilities and @command{cargo license} -checks licenses of all dependencies. - -We must have an acceptable output of @command{cargo audit} and ensure all -dependencies are licensed with our supported licenses (@pxref{Defining Packages, -@code{license}}). - -@item -Import dependencies from previously generated lockfile: - -@example -guix import --insert=gnu/packages/rust-crates.scm \ - crate --lockfile=/path/to/Cargo.lock cargo-audit -@end example - -@code{cargo-audit} used here must be consistent with the identifier used for -@code{cargo-inputs}. - -At this stage, @code{cargo-audit} is buildable. - -@item -Finally we'll unbundle vendored sources. The lockfile importer inserts -@code{TODO:} comments to dependencies with high probability of bundled sources. -@code{cargo-build-system} also performs additional check in its -@code{check-for-pregenerated-files} phase: - -@example -$ ./pre-inst-env guix build cargo-audit -@dots{} -starting phase `check-for-pregenerated-files' -Searching for binary files... -./guix-vendor/rust-addr2line-0.21.0.tar.gz/rustfmt.toml -./guix-vendor/rust-arc-swap-1.7.1.tar.gz/rustfmt.toml -./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust -./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other -./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst -./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst -@dots{} -@end example - -Although dependencies in @code{(gnu packages rust-crates)} are not exported, we -can still select them in Guix command-line interface through expression: - -@example -guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)' -@end example - -For most dependencies, a snippet is sufficient: - -@lisp -(define rust-bzip2-sys-0.1.13+1.0.8 - (crate-source "bzip2-sys" "0.1.13+1.0.8" - "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2" - #:snippet - '(begin - (delete-file-recursively "bzip2-1.0.8") - (delete-file "build.rs") - ;; Inspired by Debian's patch. - (with-output-to-file "build.rs" - (lambda _ - (format #t "fn main() @{~@@ - println!(\"cargo:rustc-link-lib=bz2\");~@@ - @}~%")))))) -@end lisp - -In a more complex case, where unbundling one dependency requires other packages, -we should package the dependency in @code{(gnu packages rust-sources)} first and -point the imported definition to a symbol with the same name. - -For example we have defined a @code{rust-ring-0.17} in @code{(gnu packages -rust-sources)}, then the imported definition in @code{(gnu packages -rust-crates)} should be modified to point to it: - -@lisp -(define rust-ring-0.17.14 'rust-ring-0.17) -@end lisp - -When one dependency can be safely removed, modify it to @code{#f}. - -@lisp -(define rust-openssl-src-300.4.2+3.4.1 #f) -@end lisp - -Such modifications are processed by @code{cargo-inputs} procedure. - -@code{cargo-inputs} can also be configured by keywoard arguments -@code{#:crates-module} (default: @code{(gnu packages rust-crates)}, module to -resolve imported definitions) and @code{#:sources-module} (default: @code{(gnu -packages rust-sources)}, module to resolve modified symbols), which are useful -to maintain an independent channel. +In this case we can use a @code{for-@var{program}} suffix, for example, +@code{rust-pipewire-for-niri} and @code{rust-pubgrub-for-uv}. @end enumerate -Example 2: @code{niri}, which is not available on crates.io and depends on -development snapshots (also Cargo workspaces in this example). +Rust dependencies are not referenced directly. @code{(guix build-sytem cargo)} +provides a @code{cargo-inputs} procedure to create an input list, in combination +with the lockfile importer. -@enumerate -@item -As we can't ensure compatibility of a development snapshot, before executing -@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it -in a known working revision. +@cindex cargo-inputs +@deffn {Procedure} cargo-inputs @var{name} @ + [#:crates-module '(gnu packages rust-crates)] @ + [#:sources-module '(gnu packages rust-sources)] +Given symbol @code{'@var{name}}, resolve variable +@code{@var{name}-cargo-inputs}, an input list, in @var{crates-module}, return +its copy with @code{#f} removed and symbols resolved to variables defined in +@var{sources-module} if the input list exists, otherwise return an empty list. -To use our packaged development snapshots, it's also necessary to modify -@file{Cargo.toml} in build environment, the substitution pattern is -package-specific. - -@code{cargo-inputs} returns a list, all list operations apply to it too. - -@lisp -(define-public niri - (package - (name "niri") - (version "25.02") - (source (origin - (method git-fetch) - (uri (git-reference - (url "https://github.com/YaLTeR/niri") - (commit (string-append "v" version)))) - (file-name (git-file-name name version)) - (sha256 - (base32 - "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r")))) - (build-system cargo-build-system) - (arguments - (list #:install-source? #f - #:phases - #~(modify-phases %standard-phases - (add-after 'unpack 'use-guix-vendored-dependencies - (lambda _ - (substitute* "Cargo.toml" - (("# version =.*") - "version = \"*\"") - (("git.*optional") - "version = \"*\", optional") - (("^git = .*") - ""))))))) - (native-inputs - (list pkg-config)) - (inputs - (cons* clang - libdisplay-info - libinput-minimal - libseat - libxkbcommon - mesa - pango - pipewire - wayland - (cargo-inputs 'niri))) - (home-page "https://github.com/YaLTeR/niri") - (synopsis "Scrollable-tiling Wayland compositor") - (description - "Niri is a scrollable-tiling Wayland compositor which arranges windows in a -scrollable format. It is considered stable for daily use and performs most -functions expected of a Wayland compositor.") - (license license:gpl3))) -@end lisp - -@item -@code{niri} also has Cargo workspace dependencies. When packaging a Cargo -workspace, build argument @code{#:cargo-package-crates} is required. - -@lisp -(define-public rust-pipewire-for-niri - (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9") - (revision "0")) - (package - (name "rust-pipewire") - (version (git-version "0.8.0" revision commit)) - (source - (origin - (method git-fetch) - (uri (git-reference - (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git") - (commit commit))) - (file-name (git-file-name name version)) - (sha256 - (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v")))) - (build-system cargo-build-system) - (arguments - (list #:skip-build? #t - #:cargo-package-crates - ''("libspa-sys" "libspa" "pipewire-sys" "pipewire"))) - (inputs (cargo-inputs 'rust-pipewire-for-niri)) - (home-page "https://pipewire.org/") - (synopsis "Rust bindings for PipeWire") - (description "This package provides Rust bindings for PipeWire.") - (license license:expat)))) -@end lisp - -Don't forget to modify all workspace members in @code{(gnu packages -rust-crates)}: - -@lisp -(define rust-pipewire-0.8.0.fd3d8f7 'rust-pipewire-for-niri) -(define rust-pipewire-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri) -@dots{} -(define rust-libspa-0.8.0.fd3d8f7 'rust-pipewire-for-niri) -(define rust-libspa-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri) -@end lisp -@end enumerate - -Example 3: @code{fish}, which combines two build systems. - -@enumerate -@item -When building Rust packages in other build systems, we need to add @code{rust} -and @code{rust:cargo} to @code{native-inputs}, import and use modules from both -build systems, and apply necessary build phases from @code{cargo-build-system}. +@var{name} must be consistent with the one used in lockfile importer invocation. +@end deffn -@lisp -(define-public fish - (package - (name "fish") - (version "4.0.0") - (source - (origin - (method url-fetch) - (uri (string-append "https://github.com/fish-shell/fish-shell/" - "releases/download/" version "/" - "fish-" version ".tar.xz")) - (sha256 - (base32 "1wv9kjwg6ax8m2f85i58l9f9cndshn1f15n8skc68w1mf3cmpnig")))) - (build-system cmake-build-system) - (inputs - (cons* fish-foreign-env ncurses pcre2 - python ;for fish_config and manpage completions - (cargo-inputs 'fish))) - (native-inputs - (list doxygen groff ;for 'fish --help' - pkg-config - procps ;for the test suite - rust - `(,rust "cargo"))) - (arguments - (list #:imported-modules - (append %cargo-build-system-modules - %cmake-build-system-modules) - #:modules - (((guix build cargo-build-system) #:prefix cargo:) - (guix build cmake-build-system) - (guix build utils)) - #:phases - #~(modify-phases %standard-phases - (add-after 'unpack 'use-guix-vendored-dependencies - (lambda _ - (substitute* "Cargo.toml" - (("git.*tag.*,") - "version = \"*\",")))) - (add-after 'unpack 'prepare-cargo-build-system - (lambda args - (for-each - (lambda (phase) - (format #t "Running cargo phase: ~a~%" phase) - (apply (assoc-ref cargo:%standard-phases phase) - args)) - '(unpack-rust-crates - configure - check-for-pregenerated-files - patch-cargo-checksums))))))) - (synopsis "The friendly interactive shell") - (description - "Fish (friendly interactive shell) is a shell focused on interactive use, -discoverability, and friendliness. Fish has very user-friendly and powerful -tab-completion, including descriptions of every completion, completion of -strings with wildcards, and many completions for specific commands. It also -has extensive and discoverable help. A special @@command@{help@} command gives -access to all the fish documentation in your web browser. Other features -include smart terminal handling based on terminfo, an easy to search history, -and syntax highlighting.") - (home-page "https://fishshell.com/") - (license license:gpl2))) -@end lisp -@end enumerate +For a more detailed packaging workflow, @pxref{Packaging Rust Crates,,, +guix-cookbook, GNU Guix Cookbook}. @node Elm Packages diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index d9b98a2ab3..1f0e5eee54 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -103,6 +103,7 @@ Top Packaging * Packaging Tutorial:: A tutorial on how to add packages to Guix. +* Packaging Workflows:: Real life examples on working with specific build systems. Packaging Tutorial @@ -127,6 +128,16 @@ Top * Automatic update:: * Inheritance:: +Packaging Workflows + +* Packaging Rust Crates:: + +Packaging Rust Crates + +* Common Workflow for Packaging Rust Crates:: +* Development Snapshots and Cargo Workspaces:: +* Rust Programs With Mixed Build Systems:: + System Configuration * Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY @@ -514,6 +525,7 @@ Packaging @menu * Packaging Tutorial:: A tutorial on how to add packages to Guix. +* Packaging Workflows:: Real life examples on working with specific build systems. @end menu @node Packaging Tutorial @@ -1593,6 +1605,346 @@ References @uref{https://www.gnu.org/software/guix/guix-ghm-andreas-20130823.pdf, ``GNU Guix: Package without a scheme!''}, by Andreas Enge @end itemize +@node Packaging Workflows +@section Packaging Workflows + +The following sections are real life examples on working with specific build +systems, serving as extensions to the concise packaging guidelines +(@pxref{Packaging Guidelines,,, guix, GNU Guix Reference Manual}). + +@menu +* Packaging Rust Crates:: +@end menu + +@node Packaging Rust Crates +@subsection Packaging Rust Crates + +In preparation, add the following packages to our environment: + +@example +$ guix shell rust rust:cargo cargo-audit cargo-license +@end example + +@menu +* Common Workflow for Packaging Rust Crates:: +* Development Snapshots and Cargo Workspaces:: +* Rust Programs With Mixed Build Systems:: +@end menu + +@node Common Workflow for Packaging Rust Crates +@subsubsection Common Workflow for Packaging Rust Crates + +In this example, we'll package @code{cargo-audit}, which is published on the +@uref{https://crates.io, crates.io} Rust package repository. All its +dependencies are on crates.io as well. + +@enumerate +@item +Since @code{cargo-audit} is available on crates.io, We can generate a draft +definition via the crates.io importer (@pxref{Invoking guix import,,, guix, GNU +Guix Reference Manual}). In the end we'll have the following definiton: + +@lisp +(define-public cargo-audit + (package + (name "cargo-audit") + (version "0.21.2") + (source + (origin + (method url-fetch) + (uri (crate-uri "cargo-audit" version)) + (file-name (string-append name "-" version ".tar.gz")) + (sha256 + (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n")))) + (build-system cargo-build-system) + (arguments (list #:install-source? #f)) + (inputs (cargo-inputs 'cargo-audit)) + (home-page "https://rustsec.org/") + (synopsis "Audit Cargo.lock for crates with security vulnerabilities") + (description + "This package provides a Cargo subcommand, @@command@{cargo audit@}, to +audit @@file@{Cargo.lock@} for crates with security vulnerabilities.") + (license (list license:asl2.0 license:expat)))) +@end lisp + +The symbol used in @code{cargo-inputs}, @code{'cargo-audit} here, must be a +unique identifier, usually matching the variable name of the package. + +@item +Unpack package source and navigate to the unpacked directory, then execute the +following commands: + +@example +$ cargo generate-lockfile +$ cargo audit +$ cargo license +@end example + +@command{cargo generate-lockfile} updates dependencies to compatible versions, +@command{cargo audit} checks known vulnerabilities and @command{cargo license} +checks licenses of all dependencies. + +We must have an acceptable output of @command{cargo audit} and ensure all +dependencies are licensed with our supported licenses (@pxref{Defining +Packages,,, guix, GNU Guix Reference Manual}). + +@item +Import dependencies from previously generated lockfile: + +@example +$ guix import --insert=gnu/packages/rust-crates.scm \ + crate --lockfile=/path/to/Cargo.lock cargo-audit +@end example + +@code{cargo-audit} used here must be consistent with the identifier used for +@code{cargo-inputs}. + +At this stage, package @code{cargo-audit} is buildable. + +@item +Finally we'll unbundle vendored sources. The lockfile importer inserts +@code{TODO:} comments to dependencies with high probability of bundled sources. +@code{cargo-build-system} also performs additional check in its +@code{check-for-pregenerated-files} phase: + +@example +$ ./pre-inst-env guix build cargo-audit +@dots{} +starting phase `check-for-pregenerated-files' +Searching for binary files... +./guix-vendor/rust-addr2line-0.21.0.tar.gz/rustfmt.toml +./guix-vendor/rust-arc-swap-1.7.1.tar.gz/rustfmt.toml +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst +@dots{} +@end example + +Although dependencies in @code{(gnu packages rust-crates)} are not exported, we +can still select them via the Guix command-line interface through expression: + +@example +$ guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)' +@end example + +For most dependencies, a snippet is sufficient: + +@lisp +(define rust-bzip2-sys-0.1.13+1.0.8 + (crate-source "bzip2-sys" "0.1.13+1.0.8" + "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2" + #:snippet + '(begin + (delete-file-recursively "bzip2-1.0.8") + (delete-file "build.rs") + ;; Inspired by Debian's patch. + (with-output-to-file "build.rs" + (lambda _ + (format #t "fn main() @{~@@ + println!(\"cargo:rustc-link-lib=bz2\");~@@ + @}~%")))))) +@end lisp + +In a more complex case, where unbundling one dependency requires other packages, +we should package the dependency in @code{(gnu packages rust-sources)} first and +reference it by a symbol in the imported definition. + +For example we have defined a @code{rust-ring-0.17} in @code{(gnu packages +rust-sources)}, then the imported definition in @code{(gnu packages +rust-crates)} should be modified to a matching symbol. + +@lisp +(define rust-ring-0.17.14 'rust-ring-0.17) +@end lisp + +When one dependency can be safely removed, modify it to @code{#f}. + +@lisp +(define rust-openssl-src-300.4.2+3.4.1 #f) +@end lisp +@end enumerate + +@node Development Snapshots and Cargo Workspaces +@subsubsection Development Snapshots and Cargo Workspaces + +In this example, we'll package @code{niri}, which depends on development +snapshots (also Cargo workspaces here). + +As we can't ensure compatibility of a development snapshot, before executing +@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it +to a known working revision. + +To use our packaged development snapshots, it's also necessary to modify +@file{Cargo.toml} in a build phase, with a package-specific substitution +pattern. + +@lisp +(define-public niri + (package + (name "niri") + (version "25.02") + (source (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/YaLTeR/niri") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r")))) + (build-system cargo-build-system) + (arguments + (list #:install-source? #f + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack 'use-guix-vendored-dependencies + (lambda _ + (substitute* "Cargo.toml" + (("# version =.*") + "version = \"*\"") + (("git.*optional") + "version = \"*\", optional") + (("^git = .*") + ""))))))) + (native-inputs + (list pkg-config)) + (inputs + (cons* clang + libdisplay-info + libinput-minimal + libseat + libxkbcommon + mesa + pango + pipewire + wayland + (cargo-inputs 'niri))) + (home-page "https://github.com/YaLTeR/niri") + (synopsis "Scrollable-tiling Wayland compositor") + (description + "Niri is a scrollable-tiling Wayland compositor which arranges windows in a +scrollable format. It is considered stable for daily use and performs most +functions expected of a Wayland compositor.") + (license license:gpl3+))) +@end lisp + +@code{niri} has Cargo workspace dependencies. When packaging a Cargo workspace, +argument @code{#:cargo-package-crates} is required. + +@lisp +(define-public rust-pipewire-for-niri + (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9") + (revision "0")) + (package + (name "rust-pipewire") + (version (git-version "0.8.0" revision commit)) + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git") + (commit commit))) + (file-name (git-file-name name version)) + (sha256 + (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v")))) + (build-system cargo-build-system) + (arguments + (list #:skip-build? #t + #:cargo-package-crates + ''("libspa-sys" "libspa" "pipewire-sys" "pipewire"))) + (inputs (cargo-inputs 'rust-pipewire-for-niri)) + (home-page "https://pipewire.org/") + (synopsis "Rust bindings for PipeWire") + (description "This package provides Rust bindings for PipeWire.") + (license license:expat)))) +@end lisp + +Don't forget to modify all workspace members in @code{(gnu packages +rust-crates)}: + +@lisp +(define rust-pipewire-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +(define rust-pipewire-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +@dots{} +(define rust-libspa-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +(define rust-libspa-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri) +@end lisp + +@node Rust Programs With Mixed Build Systems +@subsubsection Rust Programs With Mixed Build Systems + +In this example, we'll package @code{fish}, which combines two build systems. + +When building Rust packages in other build systems, we need to add @code{rust} +and @code{rust:cargo} to @code{native-inputs}, import and use modules from both +build systems, and apply necessary build phases from @code{cargo-build-system}. + +@lisp +(define-public fish + (package + (name "fish") + (version "4.0.0") + (source + (origin + (method url-fetch) + (uri (string-append "https://github.com/fish-shell/fish-shell/" + "releases/download/" version "/" + "fish-" version ".tar.xz")) + (sha256 + (base32 "1wv9kjwg6ax8m2f85i58l9f9cndshn1f15n8skc68w1mf3cmpnig")))) + (build-system cmake-build-system) + (inputs + (cons* fish-foreign-env ncurses pcre2 + python ;for fish_config and manpage completions + (cargo-inputs 'fish))) + (native-inputs + (list doxygen groff ;for 'fish --help' + pkg-config + procps ;for the test suite + rust + `(,rust "cargo"))) + (arguments + (list #:imported-modules + (append %cargo-build-system-modules + %cmake-build-system-modules) + #:modules + (((guix build cargo-build-system) #:prefix cargo:) + (guix build cmake-build-system) + (guix build utils)) + #:phases + #~(modify-phases %standard-phases + (add-after 'unpack 'use-guix-vendored-dependencies + (lambda _ + (substitute* "Cargo.toml" + (("git.*tag.*,") + "version = \"*\",")))) + (add-after 'unpack 'prepare-cargo-build-system + (lambda args + (for-each + (lambda (phase) + (format #t "Running cargo phase: ~a~%" phase) + (apply (assoc-ref cargo:%standard-phases phase) + args)) + '(unpack-rust-crates + configure + check-for-pregenerated-files + patch-cargo-checksums))))))) + (synopsis "The friendly interactive shell") + (description + "Fish (friendly interactive shell) is a shell focused on interactive use, +discoverability, and friendliness. Fish has very user-friendly and powerful +tab-completion, including descriptions of every completion, completion of +strings with wildcards, and many completions for specific commands. It also +has extensive and discoverable help. A special @@command@{help@} command gives +access to all the fish documentation in your web browser. Other features +include smart terminal handling based on terminfo, an easy to search history, +and syntax highlighting.") + (home-page "https://fishshell.com/") + (license license:gpl2))) +@end lisp + + @c ********************************************************************* @node System Configuration @chapter System Configuration diff --git a/etc/teams.scm b/etc/teams.scm index 6bddbd91fa..c4bee8d3d9 100755 --- a/etc/teams.scm +++ b/etc/teams.scm @@ -389,6 +389,7 @@ (define-team rust "guix/build/cargo-utils.scm" "guix/build-system/cargo.scm" "guix/import/crate.scm" + "guix/import/crate/cargo-lock.scm" "guix/scripts/import/crate.scm" "tests/crate.scm"))) diff --git a/etc/teams/rust/audit-rust-crates b/etc/teams/rust/audit-rust-crates new file mode 100755 index 0000000000..d5546fd1e1 --- /dev/null +++ b/etc/teams/rust/audit-rust-crates @@ -0,0 +1,70 @@ +#!/usr/bin/env -S gawk -f +# GNU Guix --- Functional package management for GNU +# Copyright © 2025 Efraim Flashner +# +# This file is part of GNU Guix. +# +# GNU Guix is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or (at +# your option) any later version. +# +# GNU Guix is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Guix. If not, see . + +# To run: +# ./etc/teams/rust/audit-rust-crates ./path/to/file.scm +# Prints the output of cargo-audit to the shell. + +# Make sure we have cargo-audit in our PATH +BEGIN { + if (system("which cargo-audit 1> /dev/null")) + exit 1; + # Parse a record at a time. + RS = "\n\n" + cargoAudit = "cargo-audit audit --file -" +} + +# Check the crate-source origin-only inputs +/crate-source/ { + for(i=3; i <= NF-2; i++) { + if($i == "(crate-source") { + cargoLock = cargoLock "[[package]]\nname = " $(i+1) "\nversion = " $(i+2) "\n" + next + } + } +} + +# Check the crates packaged from crates.io tarballs +/crate-uri/ { + for(i=3; i <= NF; i++) { + if($i == "(version") + crateVersion = $(i+1) + if($i == "(crate-uri") + crateName = $(i+1) + } + gsub(/)/, "", crateVersion) + cargoLock = cargoLock "[[package]]\nname = " crateName "\nversion = " crateVersion "\n" +} + +# The xxxx-cargo-input variables have a set style +# TODO: Replace the last dash between the name and the version with a space! +# This doesn't take into account swapping between "-" and "_" so we skip it. +#( $2 ~ /-cargo-inputs/ ) { +# sub(/-cargo-inputs/, "", $2) +# gsub(/)/, "", $0) +# gsub(/rust-/, "", $0) +# #gensub(/([[:alpha:]])-([[:digit:]]+)/, "\\1 \\2", "g", $i) +# print "[[package]]\nname = \"" $2 "\"\nversion = \"1.0.0\"\ndependencies = [" +# for (i = 4; i <= NF; i++) { +# print "\"" $i "\"," +# } +# print "]" +#} + +END { print cargoLock | cargoAudit } diff --git a/etc/teams/rust/cleanup-crates.sh b/etc/teams/rust/cleanup-crates.sh index cd4c2462b9..eca37ca00c 100755 --- a/etc/teams/rust/cleanup-crates.sh +++ b/etc/teams/rust/cleanup-crates.sh @@ -1,5 +1,4 @@ #!/bin/sh - # GNU Guix --- Functional package management for GNU # Copyright © 2025 Hilton Chain # @@ -28,7 +27,7 @@ do (begin (use-modules (guix utils)) (let ((source-properties - (find-definition-location \"$FILE\" '$crate #:definer 'define))) + (find-definition-location \"$FILE\" '$crate #:define-prefix 'define))) (and=> source-properties delete-expression)))" | guix repl -t machine fi diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm index 47d4f10969..bbfa2f933b 100644 --- a/guix/build-system/cargo.scm +++ b/guix/build-system/cargo.scm @@ -78,9 +78,11 @@ (define* (crate-source name version hash #:key (patches '()) (snippet #f)) (define* (cargo-inputs name #:key (crates-module '(gnu packages rust-crates)) (sources-module '(gnu packages rust-sources))) - "Given NAME, resolve input list 'NAME-cargo-inputs' in CRATES-MODULE, return -its copy with #f removed and symbols resolved to variables defined in -SOURCES-MODULE, if the list exists, otherwise return an empty list." + + "Given symbol NAME, resolve variable 'NAME-cargo-inputs', an input list, in +CRATES-MODULE, return its copy with #f removed and symbols resolved to +variables defined in SOURCES-MODULE if the input list exists, otherwise return +an empty list." (let loop ((inputs (catch #t (lambda () @@ -101,7 +103,7 @@ (define* (cargo-inputs name #:key (crates-module '(gnu packages rust-crates)) ((symbol? input) (cons (module-ref (resolve-interface sources-module) input) result)) - ;; Else: no change. + ;; Else: keep it. (else (cons input result))))))))) diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm index 41adc03752..1012d7f401 100644 --- a/guix/build/cargo-build-system.scm +++ b/guix/build/cargo-build-system.scm @@ -350,17 +350,16 @@ (define* (package #:key (if (null? cargo-package-crates) (apply invoke `("cargo" "package" "--offline" ,@cargo-package-flags)) - (begin - (for-each - (lambda (pkg) - (apply invoke "cargo" "package" "--offline" "--package" pkg - cargo-package-flags) - (for-each - (lambda (crate) - (invoke "tar" "xzf" crate "-C" vendor-dir)) - (find-files "target/package" "\\.crate$")) - (patch-cargo-checksums #:vendor-dir vendor-dir)) - cargo-package-crates))) + (for-each + (lambda (pkg) + (apply invoke "cargo" "package" "--offline" "--package" pkg + cargo-package-flags) + (for-each + (lambda (crate) + (invoke "tar" "xzf" crate "-C" vendor-dir)) + (find-files "target/package" "\\.crate$")) + (patch-cargo-checksums #:vendor-dir vendor-dir)) + cargo-package-crates)) ;; Then unpack the crate, reset the timestamp of all contained files, and ;; repack them. This is necessary to ensure that they are reproducible. diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index f6e2985ed9..1bddac15c6 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -126,21 +126,22 @@ (define-command (guix-import . args) (show-version-and-exit "guix import")) ((or ("-i" file importer args ...) ("--insert" file importer args ...)) - (let* ((importer-definers + (let* ((define-prefixes `(,@(if (member importer '("crate")) '(define) '()) define-public)) - (definer? (cut member <> importer-definers)) + (define-prefix? (cut member <> define-prefixes)) (find-and-insert (lambda (expr) (match expr - (((? definer? definer) term _ ...) + (((? define-prefix? define-prefix) term _ ...) ;; Skip existing definition. - (unless (find-definition-location file term #:definer definer) + (unless (find-definition-location + file term #:define-prefix define-prefix) (let ((source-properties (find-definition-insertion-location - file term #:definer definer))) + file term #:define-prefix define-prefix))) (if source-properties (insert-expression source-properties expr) (let ((port (open-file file "a"))) diff --git a/guix/utils.scm b/guix/utils.scm index 77ec6d992a..3f85320845 100644 --- a/guix/utils.scm +++ b/guix/utils.scm @@ -522,19 +522,19 @@ (define (insert-expression source-properties expr) (edit-expression source-properties insert))) (define* (find-definition-location file term - #:key (definer 'define-public) + #:key (define-prefix 'define-public) (pred string=)) - "Search in FILE for a top-level definition defined by DEFINER with defined -term comparing to TERM through PRED. Return the location if PRED returns #t, -or #f otherwise." + "Search in FILE for a top-level definition defined by DEFINE-PREFIX with +defined term comparing to TERM through PRED. Return the location if PRED +returns #t, or #f otherwise." (let ((search-term (symbol->string term)) - (definer? (cut eq? definer <>))) + (define-prefix? (cut eq? define-prefix <>))) (call-with-input-file file (lambda (port) (do ((syntax (read-syntax port) (read-syntax port))) ((match (syntax->datum syntax) - (((? definer?) current-term _ ...) + (((? define-prefix?) current-term _ ...) (pred (symbol->string current-term) search-term)) ((? eof-object?) #t) @@ -543,11 +543,13 @@ (define* (find-definition-location file term (syntax-source syntax)))))))) (define* (find-definition-insertion-location file term - #:key (definer 'define-public)) - "Search in FILE for a top-level definition defined by DEFINER with defined -term alphabetically succeeds TERM. Return the location if found, or #f + #:key + (define-prefix 'define-public)) + "Search in FILE for a top-level definition defined by DEFINE-PREFIX with +defined term alphabetically succeeds TERM. Return the location if found, or #f otherwise." - (find-definition-location file term #:definer definer #:pred string>)) + (find-definition-location + file term #:define-prefix define-prefix #:pred string>)) ;;;