From patchwork Tue Mar 18 14:26:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hilton Chain X-Patchwork-Id: 40328 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id A7C1427BBE9; Tue, 18 Mar 2025 14:28:53 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-5.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,FROM_SUSPICIOUS_NTLD,MAILING_LIST_MULTI, PDS_OTHER_BAD_TLD,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_VALIDITY_CERTIFIED, RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E903A27BBE2 for ; Tue, 18 Mar 2025 14:28:51 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tuXvQ-0004tk-MP; Tue, 18 Mar 2025 10:28:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tuXvN-0004sR-7V for guix-patches@gnu.org; Tue, 18 Mar 2025 10:28:10 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tuXvM-0001KW-7W; Tue, 18 Mar 2025 10:28:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:From:To:In-Reply-To:References:Subject; bh=LWpir8GeaddlWVQvf1QKYms/YDPoznC6UNHiRl7QF60=; b=MZr9FNwI6AVvbbl77mDUO9KMEBMkLSd9QbLDe2HroGOTv5OQBPTHSRYPBkmG3eOBmxf8NFpjhYvNGgieWHnKfeVDR3N+oPLe04kjkV2k38733HLy65uGaCEFU49rpTicPxbNBwpNzVws1lFxloxdY4qIL4XsKrjGgI9uANpjAvqNXjTsZbR79rzTyfDnsF1cew4KbRAOOVB6TWj/RZ+LGYfKDAWBrXqht+DJ3w+Q18+zdW69+GBcz78NHacwwUyYsHwKo04iAXBQmFgeD4Vl5CKPxkKjB8kpMbd56Nw6md1xoOQonYrBIAMdg3ReSi0yOQ8ZGTenhlYw7BpFOu9KeQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tuXvL-0004YG-Se; Tue, 18 Mar 2025 10:28:07 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#77093] [PATCH rust-team v2 00/17] New Rust packaging workflow based on lockfile importer. References: In-Reply-To: Resent-From: Hilton Chain Original-Sender: "Debbugs-submit" Resent-CC: ludo@gnu.org, maxim.cournoyer@gmail.com, guix-patches@gnu.org Resent-Date: Tue, 18 Mar 2025 14:28:07 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 77093 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 77093@debbugs.gnu.org Cc: Hilton Chain , Efraim Flashner , Ludovic =?utf-8?q?Court=C3=A8s?= , Maxim Cournoyer X-Debbugs-Original-Xcc: Ludovic =?utf-8?q?Court=C3=A8s?= , Maxim Cournoyer Received: via spool by 77093-submit@debbugs.gnu.org id=B77093.174230806917267 (code B ref 77093); Tue, 18 Mar 2025 14:28:07 +0000 Received: (at 77093) by debbugs.gnu.org; 18 Mar 2025 14:27:49 +0000 Received: from localhost ([127.0.0.1]:41661 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tuXuy-0004Tj-HB for submit@debbugs.gnu.org; Tue, 18 Mar 2025 10:27:48 -0400 Received: from mx.boiledscript.com ([88.99.243.112]:48544) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tuXuf-0004Qa-Vx for 77093@debbugs.gnu.org; Tue, 18 Mar 2025 10:27:30 -0400 From: Hilton Chain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ultrarare.space; s=mail; t=1742308044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=8YJ9OAP6sPQNiunvQM9CKMsUixIPuKAOpTsO0vPegxY=; b=QiscMu+EY817owp62ew8rFOupzqV/jFxxcg7L7ZSlV2QpJA1gTRd2SN7KTW+gisR5V+04k Qa3ZZaDBeVd5lFTVjBKa30yuX4wY6DH2wkue4e5isUvzDKB0g+zhKcoZcBlnjuXokwxAcy nktfh3lNT0VXVoguGNB77lB0c3bU2RJ17dvLpoV5U3T0OBx5hghx6pHOgdyYso2QRqWMJo AZlRqVWnhcLJZLMWfME2IZqjchetVJCxo2WL7mGqtMHAzr2x9uhdqDPBp0nyIXJG8EbN4c KD6kwMrGqrHXH6WDoA7PLG0DLaWUbZ7fAzhN4SMa+v8XnyddUKDAGWrSHxKVFA== Date: Tue, 18 Mar 2025 22:26:43 +0800 Message-ID: MIME-Version: 1.0 X-MS-Reactions: disallow X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches V1 -> V2: * Drop jemalloc environment variable patch. * Remove use of RUSTC_BOOTSTRAP. * Adjust documentation. Diff inserted at end. Efraim Flashner (1): build/cargo: Don't try to unpack sanity-check.py. Hilton Chain (16): build/cargo: Pass ‘--offline’ to cargo. build/cargo: Print out all non-empty binary files. build-system: cargo: Support packaging Cargo workspace. build-system: cargo: Support installing Cargo workspace. build/cargo: Set default value of arguments for build phases. build/cargo: Support non-workspace directory source inputs. scripts: import: Document argument for ‘--insert’ option in help message. scripts: import: Add two newlines for ‘--insert’ option. scripts: import: Support expressions defined by 'define. scripts: import: Pass "--insert" to importers. scripts: import: Skip existing definition for ‘--insert’ option. import: crate: crate-name->package-name: Move to (guix build-system cargo). build-system: cargo: Add ‘cargo-inputs’. import: crate: Add Cargo.lock parser. import: crate: Add ‘--lockfile’ option. doc: Document lockfile importer based Rust packaging workflow. Makefile.am | 1 + doc/contributing.texi | 415 ++++++++++++++++++++++++++++-- doc/guix.texi | 24 ++ etc/teams/rust/cleanup-crates.sh | 38 +++ etc/teams/rust/rust-crates.tmpl | 42 +++ gnu/local.mk | 2 + gnu/packages/rust-crates.scm | 42 +++ gnu/packages/rust-sources.scm | 29 +++ guix/build-system/cargo.scm | 57 +++- guix/build/cargo-build-system.scm | 151 ++++++++--- guix/import/crate.scm | 81 +++++- guix/import/crate/cargo-lock.scm | 105 ++++++++ guix/scripts/import.scm | 53 ++-- guix/scripts/import/crate.scm | 58 ++++- guix/utils.scm | 27 +- tests/crate.scm | 88 +++++++ 16 files changed, 1107 insertions(+), 106 deletions(-) create mode 100755 etc/teams/rust/cleanup-crates.sh create mode 100644 etc/teams/rust/rust-crates.tmpl create mode 100644 gnu/packages/rust-crates.scm create mode 100644 gnu/packages/rust-sources.scm create mode 100644 guix/import/crate/cargo-lock.scm base-commit: b54a9ca849f013300c633fb79d80bc754f6b28a2 --- 2.48.1 --8<---------------cut here---------------start------------->8--- --8<---------------cut here---------------end--------------->8--- diff --git a/doc/contributing.texi b/doc/contributing.texi index e7c5116a3d..837074dead 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -1607,15 +1607,15 @@ Rust Crates have switched to a different packaging model. Rust programs (binary crates) and dependencies (library crates) are treated -separately. We put main efforts into programs and only package Rust -dependencies as sources, utilizing automation with manual focus on unbundling +separately. We put our main efforts into programs and only package Rust +dependencies as sources, utilizing automation with a manual focus on unbundling vendored dependencies. The following paragraphs will explain them and give several examples. Rust programs are treated like any other package and named using the lowercase upstream name. When using the Cargo build system (@pxref{Build Systems, @code{cargo-build-system}}), Rust programs should have @code{#:install-source?} -argument set to @code{#f}, as this argument only makes sense to dependencies. +argument set to @code{#f}, as this argument only makes sense for dependencies. When the package source is a Cargo workspace, @code{#:cargo-install-paths} must be set to enable relevant support. @@ -1631,21 +1631,21 @@ Rust Crates contributing to Guix. Naturally, this module serves as a store for both sources and unbundling strategies. -This module is managed by Rust team (@pxref{Teams}) to ensure there's always one -version containing all changes from other branches, so that the maintained +This module is managed by the Rust team (@pxref{Teams}) to ensure there's always +one version containing all changes from other branches, so that the maintained version can be used directly in case of merge conflicts, thus coordination is required for other committers to modify it. Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup -script @file{etc/teams/rust/cleanup-crates.sh} for this moudle. +script @file{etc/teams/rust/cleanup-crates.sh} for this module. @item @code{(gnu packages rust-sources)}, storing more complex definitions that need to be full packages. This includes Rust dependencies requiring external inputs to unbundle and Cargo workspaces. -These dependencies should have @code{#:skip-build?} argument set to @code{#t}. -For Cargo workspaces, @code{#:cargo-package-crates} must be set. +These dependencies should have the @code{#:skip-build?} argument set to +@code{#t}. For Cargo workspaces, @code{#:cargo-package-crates} must be set. Since they are added manually, the following naming convention applies: @@ -1668,7 +1668,7 @@ Rust Crates Let's demonstrate the packaging workflow by examples, note that package-specific issues are not involved here. -In preparation, we'll adding the following packages to our environment: +In preparation, we'll add the following packages to our environment: @example guix shell rust rust:cargo cargo-audit cargo-license @@ -1747,7 +1747,7 @@ Rust Crates @code{check-for-pregenerated-files} phase: @example -$ guix build cargo-audit +$ ./pre-inst-env guix build cargo-audit @dots{} starting phase `check-for-pregenerated-files' Searching for binary files... diff --git a/doc/guix.texi b/doc/guix.texi index 029c8cf59c..218c2ba630 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9365,14 +9365,14 @@ Build Systems sources, to ease in future hacking on rust packages. This build system supports cargo workspaces. Parameter -@code{#:cargo-package-crates} (default: @code{''()}) allows specifying -names of library crates to package in @code{package} phase. Specified -crates are packaged from left to right, in case there's dependency among -them. For example, specifying @code{''("pcre2-sys" "pcre2")} will -package @code{"pcre2-sys"} first and then @code{"pcre2"}. Parameter -@code{#:cargo-install-paths} (default: @code{''()}) allows specifying -paths of binary crates to install in @code{install} phase, -@code{''("crates/atuin")}, for example. +@code{#:cargo-package-crates} (default: @code{''()}) allows specifying names of +library crates to package in the @code{package} phase. Specified crates are +packaged from left to right, in case there's dependency among them. For +example, specifying @code{''("pcre2-sys" "pcre2")} will package +@code{"pcre2-sys"} first and then @code{"pcre2"}. Parameter +@code{#:cargo-install-paths} (default: @code{''()}) allows specifying paths of +binary crates to install in the @code{install} phase, @code{''("crates/atuin")}, +for example. @end defvar @defvar chicken-build-system @@ -14705,19 +14705,18 @@ Invoking guix import a comment. @item --lockfile=@var{file} @itemx -f @var{file} -When @option{--lockfile} is specified, importer will ignore other -options and won't output package definitions, instead generating source -definition for all dependencies in @var{file}, a @file{Cargo.lock} file. -For example: +When @option{--lockfile} is specified, the importer will ignore other options +and won't output package definitions, instead generating source definitions for +all dependencies in @var{file}, a @file{Cargo.lock} file. For example: @example guix import crate --lockfile=/path/to/Cargo.lock my-package @end example -generates sources from @file{/path/to/Cargo.lock}, and a list -@code{my-package-cargo-inputs} referencing these sources. The generated -list is intended for package's base @code{inputs}, replacing -@code{#:cargo-inputs} and @code{#:cargo-development-inputs}. +generates sources from @file{/path/to/Cargo.lock} and a list +@code{my-package-cargo-inputs} referencing these sources. The generated list is +intended for the package's @code{inputs}, replacing @code{#:cargo-inputs} and +@code{#:cargo-development-inputs}. @end table @item elm diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm index b547421b88..41adc03752 100644 --- a/guix/build/cargo-build-system.scm +++ b/guix/build/cargo-build-system.scm @@ -248,12 +248,6 @@ (define* (configure #:key inputs (setenv "LIBSSH2_SYS_USE_PKG_CONFIG" "1") (setenv "SODIUM_USE_PKG_CONFIG" "1") (setenv "ZSTD_SYS_USE_PKG_CONFIG" "1") - ;; This flag is needed when not using the bundled jemalloc. - ;; https://github.com/tikv/jemallocator/issues/19 - (setenv "CARGO_FEATURE_UNPREFIXED_MALLOC_ON_SUPPORTED_PLATFORMS" "1") - (when (assoc-ref inputs "jemalloc") - (setenv "JEMALLOC_OVERRIDE" - (string-append (assoc-ref inputs "jemalloc") "/lib/libjemalloc.so"))) (when (assoc-ref inputs "openssl") (setenv "OPENSSL_DIR" (assoc-ref inputs "openssl"))) (when (assoc-ref inputs "gettext") @@ -356,23 +350,17 @@ (define* (package #:key (if (null? cargo-package-crates) (apply invoke `("cargo" "package" "--offline" ,@cargo-package-flags)) - ;; Use unstable feature ‘--registry’. (begin - (setenv "RUSTC_BOOTSTRAP" "1") (for-each (lambda (pkg) (apply invoke "cargo" "package" "--offline" "--package" pkg - "--registry" "crates-io" "-Z" "package-workspace" cargo-package-flags) (for-each (lambda (crate) (invoke "tar" "xzf" crate "-C" vendor-dir)) - (begin - (delete-file-recursively "target/package/tmp-registry") - (find-files "target/package" "\\.crate$"))) + (find-files "target/package" "\\.crate$")) (patch-cargo-checksums #:vendor-dir vendor-dir)) - cargo-package-crates) - (unsetenv "RUSTC_BOOTSTRAP"))) + cargo-package-crates))) ;; Then unpack the crate, reset the timestamp of all contained files, and ;; repack them. This is necessary to ensure that they are reproducible. diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index d8e1ed3e6f..f6e2985ed9 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -126,13 +126,12 @@ (define-command (guix-import . args) (show-version-and-exit "guix import")) ((or ("-i" file importer args ...) ("--insert" file importer args ...)) - (let* ((definer? - (cut member - <> - `(,@(if (member importer '("crate")) - '(define) - '()) - define-public))) + (let* ((importer-definers + `(,@(if (member importer '("crate")) + '(define) + '()) + define-public)) + (definer? (cut member <> importer-definers)) (find-and-insert (lambda (expr) (match expr diff --git a/tests/crate.scm b/tests/crate.scm index d46f753f9c..9b7066c3b1 100644 --- a/tests/crate.scm +++ b/tests/crate.scm @@ -478,7 +478,7 @@ (define rust-leaf-bob-3.0.2-yanked (license #f))) (define temp-file - (string-append "t-utils-" (number->string (getpid)))) + (string-append "t-crate-" (number->string (getpid)))) (test-begin "crate")