From patchwork Mon Feb 17 14:34:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Morgan Arnold X-Patchwork-Id: 38780 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 5032027BBEA; Mon, 17 Feb 2025 14:36:24 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7680527BBE2 for ; Mon, 17 Feb 2025 14:36:23 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tk2ED-0007Um-Ar; Mon, 17 Feb 2025 09:36:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tk2E7-0007Tv-BI for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tk2E6-00037b-Mo for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:From:Date:To:In-Reply-To:References:Subject; bh=m0DrksV4UMimhWHwf1Dqc7dogmkQrhluY9l5siVBdTI=; b=Qea7e1wuBYbYIozXdKpE2QnY4GvLZK80dkKdXvdhNp9koz2/7uMZvuhcxGm6tBTG1uf7/vCpYWMdN4aBZRkuKSF/ZHIWyOjgNhvDRxOtyy7CZ0M1/vV7b7GST/FdH45pPoRbSCGA3yo/BSRedPoaHQhyBqWj9667gHrMkm6T5aWM9BgXV8TlLDJ3CK4SGhI/yDehvwlDxF1do80aqaVA6tSCheKnQFDtl9WT1rkGy5Xcyd1iKgWHqGyy8Ocw9iXENJWqPqZlxkGw6rV+DxiuHrdb5JGBsnBfaFqFI+wOFdozRV7MGm7duAk8C5uRsSe9qi8BoM6mI59nnOqNA7RISw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tk2E6-0001aj-Et for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#55231] [PATCH v7 1/4] Allow copying of out-of-tree modules to the Linux initrd. References: <87wnf3pv87.fsf@ditto.jhoto.spork.org> In-Reply-To: <87wnf3pv87.fsf@ditto.jhoto.spork.org> Resent-From: Morgan Arnold Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 17 Feb 2025 14:36:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55231 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 55231@debbugs.gnu.org Cc: Maxim Cournoyer , Brian Cully Received: via spool by 55231-submit@debbugs.gnu.org id=B55231.17398029145914 (code B ref 55231); Mon, 17 Feb 2025 14:36:02 +0000 Received: (at 55231) by debbugs.gnu.org; 17 Feb 2025 14:35:14 +0000 Received: from localhost ([127.0.0.1]:45003 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tk2DI-0001XD-E3 for submit@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:13 -0500 Received: from mail-4322.protonmail.ch ([185.70.43.22]:36973) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tk2DD-0001QV-BV for 55231@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=6ojnzcdkr5fxrkuaqdocsg5vxm.protonmail; t=1739802898; x=1740062098; bh=m0DrksV4UMimhWHwf1Dqc7dogmkQrhluY9l5siVBdTI=; h=Date:To:From:Cc:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector: List-Unsubscribe:List-Unsubscribe-Post; b=BBRzogF5yKJystxh1wXsqETpF9b3WuX5D021ZHc/W/ImhTKka6NYfEsQGtjui1lFk 8NfPjzIu6VruO1VU9WWUYftAaKS0VcXKC3sYF81isnFv/VkrkHM/FV4TuX+d0It8JL giUR1l6wE1P/ovlCEkBWL5I11wWZ+ZCBm081N/RJ7xO5BRyUXT68rAv5uXhuQLT7l7 gP3XzGmxtE7bEmuzbN5zHYhCGusnPMgmvENSNb8DzU+zV1YwPeWz8BWypGy7Tc18x8 SmDl94wQ7XjMJkR0VbE/G2x4JG13nJu4d8OJsa+281S0pc9ntHKCvHiU6v6QCqB7Qv pRTa20crzRaOQ== Date: Mon, 17 Feb 2025 14:34:52 +0000 Message-ID: Feedback-ID: 45510636:user:proton X-Pm-Message-ID: ffa7461925d3790da96d5b088cc6199c3c095dd3 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Morgan Arnold X-ACL-Warn: , Morgan Arnold via Guix-patches X-Patchwork-Original-From: Morgan Arnold via Guix-patches via From: Morgan Arnold Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Brian Cully With this patch, modules for ‘initrd-modules’ will not only be searched for in the in-tree Linux modules, but also any additional modules specified in ‘kernel-loadable-modules’. * gnu/build/linux-modules.scm (find-module-file): Change DIRECTORY argument to DIRECTORIES. Now takes a list of directories to search, rather than a single one. * gnu/system/linux-initrd.scm (flat-linux-module-directory): change LINUX argument to PACKAGES. Now contains a list of file-like objects to search for modules. (raw-initrd): Add LINUX-EXTRA-MODULE-DIRECTORIES keyword argument. Pass it to (flat-linux-module-directory) along with the selected LINUX package. (base-initrd): Add LINUX-EXTRA-MODULE-DIRECTORIES keyword argument. Pass it to (raw-initrd). * gnu/system.scm (operating-system-initrd-file): Pass in operating system definition's kernel-loadable-modules into (make-initrd) as LINUX-EXTRA-MODULE-DIRECTORIES. * doc/guix.texi (Initial RAM Disk): Document how out-of-tree modules can be used. Change-Id: Ic39f2abcfabc3ec34a71acce840038396bf9c82e Signed-off-by: Maxim Cournoyer Modified-by: Maxim Cournoyer --- doc/guix.texi | 15 ++++++++++++ gnu/build/linux-modules.scm | 22 +++++++++++------ gnu/system.scm | 2 ++ gnu/system/linux-initrd.scm | 49 +++++++++++++++++++++++-------------- 4 files changed, 62 insertions(+), 26 deletions(-) base-commit: b30669e15d2e8c3d1b74b32f77e2095682aab4ca diff --git a/doc/guix.texi b/doc/guix.texi index ce78068..3c7ece7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -43346,6 +43346,21 @@ Initial RAM Disk (initrd-modules (cons "megaraid_sas" %base-initrd-modules))) @end lisp +If a module listed in @code{initrd-modules} is not included in the +Linux-libre kernel, then its location must be provided via the +@code{kernel-loadable-modules} list. + +As an example, if you need the driver for a Realtek RTL8821CE wireless +network adapter for mounting the root file system over NFS, your +configuration might include the following: + +@lisp +(operating-system + ;; @dots{} + (initrd-modules (cons "8821ce" %base-initrd-modules)) + (kernel-loadable-modules (list (list rtl8821ce-linux-module "module")))) +@end lisp + @defvar %base-initrd-modules This is the list of kernel modules included in the initrd by default. @end defvar diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm index 32baf6c..24bbe08 100644 --- a/gnu/build/linux-modules.scm +++ b/gnu/build/linux-modules.scm @@ -246,8 +246,8 @@ (define (file-name->module-name file) '.ko[.gz|.xz|.zst]' and normalizing it." (normalize-module-name (strip-extension (basename file)))) -(define (find-module-file directory module) - "Lookup module NAME under DIRECTORY, and return its absolute file name. +(define (find-module-file directories module) + "Lookup module NAME under DIRECTORIES, and return its absolute file name. NAME can be a file name with or without '.ko', or it can be a module name. Raise an error if it could not be found. @@ -255,6 +255,9 @@ (define (find-module-file directory module) module names usually (always?) use underscores as the inter-word separator, whereas file names often, but not always, use hyphens. Examples: \"usb-storage.ko\", \"serpent_generic.ko\"." + (define directories (if (pair? directories) + directories + (list directories))) ;for backward compatibility (define names ;; List of possible file names. XXX: It would of course be cleaner to ;; have a database that maps module names to file names and vice versa, @@ -268,16 +271,19 @@ (define (find-module-file directory module) (else chr))) module)))) - (match (find-files directory - (lambda (file stat) - (member (strip-extension - (basename file)) names))) + (match (append-map (lambda (directory) + (find-files directory + (lambda (file _) + (member (strip-extension + (basename file)) + names)))) + directories) ((file) file) (() - (error "kernel module not found" module directory)) + (error "kernel module not found" module directories)) ((_ ...) - (error "several modules by that name" module directory)))) + (error "several modules by that name" module directories)))) (define* (recursive-module-dependencies files #:key (lookup-module dot-ko)) diff --git a/gnu/system.scm b/gnu/system.scm index 8df871f..1921b60 100644 --- a/gnu/system.scm +++ b/gnu/system.scm @@ -1373,6 +1373,8 @@ (define (operating-system-initrd-file os) #:linux (operating-system-kernel os) #:linux-modules (operating-system-initrd-modules os) + #:linux-extra-module-directories + (operating-system-kernel-loadable-modules os) #:mapped-devices mapped-devices #:keyboard-layout (operating-system-keyboard-layout os))) diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm index dc08edc..a8df905 100644 --- a/gnu/system/linux-initrd.scm +++ b/gnu/system/linux-initrd.scm @@ -120,13 +120,19 @@ (define* (expression->initrd exp `(#:references-graphs (("closure" ,init)))) "/initrd.cpio.gz")) -(define (flat-linux-module-directory linux modules) +(define (flat-linux-module-directory packages modules) "Return a flat directory containing the Linux kernel modules listed in -MODULES and taken from LINUX." +MODULES and taken from PACKAGES." (define imported-modules (source-module-closure '((gnu build linux-modules) (guix build utils)))) + (define package-inputs + (map (match-lambda + ((p o) (gexp-input p o)) + (p (gexp-input p "out"))) + packages)) + (define build-exp (with-imported-modules imported-modules (with-extensions (list guile-zlib guile-zstd) @@ -138,8 +144,9 @@ (define (flat-linux-module-directory linux modules) (srfi srfi-26) (ice-9 match)) - (define module-dir - (string-append #$linux "/lib/modules")) + (define module-dirs + (map (cut string-append <> "/lib/modules") + '#$package-inputs)) (define builtin-modules (match (find-files module-dir (lambda (file stat) @@ -157,7 +164,7 @@ (define (flat-linux-module-directory linux modules) (lset-difference string=? '#$modules builtin-modules)) (define modules - (let* ((lookup (cut find-module-file module-dir <>)) + (let* ((lookup (cut find-module-file module-dirs <>)) (modules (map lookup modules-to-lookup))) (append modules (recursive-module-dependencies @@ -192,6 +199,7 @@ (define* (raw-initrd file-systems #:key (linux linux-libre) (linux-modules '()) + (linux-extra-module-directories '()) (pre-mount #t) (mapped-devices '()) (keyboard-layout #f) @@ -199,15 +207,16 @@ (define* (raw-initrd file-systems qemu-networking? volatile-root? (on-error 'debug)) - "Return as a file-like object a raw initrd, with kernel -modules taken from LINUX. FILE-SYSTEMS is a list of file-systems to be -mounted by the initrd, possibly in addition to the root file system specified -on the kernel command line via 'root'. LINUX-MODULES is a list of kernel -modules to be loaded at boot time. MAPPED-DEVICES is a list of device -mappings to realize before FILE-SYSTEMS are mounted. PRE-MOUNT is a -G-expression to evaluate before realizing MAPPED-DEVICES. -HELPER-PACKAGES is a list of packages to be copied in the initrd. It may include -e2fsck/static or other packages needed by the initrd to check root partition. + "Return as a file-like object a raw initrd, with kernel modules taken from +LINUX. FILE-SYSTEMS is a list of file-systems to be mounted by the initrd, +possibly in addition to the root file system specified on the kernel command +line via 'root'. LINUX-MODULES is a list of kernel modules to be loaded at +boot time. LINUX-EXTRA-MODULE-DIRECTORIES is a list of file-like objects which +will be searched for modules in addition to the linux kernel. MAPPED-DEVICES +is a list of device mappings to realize before FILE-SYSTEMS are mounted. +HELPER-PACKAGES is a list of packages to be copied in the initrd. It may +include e2fsck/static or other packages needed by the initrd to check root +partition. When true, KEYBOARD-LAYOUT is a record denoting the desired console keyboard layout. This is done before MAPPED-DEVICES are set up and @@ -243,7 +252,8 @@ (define* (raw-initrd file-systems #~()))) (define kodir - (flat-linux-module-directory linux linux-modules)) + (flat-linux-module-directory (cons linux linux-extra-module-directories) + linux-modules)) (expression->initrd (with-imported-modules (source-module-closure @@ -390,6 +400,7 @@ (define* (base-initrd file-systems #:key (linux linux-libre) (linux-modules '()) + (linux-extra-module-directories '()) (mapped-devices '()) (keyboard-layout #f) qemu-networking? @@ -410,9 +421,10 @@ (define* (base-initrd file-systems QEMU-NETWORKING? and VOLATILE-ROOT? behaves as in raw-initrd. The initrd is automatically populated with all the kernel modules necessary -for FILE-SYSTEMS and for the given options. Additional kernel -modules can be listed in LINUX-MODULES. They will be added to the initrd, and -loaded at boot time in the order in which they appear." +for FILE-SYSTEMS and for the given options. Additional kernel modules can be +listed in LINUX-MODULES. Additional directories for modules can be listed in +LINUX-EXTRA-MODULE-DIRECTORIES. They will be added to the initrd, and loaded +at boot time in the order in which they appear." (define linux-modules* ;; Modules added to the initrd and loaded from the initrd. `(,@linux-modules @@ -432,6 +444,7 @@ (define* (base-initrd file-systems (raw-initrd file-systems #:linux linux #:linux-modules linux-modules* + #:linux-extra-module-directories linux-extra-module-directories #:mapped-devices mapped-devices #:helper-packages helper-packages #:keyboard-layout keyboard-layout From patchwork Mon Feb 17 14:34:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Morgan Arnold X-Patchwork-Id: 38781 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id BF9BC27BBEA; Mon, 17 Feb 2025 14:36:28 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 4583927BBE2 for ; Mon, 17 Feb 2025 14:36:28 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tk2EF-0007Uu-Tc; Mon, 17 Feb 2025 09:36:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tk2E8-0007Tz-KW for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tk2E7-00037i-As for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:From:Date:To:Subject; bh=+hjgx7vcNiJxBA/Rsm+ZmN9LjVl5su+aa5tilK0U0zg=; b=W2a80FlkPUYz5uc2T1ao5mj/+mVio+ZqG+8TfSRYqfm5qqpaweVtadLzNw4gYadJuBvXzrav18sXp8jmqJ7nU8z2alXZSh5cOoC9qd0e+ABJVTISTBmYXYHSFzXJR9hE/3CLDtMIHVkbX+xUCYrw1GqfIm7wzKLrFjdYEi7Z8pspSSAb1DKhB5KMaJ+vgqJ+bAZ5M7nsOInjmd4gYwaw388h0uwLqfn3gOVGS3uaTjNF94BfBC5cGsIGxUazQEe1OgCv2gwGEzruwUS/PLfago31B6Clff7PXtUBrw7WoiFkgJhM9Tu0eaedthbm9u4GpJnd9wIskS2kySSjVTMdhA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tk2E7-0001ar-3H for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#55231] [PATCH v7 2/4] derivations: Fix indentation. Resent-From: Morgan Arnold Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 17 Feb 2025 14:36:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55231 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 55231@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 55231-submit@debbugs.gnu.org id=B55231.17398029195947 (code B ref 55231); Mon, 17 Feb 2025 14:36:03 +0000 Received: (at 55231) by debbugs.gnu.org; 17 Feb 2025 14:35:19 +0000 Received: from localhost ([127.0.0.1]:45007 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tk2DP-0001Xp-0X for submit@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:19 -0500 Received: from mail-10629.protonmail.ch ([79.135.106.29]:34901) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tk2DI-0001Rk-83 for 55231@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=protonmail; t=1739802903; x=1740062103; bh=+hjgx7vcNiJxBA/Rsm+ZmN9LjVl5su+aa5tilK0U0zg=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=ObMIZS6Y4QRrjlUR9EU4IsflFc5YUYYD1mXSmOgqYqfHSo9HqfbPxhzrjpxAy5GPV igaUElbw8N4/oVDRo3thAzloZJQ+bnxfZBQv4S0BmSP1348tT6aQ9FKUL3XJ2sknNQ TfIFsEq41NmD0pWlca1YD5506XmeOYAyQ1xzWxxyUDXaOnaMXoyiPqFlZK+lreC8no 0g6Xk0ZN3gwrlrCDd5GEl5lLu34iOoZrDoU0U4wMLNUXPkS2/+R2armtBetHw1iT0S n3dKYfFj+GCZeU8gyADIt2h4PSU1T5UHCsyaG3rP1+fJQgq6TDqMOTIPPdn/zlH2TF lw9FeeR1lj9Bw== Date: Mon, 17 Feb 2025 14:34:59 +0000 Message-ID: <9e47b0e6cfb99f4d5cd4a3e4c4e1d6ac13973f4f.1739802789.git.morgan.arnold@proton.me> In-Reply-To: References: Feedback-ID: 45510636:user:proton X-Pm-Message-ID: b326bd8819a2ac4109fba1ba6ce81010c8110516 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Morgan Arnold X-ACL-Warn: , Morgan Arnold via Guix-patches X-Patchwork-Original-From: Morgan Arnold via Guix-patches via From: Morgan Arnold Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Maxim Cournoyer * guix/derivations.scm (derivation): Fix indentation. Change-Id: I2407b59788ce335c21c181d9f9e3f26a359e9bf5 --- guix/derivations.scm | 46 ++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/guix/derivations.scm b/guix/derivations.scm index bef98cd..ffa69e9 100644 --- a/guix/derivations.scm +++ b/guix/derivations.scm @@ -841,30 +841,30 @@ (define* (derivation store name builder args ;; corresponding environment variable. (match drv (($ outputs inputs sources - system builder args env-vars) + system builder args env-vars) (let* ((drv-hash (derivation-hash drv)) (outputs (map (match-lambda - ((output-name . ($ - _ algo hash rec?)) - (let ((path - (if hash - (fixed-output-path name hash - #:hash-algo algo - #:output output-name - #:recursive? rec?) - (output-path output-name - drv-hash name)))) - (cons output-name - (make-derivation-output path algo - hash rec?))))) + ((output-name . ($ + _ algo hash rec?)) + (let ((path + (if hash + (fixed-output-path name hash + #:hash-algo algo + #:output output-name + #:recursive? rec?) + (output-path output-name + drv-hash name)))) + (cons output-name + (make-derivation-output path algo + hash rec?))))) outputs))) (make-derivation outputs inputs sources system builder args (map (match-lambda - ((name . value) - (cons name - (or (and=> (assoc-ref outputs name) - derivation-output-path) - value)))) + ((name . value) + (cons name + (or (and=> (assoc-ref outputs name) + derivation-output-path) + value)))) env-vars) #f))))) @@ -910,10 +910,10 @@ (define* (derivation store name builder args ;; Return a variant of ENV-VARS where each OUTPUTS is associated with an ;; empty string, even outputs that do not appear in ENV-VARS. (let ((e (map (match-lambda - ((name . val) - (if (member name outputs) - (cons name "") - (cons name val)))) + ((name . val) + (if (member name outputs) + (cons name "") + (cons name val)))) env-vars))) (fold (lambda (output-name env-vars) (if (assoc output-name env-vars) From patchwork Mon Feb 17 14:35:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Morgan Arnold X-Patchwork-Id: 38782 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2C34627BBEA; Mon, 17 Feb 2025 14:36:41 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B742C27BBE2 for ; Mon, 17 Feb 2025 14:36:40 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tk2EI-0007VD-1a; Mon, 17 Feb 2025 09:36:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tk2E8-0007U0-N5 for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tk2E8-00038D-Dz for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:From:Date:To:Subject; bh=4T040SQSODzmvJUcGOrQHSVZMddJjKsnxbNRVcuS7PU=; b=XgXdBEY+mEvzgO/OBkPCuMPlOJiK8cUsfkCQgVz1pO9mvquhv2tg4w8GO3q2fK5dR1MzmKqVKfLd93qaHuwF1+0wUGDMuuqpe+nIYo2sScBQGmgNqXwGBKSMKjTX3f5TGv/eg+Bw3kADHSK1CSTxxrriIOi9rCa2yuon/2yaFeweC9+/n7MJbO1M8cEVqGk0TGMy+ZSrfmNn63WQjHjsL+K2kYqmqsS9rC7XTbW7LoUIVO+rxOK0fa0n08BZ5owenzaYnqCvVb84B9Cf3FCUfnt1P7B/CcQ50n+5L83Ty6e8EXmx0z+t4zq7JStGlyHQkj2HthWM8+9wid/oVNQewQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tk2E8-0001b5-84 for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#55231] [PATCH v7 3/4] tests: Remove extraneous 'with-store' in derivations test. Resent-From: Morgan Arnold Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 17 Feb 2025 14:36:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55231 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 55231@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 55231-submit@debbugs.gnu.org id=B55231.17398029245964 (code B ref 55231); Mon, 17 Feb 2025 14:36:04 +0000 Received: (at 55231) by debbugs.gnu.org; 17 Feb 2025 14:35:24 +0000 Received: from localhost ([127.0.0.1]:45009 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tk2DT-0001Y8-Gf for submit@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:23 -0500 Received: from mail-10630.protonmail.ch ([79.135.106.30]:12697) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tk2DN-0001Wh-52 for 55231@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=protonmail; t=1739802910; x=1740062110; bh=4T040SQSODzmvJUcGOrQHSVZMddJjKsnxbNRVcuS7PU=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=X6dTDp333QRS3u9HoOPRkA1fbjIl9BvOSIPHGmWPVJixyWPlEgMRdg66IeMNBz3O4 Yw1f8p1oDmkeGDjoEgSnCmG3cN3VCek7KUqqn8MIfCAJ1I1bw5s37I/aHKrhJ4ZDWq 61bP5uavYGtc6HhSxxYh6w6WcVYaPuvc6u/yrlAFceWuOgmvxknxiqbJ9I3As4Ph1W PUMRkehMLcK/ZvhRmWHIGGoM9m/egO3kMNIXD7bsVCpNviAaQ46Lc8fUlkXaD2rOAP kbRImKJXvzibtDKuBfzAw5K039UOTtVmtcnAu6YMaf5cvT3v+E0YbOjw9r1Hfc8jtY XbYHmc6byQsTw== Date: Mon, 17 Feb 2025 14:35:07 +0000 Message-ID: In-Reply-To: References: Feedback-ID: 45510636:user:proton X-Pm-Message-ID: b5b9b86a54fb6cb7c93ffe0285aade37f3079826 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Morgan Arnold X-ACL-Warn: , Morgan Arnold via Guix-patches X-Patchwork-Original-From: Morgan Arnold via Guix-patches via From: Morgan Arnold Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Maxim Cournoyer * tests/derivations.scm ("derivation fails but keep going"): Remove extraneous 'with-store'. Change-Id: If30c2d457504b8524cd167f1a145fbbea61b513c --- tests/derivations.scm | 45 +++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/tests/derivations.scm b/tests/derivations.scm index efcd21f..72ea9aa 100644 --- a/tests/derivations.scm +++ b/tests/derivations.scm @@ -150,29 +150,28 @@ (define* (directory-contents dir #:optional (slurp get-bytevector-all)) (test-assert "derivation fails but keep going" ;; In keep-going mode, 'build-derivations' should fail because of D1, but it ;; must return only after D2 has succeeded. - (with-store store - (let* ((d1 (derivation %store "fails" - %bash `("-c" "false") - #:sources (list %bash))) - (d2 (build-expression->derivation %store "sleep-then-succeed" - `(begin - ,(random-text) - ;; XXX: Hopefully that's long - ;; enough that D1 has already - ;; failed. - (sleep 2) - (mkdir %output))))) - (set-build-options %store - #:use-substitutes? #f - #:keep-going? #t) - (guard (c ((store-protocol-error? c) - (and (= 100 (store-protocol-error-status c)) - (string-contains (store-protocol-error-message c) - (derivation-file-name d1)) - (not (valid-path? %store (derivation->output-path d1))) - (valid-path? %store (derivation->output-path d2))))) - (build-derivations %store (list d1 d2)) - #f)))) + (let* ((d1 (derivation %store "fails" + %bash `("-c" "false") + #:sources (list %bash))) + (d2 (build-expression->derivation %store "sleep-then-succeed" + `(begin + ,(random-text) + ;; XXX: Hopefully that's long + ;; enough that D1 has already + ;; failed. + (sleep 2) + (mkdir %output))))) + (set-build-options %store + #:use-substitutes? #f + #:keep-going? #t) + (guard (c ((store-protocol-error? c) + (and (= 100 (store-protocol-error-status c)) + (string-contains (store-protocol-error-message c) + (derivation-file-name d1)) + (not (valid-path? %store (derivation->output-path d1))) + (valid-path? %store (derivation->output-path d2))))) + (build-derivations %store (list d1 d2)) + #f))) (test-assert "identical files are deduplicated" ;; Note: DATA must be longer than %DEDUPLICATION-MINIMUM-SIZE. From patchwork Mon Feb 17 14:35:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Morgan Arnold X-Patchwork-Id: 38779 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2EBCC27BBEA; Mon, 17 Feb 2025 14:36:20 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 3B67427BBE2 for ; Mon, 17 Feb 2025 14:36:19 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tk2EF-0007Ut-LA; Mon, 17 Feb 2025 09:36:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tk2E9-0007U5-Dm for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tk2E9-00038U-3o for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:From:Date:To:Subject; bh=drWhgMRc6fLuGGK5g27afE3jprIjkVBnmhIH9R0H29Y=; b=klfG2oaAOXTNsN5XdKnqJ7DEQkkCkXoV3/wcYXcrdoR9g+LIkCbfdkl2MuUxnsg2nSUkdzzaOme1A0otft7jOR1L1XGD0dqF2By7Ke40foACcdgZFwLiNZo6+kR9JxFVmdE0r19VwJjgnAzE8vh1zwrBPUSE4efAU9FA0BnuO6WJwvQEavryc/6sgRF2gWZgV7feGgb/CfzJ5Pyas4sx8esW5c9Sq7i4R/6/3FNrVyHj9lt4p3izQ1A/Kbcmde0wtrWaIOoAWD+GuTVIxKDhO7j0UZd5mw5Br7ZjskUJfFg4VclTsbq2E8+4KCjimH7cgnK68mD8CmYTdf6/EIdvrw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tk2E8-0001bF-Sd for guix-patches@gnu.org; Mon, 17 Feb 2025 09:36:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#55231] [PATCH v7 4/4] Propagate non-substitutability of derivations. Resent-From: Morgan Arnold Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 17 Feb 2025 14:36:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55231 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 55231@debbugs.gnu.org Cc: Morgan Arnold , Maxim Cournoyer Received: via spool by 55231-submit@debbugs.gnu.org id=B55231.17398029305999 (code B ref 55231); Mon, 17 Feb 2025 14:36:04 +0000 Received: (at 55231) by debbugs.gnu.org; 17 Feb 2025 14:35:30 +0000 Received: from localhost ([127.0.0.1]:45013 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tk2DY-0001Yc-Tr for submit@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:29 -0500 Received: from mail-40131.protonmail.ch ([185.70.40.131]:58875) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tk2DU-0001Xm-Ml for 55231@debbugs.gnu.org; Mon, 17 Feb 2025 09:35:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=protonmail; t=1739802916; x=1740062116; bh=drWhgMRc6fLuGGK5g27afE3jprIjkVBnmhIH9R0H29Y=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=N1+GrsYvx9krv5dY8u/GlEMytxxgFITNqqrY+Si54kEt0tGmqRvw0tZqVnkshrOL1 YCTpp5eLyoh6yzKBP+giQLETqY4oYdaqLLfZMPtIBIo78qh/DjNogu3MMdcKGYdxwB koeLQu40V55T/ZAsSDeu0nFyu3ujFk675T/L1xFKoQRHEkU4RTnq0QxF3Zc+eAHgK6 XdvnJrgYhkA/Hg6PZZtIJSQB7rY3Gy4K2EHflOR4ssYXIPo6i4N9wb791bWhlKRaNp j6LkaPnPqbBJ38p8TkegebEB9rkRHpsJSbodTy5edLefXzOlQsWYzzXzbAahCvZRBw SI4yn+Zu23T2g== Date: Mon, 17 Feb 2025 14:35:13 +0000 Message-ID: <5b9eb2560f2602e147d7c0d4d9ecf6582b1fe3f3.1739802789.git.morgan.arnold@proton.me> In-Reply-To: References: Feedback-ID: 45510636:user:proton X-Pm-Message-ID: f5f990b964ed6f581408a831e00cb7cee0b52479 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Morgan Arnold X-ACL-Warn: , Morgan Arnold via Guix-patches X-Patchwork-Original-From: Morgan Arnold via Guix-patches via From: Morgan Arnold Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches This commit changes the conditions under which derivations, as constructed by the `derivation' procedure, are made substitutable, to prevent potential copyright violations related to the construction of substitutable initrds including non-substitutable derivations (in particular, ZFS). This change prevents such copyright violations by only marking a derivation as substitutable if it is itself marked substitutable along all of its inputs. This means that non-substitutable derivations propagate to other derivations using them as input. Change-Id: I80ba4a371ee0c55a1294aff311d4e7b151055fac Signed-off-by: Maxim Cournoyer Modified-by: Maxim Cournoyer --- gnu/system/linux-initrd.scm | 3 +- guix/derivations.scm | 59 +++++++++++++++++++++---------------- tests/derivations.scm | 16 +++++----- 3 files changed, 44 insertions(+), 34 deletions(-) diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm index a8df905..98a4c6e 100644 --- a/gnu/system/linux-initrd.scm +++ b/gnu/system/linux-initrd.scm @@ -117,7 +117,8 @@ (define* (expression->initrd exp (file-append (computed-file name builder #:options - `(#:references-graphs (("closure" ,init)))) + `(#:references-graphs (("closure" ,init)) + #:propagating-substitutable? #t)) "/initrd.cpio.gz")) (define (flat-linux-module-directory packages modules) diff --git a/guix/derivations.scm b/guix/derivations.scm index ffa69e9..7682c90 100644 --- a/guix/derivations.scm +++ b/guix/derivations.scm @@ -804,6 +804,7 @@ (define* (derivation store name builder args allowed-references disallowed-references leaked-env-vars local-build? (substitutable? #t) + (propagating-substitutable? #f) (properties '()) (%deprecation-warning? #t)) "Build a derivation with the given arguments, and return the resulting @@ -832,7 +833,9 @@ (define* (derivation store name builder args derivations where the costs of data transfers would outweigh the benefits. When SUBSTITUTABLE? is false, declare that substitutes of the derivation's -output should not be used. +output should not be used. When PROPAGATING-SUBSTITUTABLE? is true, declare +that substitutes of the derivation's output should not be used if any of the +derivation's inputs are not substitutable. PROPERTIES must be an association list describing \"properties\" of the derivation. It is kept as-is, uninterpreted, in the derivation." @@ -868,33 +871,37 @@ (define* (derivation store name builder args env-vars) #f))))) - (define (user+system-env-vars) + (define (user+system-env-vars inputs) ;; Some options are passed to the build daemon via the env. vars of ;; derivations (urgh!). We hide that from our API, but here is the place ;; where we kludgify those options. - (let ((env-vars `(,@(if local-build? - `(("preferLocalBuild" . "1")) - '()) - ,@(if (not substitutable?) - `(("allowSubstitutes" . "0")) - '()) - ,@(if allowed-references - `(("allowedReferences" - . ,(string-join allowed-references))) - '()) - ,@(if disallowed-references - `(("disallowedReferences" - . ,(string-join disallowed-references))) - '()) - ,@(if leaked-env-vars - `(("impureEnvVars" - . ,(string-join leaked-env-vars))) - '()) - ,@(match properties - (() '()) - (lst `(("guix properties" - . ,(object->string properties))))) - ,@env-vars))) + (let* ((substitutable-inputs? (every substitutable-derivation? + (map derivation-input-derivation + inputs))) + (env-vars `(,@(if local-build? + `(("preferLocalBuild" . "1")) + '()) + ,@(if (and substitutable? (or (not propagating-substitutable?) + substitutable-inputs?)) + '() + `(("allowSubstitutes" . "0"))) + ,@(if allowed-references + `(("allowedReferences" + . ,(string-join allowed-references))) + '()) + ,@(if disallowed-references + `(("disallowedReferences" + . ,(string-join disallowed-references))) + '()) + ,@(if leaked-env-vars + `(("impureEnvVars" + . ,(string-join leaked-env-vars))) + '()) + ,@(match properties + (() '()) + (lst `(("guix properties" + . ,(object->string properties))))) + ,@env-vars))) (match references-graphs (((file . path) ...) (let ((value (map (cut string-append <> " " <>) @@ -967,7 +974,7 @@ (define* (derivation store name builder args (filter-map input->derivation-input inputs)) derivation-inputoutput-path drv2)))) + ;; DRV2 is *not* available as a substitute, since it has drv1 as + ;; input, and the non-substitutability is viral to avoid + ;; distributing non-substitutable items that could have become + ;; embedded, for example in an initrd. + (and (null? download) (match build - (((= derivation-file-name build)) - (string=? build (derivation-file-name drv1)))))))))) + (((= derivation-file-name build1) + (= derivation-file-name build2)) + (string=? build1 (derivation-file-name drv1)) + (string=? build2 (derivation-file-name drv2)))))))))) (test-assert "derivation-build-plan and substitutes, local build" (with-store store