From patchwork Wed May 15 16:57:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 64464 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0933E27BBE9; Wed, 15 May 2024 18:00:56 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 345E827BBE2 for ; Wed, 15 May 2024 18:00:51 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s7HzJ-0001wM-Vl; Wed, 15 May 2024 13:00:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s7Hz7-0001q3-Uc for guix-patches@gnu.org; Wed, 15 May 2024 13:00:11 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s7Hz7-0007HC-9G; Wed, 15 May 2024 13:00:09 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1s7Hz5-0007Co-HP; Wed, 15 May 2024 13:00:07 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#70962] [PATCH 13/14] gnu: linux-libre: Enable Zstd compression of kernel modules. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, leo@famulari.name, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, w@wmeyer.eu, guix-patches@gnu.org Resent-Date: Wed, 15 May 2024 17:00:07 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70962 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 70962@debbugs.gnu.org Cc: Maxim Cournoyer , Christopher Baines , Josselin Poiret , Leo Famulari , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice , Wilko Meyer X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Leo Famulari , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice , Wilko Meyer Received: via spool by 70962-submit@debbugs.gnu.org id=B70962.171579238027552 (code B ref 70962); Wed, 15 May 2024 17:00:07 +0000 Received: (at 70962) by debbugs.gnu.org; 15 May 2024 16:59:40 +0000 Received: from localhost ([127.0.0.1]:43180 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s7Hyd-0007AD-M2 for submit@debbugs.gnu.org; Wed, 15 May 2024 12:59:40 -0400 Received: from mail-qv1-f54.google.com ([209.85.219.54]:60620) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s7HyV-000796-T4 for 70962@debbugs.gnu.org; Wed, 15 May 2024 12:59:33 -0400 Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-69b514d3cf4so78263916d6.0 for <70962@debbugs.gnu.org>; Wed, 15 May 2024 09:59:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715792304; x=1716397104; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m6cZWtRaw3GPkSQvTE2sZ5G6Gf8d/R777VYqoD9YE/I=; b=Ff/YCx211Dhs7MXY9d+0WlzeR55a7LoMZO10dCEmSj/98bq5MFuss24uVgK5fHljp0 +fVVRP3R+Y7qY1TMYOW479h/8IyxA+l+qVEzV2xJ71p11XLI8IZuAX/iMMXXe2Wgoded 500TouyCTcTci6qeJuo2iLD5LZNpnpTCZ0NQoidVeUV9RN6/olsBq63QZJG29Le9bwXP ZzmRWavf0oJ2KFif1jrhmxtxcWqmG1n24WWaEHkP/5tVz6kw3w1FpcaYQtHpLxCCwkHT /jveAarZGnskjbauYzyxnrZFSiKpzB3q5tNTLkdPHZ3cOdLgdcohIA0314b9PWDFl3cV WvGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715792304; x=1716397104; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m6cZWtRaw3GPkSQvTE2sZ5G6Gf8d/R777VYqoD9YE/I=; b=IC0EVX4LJgGklI5hZa68ps10C8SDIyxg5k1FfC61n7Mbp0GBpDtD+h03mnz5i0IB4L h8r/IatIsqTDf8Aiz55JAZ/Dq3asPApjV7eOTjD704z8aF++NCznfx+/Tbdz2qBurmmn E3UFtCQ9Ov6ec81lcNJGegI5BphXa0ADHqW+hWAiVOqo9fJWv0BF64wQXwYU5OAGeriS yH948yK1am0FHhc0Ctw2Xw0qxf0ggizjMPUvvXdNX8C3VHD8dSH8uOiPlCc6dHd7HDR5 x9PfwInr3qDVT42oyujqpk/25YPvnrBYHR9Xp9lQJZ9sYFfbbjCABa5Afg1GI7x+Difi ToZw== X-Gm-Message-State: AOJu0Yw8nCvjy8Mmy+Gsxn6oiA2ZDViuN40Pme4HjlvRm5MOh5iOMmLa G/3qARVpq+DHWD9md0BEnlvco4oLrJ5io1p8dt4OqVm0ceps+T1TDukXbA== X-Google-Smtp-Source: AGHT+IH+DH8E49i1wIGYdafVhYC8osRpMAS8j/qDjY84IZ/Kxrokvtq/KoPi+BsLu0tIQQHjx61p/A== X-Received: by 2002:a05:6214:33c1:b0:6a0:54da:b7ca with SMTP id 6a1803df08f44-6a1681956e7mr189988566d6.36.1715792303978; Wed, 15 May 2024 09:58:23 -0700 (PDT) Received: from localhost.localdomain (dsl-205-233-125-107.b2b2c.ca. [205.233.125.107]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6a15f194871sm66068896d6.59.2024.05.15.09.58.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 May 2024 09:58:23 -0700 (PDT) From: Maxim Cournoyer Date: Wed, 15 May 2024 12:57:23 -0400 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <7cf609dcf83c85b6897c9fb9af46ce8a854a15c8.1715791830.git.maxim.cournoyer@gmail.com> References: <7cf609dcf83c85b6897c9fb9af46ce8a854a15c8.1715791830.git.maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches This brings the on disk size of the kernel from 164 MiB to 144 MiB, or about 12%. * gnu/packages/linux.scm (default-extra-linux-options) [version>=5.13]: Enable CONFIG_MODULE_COMPRESS_ZSTD, else CONFIG_MODULE_COMPRESS_GZIP. (make-linux-libre*) [phases] {set-environment}: Set ZSTD_CLEVEL environment variable to 19. [native-inputs]: Add zstd. * gnu/build/linux-modules.scm (module-regex): Add .zst to regexp. Update doc. (modinfo-section-contents): Extend support to Zstd compressed module. (dot-ko): Register the 'zstd compression type. (ensure-dot-ko, file-name->module-name, load-linux-module*) (module-name->file-name/guess, write-module-name-database) (write-module-alias-database, write-module-device-database): Update doc. (module-name-lookup): Also consider zstd-compressed modules. * gnu/installer.scm (installer-program): Add guile-zstd extension to gexp. * gnu/system/linux-initrd.scm (flat-linux-module-directory): Likewise. Decompress zstd-compressed modules for use in initrd. * guix/profiles.scm (linux-module-database): Add guile-zstd extension to gexp. Change-Id: Ide899dc5c58ea5033583b1a91a92c025fc8d901a --- gnu/build/linux-modules.scm | 62 +++++++++++++++++++++---------------- gnu/installer.scm | 1 + gnu/packages/linux.scm | 13 ++++++-- gnu/system/linux-initrd.scm | 6 ++-- guix/profiles.scm | 7 +++-- 5 files changed, 57 insertions(+), 32 deletions(-) diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm index 68c32ff873..32baf6c525 100644 --- a/gnu/build/linux-modules.scm +++ b/gnu/build/linux-modules.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2018 Danny Milosavljevic ;;; Copyright © 2023 Tobias Geerinckx-Rice +;;; Copyright © 2024 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ (define-module (gnu build linux-modules) #:use-module ((guix build utils) #:select (find-files invoke)) #:use-module (guix build union) #:autoload (zlib) (call-with-gzip-input-port) + #:autoload (zstd) (call-with-zstd-input-port) #:use-module (rnrs io ports) #:use-module (rnrs bytevectors) #:use-module (srfi srfi-1) @@ -108,24 +110,29 @@ (define (key=value->pair str) (cons (string->symbol (string-take str =)) (string-drop str (+ 1 =))))) -;; Matches kernel modules, without compression, with GZIP compression or with -;; XZ compression. -(define module-regex "\\.ko(\\.gz|\\.xz)?$") +;; Matches kernel modules, without compression, with GZIP, XZ or ZSTD +;; compression. +(define module-regex "\\.ko(\\.gz|\\.xz|\\.zst)?$") (define (modinfo-section-contents file) "Return the contents of the '.modinfo' section of FILE as a list of key/value pairs.." + (define (decompress-file decompressor file) + (let ((port (open-file file "r0"))) + (dynamic-wind + (lambda () + #t) + (lambda () + (decompressor port get-bytevector-all)) + (lambda () + (close-port port))))) + (define (get-bytevector file) (cond ((string-suffix? ".ko.gz" file) - (let ((port (open-file file "r0"))) - (dynamic-wind - (lambda () - #t) - (lambda () - (call-with-gzip-input-port port get-bytevector-all)) - (lambda () - (close-port port))))) + (decompress-file call-with-gzip-input-port file)) + ((string-suffix? ".ko.zst" file) + (decompress-file call-with-zstd-input-port file)) (else (call-with-input-file file get-bytevector-all)))) @@ -213,11 +220,12 @@ (define* (dot-ko name #:optional compression) (let ((suffix (match compression ('xz ".ko.xz") ('gzip ".ko.gz") + ('zstd ".ko.zst") (else ".ko")))) (string-append name suffix))) (define (ensure-dot-ko name compression) - "Return NAME with a '.ko[.gz|.xz]' suffix appended, unless it already has + "Return NAME with a '.ko[.gz|.xz|.zst]' suffix appended, unless it already has it." (if (string-contains name ".ko") name @@ -235,7 +243,7 @@ (define (normalize-module-name module) (define (file-name->module-name file) "Return the module name corresponding to FILE, stripping the trailing -'.ko[.gz|.xz]' and normalizing it." +'.ko[.gz|.xz|.zst]' and normalizing it." (normalize-module-name (strip-extension (basename file)))) (define (find-module-file directory module) @@ -333,11 +341,11 @@ (define* (load-linux-module* file (recursive? #t) (lookup-module dot-ko) (black-list (module-black-list))) - "Load Linux module from FILE, the name of a '.ko[.gz|.xz]' file; return true -on success, false otherwise. When RECURSIVE? is true, load its dependencies -first (à la 'modprobe'.) The actual files containing modules depended on are -obtained by calling LOOKUP-MODULE with the module name. Modules whose name -appears in BLACK-LIST are not loaded." + "Load Linux module from FILE, the name of a '.ko[.gz|.xz|.zst]' file; return +true on success, false otherwise. When RECURSIVE? is true, load its +dependencies first (à la 'modprobe'.) The actual files containing modules +depended on are obtained by calling LOOKUP-MODULE with the module name. +Modules whose name appears in BLACK-LIST are not loaded." (define (black-listed? module) (let ((result (member module black-list))) (when result @@ -695,7 +703,7 @@ (define* (module-name->file-name/guess directory name "Guess the file name corresponding to NAME, a module name. That doesn't always work because sometimes underscores in NAME map to hyphens (e.g., \"input-leds.ko\"), sometimes not (e.g., \"mac_hid.ko\"). If the module is -compressed then COMPRESSED can be set to 'xz or 'gzip, depending on the +compressed then COMPRESSED can be set to 'zstd, 'xz or 'gzip, depending on the compression type." (string-append directory "/" (ensure-dot-ko name compression))) @@ -706,6 +714,8 @@ (define (module-name-lookup directory) (define (guess-file-name name) (let ((names (list (module-name->file-name/guess directory name) + (module-name->file-name/guess directory name + #:compression 'zstd) (module-name->file-name/guess directory name #:compression 'xz) (module-name->file-name/guess directory name @@ -729,8 +739,8 @@ (define (module-name-lookup directory) (define (write-module-name-database directory) "Write a database that maps \"module names\" as they appear in the relevant -ELF section of '.ko[.gz|.xz]' files, to actual file names. This format is -Guix-specific. It aims to deal with inconsistent naming, in particular +ELF section of '.ko[.gz|.xz|.zst]' files, to actual file names. This format +is Guix-specific. It aims to deal with inconsistent naming, in particular hyphens vs. underscores." (define mapping (map (lambda (file) @@ -749,8 +759,8 @@ (define (write-module-name-database directory) (pretty-print mapping port)))) (define (write-module-alias-database directory) - "Traverse the '.ko[.gz|.xz]' files in DIRECTORY and create the corresponding -'modules.alias' file." + "Traverse the '.ko[.gz|.xz|.zst]' files in DIRECTORY and create the +corresponding 'modules.alias' file." (define aliases (map (lambda (file) (cons (file-name->module-name file) (module-aliases file))) @@ -796,9 +806,9 @@ (define %not-dash (char-set-complement (char-set #\-))) (define (write-module-device-database directory) - "Traverse the '.ko[.gz|.xz]' files in DIRECTORY and create the corresponding -'modules.devname' file. This file contains information about modules that can -be loaded on-demand, such as file system modules." + "Traverse the '.ko[.gz|.xz|.zst]' files in DIRECTORY and create the +corresponding 'modules.devname' file. This file contains information about +modules that can be loaded on-demand, such as file system modules." (define aliases (filter-map (lambda (file) (match (aliases->device-tuple (module-aliases file)) diff --git a/gnu/installer.scm b/gnu/installer.scm index 3b06b4f01b..3792fc7e35 100644 --- a/gnu/installer.scm +++ b/gnu/installer.scm @@ -386,6 +386,7 @@ (define (installer-program) guile-json-3 guile-git guile-webutils guile-gnutls guile-zlib ;for (gnu build linux-modules) + guile-zstd ;for (gnu build linux-modules) (current-guix)) (with-imported-modules `(,@(source-module-closure `(,@modules diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm index d2b9344296..7147a6b41a 100644 --- a/gnu/packages/linux.scm +++ b/gnu/packages/linux.scm @@ -847,6 +847,10 @@ (define (default-extra-linux-options version) ,@(if (version>=? version "5.13") '(("BPF_UNPRIV_DEFAULT_OFF" . #t)) '()) + ;; Compress kernel modules via Zstd. + ,(if (version>=? version "5.13") + '("CONFIG_MODULE_COMPRESS_ZSTD" . #t) + '("CONFIG_MODULE_COMPRESS_GZIP" . #t)) ;; Some very mild hardening. ("CONFIG_SECURITY_DMESG_RESTRICT" . #t) ;; All kernels should have NAMESPACES options enabled @@ -1036,7 +1040,10 @@ (define* (make-linux-libre* version gnu-revision source supported-systems "EXTRAVERSION ?=")) (setenv "EXTRAVERSION" #$(and extra-version - (string-append "-" extra-version))))) + (string-append "-" extra-version))) + ;; Use the maximum compression available for Zstd-compressed + ;; modules. + (setenv "ZSTD_CLEVEL" "19"))) (replace 'configure (lambda _ (let ((config @@ -1130,7 +1137,9 @@ (define* (make-linux-libre* version gnu-revision source supported-systems ;; support. dwarves ;for pahole python-wrapper - zlib)) + zlib + ;; For Zstd compression of kernel modules. + zstd)) (home-page "https://www.gnu.org/software/linux-libre/") (synopsis "100% free redistribution of a cleaned Linux kernel") (description "GNU Linux-Libre is a free (as in freedom) variant of the diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm index d448c78918..f902637e48 100644 --- a/gnu/system/linux-initrd.scm +++ b/gnu/system/linux-initrd.scm @@ -128,7 +128,7 @@ (define (flat-linux-module-directory linux modules) (define build-exp (with-imported-modules imported-modules - (with-extensions (list guile-zlib) + (with-extensions (list guile-zlib guile-zstd) #~(begin (use-modules (gnu build linux-modules) (guix build utils) @@ -168,7 +168,9 @@ (define (flat-linux-module-directory linux modules) ;; is already gzipped as a whole. (cond ((string-contains file ".ko.gz") - (invoke #+(file-append gzip "/bin/gunzip") file)))) + (invoke #+(file-append gzip "/bin/gunzip") file)) + ((string-contains file ".ko.zst") + (invoke #+(file-append zstd "/bin/zstd") "-d" file)))) (mkdir #$output) (for-each (lambda (module) diff --git a/guix/profiles.scm b/guix/profiles.scm index d41802422b..864ed02b6d 100644 --- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -7,7 +7,7 @@ ;;; Copyright © 2016, 2017, 2018, 2019, 2021, 2022 Ricardo Wurmus ;;; Copyright © 2016 Chris Marusich ;;; Copyright © 2017 Huang Ying -;;; Copyright © 2017, 2021 Maxim Cournoyer +;;; Copyright © 2017, 2021, 2024 Maxim Cournoyer ;;; Copyright © 2019 Kyle Meyer ;;; Copyright © 2019 Mathieu Othacehe ;;; Copyright © 2020 Danny Milosavljevic @@ -1487,11 +1487,14 @@ (define* (linux-module-database manifest #:optional system) (define guile-zlib (module-ref (resolve-interface '(gnu packages guile)) 'guile-zlib)) + (define guile-zstd + (module-ref (resolve-interface '(gnu packages guile)) 'guile-zstd)) + (define build (with-imported-modules (source-module-closure '((guix build utils) (gnu build linux-modules))) - (with-extensions (list guile-zlib) + (with-extensions (list guile-zlib guile-zstd) #~(begin (use-modules (ice-9 ftw) (ice-9 match)