diff mbox series

[bug#45692,4/4] gnu: Add ZFS service.

Message ID LmlpVtErWRRqKI9LIqHvswFA3ig_rjwIk9u8S96Bp6A9VG_l0bAnQdhIQsxf_fB5CAJxMnoClNXDObuPloDX7JDDOl4ZuFGn6u54FmJ5K2o=@protonmail.com
State Accepted
Headers show
Series [bug#42193] kernel-module-configuration-service for configuring kernel parameters | expand

Checks

Context Check Description
cbaines/comparison success View comparision
cbaines/git branch success View Git branch
cbaines/applying patch fail View Laminar job
cbaines/issue success View issue

Commit Message

raid5atemyhomework Jan. 8, 2021, 3:02 p.m. UTC
This new version is dependent on https://issues.guix.info/45723 and https://issues.guix.info/45722

New features:

* We can now delay pool importation after particular mapped devices are opened or file systems are mounted.  This is useful if you want to make a ZFS pool out of LUKS containers, or mount an encrypted ZFS dataset with the encryption key in another (hopefully also encrypted) filesystem.
* We can now put other non-ZFS filesystems inside ZVOLs, block devices provided by a ZFS pool, and have them mounted after ZFS has scanned pools and opened such ZVOLs for use by the ystem.

It's now possible to have a bunch of LUKS containers in a ZPOOL that provides a ZVOL that contains an EXT4 filesystem (though why would you do that, just use builtin ZFS encryption).

This uses my [PATCH 1/4] instead of https://issues.guix.info/42193#4.  Should I use the latter instead?  What would get ZFS on Guix faster?

I decided to remove ZFS parameter options, because I don't know whether I should wait for https://issues.guix.info/42193 and how long that would take.


From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
Date: Wed, 6 Jan 2021 09:24:20 +0800
Subject: [PATCH 4/4] gnu: Add service to install ZFS.

* gnu/services/file-systems.scm: New file.
(zfs-service-type): New variable.
(<zfs-configuration>): New type.
(%zfs-zvol-dependency): New variable.
* gnu/local.mk: Add gnu/services/file-systems.scm.
* gnu/services/base.scm (dependency->shepherd-service-name): Export.
* doc/guix.texi (ZFS file system): New subsection.
---
 doc/guix.texi                 | 107 +++++++++++++++++++++
 gnu/local.mk                  |   1 +
 gnu/services/base.scm         |   4 +-
 gnu/services/file-systems.scm | 174 ++++++++++++++++++++++++++++++++++
 4 files changed, 285 insertions(+), 1 deletion(-)
 create mode 100644 gnu/services/file-systems.scm

--
2.30.0

Comments

Ludovic Courtès Feb. 10, 2021, 2:27 p.m. UTC | #1
raid5atemyhomework <raid5atemyhomework@protonmail.com> skribis:

>>From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework <raid5atemyhomework@protonmail.com>
> Date: Wed, 6 Jan 2021 09:24:20 +0800
> Subject: [PATCH 4/4] gnu: Add service to install ZFS.
>
> * gnu/services/file-systems.scm: New file.
> (zfs-service-type): New variable.
> (<zfs-configuration>): New type.
> (%zfs-zvol-dependency): New variable.
> * gnu/local.mk: Add gnu/services/file-systems.scm.
> * gnu/services/base.scm (dependency->shepherd-service-name): Export.
> * doc/guix.texi (ZFS file system): New subsection.

A bit of nitpicking on the documentation bits…

> +@node ZFS file system
> +@subsection ZFS file system

Please follow the same typographical conventions as in the rest of the
manual.  In particular, capitalize section titles and leave two spaces
after end-of-sentence periods.

Here I recommend s/ZFS file system/ZFS/ in all the text.

> +The ZFS file system has a license incompatible with the Linux kernel,
> +and thus cannot be distributed with the kernel. However, as a user
> +you have the right to do whatever you want on your own hardware,
> +including download the ZFS source code, compile it, link it to your
> +own private copy of Linux, and run it. You can even use the Guix build
> +system to automate this.

How about:

  ZFS is free software; unfortunately its license is incompatible with
  the GNU General Public License (GPL), the license of the Linux kernel,
  which means they cannot be distributed together.  However, as a user,
  you can choose to build ZFS and use it together with Linux; you can
  even rely on Guix to automate this task.  See
  @uref{https://www.fsf.org/licensing/zfs-and-linux, this analysis by
  the Free Software Foundation} for more information.

?

> +As a large and complex kernel module, ZFS on Linux has to be compiled
                                         ^
I think it’s a proper noun and is spelled “ZFS-on-Linux”.

> +with a specific version of the kernel. Often even the latest ZFS
> +package available cannot be compiled with the latest Linux kernel
                                                        ^
s/Linux/Linux-libre/

Thanks,
Ludo’.
raid5atemyhomework Feb. 10, 2021, 2:32 p.m. UTC | #2
Hi Ludo,

> raid5atemyhomework raid5atemyhomework@protonmail.com skribis:
>
> > > From d0e095136cd471f083d92416f12ca22b47301f31 Mon Sep 17 00:00:00 2001
> > > From: raid5atemyhomework raid5atemyhomework@protonmail.com
> > > Date: Wed, 6 Jan 2021 09:24:20 +0800
> > > Subject: [PATCH 4/4] gnu: Add service to install ZFS.
> >
> > -   gnu/services/file-systems.scm: New file.
> >     (zfs-service-type): New variable.
> >     (<zfs-configuration>): New type.
> >     (%zfs-zvol-dependency): New variable.
> >
> > -   gnu/local.mk: Add gnu/services/file-systems.scm.
> > -   gnu/services/base.scm (dependency->shepherd-service-name): Export.
> > -   doc/guix.texi (ZFS file system): New subsection.
>
> A bit of nitpicking on the documentation bits…
>
> > +@node ZFS file system
> > +@subsection ZFS file system
>
> Please follow the same typographical conventions as in the rest of the
> manual. In particular, capitalize section titles and leave two spaces
> after end-of-sentence periods.

This was imitated from the `@subsection Btrfs file system`.

Thanks
raid5atemyhomework
raid5atemyhomework Feb. 13, 2021, 1:49 a.m. UTC | #3
Hello Ludo',


> > +As a large and complex kernel module, ZFS on Linux has to be compiled
>
>                                          ^
>
>
> I think it’s a proper noun and is spelled “ZFS-on-Linux”.

FWIW ZFSonLinux is now folded into OpenZFS (which is now a package that includes both Linux and FreeBSD support, with plans to merge in the Illumos codebase as well) since 2.0.0.  The documentation was written when Guix was still at 0.8.5 (when it was definitely ZoL), so I should probably change this to OpenZFS, and clarify whether I am referring to the ZFS file system or the OpenZFS package.

Thanks
raid5atemyhomework
diff mbox series

Patch

diff --git a/doc/guix.texi b/doc/guix.texi
index a31d355780..714274b8c9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13886,6 +13886,113 @@  a file system declaration such as:
 compress-force=zstd,space_cache=v2"))
 @end lisp

+@node ZFS file system
+@subsection ZFS file system
+
+The ZFS file system has a license incompatible with the Linux kernel,
+and thus cannot be distributed with the kernel. However, as a user
+you have the right to do whatever you want on your own hardware,
+including download the ZFS source code, compile it, link it to your
+own private copy of Linux, and run it. You can even use the Guix build
+system to automate this.
+
+As a large and complex kernel module, ZFS on Linux has to be compiled
+with a specific version of the kernel. Often even the latest ZFS
+package available cannot be compiled with the latest Linux kernel
+package provided by Guix System. Thus, installing the @code{zfs}
+package is likely to fail.
+
+Instead, you have to use an older long-term-support Linux kernel.
+Do not use @code{linux-libre-lts} as the latest long-term-support
+kernel might be too new for the ZFS package; instead, explicitly
+select the version number, like @code{linux-libre-5.4}.
+
+Then, you have to modify your system configuration file and use the
+selected older kernel, and add the @code{zfs-service-type} service.
+
+@lisp
+(use-modules (gnu))
+(use-package-modules
+   ; @dots{}
+   linux)
+(use-service-modules
+   ; @dots{}
+   file-systems)
+
+(define my-kernel linux-libre-5.4)
+
+(operating-system
+  (kernel my-kernel)
+  ;; @dots{}
+  (services
+    (cons* (service zfs-service-type
+             (zfs-configuration
+               (kernel my-kernel)))
+           ; @dots{}
+           %desktop-services))
+  ;; @dots{}
+  )
+@end lisp
+
+@defvr {Scheme Variable} zfs-service-type
+This is the type of the service to compile and install OpenZFS to
+your operating system. It loads the ZFS module at startup, imports
+pools, mounts automounted ZFS datasets, installs the ZFS command
+line tools, and can provide module options for the ZFS module.  Its
+value must be a @code{zfs-configuration} record (see below).
+
+Here is an example use:
+
+@lisp
+(service zfs-service-type
+  (zfs-configuration
+    (kernel linux-libre-5.4)
+    (options '("zfs_arc_max=5000000000"))))
+@end lisp
+@end defvr
+
+@deftp {Data Type} zfs-configuration
+This data type represents the configuration of the ZFS service.
+The available fields are:
+
+@table @asis
+@item @code{kernel}
+The package of the Linux kernel to compile ZFS for. Required.
+
+@item @code{base-zfs} (default: @code{zfs})
+The ZFS package to use. It will be modified to use the indicated
+kernel.
+
+@item @code{options} (default: @code{'()})
+A list of string options to pass as options to the ZFS module.
+These will be put in a @file{/etc/modprobe.d/zfs.conf} file,
+for example setting this to @code{'("zfs_admin_snapshot=1"
+"zfs_trim_extent_bytes_min=0")} will create the following file:
+
+@example
+options zfs zfs_admin_snapshot=1 zfs_trim_extent_bytes_min=0
+@end example
+@end table
+@end deftp
+
+Once your system has been configured to include the ZFS service
+and you have restarted the system, you can manage ZFS pools and
+datasets with @code{zpool} and @code{zfs} commands.
+
+ZFS datasets with an appropriate @code{mountpoint} property will
+be automounted at startup after the root file system is started.
+Encrypted datasets that are automounted will cause boot to pause
+and prompt for the password to be provided on the console.
+
+It's possible to have a ZFS dataset as your @code{/home} by simply
+setting the @code{mountpoint} property. However, note that ZFS will
+refuse to mount over a non-empty directory, so if your root
+filesystem already has a non-empty @code{/home}, you should remove
+it and its contents, then restart the system.
+
+Having ZFS as a root filesystem or as @code{/boot} is not supported
+yet.
+
 @node Mapped Devices
 @section Mapped Devices

diff --git a/gnu/local.mk b/gnu/local.mk
index 1151d4642e..5aeb45c4c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -602,6 +602,7 @@  GNU_SYSTEM_MODULES =				\
   %D%/services/dict.scm				\
   %D%/services/dns.scm				\
   %D%/services/docker.scm			\
+  %D%/services/file-systems.scm			\
   %D%/services/authentication.scm		\
   %D%/services/games.scm			\
   %D%/services/ganeti.scm			\
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 13cfb6a8a2..ef3680583b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -188,7 +188,9 @@ 

             references-file

-            %base-services))
+            %base-services
+
+            dependency->shepherd-service-name))

 ;;; Commentary:
 ;;;
diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scm
new file mode 100644
index 0000000000..9061ab9582
--- /dev/null
+++ b/gnu/services/file-systems.scm
@@ -0,0 +1,174 @@ 
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services file-systems)
+  #:use-module (gnu packages file-systems)
+  #:use-module (gnu services)
+  #:use-module (gnu services base)
+  #:use-module (gnu services linux)
+  #:use-module (gnu services shepherd)
+  #:use-module (gnu system mapped-devices)
+  #:use-module (guix gexp)
+  #:use-module (guix packages)
+  #:use-module (guix records)
+  #:export (zfs-service-type
+
+            zfs-configuration
+            zfs-configuration?
+            zfs-configuration-kernel
+            zfs-configuration-base-zfs
+            zfs-configuration-dependencies
+
+            %zfs-zvol-dependency))
+
+(define-record-type* <zfs-configuration>
+  zfs-configuration make-zfs-configuration zfs-configuration?
+  ; kernel you want to compile the base-zfs module for.
+  (kernel         zfs-configuration-kernel)
+  ; base package that will be compiled for the kernel
+  (base-zfs       zfs-configuration-base-zfs
+                  (default zfs))
+  ; list of <mapped-device> | <file-system> we should wait for,
+  ; before scanning for ZFS pools.
+  (dependencies   zfs-configuration-dependencies
+                  (default '())))
+
+;; This is a synthetic and unusable MAPPED-DEVICE; its only use
+;; is to be added as a (dependency ...) of some FILE-SYSTEM.
+(define %zfs-zvol-dependency
+  (mapped-device
+    (source '())
+    ;; The /* prevents naming conflict with non-ZFS device mappings,
+    ;; since it is not a valid name for mapped devices, and also
+    ;; implies "all zvols" in terms of globs.
+    (targets '("zvol/*"))
+    (type #f)))
+
+(define (make-zfs-package conf)
+  (let ((base-zfs (zfs-configuration-base-zfs conf))
+        (kernel   (zfs-configuration-kernel conf)))
+    (package
+      (inherit base-zfs)
+      (name (string-join (list (package-name base-zfs)
+                               "for"
+                               (package-name kernel)
+                               (package-version kernel)
+                               "version")
+                         "-"))
+      (arguments (cons* #:linux kernel (package-arguments base-zfs))))))
+
+(define (zfs-loadable-module conf)
+  (list (list (make-zfs-package conf) "module")))
+
+(define (zfs-shepherd-services conf)
+  (let* ((zfs-package     (make-zfs-package conf))
+         (zpool           (file-append zfs-package "/sbin/zpool"))
+         (zfs             (file-append zfs-package "/sbin/zfs"))
+         (zvol_wait       (file-append zfs-package "/bin/zvol_wait"))
+         (scheme-modules  `((srfi srfi-1)
+                            (srfi srfi-34)
+                            (srfi srfi-35)
+                            (rnrs io ports)
+                            ,@%default-modules)))
+
+    (define zfs-scan
+      (shepherd-service
+        (provision '(zfs-scan))
+        (documentation "Scans for ZFS pools.")
+        (requirement `(kernel-module-loader
+                       root-file-system
+                       ,@(map dependency->shepherd-service-name
+                              (zfs-configuration-dependencies conf))))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error importing zpools: ~a~%"
+                                      (condition-message c))
+                              #f))
+                     ; TODO: optionally use a cachefile, for systems with dozens or
+                     ; hundreds of devices.
+                     (invoke/quiet #$zpool "import" "-a" "-N"))))
+        (stop #~(const #t))))
+
+    (define device-mapping-zvol/*
+      (shepherd-service
+        (provision '(device-mapping-zvol/*))
+        (documentation "Waits for ZFS ZVOL devices to appear.")
+        (requirement '(zfs-scan))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error waiting for zvols: ~a~%"
+                                      (condition-message c))
+                              #f))
+                     (invoke/quiet #$zvol_wait))))
+        (stop #~(const #t))))
+
+    (define zfs-automount
+      (shepherd-service
+        (provision '(zfs-automount))
+        (documentation "Automounts ZFS datasets.")
+        (requirement '(zfs-scan))
+        (modules scheme-modules)
+        (start #~(lambda _
+                   (guard (c ((message-condition? c)
+                              (format (current-error-port)
+                                      "error automounting zfs: ~a~$")
+                              #f))
+                     ; (current-output-port) is typically connected to /dev/klog,
+                     ; so redirect it to (current-error-port) so that user can see
+                     ; prompts for passphrases on console
+                     (with-output-to-port (current-error-port)
+                       (lambda ()
+                         (invoke #$zfs "mount" "-a" "-l"))))))
+        (stop #~(lambda _
+                  ;; make sure we don't keep any ZFS mountpoints busy.
+                  (chdir "/")
+                  ;; unmount everything.
+                  (invoke/quiet #$zfs "unmount" "-a" "-f")))))
+
+    (list zfs-scan
+          device-mapping-zvol/*
+          zfs-automount)))
+
+(define zfs-service-type
+  (service-type (name 'zfs)
+                (extensions
+                  (list
+                    ; install the kernel module
+                    (service-extension kernel-loadable-module-service-type
+                                       zfs-loadable-module)
+                    ; load the kernel module
+                    (service-extension kernel-module-loader-service-type
+                                       (const '("zfs")))
+                    ; scan ZFS pools, automount filesystem, wait for zvols.
+                    (service-extension shepherd-root-service-type
+                                       zfs-shepherd-services)
+                    ; make sure automount occurs before file-systems target is reached
+                    (service-extension file-systems-target-service-type
+                                       (const '(zfs-automount)))
+                    ; install ZFS management tools
+                    (service-extension profile-service-type
+                                       (compose list make-zfs-package))
+                    ; install ZFS udev rules
+                    (service-extension udev-service-type
+                                       (compose list make-zfs-package))))
+                (description
+                  "Install ZFS, an advanced filesystem and volume manager.")))