[bug#41011] gnu: grub: Support for network boot via tftp/nfs.

* gnu/bootloader/grub.scm (grub-net-bootloader): New bootloader for
network booting via tftp.
(install-grub-net): New bootloader installer for tftp.
---
 gnu/bootloader/grub.scm | 79 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 75 insertions(+), 4 deletions(-)

Hi!

This debbugs thread got already very long. Therefore I would like to focus on the grub changes in this ticket done with my last patch, see <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41011#95>

This patch only introduces a new grub-net bootloader, which basically installs grub via its grub-mknetdir command.

There is no modification of other bootloaders. The restriction of the “/boot/grub/grub.cfg” file remains.

As usually the DHCP option 67 “Bootfile name” is involved to point the EFI of a machine to the file to boot via TFTP, I chose to export (install-grub-net subdir) as well to be able to modify that path, whose default is /boot/efi/Guix/boot[x64|aa64|…].efi, to something else, e.g.

(bootloader (inherit grub-net-bootloader (installer (install-grub-net "efi/machine-1"))))

To be able to boot different machines with their own guix installation, however, there is still the problem to provide to each an own grub.cfg file via TFTP. This file – as you know – has still a hard coded path of /boot/grub/grub.cfg. But this is a different issue and should not be tackled with this patch.


Bye

Stefan

Hi Stefan,

I think this looks good in general.

I'd like to do some nitpicking on the names--especially since the procedure is
exported and thus presumably can't have its signature modified later without
breaking backward compatibility.

In this case, the man page grub-mknetdir(8) mentions "netboot" ?
Do you think "net" or "netboot" is a better name for this functionality ?

On Sat, 5 Sep 2020 13:25:24 +0200
Stefan <stefan-guix@vodafonemail.de> wrote:

> +            install-grub-net

I'm fine with whatever--but the man page says "netboot".  If that's the usual
name, let's use it.  If "net"'s the usual name, let's use that.

> +  (let* ((arch (car (string-split (or (%current-target-system)
> +                                      (%current-system))
> +                                  #\-)))

Let's not use arcane Scheme anachronisms like "car".  I know most Scheme
programmers probably know what it does--but still, better not to use
names of registers of a machine no one uses anymore.

Better something like this:

(let* ((system-parts (string-split (or (%current-target-system)
                                       (%current-system))
                            #\-)))

> +         (efi-bootloader-link (string-append "/boot"

> +                                             (match arch
> +                                               ("i686" "ia32")
> +                                               ("x86_64" "x64")
> +                                               ("arm" "arm")
> +                                               ("armhf" "arm")
> +                                               ("aarch64" "aa64")
> +                                               ("riscv" "riscv32")
> +                                               ("riscv64" "riscv64"))
> +                                             ".efi"))

Also, I have a slight preference for greppable file names even when it's a
little more redundant, so more like that:

(match system-parts
 (("i686" _ ...) "ia32.efi")
 (("x86_64" _ ...) "x64.efi")
 (("arm" _ ...) "arm.efi")
 (("armhf" _ ...) "arm.efi")
 (("aarch64" _ ...) "aa64.efi")
 (("riscv" _ ...) "riscv32.efi")
 (("riscv64" _ ...) "riscv64.efi"))

> +         (efi-bootloader (string-append (match arch
> +                                          ("i686" "i386")
> +                                          ("x86_64" "x86_64")
> +                                          ("arm" "arm")
> +                                          ("armhf" "arm")
> +                                          ("aarch64" "arm64")
> +                                          ("riscv" "riscv32")
> +                                          ("riscv64" "riscv64"))
> +                                        "-efi/core.efi")))

Likewise:

         (efi-bootloader (match system-parts
                          (("i686" _ ...) "i386-efi/core.efi")
                          (("x86_64" _ ...) "x86_64-efi/core.efi")
                          (("arm" _ ...) "arm-efi/core.efi")
                          (("armhf" _ ...) "arm-efi/core.efi")
                          (("aarch64" _ ...) "arm64-efi/core.efi")
                          (("riscv" _ ...) "riscv32-efi/core.efi")
                          (("riscv64" _ ...) "riscv64-efi/core.efi"))))

> +    #~(lambda (bootloader target mount-point)

> +        "Install GRUB as e.g. \"bootx64.efi\" or \"bootarm64.efi\" \"into
> +SUBDIR, which is usually \"efi/boot\" or \"efi/Guix\" below the directory TARGET
> +for the system whose root is mounted at MOUNT-POINT."

I think you mean:

> +        "Install GRUB as e.g. \"bootx64.efi\" or \"bootarm64.efi\" \"into
> +SUBDIR (which is usually \"efi/boot\" or \"efi/Guix\") below the directory TARGET
> +for the system whose root is mounted at MOUNT-POINT."

> +        (let* (;; Use target-depth and subdir-depth to construct links to
> +               ;; "../gnu" and "../../../boot/grub/grub.cfg" with the correct
> +               ;; number of "../". Note: This doesn't consider ".." or ".",
> +               ;; which may appear inside target or subdir.

Uhhhh... that could use some more explanationary comments in the source code
of why it is done in the first place.

Also, is TARGET itself assumed to be an absolute path or is it relative to
something else ?  According to the rest of the patch it's relative to
MOUNT-POINT--but please state this explicitly in the docstring.

> +               (target-depth (length (delete "" (string-split target #\/))))

> +               (subdir-depth (length (delete "" (string-split #$subdir #\/))))

> +               (up1 (string-join (make-list target-depth "..") "/" 'suffix))

Maybe better name: escape-target or something.

> +               (up2 (string-join (make-list subdir-depth "..") "/" 'suffix))

Maybe better name: escape-subdir or something.

So this is in order to get out of (string-append TARGET "/" SUBDIR), correct?
Does the (string-append TARGET "/" SUBDIR) have an official name ?
If not, fine.

> +               (net-dir (string-append mount-point target "/"))

So TARGET is relative to MOUNT-POINT ?
And MOUNT-POINT is assumed to have a slash at the end ?

> +               (store-name (car (delete "" (string-split bootloader #\/))))

Maybe use match.

Also isn't there an official way to find out how the store is called ?
(%store-prefix) ?

> +               (store (string-append up1 store-name))

(string-append escape-target store-name)

> +               (store-link (string-append net-dir store-name))

*mumbles to self* (string-append MOUNT-POINT TARGET) is net-dir.
So it tries to get to (string-append MOUNT-POINT "/gnu").

I vaguely remember our docker pack adding some serious plumbing to support
symlinks like that.

I'll try to find it.  I just wanted to send this E-Mail because of the following:

>  ;;;
>  ;;; Bootloader definitions.
>  ;;;
> +;;; For all these grub-bootloader variables the path to /boot/grub/grub.cfg
> +;;; is fixed.  Inheriting and overwriting the field 'configuration-file' will
> +;;; break 'guix system delete-generations', 'guix system switch-generation',
> +;;; and 'guix system roll-back'.

I've added that comment to the source code in an extra commit
3f2bd9df410e85795ec656052f44d2cddec2a060 in guix master.
Thank you very much for it.

> -(define* grub-minimal-bootloader
> +(define grub-minimal-bootloader
>    (bootloader

> -(define* grub-efi-bootloader
> +(define grub-efi-bootloader
>    (bootloader

> -(define* grub-mkrescue-bootloader
> +(define grub-mkrescue-bootloader

I've applied this hunk to guix master as commit
8664c35d6d7fd6e9ce1ca8adefa8070a8e556db4.

Thanks.

> I vaguely remember our docker pack adding some serious plumbing to support
> symlinks like that.

                       ((guix build union) #:select (relative-file-name))

symlink-relative in (guix build union):

  "Assuming both OLD and NEW are absolute file names, make NEW a symlink to
OLD, but using a relative file name."

Hi Danny!

> In this case, the man page grub-mknetdir(8) mentions "netboot" ?
> Do you think "net" or "netboot" is a better name for this functionality ?

At <https://www.gnu.org/software/grub/manual/grub/grub.html> there is only a single hit for ‘netboot’ which is in “To generate a netbootable directory, run:”.

All GRUB variables and commands have the prefix ‘net_’.

But I agree, grub-netboot seems to be a more describing name.

In the end this is a grub-efi for booting over network. Would grub-efi-netboot be an even better name? It will not work with BIOS machines.

> Let's not use arcane Scheme anachronisms like "car".  I know most Scheme
> programmers probably know what it does--but still, better not to use
> names of registers of a machine no one uses anymore.
> 
> Better something like this:
> 
> (let* ((system-parts (string-split (or (%current-target-system)
>                                       (%current-system))
>                            #\-)))

I only need the first list element here. I will use (first …).

>> +         (efi-bootloader-link (string-append "/boot"
> 
>> +                                             (match arch
>> +                                               ("i686" "ia32")
>> +                                               ("x86_64" "x64")
>> +                                               ("arm" "arm")
>> +                                               ("armhf" "arm")
>> +                                               ("aarch64" "aa64")
>> +                                               ("riscv" "riscv32")
>> +                                               ("riscv64" "riscv64"))
>> +                                             ".efi"))
> 
> Also, I have a slight preference for greppable file names even when it's a
> little more redundant, so more like that:
> 
> (match system-parts
> (("i686" _ ...) "ia32.efi")
> (("x86_64" _ ...) "x64.efi")
> (("arm" _ ...) "arm.efi")
> (("armhf" _ ...) "arm.efi")
> (("aarch64" _ ...) "aa64.efi")
> (("riscv" _ ...) "riscv32.efi")
> (("riscv64" _ ...) "riscv64.efi"))

I’m not familiar with the match syntax yet. For me using the first element as arch seems simpler.

>> +         (efi-bootloader (string-append (match arch
>> +                                          ("i686" "i386")
>> +                                          ("x86_64" "x86_64")
>> +                                          ("arm" "arm")
>> +                                          ("armhf" "arm")
>> +                                          ("aarch64" "arm64")
>> +                                          ("riscv" "riscv32")
>> +                                          ("riscv64" "riscv64"))
>> +                                        "-efi/core.efi")))
> 
> Likewise:
> 
>         (efi-bootloader (match system-parts
>                          (("i686" _ ...) "i386-efi/core.efi")
>                          (("x86_64" _ ...) "x86_64-efi/core.efi")
>                          (("arm" _ ...) "arm-efi/core.efi")
>                          (("armhf" _ ...) "arm-efi/core.efi")
>                          (("aarch64" _ ...) "arm64-efi/core.efi")
>                          (("riscv" _ ...) "riscv32-efi/core.efi")
>                          (("riscv64" _ ...) "riscv64-efi/core.efi"))))

I’d prefer to keep the still grepable “/core.efi” separate.

>> +    #~(lambda (bootloader target mount-point)
> 
>> +        "Install GRUB as e.g. \"bootx64.efi\" or \"bootarm64.efi\" \"into
>> +SUBDIR, which is usually \"efi/boot\" or \"efi/Guix\" below the directory TARGET
>> +for the system whose root is mounted at MOUNT-POINT."
> 
> I think you mean:
> 
>> +        "Install GRUB as e.g. \"bootx64.efi\" or \"bootarm64.efi\" \"into
>> +SUBDIR (which is usually \"efi/boot\" or \"efi/Guix\") below the directory TARGET
>> +for the system whose root is mounted at MOUNT-POINT."

Yes.

>> +        (let* (;; Use target-depth and subdir-depth to construct links to
>> +               ;; "../gnu" and "../../../boot/grub/grub.cfg" with the correct
>> +               ;; number of "../". Note: This doesn't consider ".." or ".",
>> +               ;; which may appear inside target or subdir.
> 
> Uhhhh... that could use some more explanationary comments in the source code
> of why it is done in the first place.

I’ll put an explanation into the doc-string. This is because the grub.cfg and the store both need to be accessible to GRUB via TFTP, but the TFTP root is TARGET, which is usually /boot.

> Also, is TARGET itself assumed to be an absolute path or is it relative to
> something else ?  According to the rest of the patch it's relative to
> MOUNT-POINT--but please state this explicitly in the docstring.

TARGET is the (operating-system (boot-loader (target "/boot") …) …). I think this has to be an absolute path, but I didn’t find any checks for this. But the manual doesn’t mention this, just all examples are using absolute paths.

And yes, when using ‘guix system init /etc/config.scm /mnt/here’, then MOUNT-POINT and TARGET are concatenated. But this is nothing specific to the new installer, this is the usual behaviour of Guix and the reason for the two parameters TARGET and MOUNT-POINT to any bootloader installer. I don’t think stating this inside the new doc-string is the right place.

>> +               (target-depth (length (delete "" (string-split target #\/))))
> 
>> +               (subdir-depth (length (delete "" (string-split #$subdir #\/))))
> 
>> +               (up1 (string-join (make-list target-depth "..") "/" 'suffix))
> 
> Maybe better name: escape-target or something.
> 
>> +               (up2 (string-join (make-list subdir-depth "..") "/" 'suffix))
> 
> Maybe better name: escape-subdir or something.
> 
> So this is in order to get out of (string-append TARGET "/" SUBDIR), correct?

Yes, correct. I’ll rework this with the (symlink-relative) function you mentioned.

> Does the (string-append TARGET "/" SUBDIR) have an official name ?
> If not, fine.

No. The TARGET becomes the TFTP root for GRUB, the SUBDIR becomes the ‘prefix’ variable for GRUB.

>> +               (net-dir (string-append mount-point target "/"))
> 
> So TARGET is relative to MOUNT-POINT ?
> And MOUNT-POINT is assumed to have a slash at the end ?

MOUNT-POINT is either ‘/’ or depends on the argument to ‘guix system init’. On the other side TARGET has to be an absolute path, so it should be safe. At least (install-grub-efi) makes the same mistake. What do you think?

>> +               (store-name (car (delete "" (string-split bootloader #\/))))
> 
> Maybe use match.

I’ll use (first …).

> Also isn't there an official way to find out how the store is called ?
> (%store-prefix) ?

I only need the first path element to the store, which is usually /gnu. The %store-prefix contains /gnu/store then. So it makes no difference.

>> +               (store (string-append up1 store-name))
> 
> (string-append escape-target store-name)
> 
>> +               (store-link (string-append net-dir store-name))
> 
> *mumbles to self* (string-append MOUNT-POINT TARGET) is net-dir.
> So it tries to get to (string-append MOUNT-POINT "/gnu").

The trouble is that GRUB shall load a file like /gnu/store/…-linux…/Image via TFTP, but the TFTP root is actually Guix’ final /boot folder.

In the end this creates a relative symlink as ../gnu pointing from /mnt/here/boot/gnu to /mnt/here/gnu.

And GRUB’s “working directory” to search for its modules and the grub.cfg is defined by its ‘prefix’ variable, which is set through the SUBDIR argument, which defaults to Guix’ final /boot/efi/Guix.

This requires a relative symlink as ../../../boot/grub/grub.cfg pointing from /mnt/here/boot/efi/Guix/grub.cfg to /mnt/here/boot/grub/grub.cfg.

And be aware that TARGET may be /boot, but could be something else like /tftp-root. Then the symlink would point from /mnt/here/tftp-root/efi/Guix/grub.cfg to /mnt/here/boot/grub/grub.cfg, as the later is kind of hard-coded.

>> ;;; Bootloader definitions.
>> ;;;
>> +;;; For all these grub-bootloader variables the path to /boot/grub/grub.cfg
>> +;;; is fixed.  Inheriting and overwriting the field 'configuration-file' will
>> +;;; break 'guix system delete-generations', 'guix system switch-generation',
>> +;;; and 'guix system roll-back'.
> 
> I've added that comment to the source code in an extra commit
> 3f2bd9df410e85795ec656052f44d2cddec2a060 in guix master.
> Thank you very much for it.
> 
>> -(define* grub-minimal-bootloader
>> +(define grub-minimal-bootloader
>>   (bootloader
> 
>> -(define* grub-efi-bootloader
>> +(define grub-efi-bootloader
>>   (bootloader
> 
>> -(define* grub-mkrescue-bootloader
>> +(define grub-mkrescue-bootloader
> 
> I've applied this hunk to guix master as commit
> 8664c35d6d7fd6e9ce1ca8adefa8070a8e556db4.
> 
> Thanks.


Thanks!


Bye

Stefan

Hi Stefan,

On Tue, 8 Sep 2020 00:59:38 +0200
Stefan <stefan-guix@vodafonemail.de> wrote:

> In the end this is a grub-efi for booting over network. 

>Would grub-efi-netboot be an even better name? It will not work with BIOS machines.

Oh, then definitely let's use that name.

> I only need the first list element here. I will use (first …).

Okay.

(I leave it to the others to comment on here if they have a problem with it--I
see no downside in this case)

> >> +         (efi-bootloader-link (string-append "/boot"  
> >   
> >> +                                             (match arch
> >> +                                               ("i686" "ia32")
> >> +                                               ("x86_64" "x64")
> >> +                                               ("arm" "arm")
> >> +                                               ("armhf" "arm")
> >> +                                               ("aarch64" "aa64")
> >> +                                               ("riscv" "riscv32")
> >> +                                               ("riscv64" "riscv64"))
> >> +                                             ".efi"))  
> > 
> > Also, I have a slight preference for greppable file names even when it's a
> > little more redundant, so more like that:
> > 
> > (match system-parts
> > (("i686" _ ...) "ia32.efi")
> > (("x86_64" _ ...) "x64.efi")
> > (("arm" _ ...) "arm.efi")
> > (("armhf" _ ...) "arm.efi")
> > (("aarch64" _ ...) "aa64.efi")
> > (("riscv" _ ...) "riscv32.efi")
> > (("riscv64" _ ...) "riscv64.efi"))  
> 
> I’m not familiar with the match syntax yet. For me using the first element as arch seems simpler.

Match just does pattern matching.  The pattern here is for example ("i686" _ ...).
That means it will match anything that is a list that is starting with "i686".
It will put the remainder (...) into the variable "_" (which is customary to
use as "don't care" variable).

The major advantage of using "match" is its failure mode.  If the thing matched
on is not a list (for some unfathomable reason) or if the first element is not
matched on (!) then you get an exception--which is much better than doing weird
unknown stuff.

You have used "match" before--but only on parts of the list.  Why not use it
on the whole list?  It makes little sense to do manual destructuring and then
use match--when match would have done the destructuring bind anyway.

> > Likewise:
> > 
> >         (efi-bootloader (match system-parts
> >                          (("i686" _ ...) "i386-efi/core.efi")
> >                          (("x86_64" _ ...) "x86_64-efi/core.efi")
> >                          (("arm" _ ...) "arm-efi/core.efi")
> >                          (("armhf" _ ...) "arm-efi/core.efi")
> >                          (("aarch64" _ ...) "arm64-efi/core.efi")
> >                          (("riscv" _ ...) "riscv32-efi/core.efi")
> >                          (("riscv64" _ ...) "riscv64-efi/core.efi"))))  
> 
> I’d prefer to keep the still grepable “/core.efi” separate.

Sure.

> And yes, when using ‘guix system init /etc/config.scm /mnt/here’, then MOUNT-POINT and TARGET are concatenated. But this is nothing specific to the new installer, this is the usual behaviour of Guix and the reason for the two parameters TARGET and MOUNT-POINT to any bootloader installer. I don’t think stating this inside the new doc-string is the right place.

Ah, so that's what it means.

Well, it should be stated *somewhere* at least.  It probably is and I just
didn't see it.

> Yes, correct. I’ll rework this with the (symlink-relative) function you mentioned.

Thanks!

> > So TARGET is relative to MOUNT-POINT ?
> > And MOUNT-POINT is assumed to have a slash at the end ?  
> 
> MOUNT-POINT is either ‘/’ or depends on the argument to ‘guix system init’. On the other side TARGET has to be an absolute path, so it should be safe. At least (install-grub-efi) makes the same mistake. What do you think?

If grub-efi does it then it seems to be fine to do it--at least we didn't get
bug reports caused by it.  Let's just keep using it for the time being.

> >> +               (store-name (car (delete "" (string-split bootloader #\/))))  
> > 
> > Maybe use match.  
> 
> I’ll use (first …).
> 
> > Also isn't there an official way to find out how the store is called ?
> > (%store-prefix) ?  
> 
> I only need the first path element to the store, which is usually /gnu. The %store-prefix contains /gnu/store then. So it makes no difference.

I have no strong opinion either way, except please add a comment that you
are extracting part of the store prefix (or whatever) from the in-store
name of the bootloader store item.  It seems weird to me to do that--but
then again I don't get why Guix has two directories (/gnu and /gnu/store)
to the store anyway.

Fine, I guess.

I'm not sure whether it would be technically possible to have a custom
store directory like "/foo" without "/gnu" as the store.  That would be
a problem--and I'm sure someone somewhere does that--otherwise, why have
%store-prefix as a variable otherwise?

> >> +               (store (string-append up1 store-name))  
> > 
> > (string-append escape-target store-name)
> >   
> >> +               (store-link (string-append net-dir store-name))  
> > 
> > *mumbles to self* (string-append MOUNT-POINT TARGET) is net-dir.
> > So it tries to get to (string-append MOUNT-POINT "/gnu").  
> 
> The trouble is that GRUB shall load a file like /gnu/store/…-linux…/Image via TFTP, but the TFTP root is actually Guix’ final /boot folder.
> 
> In the end this creates a relative symlink as ../gnu pointing from /mnt/here/boot/gnu to /mnt/here/gnu.
> 
> And GRUB’s “working directory” to search for its modules and the grub.cfg is defined by its ‘prefix’ variable, which is set through the SUBDIR argument, which defaults to Guix’ final /boot/efi/Guix.
> 
> This requires a relative symlink as ../../../boot/grub/grub.cfg pointing from /mnt/here/boot/efi/Guix/grub.cfg to /mnt/here/boot/grub/grub.cfg.
> 
> And be aware that TARGET may be /boot, but could be something else like /tftp-root. Then the symlink would point from /mnt/here/tftp-root/efi/Guix/grub.cfg to /mnt/here/boot/grub/grub.cfg, as the later is kind of hard-coded.

Please add that to comments in the source code.  Otherwise, it would be
very probable to be broken by further maintenance.