[bug#41785] DRAFT services: Add 'hurd-in-vm service-type'.

TODO: Figure-out how to run this hurd VM inside a VM.

Using

--8<---------------cut here---------------start------------->8---

Hello Jan,

> TODO: Figure-out how to run this hurd VM inside a VM.

Just to clarify things here, we now have the "disk-image" command that
produces a Hurd bootable disk-image. The "vm-image" command is almost
working, and should produce a very similar result.

The "vm" command that produces a vm-image and spawns a VM isn't working
yet with the Hurd. This command is for now passing directly "kernel" and
"initrd" arguments to QEMU, so a few adaptations will be needed.

Now the goad of hurd-in-vm-service-type is to provide a service that
does almost the same thing as the "vm" command, but as a service, right?

So, I don't get why would we need to run a Hurd VM inside a VM. I've
been struggling a lot with running nested layers of virtualization (for
system generation before the recent patches), and the result is often
too slow to be really usable.

Thanks,

Mathieu

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

> Just to clarify things here, we now have the "disk-image" command that
> produces a Hurd bootable disk-image. The "vm-image" command is almost
> working, and should produce a very similar result.
>
> The "vm" command that produces a vm-image and spawns a VM isn't working
> yet with the Hurd. This command is for now passing directly "kernel" and
> "initrd" arguments to QEMU, so a few adaptations will be needed.
>
> Now the goad of hurd-in-vm-service-type is to provide a service that
> does almost the same thing as the "vm" command, but as a service, right?

That’s my understanding.

> So, I don't get why would we need to run a Hurd VM inside a VM. I've
> been struggling a lot with running nested layers of virtualization (for
> system generation before the recent patches), and the result is often
> too slow to be really usable.

I guess it’s useful if you want to test a GNU/Linux system that uses the
VM service, as in:

  guix system vm my-gnu+linux.scm

where ‘my-gnu+linux.scm’ contains:

   (services (cons (service hurd-vm-service-type)
                   …))

I suppose it should just work, though probably without ‘-enable-kvm’;
janneke?

Thanks,
Ludo’.

Hey!

That was fast!  :-)

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

> and doing something like
>
>     ./pre-inst-env guix system vm gnu/system/examples/bare-bones.tmpl --no-offload
>     /gnu/store/96wh3jwsla4p6d4s547mmqxsi4qbbc0r-run-vm.sh -m 2G \
>       --device rtl8139,netdev=net0 \
>       --netdev user,id=net0,hostfwd=tcp:127.0.0.1:10022-:2222,hostfwd=tcp:127.0.0.1:5900-:5900
>
> nicely starts a bare-bones VM with the the hurd-in-vm service inside, but I
> cannot seem to connect to the Hurd VM it in any way.  Appending
> ",hostfwd=tcp:127.0.0.1:20022-:20022" (to directly ssh into the Hurd) even
> blocks me from ssh'ing into the GNU/linux host VM.

Weird.

> hurd-in-vm works beautifully when added to my system configuration and
> reconfiguring.
>
> * gnu/services/virtualization.scm (disk-image, hurd-in-vm-shepherd-service,
> hurd-vm-disk-image): New procedures.
> (%hurd-in-vm-operating-system, hurd-in-vm-service-type): New variable.
> (<hurd-in-vm-configuration>): New record type.
> * doc/guix.texi (Virtualization Services): Document it.

[…]

> +@subsubheading The Hurd in a Virtual Machine
> +
> +@cindex @code{hurd}
> +@cindex the Hurd
> +
> +Service @code{hurd-in-vm} provides support for running a Virtual Machine
> +with the GNU@tie{}Hurd.

“… support for running GNU/Hurd in a virtual machine (VM).  The virtual
machine is a Shepherd service that can be controlled with commands such
as:

@example
herd stop hurd-vm
@end example

The given GNU/Hurd operating system configuration is cross-compiled.”

Nitpick: I’d call it “hurd-vm”, because it runs a Hurd VM.  :-)

It’s a volatile VM, due to the use of ‘-snapshot’, right?

(The Hurd actually has “sub-Hurds”¹ and “neighborhurds”².  I wonder if
it’s our duty to coin another term… a guesthurd? a visithurd?)

¹ https://www.gnu.org/software/hurd/hurd/subhurd.html
² https://www.gnu.org/software/hurd/hurd/neighborhurd.html

> +(define* (disk-image os #:key (image-size 'guess) target)
> +  "Return a disk-image for OS with size IMAGE-SIZE, built for TARGET."
> +  (with-store store
      ^
In general, procedures should talk to the user-provided store and never
open a new connection.  They should also never call ‘build-derivations’
explicitly, the only exception so far being the graft implementation.

So you can drop ‘with-store’ here, and then:

> +    (run-with-store store
> +      (let ((file-system-type "ext2"))
> +        (mlet* %store-monad
> +            ((base-image (find-image file-system-type))
> +             (sys        (lower-object
> +                          (system-image
> +                           (image
> +                            (inherit base-image)
> +                            (size image-size)
> +                            (operating-system os)))))
> +             (drvs       (mapm/accumulate-builds lower-object (list sys)))
> +             (%          (built-derivations drvs)))
> +          (let ((output (derivation->output-path sys)))
> +            (return output))))

Mathieu, can we make ‘find-image’ non-monadic?  It really shouldn’t be
because it doesn’t interact with the store.  It can take an optional
‘system’ parameter if we want.

So, assuming ‘find-image’ is non-monadic, the code above becomes
something like:

  (system-image
    (image (inherit base-image)
           (size image-size)
           (operating-system
            (with-parameters ((%current-target-system "i586-pc-gnu"))
              os))))

> +(define %hurd-in-vm-operating-system
> +  (operating-system
> +    (inherit %hurd-default-operating-system)
> +    (host-name "guixydevel")
> +    (timezone "Europe/Amsterdam")
> +    (bootloader (bootloader-configuration
> +                 (bootloader grub-minimal-bootloader)
> +                 (target "/dev/vda")
> +                 (timeout 0)))
> +    (services (cons*
> +               (service openssh-service-type
> +                        (openssh-configuration
> +                         (openssh openssh-sans-x)
> +                         (use-pam? #f)
> +                         (port-number 2222)
> +                         (permit-root-login #t)
> +                         (allow-empty-passwords? #t)
> +                         (password-authentication? #t)))
> +               %base-services/hurd))))

I understand the need to factorize useful configs, but IMO it doesn’t
belong here.  So I’d just leave it out.  There’s already
‘%hurd-default-operating-system’ that does the heavy lifting anyway.

> +(define hurd-in-vm-service-type
> +  (service-type
> +   (name 'hurd-in-vm)
> +   (extensions (list (service-extension shepherd-root-service-type
> +                                        hurd-in-vm-shepherd-service)))
> +   (default-value (hurd-in-vm-configuration))
> +   (description
> +    "Provide a Virtual Machine running the GNU Hurd.")))

Being pedantic: s|the GNU Hurd|GNU/Hurd|.  :-)

Otherwise looks great to me, thank you!

Ludo’.

Ludovic Courtès writes:

Hello,

> Mathieu Othacehe <othacehe@gnu.org> skribis:
>
>> Just to clarify things here, we now have the "disk-image" command that
>> produces a Hurd bootable disk-image. The "vm-image" command is almost
>> working, and should produce a very similar result.

(yes)

>> The "vm" command that produces a vm-image and spawns a VM isn't working
>> yet with the Hurd. This command is for now passing directly "kernel" and
>> "initrd" arguments to QEMU, so a few adaptations will be needed.
>>
>> Now the goad of hurd-in-vm-service-type is to provide a service that
>> does almost the same thing as the "vm" command, but as a service, right?
>
> That’s my understanding.

Indeed, mine too.

>> So, I don't get why would we need to run a Hurd VM inside a VM. I've
>> been struggling a lot with running nested layers of virtualization (for
>> system generation before the recent patches), and the result is often
>> too slow to be really usable.
>
> I guess it’s useful if you want to test a GNU/Linux system that uses the
> VM service, as in:
>
>   guix system vm my-gnu+linux.scm
>
> where ‘my-gnu+linux.scm’ contains:
>
>    (services (cons (service hurd-vm-service-type)
>                    …))
>
> I suppose it should just work, though probably without ‘-enable-kvm’;
> janneke?

Well -- yes.  That was my thinking.

We would not normally use such a VM-in-VM thing; when you want a Hurd VM
using the "vm" command would be great (or as now, create a disk image
and start QEMU).

However, for writing and debugging (testing?) this Hurd-in-VM service, I
thought adding it to a linux bare-bones.tmpl and using guix vm on that
was a nice and "trivial" way to test it.  That proved so troublesome
that I added it to my system.

As I understand it the intended use for this service is to have a real
simple way to add a Hurd VM to some build machines, so that we can
easily create a herd of Hurd-VMs to aid the Hurd work.  Ludo?

Greetings,
Janneke

Mathieu Othacehe <othacehe@gnu.org> writes:

> So, I don't get why would we need to run a Hurd VM inside a VM. I've
> been struggling a lot with running nested layers of virtualization (for
> system generation before the recent patches), and the result is often
> too slow to be really usable.

Note that recent processors support nested layers of virtualization
natively with little overhead, but it's disabled by default.

For an Intel processor, it can be enabled by adding this to your system
configuration:

  (kernel-arguments (cons "kvm_intel.nested=1" %default-kernel-arguments))

The corresponding AMD kernel module is called "kvm_amd".

Marius Bakke writes:

Hello,

> Mathieu Othacehe <othacehe@gnu.org> writes:
>
>> So, I don't get why would we need to run a Hurd VM inside a VM. I've
>> been struggling a lot with running nested layers of virtualization (for
>> system generation before the recent patches), and the result is often
>> too slow to be really usable.
>
> Note that recent processors support nested layers of virtualization
> natively with little overhead, but it's disabled by default.

Ah!

> For an Intel processor, it can be enabled by adding this to your system
> configuration:
>
>   (kernel-arguments (cons "kvm_intel.nested=1" %default-kernel-arguments))

Is there an obvious downside to enabling this?

Great...So on the host I did

--8<---------------cut here---------------start------------->8---
root@dundal ~# rmmod kvm_intel
root@dundal ~# modprobe kvm_intel kvm_intel.nested=1
root@dundal ~# cat /sys/module/kvm_intel/parameters/nested
Y
--8<---------------cut here---------------end--------------->8---

and the interwebs told me that to start the VM, you have to add "-cpu
host"; so I started it using

--8<---------------cut here---------------start------------->8---
/gnu/store/k2b7nx34cwyi6yk49wgy4hg9mrwcmll5-run-vm.sh -cpu host -m 2G -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp:127.0.0.1:10022-:2222,hostfwd=tcp:127.0.0.1:25900-:25900
--8<---------------cut here---------------end--------------->8---

and trying to "ssh -p 20022 localhost" from inside the bare-bones VM now
prints

--8<---------------cut here---------------start------------->8---
qemu-system-i386: Slirp: Failed to send package, ret: -1
qemu-system-i386: Slirp: Failed to send package, ret: -1
qemu-system-i386: Slirp: Failed to send package, ret: -1
qemu-system-i386: Slirp: Failed to send package, ret: -1
qemu-system-i386: Slirp: Failed to send package, ret: -1
qemu-system-i386: Slirp: Failed to send package, ret: -1
key_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 20022
--8<---------------cut here---------------end--------------->8---

...something networky with QEMU.  Ideas?

Janneke

Hey,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

[...]

> and the interwebs told me that to start the VM, you have to add "-cpu
> host"; so I started it using
>
> /gnu/store/k2b7nx34cwyi6yk49wgy4hg9mrwcmll5-run-vm.sh -cpu host -m 2G -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp:127.0.0.1:10022-:2222,hostfwd=tcp:127.0.0.1:25900-:25900
>
>
> and trying to "ssh -p 20022 localhost" from inside the bare-bones VM now
> prints
>
> qemu-system-i386: Slirp: Failed to send package, ret: -1
> qemu-system-i386: Slirp: Failed to send package, ret: -1
> qemu-system-i386: Slirp: Failed to send package, ret: -1
> qemu-system-i386: Slirp: Failed to send package, ret: -1
> qemu-system-i386: Slirp: Failed to send package, ret: -1
> qemu-system-i386: Slirp: Failed to send package, ret: -1
> key_exchange_identification: read: Connection reset by peer
> Connection reset by 127.0.0.1 port 20022
>
> ...something networky with QEMU.  Ideas?

I've recently had intermittent issues with 'guix system vm' and hostfwd.
Does it help if you remove the '-nic user,model=virtio-net-pci' option
from (a copy of) the run-vm.sh script so that there's only one netdev?

Regards,

Diego

Hello Ludo & janneke,

> Mathieu, can we make ‘find-image’ non-monadic?  It really shouldn’t be
> because it doesn’t interact with the store.  It can take an optional
> ‘system’ parameter if we want.

Yes, you're right, passing 'target' to 'find-image' should be enough to
make it non-monadic.

> So, assuming ‘find-image’ is non-monadic, the code above becomes
> something like:
>
>   (system-image
>     (image (inherit base-image)
>            (size image-size)
>            (operating-system
>             (with-parameters ((%current-target-system "i586-pc-gnu"))
>               os))))

I would prefer 'target' to be part of the image itself, as I proposed
here: https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00417.html.

There's no way for now, that the image is built without cross-compiling
for "i586-pc-gnu", so I think it could be part of the "image" record
itself.

WDYT?

>> +(define hurd-in-vm-service-type
>> +  (service-type
>> +   (name 'hurd-in-vm)
>> +   (extensions (list (service-extension shepherd-root-service-type
>> +                                        hurd-in-vm-shepherd-service)))
>> +   (default-value (hurd-in-vm-configuration))
>> +   (description
>> +    "Provide a Virtual Machine running the GNU Hurd.")))
>
> Being pedantic: s|the GNU Hurd|GNU/Hurd|.  :-)
>
> Otherwise looks great to me, thank you!

Looks really nice to me too :)

Thanks,

Mathieu

Hi Mathieu!

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> So, assuming ‘find-image’ is non-monadic, the code above becomes
>> something like:
>>
>>   (system-image
>>     (image (inherit base-image)
>>            (size image-size)
>>            (operating-system
>>             (with-parameters ((%current-target-system "i586-pc-gnu"))
>>               os))))
>
> I would prefer 'target' to be part of the image itself, as I proposed
> here: https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00417.html.
>
> There's no way for now, that the image is built without cross-compiling
> for "i586-pc-gnu", so I think it could be part of the "image" record
> itself.
>
> WDYT?

Yes, why not, a ‘target’ field in <image> sounds fine.

In general, I think lowerable objects should not be parameterized by
target/system.  However, <image> is an exception because it needs to
access the operating system record, and still use the desired target, if
any.

Thanks,
Ludo’.

Diego Nicola Barbato writes:

Hi,

> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>
> [...]
>
>> and the interwebs told me that to start the VM, you have to add "-cpu
>> host"; so I started it using
>>
>> /gnu/store/k2b7nx34cwyi6yk49wgy4hg9mrwcmll5-run-vm.sh -cpu host -m
>> 2G -device rtl8139,netdev=net0 -netdev
>> user,id=net0,hostfwd=tcp:127.0.0.1:10022-:2222,hostfwd=tcp:127.0.0.1:25900-:25900
>>
>>
>> and trying to "ssh -p 20022 localhost" from inside the bare-bones VM now
>> prints
>>
>> qemu-system-i386: Slirp: Failed to send package, ret: -1
>> qemu-system-i386: Slirp: Failed to send package, ret: -1
>> qemu-system-i386: Slirp: Failed to send package, ret: -1
>> qemu-system-i386: Slirp: Failed to send package, ret: -1
>> qemu-system-i386: Slirp: Failed to send package, ret: -1
>> qemu-system-i386: Slirp: Failed to send package, ret: -1
>> key_exchange_identification: read: Connection reset by peer
>> Connection reset by 127.0.0.1 port 20022
>>
>> ...something networky with QEMU.  Ideas?
>
> I've recently had intermittent issues with 'guix system vm' and hostfwd.
> Does it help if you remove the '-nic user,model=virtio-net-pci' option
> from (a copy of) the run-vm.sh script so that there's only one netdev?

That's helpful!  I now noticed that the QEMU command above I used was
the one I have been using to start the Hurd, networking wise.

After running

--8<---------------cut here---------------start------------->8---
./run-vm.sh -cpu host -m 2G -net nic -net user,id=net0,hostfwd=tcp:127.0.0.1:10022-:2222,hostfwd=tcp:127.0.0.1:5900-:25900,hostfwd=tcp:127.0.0.1:20022-:20022
--8<---------------cut here---------------end--------------->8---

I noticed that instead of two network interfaces, there is now just one,
as expected; much better.

No luck ssh'ing into the second VM, directly or indirectly.  I'm not
exploring this further as we don't have an actually need for this
vm-in-vm thing.

Greetings,
Janneke