[bug#66156,00/12] Introducing Smart Hurdloading

This patch series, my friends, sets up Smart Hurdloading™, sometimes
referred to as “Hurd auto-offloading”.  (Yeah I worked hard on branding…)

Concretely, if you add (service hurd-vm-service-type), you can now run:

  guix build libreoffice -s i586-gnu

on your machine, and the childhurd will take care of keeping one CPU core
busy for a while.  There’s no configuration involved beyond this one
(service hurd-vm-service-type) line: no need to exchange SSH keys,
substitute keys, write /etc/guix/machines.scm, and all that.  It’s all
taken care off automatically.

One key element here is support for declarative and extensible offloading
configuration—meaning that /etc/guix/machines.scm is now optionally generated
straight from your ‘operating-system’ config.  The rest is about generating
keys, authorizing them, etc.

The series is a bit of a potpourri: I fixed Hurd issues here and there
(currently offloading to a childhurd doesn’t work on ‘master’, because
of the locale issue, for instance), improved documentation, etc.  I also
took a couple of long detours not shown here that might lead to further
improvements in the future.

My goal is to extend this mechanism beyond the Hurd, to have a generic
mechanism to spin up Guix System VMs we can easily offload to.

Thoughts?

Ludo’.

Ludovic Courtès (12):
  system: vm: Remove unused variable.
  secret-service: Increase default handshake timeout.
  services: hurd-vm: Use the default SSH port number.
  gnu: glibc-utf8-locales: Reintroduce input labels.
  services: guix: Use the right locale package on GNU/Hurd.
  services: guix: Support declarative offloading setup.
  services: childhurd: Authorize the childhurd’s key on the host.
  services: hurd-vm: ‘image’ field has to be an <image> record.
  tests: hurd-vm: Remove custom disk image configuration.
  services: hurd-vm: Disable password-based authentication for root.
  doc: Give an example showing how to add an account in the childhurd.
  services: hurd-vm: Implement zero-configuration offloading.

 doc/guix.texi                   | 167 ++++++++++++++++++++++++++------
 gnu/build/secret-service.scm    |   2 +-
 gnu/packages/base.scm           |   4 +-
 gnu/services/base.scm           |  58 ++++++++++-
 gnu/services/virtualization.scm | 153 ++++++++++++++++++++++++++---
 gnu/system/vm.scm               |  41 ++------
 gnu/tests/virtualization.scm    |  68 ++++++++++---
 7 files changed, 394 insertions(+), 99 deletions(-)


base-commit: 3d8d67ef6928f5d81118c97f03372cd341eab8b0
  

Ludovic Courtès writes:

Hi Ludo,

> This patch series, my friends, sets up Smart Hurdloading™, sometimes
> referred to as “Hurd auto-offloading”.  (Yeah I worked hard on branding…)
>
> Concretely, if you add (service hurd-vm-service-type), you can now run:
>
>   guix build libreoffice -s i586-gnu

Beautiful!  I'm wondering how this works wrt disk size; the default is
still 'guess?  In my childhurds I have

   (hurd-vm-configuration
    ..
    (disk-size (* 16 1024 (expt 2 20))) ;16GiB

which is pretty cheap now that we have qcow2.  I'm not sure how
expensive it is to set memory-size if you don't use it?

> The series is a bit of a potpourri: I fixed Hurd issues here and there
> (currently offloading to a childhurd doesn’t work on ‘master’, because
> of the locale issue, for instance), improved documentation, etc.  I also
> took a couple of long detours not shown here that might lead to further
> improvements in the future.

That's great, very nice.  And quite understandable afaic, the hurd-team
branch has also been quite a mixed set of work.

> My goal is to extend this mechanism beyond the Hurd, to have a generic
> mechanism to spin up Guix System VMs we can easily offload to.
>
> Thoughts?

I'm wondering if there is a way to/if we need a way to set the default
priority of the childhurd.  I'm usually offloading to another machine's
childhurd and will probably want to prioritize that when I'm at home.

(Making a copy of /etc/guix/machines.scm and editing that as required
would probably mork for me.)

I've added this patch set to the hurd-team branch and am testing it
now.  The only thing I found without testing:

> Subject: [bug#66156] [PATCH 06/12] services: guix: Support declarative offloading setup.
> To: 66156@debbugs.gnu.org
> Cc: Ludovic Courtès <ludo@gnu.org>
> Date: Fri, 22 Sep 2023 14:54:07 +0200 (44 minutes, 21 seconds ago)
> Resent-From: Ludovic Courtès <ludo@gnu.org>
> 
> * gnu/services/base.scm (guix-machines-files-installation): New
> procedure.
> (<guix-configuration>)[build-machines]: New field.
> (guix-activation): Call ‘ guix-machines-files-installation’.
                           ^
Remove space.

Thanks for this amazing piece of work and bug-fixing!

Greetings,
Janneke
  

Hello!

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

> Beautiful!  I'm wondering how this works wrt disk size; the default is
> still 'guess?  In my childhurds I have
>
>    (hurd-vm-configuration
>     ..
>     (disk-size (* 16 1024 (expt 2 20))) ;16GiB
>
> which is pretty cheap now that we have qcow2.  I'm not sure how
> expensive it is to set memory-size if you don't use it?

I guess setting ‘disk-size’ should still work (and as you write: with
QCOW2, it doesn’t cost much to ask for extra space, until you actually
use it); nothing has changed here.

> I'm wondering if there is a way to/if we need a way to set the default
> priority of the childhurd.  I'm usually offloading to another machine's
> childhurd and will probably want to prioritize that when I'm at home.

Good question.  Perhaps you could declare that other machine with a
higher ‘speed’ value?

Thanks for your feedback!

Ludo’.
  

Ludovic Courtès writes:

Hello!

Okay, after overcoming completely unreleated troubles I finally managed
to reconfigure a laptop to hurd-team to test it.  It works great!

> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Beautiful!  I'm wondering how this works wrt disk size; the default is
>> still 'guess?  In my childhurds I have
>>
>>    (hurd-vm-configuration
>>     ..
>>     (disk-size (* 16 1024 (expt 2 20))) ;16GiB
>>
>> which is pretty cheap now that we have qcow2.  I'm not sure how
>> expensive it is to set memory-size if you don't use it?
>
> I guess setting ‘disk-size’ should still work (and as you write: with
> QCOW2, it doesn’t cost much to ask for extra space, until you actually
> use it); nothing has changed here.

Yes, works beautifully.

>> I'm wondering if there is a way to/if we need a way to set the default
>> priority of the childhurd.  I'm usually offloading to another machine's
>> childhurd and will probably want to prioritize that when I'm at home.
>
> Good question.  Perhaps you could declare that other machine with a
> higher ‘speed’ value?

That also works nicely.

I found that by default, only root can offload.  Is that intentional,
how would that work?  I changed permissions on
`/etc/guix/offload/ssh/childhurd' like so

--8<---------------cut here---------------start------------->8---
$ l /etc/guix/offload/ssh
total 8
-rw-r--r-- 1 root root   93 Sep 23 14:47 childhurd.pub
-rw-r----- 1 root wheel 399 Sep 23 14:47 childhurd
--8<---------------cut here---------------end--------------->8---

to "fix" that. WDYT?

Found another two commit message nitpicks

> Subject: Re: [bug#66156] [PATCH 12/12] services: hurd-vm: Implement zero-configuration offloading.
[..]

> > This allows for zero-configuration offloading to a childhurd.
>
> * gnu/services/virtualization.scm (operating-system-with-offloading-account):
> New procedure.
> (<hurd-vm-configuration>)[offloading?]: New field.
> (hurd-vm-disk-image): Define ‘transform’ and use it.
> (hurd-vm-activation): Generate SSH key for user ‘offloading’ and add
> authorize it via /etc/childhurd/etc/ssh/authorized_keys.d.

Remove "add".

> (hurd-vm-configuration-offloading-ssh-key)
> (hurd-vm-guix-extension): New procedures.

Rather use 

(hurd-vm-configuration-offloading-ssh-key,
hurd-vm-guix-extension): New procedures.

?

Greetings,
Janneke
  

Hello,

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

> Okay, after overcoming completely unreleated troubles I finally managed
> to reconfigure a laptop to hurd-team to test it.  It works great!

Yay, thanks for testing!

> I found that by default, only root can offload.  Is that intentional,
> how would that work?

It’s is intentional: ‘guix offload’ is only ever invoked by
guix-daemon¹, as root.

¹ Except for ‘guix offload test’, which is meant to be spawned by users,
  as root too.

> I changed permissions on `/etc/guix/offload/ssh/childhurd' like so
>
> $ l /etc/guix/offload/ssh
> total 8
> -rw-r--r-- 1 root root   93 Sep 23 14:47 childhurd.pub
> -rw-r----- 1 root wheel 399 Sep 23 14:47 childhurd
>
> to "fix" that. WDYT?

What would be the use case?

> Found another two commit message nitpicks

Noted!

I’d like to see if there are other opinions in particular about
/etc/guix/machines.scm generation, then I guess we can merge?

Thanks,
Ludo’.
  

Ludovic Courtès writes:

Hi!

> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Okay, after overcoming completely unreleated troubles I finally managed
>> to reconfigure a laptop to hurd-team to test it.  It works great!
>
> Yay, thanks for testing!

Happy to :)

>> I found that by default, only root can offload.  Is that intentional,
>> how would that work?
>
> It’s is intentional: ‘guix offload’ is only ever invoked by
> guix-daemon¹, as root.

...ah.  I missed...

> ¹ Except for ‘guix offload test’, which is meant to be spawned by users,
>   as root too.

...this bit.

>> I changed permissions on `/etc/guix/offload/ssh/childhurd' like so
>>
>> $ l /etc/guix/offload/ssh
>> total 8
>> -rw-r--r-- 1 root root   93 Sep 23 14:47 childhurd.pub
>> -rw-r----- 1 root wheel 399 Sep 23 14:47 childhurd
>>
>> to "fix" that. WDYT?
>
> What would be the use case?

Well, running `guix offload test' as "janneke" :)

I didn't "dare" (should I say "proceed"?) to test a real offload before
seeing "guix offload test" work.  That's how I usually go about
offloading.

Reading the manual, especially if you know it, I guess that it does say
so in a subtle way

--8<---------------cut here---------------start------------->8---
   To test whether your setup is operational, run this command on the
master node:

     # guix offload test
--8<---------------cut here---------------end--------------->8---

And sure enough, offloading does work when I revert permessions on the
private key, so yeah...

>> Found another two commit message nitpicks
>
> Noted!
>
> I’d like to see if there are other opinions in particular about
> /etc/guix/machines.scm generation, then I guess we can merge?

Sure!

Greetings,
Janneke
  

Hello,

I pushed these as b9fae146d6cc4a6968a8eb18beef29aa1414a31e.

I forgot to amend the commit logs as you had suggested; apologies!

I also forgot to submit a news entry with the initial patch set, but
then thought it’d be nice to have, so I pushed the following as
b9fae146d6cc4a6968a8eb18beef29aa1414a31e (with French translation):

   (entry (commit "953c65ffdd43c02c934518fb7a1c68542584b223")
          (title
           (en "Declarative offloading on Guix System and childhurds"))
          (body
           (en "When configuring Guix System, it is now possible to declare
  builds machines to offload to directly from the @code{operating-system}
  declaration by specifying the @code{build-machines} field of
  @code{guix-configuration}.  When you do this, @command{guix system} generates
  a @file{/etc/guix/machines.scm} file by concatenating the @code{(build-machine
  @dots{})} expressions you specified.

  This mechanism is used by @code{hurd-vm-service-type}, also known as
  ``childhurd'', to create virtual machines running the GNU/Hurd operating
  system one can offload to, without additional configuration steps.

  Run @command{info \"(guix) Daemon Offload Setup\"} for more info on
  offloading; run @command{info \"(guix) Virtualization Services\"} for info on
  @code{hurd-vm-service-type}.")))

Thanks,
Ludo’.