mbox series

[bug#41350,0/3] Use native qemu to build vm-image.

Message ID 87mu66q3rt.fsf@gnu.org
Headers show
Series Use native qemu to build vm-image. | expand

Message

Jan Nieuwenhuizen May 17, 2020, 10:01 a.m. UTC
Hi!

Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
That does not work for the Hurd, as there is no qemu-HURD.

This patch switches to cross building vm-images using a native qemu vm.  If
there a reason for using qemu-TARGET we may want to make this switch
conditional for cross building to the Hurd?

There is a tricky aspect to this: the "user-builder" includes all the
cross-built packages for TARGET, but we need to hard the native qemu a
native #+LOADER, loading native #+LINUX and #+INITRD.

To accomplish this NATIVE/CROSS mixt, the builder is then run with a native
guile, like so

--8<---------------cut here---------------start------------->8---
(system* #+(file-append (default-guile) "/bin/guile")
         "--no-auto-compile"
         #$(preserve-target user-builder))
--8<---------------cut here---------------end--------------->8---

while the packages in USER-BUILDER will still be cross-built.  However,
because the builder has

    (with-extensions gcrypt-sqlite3&co
         (use-modules (gnu store database))
      ...)

this makes the native guile "see" the cross-built .GO files.  That would
"only" be a secondary problem except that loading (sqlite3) throws an
exception when dynamic-loading the .SO fails.  Seeing it like this I am not so
sure anymore this is a bug, WDYT?  ...so instead

...the problem is now avoided by removing the sqlite dependency when
cross-building by not registering closures and postponing the loading of (gnu
store database) and thus (sqlite3).

I have reset wip-hurd-vm onto these changes, so you can also look there.
Doing

    ./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts \
        gnu/system/examples/bare-hurd.tmpl

now produces a pretty nice hurd VM :-)

Greetings,
Janneke

Jan (janneke) Nieuwenhuizen (3):
  utils: Move 'reset-timestamps' out of database.
  system: vm: Do not register-closures when cross-building.
  system: vm: Build vm-image using native qemu.

 gnu/bootloader/grub.scm |  4 +--
 gnu/build/vm.scm        |  3 ++-
 gnu/system/vm.scm       | 54 ++++++++++++++++++++++++++---------------
 guix/store/database.scm | 41 +++----------------------------
 guix/utils.scm          | 41 ++++++++++++++++++++++++++++---
 5 files changed, 79 insertions(+), 64 deletions(-)

Comments

Mathieu Othacehe May 18, 2020, 9:10 a.m. UTC | #1
Hello Jan,

> Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
> That does not work for the Hurd, as there is no qemu-HURD.

Yes that's an issue.

>
> This patch switches to cross building vm-images using a native qemu vm.  If
> there a reason for using qemu-TARGET we may want to make this switch
> conditional for cross building to the Hurd?

Well first this 'vm-image' thing should be done in (gnu system image) as
soon as I have sorted out the bootloader issue. So the solution we will
find to overcome this Hurd issue will be temporary I hope.

About using qemu-TARGET, let say you are on an armhf machine and you
want to cross-build an x86-64 image. I fear that using a native, armhf
Grub to install an x86-64 Grub won't work.

So maybe, it would be better to do that only for the architectures
without an available qemu-TARGET (only the Hurd for now)?

> ...the problem is now avoided by removing the sqlite dependency when
> cross-building by not registering closures and postponing the loading of (gnu
> store database) and thus (sqlite3).

When I produce a system without register closures, once booted, "guix
build" something does not work. I don't know if its possible for
guix-daemon to work without its database. An option could be to generate
this database if its absent, at first boot?

>     ./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts \
>         gnu/system/examples/bare-hurd.tmpl
>
> now produces a pretty nice hurd VM :-)

Great to see you so close :)

Thanks,

Mathieu
Jan Nieuwenhuizen May 19, 2020, 7:22 a.m. UTC | #2
Mathieu Othacehe writes:

Hello Mathieu,

Thanks for chiming in, I meant to CC you...  :-)

>> Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
>> That does not work for the Hurd, as there is no qemu-HURD.
>
> Yes that's an issue.

Yes...luckily I think we found a way to use qemu-NATIVE to build
packages for the Hurd.  Now to see if this can be put into digestible
form.

>> This patch switches to cross building vm-images using a native qemu vm.  If
>> there a reason for using qemu-TARGET we may want to make this switch
>> conditional for cross building to the Hurd?
>
> Well first this 'vm-image' thing should be done in (gnu system image) as
> soon as I have sorted out the bootloader issue. So the solution we will
> find to overcome this Hurd issue will be temporary I hope.

Can you elaborate a bit on that?  Do you think it makes sense to continue
this temporary path some more (as it starts to work right now), or can we
better abandon it and work the permanent solution?  How could I contribute?

> About using qemu-TARGET, let say you are on an armhf machine and you
> want to cross-build an x86-64 image. I fear that using a native, armhf
> Grub to install an x86-64 Grub won't work.

Ah...right, that's helpful information.

> So maybe, it would be better to do that only for the architectures
> without an available qemu-TARGET (only the Hurd for now)?

OK -- I have changed patch 2 and 3, and am sending a new patch set.

>> ...the problem is now avoided by removing the sqlite dependency when
>> cross-building by not registering closures and postponing the loading of (gnu
>> store database) and thus (sqlite3).
>
> When I produce a system without register closures, once booted, "guix
> build" something does not work. I don't know if its possible for
> guix-daemon to work without its database. An option could be to generate
> this database if its absent, at first boot?

Ah, OK.  I'm using this "solution" to use #:register-closures? #f now
for the Hurd only and we'll hit the problem that causes later.  We can
then, s discussed on IRC, sqlite3 databases are indeed platform
independent and we could use something like guix/scripts/pack.scm's
store-database.  WDYT?

>>     ./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts \
>>         gnu/system/examples/bare-hurd.tmpl
>>
>> now produces a pretty nice hurd VM :-)
>
> Great to see you so close :)

Yes...

Greetings,
janneke
Mathieu Othacehe May 19, 2020, 10:02 a.m. UTC | #3
Hello Jan,

>> Well first this 'vm-image' thing should be done in (gnu system image) as
>> soon as I have sorted out the bootloader issue. So the solution we will
>> find to overcome this Hurd issue will be temporary I hope.
>
> Can you elaborate a bit on that?  Do you think it makes sense to continue
> this temporary path some more (as it starts to work right now), or can we
> better abandon it and work the permanent solution?  How could I contribute?

Yes. Besides offering a more modular image creation API, the vocation of
(gnu system image) is to offer a way to generate all kind of Guix system
images (raw disk-images, ISO9660 images, Qemu images) without resorting
to VM for image creation.

Dropping VM, means that the image need to be build on the host, without
root permissions. This brings several limitations. If is no longer
possible to use "mount" for instance, or to call "grub-install".

To get around these limitations, I used the same strategy as Buildroot,
Yocto and OpenWrt that do not require root permissions to generate
disk-images.

For ISO9660 images:

* I first build the "image-root" derivation. It's a store directory that
contains the root file-system.

* Then, I call make-iso9660-image that, run GNU Xorriso on this
directory.

For raw disk-images:

* For each partition, I build the "partition-image-root"
  derivation. This is very similar to "image-root" but for the specified
  partition.

* For each partition root directory, I create the corresponding
  partition image, using tools such as mke2fs or mkdosfs depending on
  the partition file-system type.
  
* Then, I need to "assemble" the partitions in a disk-image. For that, I
  use "genimage" that will roughly use 'dd' to create a final disk-image
  with the partitions copied at the right offsets.
  
* The missing part here is the bootloader installation. As I mentioned,
  grub-install refuses to take a disk-image as argument (it requires a
  mounted partition). For EFI systems there's a work-around. The idea is
  to call grub-mkstandalone to create a Grub binary in the ESP
  partition. This Grub is configured to load the Grub configuration file
  located on the root file-system at /boot/grub/grub.cfg path.

Now back to the Hurd. I see that Debian is producing Hurd ISO images. We
could try to call `guix system disk-image --target=i586-pc-gnu
--file-system-type=iso9660 hurd.scm` and see if it works.

Regarding raw disk-image, I think we could try to produce EFI compatible
Hurd images. We could set the bootloader to grub-efi-bootloader in
%hurd-default-operating-system.

Finally, to produce raw disk-images with grub-minimal-bootloader or
grub-bootloader (what you are trying to do), we need to find a way to
make grub-install work on disk-images (MBR installation and so on). That
true for the Hurd but that's also true for Linux.

Sorry for the long explanation. Please tell me if something is not
clear.

The conclusion here is that, I think that we very few adaptations to
your branch, we should be able to produce Hurd ISO images or Hurd EFI
compatible disk-images. Maybe it would be a first step.

Then, we could find a way to create "MBR compatible" Hurd and Linux
disk-images in (gnu system image).

WDYT?

Thanks,

Mathieu
Mathieu Othacehe May 20, 2020, 2:03 p.m. UTC | #4
Hey,

> The conclusion here is that, I think that we very few adaptations to
> your branch, we should be able to produce Hurd ISO images or Hurd EFI
> compatible disk-images. Maybe it would be a first step.
>
> Then, we could find a way to create "MBR compatible" Hurd and Linux
> disk-images in (gnu system image).

Ok, I made further progress. Turns out the Hurd EFI solution was a
dead-end, because I have a "no console will be available to os" message
in Qemu, instead of Hurd console output. I don't feel like debugging
this.

So, back to the MBR solution. I had another look to what OpenWrt is
doing. They found a really nice work-around! As I stated, we cannot use
"grub-install", but this command is in reality a wrapper around
"grub-mkimage" and "grub-bios-setup".

--8<---------------cut here---------------start------------->8---
#######################################
#     #            #                  #
# MBR # MBR-GAP    # FIRST PARTITION  #
#     #            #                  #
#######################################
--8<---------------cut here---------------end--------------->8---

grub-mkimage generates a Grub image, small enough so that it can fit in
the MBR-GAP (the space between the MBR and the first partition).

It cannot contain all Grub modules, but that's not an issue, because
Grub will be able to find missing modules from the first partition when
started.

They also patched grub-bios-setup[1] so that it can work on a raw
disk-image, and install the previously generated "grub.img".

So with the following commands:

--8<---------------cut here---------------start------------->8---
grub-mkimage -O i386-pc -o core.img  biosdisk part_msdos ext2
echo "(hd0) /tmp/my-disk-image" > device.map
grub-bios-setup -m device.map -r "hd0,msdos1" -d tmp /tmp/qemu-image2
--8<---------------cut here---------------end--------------->8---

I'm able to make a Guix system image bootable, without root
permissions. It then starts the Hurd kernel and crashes, but that's
another story :p

So, if it's ok for you, I can integrate this stuff cleanly, and we won't
need the vm-image stuff anymore.

WDYT?

Thanks,

Mathieu

[1]:
https://github.com/openwrt/openwrt/blob/master/package/boot/grub2/patches/100-grub_setup_root.patch
Mathieu Othacehe May 22, 2020, 7:24 p.m. UTC | #5
Hey Ludo!

>   1. When cross-compiling, can the ‘qemu-image’ procedure to its job by
>      running exclusively native software (in particular using a native
>      QEMU, native kernel, etc.)?

I think the answer is yes, but I raised a concern about being able to
run grub-install from an ARM system to build a cross-compiled x86-64
system (for instance).

For now, running this command shows:

--8<---------------cut here---------------start------------->8---
ls $(guix build --system=aarch64-linux grub)/lib/grub/
arm64-efi
--8<---------------cut here---------------end--------------->8---

So, the native aarch64-linux is only able to install itself on the same
system. I think can be fixed though (same as for grub-hybrid package).

> As for (2), I’d say that when cross-compiling, it should just run native
> software but simply preserve references to cross-compiled software,
> which is what janneke’s patch does.

Yes, I agree.

However, I think I found a way to install Grub, without root
permissions, from the host system (see:
https://lists.gnu.org/archive/html/guix-patches/2020-05/msg00988.html).

This should allow to deprecate the whole (gnu system vm) module.

Thanks for having a look :)

Mathieu
Mathieu Othacehe May 24, 2020, 6:11 p.m. UTC | #6
> Ah, qemu-img convert -- nice.  So, we get different type of images.
> Wonder if that could play some role?

Hope not!

> The most interesting differences I see are wrt Grub, e.g.
>
> -/mnt/boot/grub/fonts
> -/mnt/boot/grub/fonts/unicode.pf2
>
> could that still play a role?  I'm having a look why this could be missing.

It's because I just copy "lib/grub" folder in (gnu bootloader grub). I
should also do that for the fonts I guess. But I doubt the issue comes
from here.

> --- vm-image.lst-s	2020-05-24 15:41:02.051314009 +0200
> +++ disk-image.lst-s	2020-05-24 15:41:08.407414141 +0200
> @@ -3,293 +3,577 @@
>  /mnt/boot
>  /mnt/boot/activation
>  /mnt/boot/grub
> -/mnt/boot/grub/fonts

At some point, if we have the same files inside, it should work :p I
noticed that doing a cfdisk on the raw converted, vm-image root
partition reports:

Attributes: 80

which doesn't appear on the disk-image.

However, tune2fs does not show any noticeable difference (both ext_attr
and Hurd OS type are set).

Mathieu
Jan Nieuwenhuizen May 24, 2020, 6:40 p.m. UTC | #7
Mathieu Othacehe writes:

>> Ah, qemu-img convert -- nice.  So, we get different type of images.
>> Wonder if that could play some role?
>
> Hope not!

Okay...let's continue searching elsewhere then.

>> The most interesting differences I see are wrt Grub, e.g.
>>
>> -/mnt/boot/grub/fonts
>> -/mnt/boot/grub/fonts/unicode.pf2
>>
>> could that still play a role?  I'm having a look why this could be missing.
>
> It's because I just copy "lib/grub" folder in (gnu bootloader grub). I
> should also do that for the fonts I guess. But I doubt the issue comes
> from here.

Meanwhile I tried adding that by hand; indeed that's not it.

>> --- vm-image.lst-s	2020-05-24 15:41:02.051314009 +0200
>> +++ disk-image.lst-s	2020-05-24 15:41:08.407414141 +0200
>> @@ -3,293 +3,577 @@
>>  /mnt/boot
>>  /mnt/boot/activation
>>  /mnt/boot/grub
>> -/mnt/boot/grub/fonts
>
> At some point, if we have the same files inside, it should work :p I
> noticed that doing a cfdisk on the raw converted, vm-image root
> partition reports:
>
> Attributes: 80
>
> which doesn't appear on the disk-image.

I am not aware that we would be using attributes, at least I reverted
the xattr trick on wip-hurd-vm for the /servers.  Can we find out where
and what they are?

> However, tune2fs does not show any noticeable difference (both ext_attr
> and Hurd OS type are set).

Indeed, I looked at that too.  This may be so obvious once we find it...

Janneke