[bug#62153,0/2] Add Docker layered image for pack and system

Hi Guix,

This patch series add to 'guix pack' and 'guix system image' formats with a
layered Docker image, which dicreases images size by sharing same layers on a
host.

The folling commands show an example for new image formats:

./pre-inst-env guix system image --image-type=docker-layered config.scm
docker load -i result

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello
docker load -i result

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

The gnu/packages/aux-files/python/stream-layered-image.py Python script is a
copy of github.com/NixOS/nixpkgs/pkgs/build-support/docker/stream_layered_image.py
with only a simple replacement "/nix" to "/gnu" string.

Oleg Pykhalov (2):
  guix: docker: Build layered image.
  news: Add entry for the new 'docker-layered' distribution format.

 Makefile.am                                   |   3 +-
 doc/guix.texi                                 |  16 +-
 etc/news.scm                                  |  38 ++
 gnu/image.scm                                 |   3 +-
 .../aux-files/python/stream-layered-image.py  | 391 ++++++++++++++++++
 gnu/system/image.scm                          |  84 +++-
 gnu/tests/docker.scm                          |  20 +-
 guix/docker.scm                               | 182 ++++++--
 guix/scripts/pack.scm                         | 103 +++--
 guix/scripts/system.scm                       |  11 +-
 tests/pack.scm                                |  48 +++
 11 files changed, 813 insertions(+), 86 deletions(-)
 create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 60a211ec705ac98483d76da7f2523f2b8966343a
  

The cover lever guix pack example should be:

    ./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

instead of

    ./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello

Apologies,
Oleg.
  

Hi, Guix.

These patches series is rebased on origin/master. Also, the Python script is
replaced in favour of calls to GNU Tar and GNU Gzip programs.  Passed tests:
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system"
make check-system TESTS="docker-layered-system"
  

Oleg Pykhalov (2):
  guix: docker: Build layered image.
  news: Add entry for the new 'docker-layered' distribution format.

 doc/guix.texi           |  18 +++-
 etc/news.scm            |  58 ++++++++++++
 gnu/image.scm           |   3 +-
 gnu/system/image.scm    |  76 +++++++++++----
 gnu/tests/docker.scm    |  20 +++-
 guix/docker.scm         | 205 +++++++++++++++++++++++++++++++---------
 guix/scripts/pack.scm   |  62 ++++++++++--
 guix/scripts/system.scm |  11 ++-
 tests/pack.scm          |  48 ++++++++++
 9 files changed, 424 insertions(+), 77 deletions(-)


base-commit: 77f52db416a13e195d090cad4e9e7658feb2e86b
  

On Wed, May 31, 2023 at 4:46 AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
>
> Hi, Guix.
>
> These patches series is rebased on origin/master. Also, the Python script is
> replaced in favour of calls to GNU Tar and GNU Gzip programs.  Passed tests:
> make check TESTS="tests/pack.scm"
> make check-system TESTS="docker-system"
> make check-system TESTS="docker-layered-system"
>
>
> Oleg Pykhalov (2):
>   guix: docker: Build layered image.
>   news: Add entry for the new 'docker-layered' distribution format.

Why not use layered images for all docker packs?
  

Greg Hogan <code@greghogan.com> writes:

> On Wed, May 31, 2023 at 4:46 AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
>>
>> Hi, Guix.
>>
>> These patches series is rebased on origin/master. Also, the Python script is
>> replaced in favour of calls to GNU Tar and GNU Gzip programs.  Passed tests:
>> make check TESTS="tests/pack.scm"
>> make check-system TESTS="docker-system"
>> make check-system TESTS="docker-layered-system"
>>
>>
>> Oleg Pykhalov (2):
>>   guix: docker: Build layered image.
>>   news: Add entry for the new 'docker-layered' distribution format.
>
> Why not use layered images for all docker packs?

Do you mean use layered images by default without ability to build all
in a single layer?  Current layered implementation is slow to build
because it needs to calculate a size of each layer, pack, and compress.
So if user wants a faster build, a non-layered image is still an option.

Regards,
Oleg.
  

On Wed, May 31, 2023 at 9:14 AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
[...]
> Do you mean use layered images by default without ability to build all
> in a single layer?  Current layered implementation is slow to build
> because it needs to calculate a size of each layer, pack, and compress.
> So if user wants a faster build, a non-layered image is still an option.
>
> Regards,
> Oleg.

I am trying out your patch, and wanted to benchmark the runtime
difference between docker and docker-layered packs, but the latter
looks to be failing with any compression other than the default gzip.
In particular, I was looking to disable compression with
'--compression=none'.
  

Greg Hogan <code@greghogan.com> writes:

> On Wed, May 31, 2023 at 9:14 AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
> [...]
>> Do you mean use layered images by default without ability to build all
>> in a single layer?  Current layered implementation is slow to build
>> because it needs to calculate a size of each layer, pack, and compress.
>> So if user wants a faster build, a non-layered image is still an option.
>>
>> Regards,
>> Oleg.
>
> I am trying out your patch, and wanted to benchmark the runtime
> difference between docker and docker-layered packs, but the latter
> looks to be failing with any compression other than the default gzip.
> In particular, I was looking to disable compression with
> '--compression=none'.

I'll send a fixed v4 revision for '--compression=none'. Unfortunately,
because we cannot append to an existing compressed tarball:

    tar: Cannot update compressed archives
    Try 'tar --help' or 'tar --usage' for more information.

adding more compression types requires to write a handler for every
compressor separately in guix/docker.scm file:
--8<---------------cut here---------------start------------->8---
      (if layered-image?
          (begin
            (invoke "tar" "-rf" "image.tar" "config.json")
            (if compressor
                (begin
                  (apply invoke `(,@compressor "image.tar"))
                  (copy-file "image.tar.gz" image))
                (copy-file "image.tar" image)))
--8<---------------cut here---------------end--------------->8---

I would like to vote that addional compressors could be added later if
needed.

Regards,
Oleg.
  

Hi Guix,

I would like to merge 62153.  After 64173 will be merge, merging 62153
is not possible without conflict resolving with Git.

64173 introduces ‘%docker-format-options’ variable.  With this variable
it's possible in 62153 to replace ‘--image-type=docker-layered’ with
‘--docker-layers=N’ option, where:

    if ‘N’ is zero, then use current non layered format
    if ‘N’ is bigger than zero, then use layered format

Number of layers specification is nice to have, because Docker layers
are limited.  So if user would like to modify a Docker image by adding
more layers on top, then hacks like squashing layers are not required.
Also, it will be possible to delete code which builds non layered Docker
image without deprecating command line options.

Is it possible to partially merge 64173, specifically
‘%docker-format-options’ variable and it requirements, so it can be used
in 62153 for ‘--docker-layers=N’ option?

[1]: https://issues.guix.gnu.org/issue/62153
[2]: https://issues.guix.gnu.org/64173


Regards,
Oleg.