diff mbox series

[bug#78210] guix: include store parent dirs in docker layer

Message ID c2da04d9-1509-4eec-8248-ffe484953154@1729.org.uk
State New
Headers
Series [bug#78210] guix: include store parent dirs in docker layer |

Commit Message

Ray Miller May 2, 2025, 10:43 a.m. UTC 
This patch adds the /gnu and /gnu/store directories to the Docker layer
created by `guix pack -f docker ...` which enables the Docker image to be
used to create an AWS Lambda function. Without the patch, creating the AWS
Lambda function fails with this error:

"MissingParentDirectory: Parent directory does not exist for file: 
gnu/store/zic27jikg36d6wjj4cz8hyriyfl3ygiz-info-dir/"

My first attempt to fix this was just to add the /gnu and /gnu/store 
directories
to `directives` but the Docker image failed in AWS Lambda with the same 
error. These
directories need to appear in the tarball for the layer *before* the 
packages,
so the change to the order of the tar arguments is also needed.

* guix/scripts/pack.scm: add /gnu and /gnu/store directories to the
docker layer.
* guix/docker.scm: change order of arguments to tar so parent
directories are added before their contents.

Change-Id: I2b103c59981e828c965564ccc5d2415b00a7e52e
---
guix/docker.scm       | 4 ++--
guix/scripts/pack.scm | 6 ++++--
2 files changed, 6 insertions(+), 4 deletions(-)

entry-point-argument)

base-commit: 4fe4cf9fdd959126d3c53c3df4504d851e7b736a
-- 
2.47.1

Comments

Ludovic Courtès May 18, 2025, 8:34 p.m. UTC | #1 
Hi Ray,

Ray Miller <ray@1729.org.uk> writes:

> This patch adds the /gnu and /gnu/store directories to the Docker layer
> created by `guix pack -f docker ...` which enables the Docker image to be
> used to create an AWS Lambda function. Without the patch, creating the AWS
> Lambda function fails with this error:
>
> "MissingParentDirectory: Parent directory does not exist for file: 
> gnu/store/zic27jikg36d6wjj4cz8hyriyfl3ygiz-info-dir/"
>
> My first attempt to fix this was just to add the /gnu and /gnu/store 
> directories
> to `directives` but the Docker image failed in AWS Lambda with the same 
> error. These
> directories need to appear in the tarball for the layer *before* the 
> packages,
> so the change to the order of the tar arguments is also needed.
>
> * guix/scripts/pack.scm: add /gnu and /gnu/store directories to the
> docker layer.
> * guix/docker.scm: change order of arguments to tar so parent
> directories are added before their contents.
>
> Change-Id: I2b103c59981e828c965564ccc5d2415b00a7e52e

Neat!

Could you include in the commit log a line like:

  Fixes <https://issues.guix.gnu.org/XYZ>.

… so we can keep track of where this was reported and discussed?

> +++ b/guix/scripts/pack.scm
> @@ -580,9 +580,11 @@ (define* (docker-image name profile
>                       (,source -> ,target))))))
>
>              (define directives
> -              ;; Create a /tmp directory, as some programs expect it, and
> -              ;; create SYMLINKS.
> +              ;; Create /tmp, /gnu, and /gnu/store directories, as some
> +              ;; programs expect them, and create SYMLINKS.
>                `((directory "/tmp" ,(getuid) ,(getgid) #o1777)
> +                (directory "/gnu" ,(getuid) ,(getgid) #o755)
> +                (directory "/gnu/store" ,(getuid) ,(getgid) #o755)

It’s a bit trickier, because “/gnu/store” is not hardcoded.

Instead, you need to recurse over the components of (%store-prefix), so
something like:

  `((directory "/tmp" …)
    #$@(map (lambda (component)
              #~(directory #$component …))
            (string-tokenize (%store-prefix)
                             (char-set-complement (char-set #\/)))))

But perhaps this should actually be done in (guix docker) so that ‘guix
system image -t docker’ also benefits from it?

Thanks,
Ludo’.
Ray Miller May 24, 2025, 4:53 p.m. UTC | #2 
Hi Ludo,

Thank you for the feedback and suggestions.

On 18/05/2025 21:34, Ludovic Courtès wrote:
> Neat!
>
> Could you include in the commit log a line like:
>
>    Fixes <https://issues.guix.gnu.org/XYZ>.
>
> … so we can keep track of where this was reported and discussed?
>
Done.

>> +++ b/guix/scripts/pack.scm
>> @@ -580,9 +580,11 @@ (define* (docker-image name profile
>>                        (,source -> ,target))))))
>>
>>               (define directives
>> -              ;; Create a /tmp directory, as some programs expect it, and
>> -              ;; create SYMLINKS.
>> +              ;; Create /tmp, /gnu, and /gnu/store directories, as some
>> +              ;; programs expect them, and create SYMLINKS.
>>                 `((directory "/tmp" ,(getuid) ,(getgid) #o1777)
>> +                (directory "/gnu" ,(getuid) ,(getgid) #o755)
>> +                (directory "/gnu/store" ,(getuid) ,(getgid) #o755)
> It’s a bit trickier, because “/gnu/store” is not hardcoded.
Ah, OK. Good catch!
> Instead, you need to recurse over the components of (%store-prefix), so
> something like:
>
>    `((directory "/tmp" …)
>      #$@(map (lambda (component)
>                #~(directory #$component …))
>              (string-tokenize (%store-prefix)
>                               (char-set-complement (char-set #\/)))))

It turns out to be a bit simpler because these directives are processed by
evaluate-populate-directive which calls mkdir-p. (I discovered this after
implementing a recursive solution then thinking "there must be a better
way" and reading some source code...)

> But perhaps this should actually be done in (guix docker) so that ‘guix
> system image -t docker’ also benefits from it?
I tested this by running 'guix systemimage -t docker' and inspecting the
layer it generates. It turns out not to suffer from the same problem: the
store directory is present in the layer.

Updated patch to follow.

Ray.
diff mbox series

Patch

diff --git a/guix/docker.scm b/guix/docker.scm
index 60ce13cbde..9911bb84bb 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -365,10 +365,10 @@  (define* (build-docker-image image paths prefix
                 (apply invoke "tar" "-cf" "../layer.tar"
                        `(,@transformation-options
                          ,@(tar-base-options)
-                         ,@(if max-layers '() paths)
                          ,@(scandir "."
                                     (lambda (file)
-                                      (not (member file '("." ".."))))))))
+                                      (not (member file '("." "..")))))
+                         ,@(if max-layers '() paths))))
               (delete-file-recursively "extra")))

         ;; It is possible for "/" to show up in the archive, especially 
when
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 7ab2c0d447..5cb9cd0b48 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -580,9 +580,11 @@  (define* (docker-image name profile
                      (,source -> ,target))))))

             (define directives
-              ;; Create a /tmp directory, as some programs expect it, and
-              ;; create SYMLINKS.
+              ;; Create /tmp, /gnu, and /gnu/store directories, as some
+              ;; programs expect them, and create SYMLINKS.
               `((directory "/tmp" ,(getuid) ,(getgid) #o1777)
+                (directory "/gnu" ,(getuid) ,(getgid) #o755)
+                (directory "/gnu/store" ,(getuid) ,(getgid) #o755)
                 ,@(append-map symlink->directives '#$symlinks)))

             (define (form-entry-point prefix entry-point