[bug#66160] gnu: Add oci-container-service-type.

* gnu/services/docker.scm (oci-container-configuration): New variable;
(oci-container-shepherd-service): new variable;
(oci-container-service-type): new variable.
* doc/guix.texi: Document it.
---
 doc/guix.texi           | 108 ++++++++++++++++++++++++++
 gnu/services/docker.scm | 167 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 274 insertions(+), 1 deletion(-)


base-commit: f45c0c82289d409b4fac00464ea8b323839ba53f

Hi Giacomo,

Giacomo Leidi <goodoldpaul@autistici.org> skribis:

> * gnu/services/docker.scm (oci-container-configuration): New variable;
> (oci-container-shepherd-service): new variable;
> (oci-container-service-type): new variable.
> * doc/guix.texi: Document it.

We’re almost there!  There’s a couple of things I overlooked before (my
apologies), so here we go:

> +@table @asis
> +@item @code{command} (default: @code{()}) (type: list-of-strings)
> +Overwrite the default command (@code{CMD}) of the image.
> +
> +@item @code{entrypoint} (default: @code{""}) (type: string)
> +Overwrite the default entrypoint (@code{ENTRYPOINT}) of the image.

Apparently this doesn’t match the docstring that’s in
‘define-configuration’.

Could you make sure the docstring is the canonical source?  Then you can
use ‘generate-documentation’ to generate the bit that you’ll paste in
guix.texi (info "(guix) Complex Configurations").

> +  (entrypoint
> +   (string "")
> +   "Overwrite the default ENTRYPOINT of the image.")
> +  (environment
> +   (list '())
> +   "Set environment variables."
> +   (sanitizer oci-sanitize-environment))
> +  (image
> +   (string)
> +   "The image used to build the container.")
> +  (name
> +   (string "")
> +   "Set a name for the spawned container.")

Please use ‘maybe-string’ in cases where it’s either the Docker default
(default ENTRYPOINT, default CMD, etc.) or some user-provided value.
I find it clearer or at least more conventional than using the empty
string to denote default values.

> +(define oci-container-configuration->options
> +  (lambda (config)
> +    (let ((entrypoint
> +           (oci-container-configuration-entrypoint config))
> +          (network
> +           (oci-container-configuration-network config)))
> +      (apply append
> +             (filter (compose not unspecified?)
> +                     `(,(when (not (string-null? entrypoint))
> +                          (list "--entrypoint" entrypoint))
> +                       ,(append-map
> +                         (lambda (spec)
> +                           (list "--env" spec))
> +                         (oci-container-configuration-environment config))
> +                       ,(when (not (string-null? network))
> +                          (list "--network" network))

This would thus become:

   `(,@(if entrypoint
           `("--entrypoint" ,entrypoint)
           '())
     …)

> +                       #~(make-forkexec-constructor
> +                          ;; docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
> +                          (list #$docker-command
> +                                "run"
> +                                "--rm"
> +                                "--name" #$name
> +                                #$@(oci-container-configuration->options config)
> +                                #$(oci-container-configuration-image config)
> +                                #$@(oci-container-configuration-command config))
> +                          #:user "root"
> +                          #:group "root"))

Does ‘docker run’ necessarily need to run as root, or are there cases
where one might want to run it as non-root?  (I expect the latter.)

> +(define oci-container-service-type
> +  (service-type (name 'oci-container)
> +                (extensions (list (service-extension profile-service-type
> +                                                     (lambda _ (list docker-cli)))
> +                                  (service-extension shepherd-root-service-type
> +                                                     configs->shepherd-services)))
> +                (default-value '())

I wonder if it should take a list of configs and be extensible, or
simply take a single config.  Users would write:

  (service oci-container-service-type
           (oci-container-configuration …))

WDYT?

Last thing: there’s no system test (something we normally require), but
since I forgot about it before and I’m already asking for more than I
should :-) I propose to leave it for later.


Thanks!

Ludo’.

Hi Ludo’ ,


> We’re almost there!  There’s a couple of things I overlooked before (my
> apologies), so here we go:
Thank you for your help and the time you spent reviewing this!
>> +@table @asis
>> +@item @code{command} (default: @code{()}) (type: list-of-strings)
>> +Overwrite the default command (@code{CMD}) of the image.
>> +
>> +@item @code{entrypoint} (default: @code{""}) (type: string)
>> +Overwrite the default entrypoint (@code{ENTRYPOINT}) of the image.
> Apparently this doesn’t match the docstring that’s in
> ‘define-configuration’.
>
> Could you make sure the docstring is the canonical source?  Then you can
> use ‘generate-documentation’ to generate the bit that you’ll paste in
> guix.texi (info "(guix) Complex Configurations").
I should have aligned the code and documentation.
>
>> +  (entrypoint
>> +   (string "")
>> +   "Overwrite the default ENTRYPOINT of the image.")
>> +  (environment
>> +   (list '())
>> +   "Set environment variables."
>> +   (sanitizer oci-sanitize-environment))
>> +  (image
>> +   (string)
>> +   "The image used to build the container.")
>> +  (name
>> +   (string "")
>> +   "Set a name for the spawned container.")
> Please use ‘maybe-string’ in cases where it’s either the Docker default
> (default ENTRYPOINT, default CMD, etc.) or some user-provided value.
> I find it clearer or at least more conventional than using the empty
> string to denote default values.
I don't know why I didn't do it in the first place, thank you. fixed
>> +                       #~(make-forkexec-constructor
>> +                          ;; docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
>> +                          (list #$docker-command
>> +                                "run"
>> +                                "--rm"
>> +                                "--name" #$name
>> +                                #$@(oci-container-configuration->options config)
>> +                                #$(oci-container-configuration-image config)
>> +                                #$@(oci-container-configuration-command config))
>> +                          #:user "root"
>> +                          #:group "root"))
> Does ‘docker run’ necessarily need to run as root, or are there cases
> where one might want to run it as non-root?  (I expect the latter.)

yes you are right, it's only required to be in the docker group or in 
general have enough permission to operate on the docker daemon socket. I 
added a new service extension setting up an oci-container user, that 
it's just in the docker group and can not login, that runs oci backed 
services. it is also overridable by the user


>
>> +(define oci-container-service-type
>> +  (service-type (name 'oci-container)
>> +                (extensions (list (service-extension profile-service-type
>> +                                                     (lambda _ (list docker-cli)))
>> +                                  (service-extension shepherd-root-service-type
>> +                                                     configs->shepherd-services)))
>> +                (default-value '())
> I wonder if it should take a list of configs and be extensible, or
> simply take a single config.  Users would write:
>
>    (service oci-container-service-type
>             (oci-container-configuration …))
>
> WDYT?

I get that it's not super consistent with other services but for example 
Nextcloud in some case require 3 containers: nextcloud itself, redis and 
a cron container. in this case i suppose one would define an 
oci-nextcloud-service-type which maybe extends an 
oci-redis-service-type. in this case the oci-nextcloud-service-type 
would define the 2 shepherd services for nextcloud and the cron process 
and the oci-redis-service-type would define one redis service.

Right now we can already do:

(service oci-container-service-type
            (list
              (oci-container-configuration …)))

and i updated the documentation accordingly. do you have any suggestion 
for changing the api of oci-container-configuration to support having 
different shepherd oci backed services behind a guix system service? 
This way we could remove the (list) call.

>
> Last thing: there’s no system test (something we normally require), but
> since I forgot about it before and I’m already asking for more than I
> should :-) I propose to leave it for later.

I actually looked around but didn't find them, but it was a long time 
ago and I certainly didn't look very well. I'm definitely up for testing 
this (maybe it's possible to use swineherd? could we use it for 
automating integration tests?), could you point me to something similar 
i can look to as an example?


Thank you for your time and effort, i'm sending an updated patch


giacomo

Hello,

paul <goodoldpaul@autistici.org> skribis:


[...]

>> Does ‘docker run’ necessarily need to run as root, or are there cases
>> where one might want to run it as non-root?  (I expect the latter.)
>
> yes you are right, it's only required to be in the docker group or in
> general have enough permission to operate on the docker daemon
> socket. I added a new service extension setting up an oci-container
> user, that it's just in the docker group and can not login, that runs
> oci backed services. it is also overridable by the user

In that case, maybe create an “oci-service” account part of the “docker”
group, and run ‘docker run’ as that user instead of running it as root?
Would that be OK or am I overlooking something?

>>> +(define oci-container-service-type
>>> +  (service-type (name 'oci-container)
>>> +                (extensions (list (service-extension profile-service-type
>>> +                                                     (lambda _ (list docker-cli)))
>>> +                                  (service-extension shepherd-root-service-type
>>> +                                                     configs->shepherd-services)))
>>> +                (default-value '())
>> I wonder if it should take a list of configs and be extensible, or
>> simply take a single config.  Users would write:
>>
>>    (service oci-container-service-type
>>             (oci-container-configuration …))
>>
>> WDYT?

[...]

> Right now we can already do:
>
> (service oci-container-service-type
>            (list
>              (oci-container-configuration …)))
>
> and i updated the documentation accordingly. do you have any
> suggestion for changing the api of oci-container-configuration to
> support having different shepherd oci backed services behind a guix
> system service? This way we could remove the (list) call.

What I’m suggesting above is that one would build a list of
‘oci-container-service-type’ instances, like:

  (list (service oci-container-service-type
                 (oci-container-configuration …))
        (service oci-container-service-type
                 (oci-container-configuration …))
        …)

Each instance above would correspond to exactly one program in a Docker
image.

I feel it’s slightly more natural than having a service type that
implements support for multiple OCI services at once.

> I actually looked around but didn't find them, but it was a long time
> ago and I certainly didn't look very well. I'm definitely up for
> testing this (maybe it's possible to use swineherd? could we use it
> for automating integration tests?), could you point me to something
> similar i can look to as an example?

Check out under gnu/tests/*.scm, in particular (gnu tests docker).

HTH!

Ludo’.

Hello Ludo’ ,

On 10/19/23 22:13, Ludovic Courtès wrote:
> Hello,
>
> paul<goodoldpaul@autistici.org>  skribis:
>
>
> [...]
>
>>> Does ‘docker run’ necessarily need to run as root, or are there cases
>>> where one might want to run it as non-root?  (I expect the latter.)
>> yes you are right, it's only required to be in the docker group or in
>> general have enough permission to operate on the docker daemon
>> socket. I added a new service extension setting up an oci-container
>> user, that it's just in the docker group and can not login, that runs
>> oci backed services. it is also overridable by the user
> In that case, maybe create an “oci-service” account part of the “docker”
> group, and run ‘docker run’ as that user instead of running it as root?
> Would that be OK or am I overlooking something?
I already added such user in the latest version of my patch. I probably 
made a mess with patch subjects.
> What I’m suggesting above is that one would build a list of
> ‘oci-container-service-type’ instances, like:
>
>    (list (service oci-container-service-type
>                   (oci-container-configuration …))
>          (service oci-container-service-type
>                   (oci-container-configuration …))
>          …)
>
> Each instance above would correspond to exactly one program in a Docker
> image.
>
> I feel it’s slightly more natural than having a service type that
> implements support for multiple OCI services at once.
I agree it's more natural but (list service-a service-b ...) it's the 
same interface exposed by the shepherd-root-service-type, I believe for 
the same reasons I need the oci-nextcloud-service-type to instantiate 3 
shepherd services but only create a single account, activate a single 
data dir under /var/lib, something like this:

(defineoci-nextcloud-service-type
(service-type(name'nextcloud)
(extensions(list(service-extensionoci-container-service-type
(lambda (config) (make-nextcloud-container config) 
(make-nextcloud-cron-container config)))
(service-extensionaccount-service-type
(const%nextcloud-accounts))
(service-extensionactivation-service-type
%nextcloud-activation)))
(default-value(nextcloud-configuration))
(description
"This service provides the Nextcloud service as an OCI-backed container.")))

The only way where oci-container-service-type could support this use 
case by accepting a single configuration is I guess if multiple 
(service-extension oci-container-service-type ...) where allowed, am I 
understanding correctly? Is it legal in Guix to write somthing like:

(extensions(list(service-extensionoci-container-service-type
make-nextcloud-container) 
(service-extensionoci-container-service-typemake-nextcloud-cron-container) 
(service-extensionaccount-service-type
(const%nextcloud-accounts))
(service-extensionactivation-service-type
%nextcloud-activation)))

> Check out under gnu/tests/*.scm, in particular (gnu tests docker).

Thank you for the pointer, I'll look into those.

giacomo

Hi,

paul <goodoldpaul@autistici.org> skribis:


[...]

>> In that case, maybe create an “oci-service” account part of the “docker”
>> group, and run ‘docker run’ as that user instead of running it as root?
>> Would that be OK or am I overlooking something?
> I already added such user in the latest version of my patch. I
> probably made a mess with patch subjects.

Oh, my bad; perfect then.

>> What I’m suggesting above is that one would build a list of
>> ‘oci-container-service-type’ instances, like:
>>
>>    (list (service oci-container-service-type
>>                   (oci-container-configuration …))
>>          (service oci-container-service-type
>>                   (oci-container-configuration …))
>>          …)
>>
>> Each instance above would correspond to exactly one program in a Docker
>> image.
>>
>> I feel it’s slightly more natural than having a service type that
>> implements support for multiple OCI services at once.
> I agree it's more natural but (list service-a service-b ...) it's the
> same interface exposed by the shepherd-root-service-type, I believe
> for the same reasons I need the oci-nextcloud-service-type to
> instantiate 3 shepherd services but only create a single account,
> activate a single data dir under /var/lib, something like this:
>
> (defineoci-nextcloud-service-type
> (service-type(name'nextcloud)
> (extensions(list(service-extensionoci-container-service-type
> (lambda (config) (make-nextcloud-container config)
> (make-nextcloud-cron-container config)))

[...]

> The only way where oci-container-service-type could support this use
> case by accepting a single configuration is I guess if multiple
> (service-extension oci-container-service-type ...) where allowed, am I
> understanding correctly? Is it legal in Guix to write somthing like:
>
> (extensions(list(service-extensionoci-container-service-type
> make-nextcloud-container)
> (service-extensionoci-container-service-typemake-nextcloud-cron-container)
> (service-extensionaccount-service-type
> (const%nextcloud-accounts))
> (service-extensionactivation-service-type
> %nextcloud-activation)))

If you take the route of one ‘oci-container-service-type’ per
daemon/server that you want to run, then <oci-container-configuration>
should probably have a ‘user’ field to specify under which user to run
the container.  ‘oci-container-service-type’ would create exactly one
Shepherd service so, likewise, <oci-container-configuration> would need
a ‘provision’ field to specify the Shepherd service name (the
“provisions”).  Likewise, perhaps a field to specify the data directory
is needed.

Does that make sense?

Thanks,
Ludo’.

Hi Ludo’ ,

thank you for your explanation.

On 10/24/23 17:41, Ludovic Courtès wrote:
> If you take the route of one ‘oci-container-service-type’ per
> daemon/server that you want to run,
what you mention below is already implemented in my latest patch [0]:
> then <oci-container-configuration>
> should probably have a ‘user’ field to specify under which user to run
> the container.
this is oci-container-configuration-user
> <oci-container-configuration> would need
> a ‘provision’ field to specify the Shepherd service name (the
> “provisions”).
this is oci-container-configuration-name. I now realize that "name" it's 
not the best field name, so I'm sending a patch with this renamed to 
oci-container-configuration-provision .
>   Likewise, perhaps a field to specify the data directory
> is needed.
I don't think oci-container-configuration should concern about a data 
directory since oci containers themselves only have a volume concept 
which is covered. what you brought into my mind is that docker supports 
-w/--workdir so I implemented it and added it to the patch I'm about to 
send.
> Does that make sense?

Yes, thank you :) My doubts come from shepherd-root-service-type 
accepting a list of services. What would be the reason to break 
consistency with it? I think we would add the friction of having to write


(service nextcloud-cron-oci-service-type)

(service nextcloud-oci-service-type)


instead of simply


(service nextcloud-cron-oci-service-type)


One way out of this if you think is a good solution could be having an 
oci-containers-service-type that's supposed to be only extended whose 
value would be a list of <oci-container-configuration> and an 
oci-single-service-type that could not be extended whose value would be 
a single <oci-container-configuration> . The oci-single-service-type 
would simply extend the oci-containers-service-type and maintenance 
would be free.

What do you think?


Thank you for your help,

giacomo


[0]: https://issues.guix.gnu.org/66160#10-lineno69