From patchwork Tue Sep 27 17:16:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "vasilii.smirnov--- via Guix-patches\" via" X-Patchwork-Id: 43009 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 04EDF27BBEB; Tue, 27 Sep 2022 20:19:36 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 46A1727BBEA for ; Tue, 27 Sep 2022 20:19:34 +0100 (BST) Received: from localhost ([::1]:46992 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1odG7B-0004lt-8U for patchwork@mira.cbaines.net; Tue, 27 Sep 2022 15:19:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47106) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1odG6g-0004iy-CW for guix-patches@gnu.org; Tue, 27 Sep 2022 15:19:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:57756) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1odG6g-0002G7-4m for guix-patches@gnu.org; Tue, 27 Sep 2022 15:19:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1odG6f-0004NW-VY for guix-patches@gnu.org; Tue, 27 Sep 2022 15:19:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#58123] [PATCH] gnu: services: docker: Add docker-container-service-type Resent-From: =?utf-8?b?TcOhamEgVG9tw6HFoWVr?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 27 Sep 2022 19:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 58123 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 58123@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.166430630616782 (code B ref -1); Tue, 27 Sep 2022 19:19:01 +0000 Received: (at submit) by debbugs.gnu.org; 27 Sep 2022 19:18:26 +0000 Received: from localhost ([127.0.0.1]:56834 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1odG65-0004Mb-S3 for submit@debbugs.gnu.org; Tue, 27 Sep 2022 15:18:26 -0400 Received: from lists.gnu.org ([209.51.188.17]:51306) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1odEn2-0002DH-Az for submit@debbugs.gnu.org; Tue, 27 Sep 2022 13:54:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39160) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1odEn2-0000yu-57 for guix-patches@gnu.org; Tue, 27 Sep 2022 13:54:40 -0400 Received: from knopi.disroot.org ([178.21.23.139]:47034) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1odEmz-0005Nc-62 for guix-patches@gnu.org; Tue, 27 Sep 2022 13:54:39 -0400 Received: from localhost (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 831584C38F for ; Tue, 27 Sep 2022 19:54:10 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from knopi.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VjWtnDYMMhev for ; Tue, 27 Sep 2022 19:54:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1664301248; bh=KsnGyHiu2wE5h/5N+3HziBpCVGhsZHHOvBe14pPFf9M=; h=From:To:Subject:Date; b=EQ7mUb6J7Pt/mnWpKG4gRIecfhcGIfiS5C0Ct5i86pR6TWD0Mcrwa395O9CYULepw Qn7qcVL3ZwsaNwrhVHg1C8J/ow4zEzRXRfP/k7jk8EsGpLDfVBMjSuYD3IEVzS4T10 aWjvwN5xXDTLTKC0lTHENvA08BzUIqr1NTvnPM121Hycf5in4kgTilMx8y/7VtDHhC a+w+XBQuRS1SJVFCCu2VmWZ7Fk2051UwT/eDyjwEQ2y4DxxClkDZpy5IjryOg0Y6d3 h0791xOGxMn6n4YhJTXTpeUfAFsKrOuPCjplkdZ3PIc6Fzn+JbXswKLvS5KH3xdHcM irz7ntTyZKJ/g== Date: Tue, 27 Sep 2022 19:16:32 +0200 Message-ID: <87r0zwr9dv.fsf@disroot.org> Mime-Version: 1.0 Received-SPF: pass client-ip=178.21.23.139; envelope-from=maya.tomasek@disroot.org; helo=knopi.disroot.org X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Tue, 27 Sep 2022 15:18:23 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" Reply-to: =?utf-8?b?TcOhamEgVG9tw6HFoWVr?= X-ACL-Warn: , =?utf-8?q?M=C3=A1ja_Tom=C3=A1=C5=A1ek_via_Guix-patches?= X-Patchwork-Original-From: guix-patches--- via From: "vasilii.smirnov--- via Guix-patches\" via" X-getmail-retrieved-from-mailbox: Patches This patch provides a new service type, which allows the creation of shepherd services from docker containers. --- Hi, I have written a definition of the docker-container-service-type. It is a service that allows you to convert docker containers into shepherd services. It has some limitations: 1. Containers are deleted when stopping. This makes it easier to manage them from shepherd. You can achieve persistence with volume binds. 2. Containers must be either attached or detached. Some containers simply run a command inside the container, and when run with docker run, they stay attached and the docker run command therefore behaves like a normal command. However, some containers use docker's "init system", that means that they won't block the docker run command. Sadly attached containers don't properly report that they are initialized, so to docker they are in the "starting" state until termination. This means that docker doesn't create a cid file (pid file for containers). To sum it up, there is no way to tell how will the container report it's state, and this process must be specified by the user, if the container runs attached or detached. 3. Images are not managed. Docker does manage images in a really stupid way. My original idea was to have a file-like object to define an image (from an image archive), but sadly it has been more complex than I initially thought. Docker uses 2 versions of archive manifests, and each archive can have multiple images, depending on the architecture. And those images can be composed from layers, which are also images. The daemon determines ad-hoc which images from the archive it will use, and there is no official tool (at leats when I looked for it) to get image ids reliably. As the docker load command can return multiple images. I will expand on the process on how the images could be used in a managed way, but I have to say something to the current interface. It works by expecting an image by name from the image-name field. That means that docker must already have the image in its database. There is currently no way to ensure that images will be in docker database before running shepherd services, but I expect they should simply fail to start. There is currently no documentation outside of docstrings, I plan to write it, but first I would welcome comments from you, maybe this service isn't suitable for guix, as it does imply breaking of the declarative guix model, but that goes for docker and flatpak too, so I thought I can try it. Now finally to images. The idea is that all images belonging to a docker-container-configuration are tagged (via docker tag) with the name o the docker-container-configuration-name (as this is unique). The activation service would get the propper image-id (which is not easy to get from the manifest, but with some json parsing, it can be done). Then it would load the image into the docker database with docker load, and tag the new image with the docker-configuration-name tag. This would automatically update the image from which the container is running without having to modify the shepherd service. Sadly docker does not allow for containers to be ran directly from the image archive and this is the most straightforward workaround I could think of. I hope this patch finds you in good mood, Maya PS. I found out that the original docker-service-type wasn't indent-region. And it got snuck into the patch, I hope this will still be a valid patch. gnu/services/docker.scm | 289 +++++++++++++++++++++++++++++++++------- 1 file changed, 242 insertions(+), 47 deletions(-) diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm index 741bab5a8c..b05804aa16 100644 --- a/gnu/services/docker.scm +++ b/gnu/services/docker.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2020 Efraim Flashner ;;; Copyright © 2020 Jesse Dowell ;;; Copyright © 2021 Brice Waegeneire +;;; Copyright © 2022 Maya Tomasek ;;; ;;; This file is part of GNU Guix. ;;; @@ -37,6 +38,8 @@ (define-module (gnu services docker) #:export (docker-configuration docker-service-type + docker-container-configuration + docker-container-service-type singularity-service-type)) (define-configuration docker-configuration @@ -164,6 +167,198 @@ (define docker-service-type (const %docker-accounts)))) (default-value (docker-configuration)))) +(define (pair-of-strings? val) + (and (pair val) + (string? (car val)) + (string? (cdr val)))) + +(define (list-of-pair-of-strings? val) + (list-of pair-of-strings?)) + +(define-configuration/no-serialization docker-container-configuration + (name + (symbol '()) + "Name of the docker container. Will be used to denote service to Shepherd and must be unique! +We recommend, that the name of the container is prefixed with @code{docker-}.") + (comment + (string "") + "A documentation on the docker container.") + (image-name + (string) + "A name of the image that will be used. (Note that the existence of the image +is not guaranteed by this daemon.)") + (volumes + (list-of-pair-of-strings '()) + "A list of volume binds. In (HOST_PATH CONTAINER_PATH) format.") + (ports + (list-of-pair-of-strings '()) + "A list of port binds. In (HOST_PORT CONTAINER_PORT) or (HOST_PORT CONTAINER_PORT OPTIONS) format. +For example, both port bindings are valid: + +@lisp +(ports '((\"2222\" \"22\") (\"21\" \"21\" \"tcp\"))) +@end lisp" + (environments + (list-of-pair-of-strings '()) + "A list of variable binds, inside the container enviornment. In (VARIABLE VALUE) format.")) + (network + (string "none") + "Network type.") + (additional-arguments + (list-of-strings '()) + "Additional arguments to the docker command line interface.") + (container-command + (list-of-strings '()) + "Command to send into the container.") + (attached? + (boolean #t) + "Run the container as an normal attached process (sending SIGTERM). +Or run the container as a isolated environment that must be stopped with @code{docker stop}. + +Please verify first, that you container is indeed not attached, otherwise @code{shepherd} might +assume the process is dead, even when it is not. + +You can do that, by first running your container with @code{docker run image-name}. + +Then check @code{docker ps}, if the command shows beside your container the word @code{running}. +Your container is indeed detached, but if it shows @code{starting}, and it doesn't flip to +@code{running} after a while, it means that you container is attached, and you need to keep this +option turned @code{#t}.")) + +(define (serialize-volumes config) + "Serialize list of pairs into flat list of @code{(\"-v\" \"HOST_PATH:CONTAINER_PATH\" ...)}" + (append-map + (lambda (volume-bind) + (list "-v" (format #f "~?" "~a:~a" volume-bind))) + (docker-container-configuration-volumes config))) + +(define (serialize-ports config) + "Serialize list of either pairs, or lists into flat list of +@code{(\"-p\" \"NUMBER:NUMBER\" \"-p\" \"NUMBER:NUMBER/PROTOCOL\" ...)}" + (append-map + (lambda (port-bind) + (list "-p" (format #f "~?" "~a:~a~^/~a" port-bind))) + (docker-container-configuration-ports config))) + +(define (serialized-environments config) + "Serialize list of pairs into flat list of @code{(\"-e\" \"VAR=val\" \"-e\" \"VAR=val\" ...)}." + (append-map + (lambda (env-bind) + (list "-e" (format #f "~?" "~a=~a" env-bind))) + (docker-container-configuration-environments config))) + +(define (docker-container-startup-script docker-cli container-name config) + "Return a program file, that executes the startup sequence of the @code{docker-container-shepherd-service}." + (let* ((attached? (docker-container-configuration-attached? config)) + (image-name (docker-container-configuration-image config)) + (volumes (serialize-volumes config)) + (ports (serialize-ports config)) + (envs (serialize-environments config)) + (network (docker-container-configuration-network config)) + (additional-arguments (docker-container-configuration-additional-arguments config)) + (container-command (docker-container-configuration-container-command config))) + (program-file + (string-append "start-" container-name "-container") + #~(let ((docker (string-append #$docker-cli "/bin/docker"))) + (system* docker "stop" #$container-name) + (system* docker "rm" #$container-name) + (apply system* `(,docker + "run" + ,(string-append "--name=" #$container-name) + ;; Automatically remove the container when stopping + ;; If you want persistent data, you need to use + ;; volume binds or other methods. + "--rm" + ,(string-append "--network=" #$network) + ;; TODO: + ;; Write to a cid file the container id, this allows + ;; for shepherd to manage container even when the process + ;; itself gets detached from the container + ,@(if (not #$attached) '("--cidfile" #$cid-file) '()) + #$@volumes + #$@ports + #$@envs + #$@additional-arguments + ,#$image-name + #$@container-command)))))) + +(define (docker-container-shepherd-service docker-cli config) + "Return a shepherd-service that runs CONTAINER." + (let* ((container-name (symbol->string (docker-container-configuration-name config))) + (cid-file (string-append "/var/run/docker/" container-name ".pid")) + (attached? (docker-container-configuration-attached? config))) + (shepherd-service + (provision (list (docker-container-configuration-name config))) + (requirement `(dockerd)) + (start #~(make-forkexec-constructor + (list #$(docker-container-startup-script docker-cli container-name config)) + ;; Watch the cid-file instead of the docker run command, as the daemon can + ;; still be running even when the command terminates + (if (not #$attached?) + #:pid-file #$cid-file))) + (stop (if #$attached? + #~(make-kill-destructor) + #~(lambda _ + (exec-command (list + (string-append #$docker-cli "/bin/docker") + "stop" #$container-name)) + #f)))))) + + +(define (list-of-docker-container-configurations? val) + (list-of docker-container-configuration?)) + +(define-configuration/no-serialization docker-container-service-configuration + (docker-cli + (file-like docker-cli) + "The docker package to use.") + (containers + (list-of-docker-container-configurations '()) + "The docker containers to run.")) + +(define (docker-container-shepherd-services config) + "Return shepherd services for all containers inside config." + (let ((docker-cli (docker-container-service-configuration-docker-cli config))) + (map + (lambda (container) + (docker-container-shepherd-service + docker-cli + container)) + (docker-container-service-configuration-containers config)))) + +(define docker-container-service-type + "This is the type of the service that runs docker containers using GNU Shepherd. +It allows for declarative management of running containers inside the Guix System. + +This service can be extended with list of @code{} objects. + +The following is an example @code{docker-container-service-type} configuration. + +@lisp +(service docker-container-service-type + (containers (list + (docker-container-configuration + (name 'docker-example) + (comment \"An example docker container configuration\") + (image-name \"example/example:latest\") ;; Note that images must be provided separately. + (volumes '((\"/mnt\" \"/\") (\"/home/example\" \"/home/user\"))) + (ports '((\"21\" \"21\" \"tcp\") (\"22\" \"22\"))) + (network \"host\"))))) +@end lisp" + (service-type + (name 'docker-container) + (description "Manage docker containers with shepherd.") + (extensions + (list (service-extension shepherd-root-service-type docker-container-shepherd-services))) + (compose concatenate) + (extend (lambda (config containers) + (let ((docker-cli (docker-container-service-configuration-docker-cli config)) + (initial-containers (docker-container-service-configuration-containers config))) + (docker-container-service-configuration + (docker-cli docker-cli) + (containers (append initial-containers containers)))))) + (default-value (docker-container-service-configuration)))) + ;;; ;;; Singularity.