From patchwork Wed Oct 12 22:24:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Wurmus X-Patchwork-Id: 43355 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id CD24427BBE9; Wed, 12 Oct 2022 23:25:21 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 2832127BBEA for ; Wed, 12 Oct 2022 23:25:18 +0100 (BST) Received: from localhost ([::1]:48280 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oikA9-000245-AV for patchwork@mira.cbaines.net; Wed, 12 Oct 2022 18:25:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35822) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oik9u-00023w-In for guix-patches@gnu.org; Wed, 12 Oct 2022 18:25:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:59511) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oik9u-00046r-88 for guix-patches@gnu.org; Wed, 12 Oct 2022 18:25:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oik9u-00039F-1b for guix-patches@gnu.org; Wed, 12 Oct 2022 18:25:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#58477] [PATCH] doc: Add chapter on containers to Cookbook. Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 12 Oct 2022 22:25:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 58477 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 58477@debbugs.gnu.org Cc: Ricardo Wurmus X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.166561348312069 (code B ref -1); Wed, 12 Oct 2022 22:25:01 +0000 Received: (at submit) by debbugs.gnu.org; 12 Oct 2022 22:24:43 +0000 Received: from localhost ([127.0.0.1]:58589 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oik9a-00038a-IJ for submit@debbugs.gnu.org; Wed, 12 Oct 2022 18:24:43 -0400 Received: from lists.gnu.org ([209.51.188.17]:50650) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oik9Y-00038Q-4G for submit@debbugs.gnu.org; Wed, 12 Oct 2022 18:24:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54134) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oik9X-00023d-UF for guix-patches@gnu.org; Wed, 12 Oct 2022 18:24:39 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21177) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oik9T-00043U-W5 for guix-patches@gnu.org; Wed, 12 Oct 2022 18:24:39 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1665613469; cv=none; d=zohomail.com; s=zohoarc; b=LIya1avGfhdBRWfodzWRMS2gOYtkedUqTuzw0kcByRl5wMmSQ5NR3hXhOb+h4umL5+lLNHtF680crot1cU8WvawUfstjXLmS7DLcpTUEd2xfberUnPdzU7EFzmhBjAz/rwjAck0pBYqmVfSXm15k4Z4dl929OprLzyow2tAPvQI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1665613469; h=Content-Transfer-Encoding:Cc:Date:From:MIME-Version:Message-ID:Subject:To; bh=J+hrYcvRa6XJU4L15V+gkZwe/Wfn6podQxepfl3QdEc=; b=AQ4tdE5fnyWxpUsDofOAKtCaoblMNBFDhihO39xkaZ4zEc6G5OfszR0zAD7Ih/UO73Xb1YaJhBz5/guJ1DUAH8lfCkuf5opLyzubS0n6jfww3BAwutdOrlOB8tSTYKjYsIgnTQFOqEjMe0c5EjqmDxTYfL/HC657MMwOmwWdriU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1665613469; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:From:To:To:Cc:Cc:Subject:Subject:Date:Date:Message-Id:Message-Id:MIME-Version:Content-Transfer-Encoding:Reply-To; bh=J+hrYcvRa6XJU4L15V+gkZwe/Wfn6podQxepfl3QdEc=; b=V/cW2Ea2owpU16IoF1ruOTYw+sFFCBr34OFLSMc6woStaZClxURj89GMjbRLS2D/ ERaFPGLWdrQFfruzO4a6u4kwf8uVswPzFiTSILhzPlNfASowPo3XfL/waAL8dbvACgY NK58Owszod40kGi5Q/5rRSxvFFzLjLnv2c02nr5U= Received: from localhost (73-110-142-46.pool.kielnet.net [46.142.110.73]) by mx.zohomail.com with SMTPS id 1665613467036373.5865225771146; Wed, 12 Oct 2022 15:24:27 -0700 (PDT) From: Ricardo Wurmus Date: Thu, 13 Oct 2022 00:24:20 +0200 Message-Id: <20221012222420.12338-1-rekado@elephly.net> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * doc/guix-cookbook.texi (Containers): New chapter. --- doc/guix-cookbook.texi | 401 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 400 insertions(+), 1 deletion(-) diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index b61adc06da..3e219c9a7c 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -11,7 +11,7 @@ @set SUBSTITUTE-TOR-URL https://4zwzi66wwdaalbhgnix55ea3ab4pvvw66ll2ow53kjub6se4q2bclcyd.onion @copying -Copyright @copyright{} 2019 Ricardo Wurmus@* +Copyright @copyright{} 2019, 2022 Ricardo Wurmus@* Copyright @copyright{} 2019 Efraim Flashner@* Copyright @copyright{} 2019 Pierre Neidhardt@* Copyright @copyright{} 2020 Oleg Pykhalov@* @@ -71,6 +71,7 @@ Weblate} (@pxref{Translating Guix,,, guix, GNU Guix reference manual}). * Scheme tutorials:: Meet your new favorite language! * Packaging:: Packaging tutorials * System Configuration:: Customizing the GNU System +* Containers:: Isolated environments and nested systems * Advanced package management:: Power to the users! * Environment management:: Control environment @@ -2454,6 +2455,404 @@ ngx.say(stdout) #$(local-file "index.lua")))))))))))))) @end lisp +@c ********************************************************************* +@node Containers +@chapter Containers + +The kernel Linux provides a number of shared facilities that are +available to processes in the system. These facilities include a shared +view on the file system, other processes, network devices, user and +group identities, and a few others. Since Linux 3.19 a user can choose +to @emph{unshare} some of these shared facilities for selected +processes, providing them (and their child processes) with a different +view on the system. + +A process with an unshared @code{mount} namespace, for example, has its +own view on the file system --- it will only be able to see directories +that have been explicitly bound in its mount namespace. A process with +its own @code{proc} namespace will consider itself to be the only +process running on the system, running as PID 1. + +Guix uses these kernel features to provide fully isolated environments +and even complete Guix System containers, lightweight virtual machines +that share the host system's kernel. This feature comes in especially +handy when using Guix on a foreign distribution to prevent interference +from foreign libraries or configuration files that are available +system-wide. + +@menu +* Guix Containers:: Perfectly isolated environments +* Guix System Containers:: A system inside your system +@end menu + +@node Guix Containers +@section Guix Containers + +The easiest way to get started is to use @command{guix shell} with the +@option{--container} option. @xref{Invoking guix shell,,, guix, GNU +Guix Reference Manual} for a reference of valid options. + +The following snippet spawns a minimal shell process with most +namespaces unshared from the system. The current working directory is +visible to the process, but anything else on the file system is +unavailable. This extreme isolation can be very useful when you want to +rule out any sort of interference from environment variables, globally +installed libraries, or configuration files. + +@example +guix shell --container +@end example + +It is a bleak environment, barren, desolate. You will find that not +even the GNU coreutils are available here, so to explore this deserted +wasteland you need to use built-in shell commands. Even the usually +gigantic @file{/gnu/store} directory is reduced to a faint shadow of +itself. + +@example sh +$ echo /gnu/store/* +/gnu/store/@dots{}-gcc-10.3.0-lib +/gnu/store/@dots{}-glibc-2.33 +/gnu/store/@dots{}-bash-static-5.1.8 +/gnu/store/@dots{}-ncurses-6.2.20210619 +/gnu/store/@dots{}-bash-5.1.8 +/gnu/store/@dots{}-profile +/gnu/store/@dots{}-readline-8.1.1 +@end example + +@cindex exiting a container +There isn't much you can do in an environment like this other than +exiting it. You can use @key{^D} or @command{exit} to terminate this +limited shell environment. + +@cindex exposing directories, container +@cindex sharing directories, container +@cindex mapping locations, container +You can make other directories available inside of the container +environment; use @option{--expose=DIRECTORY} to bind-mount the given +directory as a read-only location inside the container, or use +@option{--share=DIRECTORY} to make the location writable. With an +additional mapping argument after the directory name you can control the +name of the directory inside the container. In the following example we +map @file{/etc} on the host system to @file{/the/host/etc} inside a +container in which the GNU coreutils are installed. + +@example sh +$ guix shell --container --share=/etc=/the/host/etc coreutils +$ ls /the/host/etc +@end example + +Similarly, you can prevent the current working directory from being +mapped into the container with the @option{--no-cwd} option. Another +good idea is to create a dedicated directory that will serve as the +container's home directory, and spawn the container shell from that +directory. + +@cindex hide system libraries, container +@cindex avoid ABI mismatch, container +On a foreign system a container environment can be used to compile +software that cannot possibly be linked with system libraries or with +the system's compiler toolchain. A common use-case in a research +context is to install packages from within an R session. Outside of a +container environment there is a good chance that the foreign compiler +toolchain and incompatible system libraries are found first, resulting +in incompatible binaries that cannot be used by R. In a container shell +this problem disappears, as system libraries and executables simply +aren't available due to the unshared @code{mount} namespace. + +Let's take a comprehensive manifest providing a comfortable development +environment for use with R: + +@lisp +(specifications->manifest + (list "r-minimal" + + ;; base packages + "bash-minimal" + "glibc-locales" + "nss-certs" + + ;; Common command line tools lest the container is too empty. + "coreutils" + "grep" + "which" + "wget" + "sed" + + ;; R markdown tools + "pandoc" + + ;; Toolchain and common libraries for "install.packages" + "gcc-toolchain@@10" + "gfortran-toolchain" + "gawk" + "tar" + "gzip" + "unzip" + "make" + "cmake" + "pkg-config" + "cairo" + "libxt" + "openssl" + "curl" + "zlib")) +@end lisp + +Let's use this to run R inside a container environment. For convenience +we share the @code{net} namespace to use the host system's network +interfaces. Now we can build R packages from source the traditional way +without having to worry about ABI mismatch or incompatibilities. + +@example sh +$ guix shell --container --network --manifest=manifest.scm -- R + +R version 4.2.1 (2022-06-23) -- "Funny-Looking Kid" +Copyright (C) 2022 The R Foundation for Statistical Computing +@dots{} +> e <- Sys.getenv("GUIX_ENVIRONMENT") +> Sys.setenv(GIT_SSL_CAINFO=paste0(e, "/etc/ssl/certs/ca-certificates.crt")) +> Sys.setenv(SSL_CERT_FILE=paste0(e, "/etc/ssl/certs/ca-certificates.crt")) +> Sys.setenv(SSL_CERT_DIR=paste0(e, "/etc/ssl/certs")) +> install.packages("Cairo", lib=paste0(getwd())) +@dots{} +* installing *source* package 'Cairo' ... +@dots{} +* DONE (Cairo) + +The downloaded source packages are in + '/tmp/RtmpCuwdwM/downloaded_packages' +> library("Cairo", lib=getwd()) +> # success! +@end example + +Using container shells is fun, but they can become a little cumbersome +when you want to go beyond just a single interactive process. Some +tasks become a lot easier when they sit on the rock solid foundation of +a proper Guix System and its rich set of system services. The next +section shows you how to launch a complete Guix System inside of a +container. + + +@node Guix System Containers +@section Guix System Containers + +The Guix System provides a wide array of interconnected system services +that are configured declaratively to form a dependable stateless GNU +System foundation for whatever tasks you throw at it. Even when using +Guix on a foreign distribution you can benefit from the design of Guix +System by running a system instance as a container. Using the same +kernel features of unshared namespaces mentioned in the previous +section, the resulting Guix System instance is isolated from the host +system and only shares file system locations that you explicitly +declare. + +A Guix System container differs from the shell process created by +@command{guix shell --container} in a number of important ways. While +in a container shell the containerized process is a Bash shell process, +a Guix System container runs the Shepherd as PID 1. In a system +container all system services (@pxref{Services,,, guix, GNU Guix +Reference Manual}) are set up just as they would be on a Guix System in +a virtual machine or on bare metal---this includes daemons managed by +the GNU@tie{}Shepherd (@pxref{Shepherd Services,,, guix, GNU Guix +Reference Manual}) as well as other kinds of extensions to the operating +system (@pxref{Service Composition,,, guix, GNU Guix Reference Manual}). + +The perceived increase in complexity of running a Guix System container +is easily justified when dealing with more complex applications that +have higher or just more rigid requirements on their execution +contexts---configuration files, dedicated user accounts, directories for +caches or log files, etc. In Guix System the demands of this kind of +software are satisfied through the deployment of system services. + + +@node A Database Container +@subsection A Database Container + +A good example might be a PostgreSQL database server. Much of the +complexity of setting up such a database server is encapsulated in this +deceptively short service declaration: + +@lisp +(service postgresql-service-type + (postgresql-configuration + (postgresql postgresql-14))) +@end lisp + +A complete operating system declaration for use with a Guix System +container would look something like this: + +@lisp +(use-modules (gnu)) +(use-package-modules databases) +(use-service-modules databases) + +(operating-system + (host-name "container") + (timezone "Europe/Berlin") + (file-systems (cons (file-system + (device (file-system-label "does-not-matter")) + (mount-point "/") + (type "ext4")) + %base-file-systems)) + (bootloader (bootloader-configuration + (bootloader grub-bootloader) + (targets '("/dev/sdX")))) + (services + (cons* (service postgresql-service-type + (postgresql-configuration + (postgresql postgresql-14) + (config-file + (postgresql-config-file + (log-destination "stderr") + (hba-file + (plain-file "pg_hba.conf" + "\ +local all all trust +host all all 10.0.0.1/32 trust")) + (extra-config + '(("listen_addresses" "*") + ("log_directory" "/var/log/postgresql"))))))) + (service postgresql-role-service-type + (postgresql-role-configuration + (roles + (list (postgresql-role + (name "test") + (create-database? #t)))))) + %base-services))) +@end lisp + +With @code{postgresql-role-service-type} we define a role ``test'' and +create a matching database, so that we can test right away without any +further manual setup. The @code{postgresql-config-file} settings allow +a client from IP address 10.0.0.1 to connect without requiring +authentication---a bad idea in production systems, but convenient for +this example. + +Let's build a script that will launch an instance of this Guix System as +a container. Write the @code{operating-system} declaration above to a +file @file{os.scm} and then use @command{guix system container} to build +the launcher. (@pxref{Invoking guix system,,, guix, GNU Guix Reference +Manual}). + +@example +$ guix system container os.scm +The following derivations will be built: + /gnu/store/@dots{}-run-container.drv + @dots{} +building /gnu/store/@dots{}-run-container.drv... +/gnu/store/@dots{}-run-container +@end example + +Now that we have a launcher script we can run it to spawn the new system +with a running PostgreSQL service. Note that due to some as yet +unresolved limitations we need to run the launcher as the root user (or +with @command{sudo}). + +@example +$ sudo /gnu/store/@dots{}-run-container +system container is running as PID 5983 +@dots{} +@end example + +Background the process with @key{Ctrl-z} followed by @command{bg}. Note +the process ID in the output; we will need it to connect to the +container later. You know what? Let's try attaching to the container +right now. We will use @command{nsenter}, a tool provided by the +@code{util-linux} package: + +@example +$ guix shell util-linux +$ sudo nsenter -a -t 5983 +root@@container /# pgrep -a postgres +49 /gnu/store/@dots{}-postgresql-14.4/bin/postgres -D /var/lib/postgresql/data --config-file=/gnu/store/@dots{}-postgresql.conf -p 5432 +51 postgres: checkpointer +52 postgres: background writer +53 postgres: walwriter +54 postgres: autovacuum launcher +55 postgres: stats collector +56 postgres: logical replication launcher +root@@container /# exit +@end example + +The PostgreSQL service is running in the container! + + +@node Container Networking +@subsection Container Networking +@cindex container networking + +What good is a Guix System running a PostgreSQL as a container when we +can only talk to it with processes originating in the container? It +would be much better if we could talk to the database over the network. + +The easiest way to do this is to create a pair of connected virtual +Ethernet devices (known as @code{veth}). We move one of the devices +(@code{ceth-test}) into the @code{net} namespace of the container and +leave the other end (@code{veth-test}) of the connection on the host +system. + +@example +pid=5983 +ns="guix-test" +host="veth-test" +client="ceth-test" + +# Attach the new net namespace "guix-test" to the container PID. +sudo ip netns attach $ns $pid + +# Create the pair of devices +sudo ip link add $host type veth peer name $client + +# Move the client device into the container's net namespace +sudo ip link set $client netns $ns +@end example + +Then we configure the host side: + +@example +sudo ip link set $host up +sudo ip addr add 10.0.0.1/24 dev $host +@end example + +@dots{}and then we configure the client side: + +@example +sudo ip netns exec $ns ip link set lo up +sudo ip netns exec $ns ip link set $client up +sudo ip netns exec $ns ip addr add 10.0.0.2/24 dev $client +@end example + +At this point the host can reach the container at IP address 10.0.0.2, +and the container can reach the host at IP 10.0.0.1. This is all we +need to talk to the database server inside the container from the host +system on the outside. + +@example +$ psql -h 10.0.0.2 -U test +psql (14.4) +Type "help" for help. + +test=> CREATE TABLE hello (who TEXT NOT NULL); +CREATE TABLE +test=> INSERT INTO hello (who) VALUES ('world'); +INSERT 0 1 +test=> SELECT * FROM hello; + who +------- + world +(1 row) +@end example + +Now that we're done with this little demonstration let's clean up: + +@example +sudo kill $pid +sudo ip netns del $ns +sudo ip link del $host +@end example + + @c ********************************************************************* @node Advanced package management @chapter Advanced package management