[bug#75810,v3,00/11] Rootless guix-daemon

  Hello!

Here’s an updated version, addressing most issues brought up
by Reepca, also available from
<https://codeberg.org/civodul/guix/src/branch/wip-rootless-daemon>.
Main changes compared to v2:

  • Derivation inputs and / are mounted read-only; additional
    tests check the ability to write to these, to /tmp, to
    /dev/{full,null}, and to remount any of these as read-write.

  • Unit files for systemd tweaked so that (1) guix-daemon sees
    a private read-write mount of the store, and (2) gnu-store.mount
    actually remounts the store read-only after guix-daemon has
    started.

  • ‘DerivationGoal::deleteTmpDir’ bails out when it fails to
    chown ‘tmpDir’ (i.e., it does not try to “pivot” the /top
    sub-directory).

Did I forget anything, Reepca?

The one observable difference compared to current guix-daemon
operational mode is that, in the build environment, writing to
the root file system results in EROFS instead of EPERM, as you
pointed out earlier.  That’s not great but probably acceptable.
We’ll only know whether this is a problem in practice once we’ve
run the test suites of tens of thousands of packages.

I tested this patch series by:

  • running ‘make check’;

  • manually running ‘guix-install.sh’ in a Debian VM, as
    explained before.

Next up:

  • automating ‘guix-install.sh’ VM tests;

  • updating ‘guix-service-type’ to optionally support
    unprivileged guix-daemon.

I think these two bits can come later though.

Thoughts?

Ludo’.

Ludovic Courtès (11):
  daemon: Use ‘close_range’ where available.
  daemon: Bind-mount all the inputs, not just directories.
  daemon: Remount inputs as read-only.
  daemon: Remount root directory as read-only.
  daemon: Allow running as non-root with unprivileged user namespaces.
  tests: Run in a chroot and unprivileged user namespaces.
  daemon: Create /var/guix/profiles/per-user unconditionally.
  daemon: Drop Linux ambient capabilities before executing builder.
  daemon: Move comments where they belong.
  etc: systemd services: Run ‘guix-daemon’ as an unprivileged user.
  guix-install.sh: Support the unprivileged daemon where possible.

 build-aux/test-env.in       |  14 ++-
 config-daemon.ac            |   5 +-
 etc/gnu-store.mount.in      |   3 +-
 etc/guix-daemon.service.in  |  20 +++-
 etc/guix-install.sh         | 108 ++++++++++++++----
 guix/substitutes.scm        |   4 +-
 nix/libstore/build.cc       | 219 ++++++++++++++++++++++++++----------
 nix/libstore/local-store.cc |  30 +++--
 nix/libutil/util.cc         |  23 +++-
 tests/processes.scm         |   9 +-
 tests/store.scm             | 206 +++++++++++++++++++++++++++------
 11 files changed, 494 insertions(+), 147 deletions(-)


base-commit: 00787cd61611d74d3e54b160e94176905d36ef39
  

Hi Ludo,

On Fri, 21 Feb 2025 at 14:05, Ludovic Courtès <ludo@gnu.org> wrote:

> The one observable difference compared to current guix-daemon
> operational mode is that, in the build environment, writing to
> the root file system results in EROFS instead of EPERM, as you
> pointed out earlier.  That’s not great but probably acceptable.
> We’ll only know whether this is a problem in practice once we’ve
> run the test suites of tens of thousands of packages.

Clearly, I do not fully understand all the deep details of all the
series.

Quoting Janneke [1]:

        I'm kind of afraid that having a writable /gnu/store, even if it's just
        on foreign distributions, is going to cause a whole lot of problems/bug
        reports with people changing files in the store.  When I came to guix I
        ran it on Debian for a couple of months and I certainly changed files in
        the store, even with the read-only mount hurdle, to "get stuff to
        build".  Only later to realise that by doing so I was making things much
        more difficult for myself.

        Hopefully I'm either misunderstanding this patch set, or else too
        pessimistict, and maybe other people aren't as stupid as I was when I
        first came to Guix?

I’m not sure to get what’s the answer now with the v3?  Especially when
connected to this other question:

                           Will there be an option for users to choose between
        a non-root guix-daemon or a read-only store?

Where the answer, IIUC, is no.

Could you clarify the status about the store when running guix-daemon as
root on foreign distros?  Or maybe now, will guix-daemon always run as a
regular user on foreign distros?

From an user perspective, instead of running guix-daemon as root, now
guix-daemon will run as the regular user named ’guix-daemon’ without any
special privileges, right?

User still need root privileges once at guix-install.sh time but not
more.  Therefore, for updating the guix-daemon, the user guix-daemon
needs to run “guix pull“ and restart the service, right?

If yes, cool!  It’ll be a booster for cluster sysadmins. :-)

Cheers,
simon

1: [bug#75810] [PATCH 0/6] Rootless guix-daemon
Janneke Nieuwenhuizen <janneke@gnu.org>
Fri, 24 Jan 2025 20:20:42 +0100
id:87ikq49fxx.fsf@gnu.org
https://issues.guix.gnu.org/75810
https://issues.guix.gnu.org/msgid/87ikq49fxx.fsf@gnu.org
https://yhetil.org/guix/87ikq49fxx.fsf@gnu.org
  

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Here’s an updated version, addressing most issues brought up
> by Reepca, also available from
> <https://codeberg.org/civodul/guix/src/branch/wip-rootless-daemon>.
> Main changes compared to v2:
>
>   • Derivation inputs and / are mounted read-only; additional
>     tests check the ability to write to these, to /tmp, to
>     /dev/{full,null}, and to remount any of these as read-write.
>
>   • Unit files for systemd tweaked so that (1) guix-daemon sees
>     a private read-write mount of the store, and (2) gnu-store.mount
>     actually remounts the store read-only after guix-daemon has
>     started.

I'm not familiar with how systemd does service dependencies, but does
this mean that the store becomes writable when the daemon is stopped?

>
>   • ‘DerivationGoal::deleteTmpDir’ bails out when it fails to
>     chown ‘tmpDir’ (i.e., it does not try to “pivot” the /top
>     sub-directory).
>
> Did I forget anything, Reepca?

I believe that if you try a "--keep-failed" build that fails in the
CAP_CHOWN case, you'll find that only root or the guix-daemon user can
delete the kept build directory, though the user that started the build
can delete everything inside it.  This is because in that case the build
directory was chown'ed back to guix-daemon so that it could be moved,
but wasn't chown'ed to the client user afterward.  If I recall correctly
there was code included to perform this extra chown in the (getuid() !=
0) case in the v2 series - was it accidentally forgotten?

Also, there are potential issues with how wide the scope of the try
block in DerivationGoal::deleteTmpDir is - _chown isn't the only place
within it that can raise a SysError, and there are failure modes present
that may merit more user attention than lvlInfo.  For example, if

rename((pivot + "/top").c_str(), top.c_str())

fails (which can be rather easily arranged by a local attacker), then
the build directory path reported in the "note: keeping build directory"
message remains up for grabs by anyone.  If the user doesn't go out of
their way to verify that the build directory isn't attacker-controlled,
they could be rather easily tricked into executing malicious code.  But
currently the exception from this rename failing will be turned into a
lvlInfo message, and I'm not sure how that interacts with the verbosity
defaults in the various CLI programs.

This does somewhat raise the question of why we're even doing the
pivoting in a way that creates a window during which failure can be
induced.  For example, we could move the inner build directory to the
pivot path, at which point the outer build directory should become
empty, so it should work to then rename the pivot path to the outer
build directory path, thereby atomically replacing it.

Also, in the unprivileged case (non-root, no CAP_CHOWN), the build
directory never gets pivoted out.  This is better for security than the
previous situation (which allowed setuid programs to be exposed), but it
should be quite doable to simply secure the file permissions first and
then carry on with the pivot.  I believe I previously mentioned perhaps
using secureFilePerms to do this?

It may work well to use the v2 patch for this with a call to
secureFilePerms added right before the try block and a have_cap_chown
boolean flag being saved for later recall after the pivot instead of the
(getuid() != 0) check.  That way in the fully-unprivileged case it
doesn't successfully pivot the now-sanitized build directory only to
immediately fail to chown it.  Actually, because that chown call doesn't
result in an exception on failure, it would also work to only add the
secureFilePerms call.


Also, I've discovered that while mount(2) uses EPERM for both a locked
mount point and insufficient privileges, umount(2) uses EINVAL for the
former and EPERM for the latter.  This may be a good way to test that
we're triggering the mount-locking behavior as intended.

> The one observable difference compared to current guix-daemon
> operational mode is that, in the build environment, writing to
> the root file system results in EROFS instead of EPERM, as you
> pointed out earlier.  That’s not great but probably acceptable.
> We’ll only know whether this is a problem in practice once we’ve
> run the test suites of tens of thousands of packages.

Strictly speaking, it's also observable that the root file system,
store, /tmp, etc is not owned by uid 0, and that the input store items
are all mounted read-only.

- reepca
  

Ludovic Courtès <ludo@gnu.org> skribis:

> Next up:
>
>   • automating ‘guix-install.sh’ VM tests;

Done in <https://issues.guix.gnu.org/76488>.

Ludo’.
  

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Quoting Janneke [1]:
>
>         I'm kind of afraid that having a writable /gnu/store, even if it's just
>         on foreign distributions,

This problem is fixed in v3: the store will be remounted readonly as is
currently the case.

> Could you clarify the status about the store when running guix-daemon as
> root on foreign distros?  Or maybe now, will guix-daemon always run as a
> regular user on foreign distros?

As currently written, guix-daemon will always run as non-root on foreign
distros (on systemd-based distros specifically.)

>>From an user perspective, instead of running guix-daemon as root, now
> guix-daemon will run as the regular user named ’guix-daemon’ without any
> special privileges, right?

Correct.

> User still need root privileges once at guix-install.sh time but not
> more.  Therefore, for updating the guix-daemon, the user guix-daemon
> needs to run “guix pull“ and restart the service, right?

The upgrade procedure remains unchanged: you would run ‘guix pull’ as
root and restart the service¹ (the service itself runs as user
‘guix-daemon’).

> If yes, cool!  It’ll be a booster for cluster sysadmins. :-)

Yup!

Ludo’.

¹ https://guix.gnu.org/manual/devel/en/html_node/Upgrading-Guix.html
  

Hi,

Reepca Russelstein <reepca@russelstein.xyz> skribis:

>>   • Unit files for systemd tweaked so that (1) guix-daemon sees
>>     a private read-write mount of the store, and (2) gnu-store.mount
>>     actually remounts the store read-only after guix-daemon has
>>     started.
>
> I'm not familiar with how systemd does service dependencies, but does
> this mean that the store becomes writable when the daemon is stopped?

I had to check because it’s not crystal clear.

‘systemctl stop guix-daemon’ also stops ‘gnu-store.mount’.

But then you can do ‘systemctl start gnu-store.mount’, which does *not*
start guix-daemon; at that point, ‘systemctl start guix-daemon’ spawns
guix-daemon, but it cannot write to the store.

It’s messy, but I don’t know how to do better.

[...]

> It may work well to use the v2 patch for this with a call to
> secureFilePerms added right before the try block and a have_cap_chown
> boolean flag being saved for later recall after the pivot instead of the
> (getuid() != 0) check.  That way in the fully-unprivileged case it
> doesn't successfully pivot the now-sanitized build directory only to
> immediately fail to chown it.  Actually, because that chown call doesn't
> result in an exception on failure, it would also work to only add the
> secureFilePerms call.

I went back to v2 + ‘secureFilePerms’ call.

> Also, I've discovered that while mount(2) uses EPERM for both a locked
> mount point and insufficient privileges, umount(2) uses EINVAL for the
> former and EPERM for the latter.  This may be a good way to test that
> we're triggering the mount-locking behavior as intended.

The tests try to MS_REMOUNT the inputs, which is exactly what we want to
prevent; we could test the low-level semantics you describe, but it’s
quite obscure and maybe unnecessary given that we test MS_REMOUNT?

>> The one observable difference compared to current guix-daemon
>> operational mode is that, in the build environment, writing to
>> the root file system results in EROFS instead of EPERM, as you
>> pointed out earlier.  That’s not great but probably acceptable.
>> We’ll only know whether this is a problem in practice once we’ve
>> run the test suites of tens of thousands of packages.
>
> Strictly speaking, it's also observable that the root file system,
> store, /tmp, etc is not owned by uid 0, and that the input store items
> are all mounted read-only.

Right.

I’ll send v4 shortly.  Thanks again for your feedback!

Ludo’.