[bug#54375] Mutable guix shell environments

sample useage:
$ cd project
$ guix package --manifest=manifest.scm --profile=.guix-profile
$ guix shell # --profile=.guix-profile is implicit

do some stuff. realize that you want to bring in another package.

$ guix install --profile=.guix-profile additional-package

Then additional package is instantly available. This is especially useful to lisp programmers because, currently, bringing in an additional library involves restarting shell, lisp process, reloading source files, regenerating process state.

Am Sonntag, dem 13.03.2022 um 18:21 +0000 schrieb Charles:

> Subject: [PATCH 2/2] guix: shell: Implicitly use a .guix-profile as -
> -profile option.
> 
> * guix/scripts/shell.scm (auto-detect-manifest): Add ".guix-profile"
> to matches as --profile option
This one LGTM and it even avoids some problems we're having with
caches.  I'll keep it open for bikeshedding suggestions w.r.t. the name
of the directory to detect, though.

> Subject: [PATCH 1/2] guix: environment: Enable mutable environments.
> 
> * guix/scripts/environment.scm (launch-environment launch-
> environment/fork
> launch-environment/container guix-environment*): Add #:set-profile?
> parameter set GUIX_PROFILE
> when --profile option is used.
> * guix/scripts/environment.scm (guix-environment*): Profile could
> point to a
> profile directory instead of a store directory.
This one not so much.  You already have GUIX_ENVIRONMENT set, which
should be enough for lookup purposes.  If it's about manipulating PATH
and other environment variables, I think it'd better to do this
manually – you could even spawn a lightweight shell on your own by
simply doing the following:

$ $SHELL
$ GUIX_PROFILE=/path/to/profile
$ source $GUIX_PROFILE/etc/profile

If you want mutable environments, that's the easiest and imho best way
of achieving such a thing.  Guix is functional by design and we should
not compromise on that.

As far as using $GUIX_PROFILE together with `guix shell' is concerned,
if anything is even holding it back, it's that we don't respect
GUIX_PROFILE in the Scheme code that adds those variables.  Which to be
fair might be for the best, as GUIX_PROFILE is not guaranteed to
correspond to this invocation of `guix shell' when people are careless.
That's not to say that support for this couldn't be added, but at the
very least we'd have to be more careful about it.

Cheers

Thanks for taking a look Liliana. I'm glad you like guix shell detecting a local profile.

> $ $SHELL
> $ GUIX_PROFILE=/path/to/profile
> $ source $GUIX_PROFILE/etc/profile

This is basically what I was going for. I just find this useful to be built in. When I think "guix shell", I think: "make an environment where I have everything I need". If I want to adjust what I need, why do I need to restart everything from scratch. If I did not know about guix shell and did all my development using ~/.guix-profile, this functionallity would already be available.

> Guix is functional by design and we should
not compromise on that.

I totally agree, and I do not think this does compromise it. With my patch, guix shells using guix.scm or manifest, would be unchanged. Currently, guix shell --profile, has strickly less functionallity than using the profile otherwise, this could be confusing (as it is to me). Profiles are not really mutable since they use generations; the subject line is a bit of a misnomer.

> As far as using $GUIX_PROFILE tog...

Yeah, before my patch, guix shell --profile, would not be set to the profile I'm using. This actually makes sense, because, we are not really using it. Guix shell just happens to be using all the same packages that were available in that profile. Confusingly, If the profile, is updated, the shell is now out of sync with it. My patch should make using guix shell --profile more intuitive.


------- Original Message -------

On Sunday, March 13th, 2022 at 2:51 PM, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

> Am Sonntag, dem 13.03.2022 um 18:21 +0000 schrieb Charles:
>
> > Subject: [PATCH 2/2] guix: shell: Implicitly use a .guix-profile as -
> >
> > -profile option.
> >
> > * guix/scripts/shell.scm (auto-detect-manifest): Add ".guix-profile"
> >
> > to matches as --profile option
>
> This one LGTM and it even avoids some problems we're having with
>
> caches. I'll keep it open for bikeshedding suggestions w.r.t. the name
>
> of the directory to detect, though.
>
> > Subject: [PATCH 1/2] guix: environment: Enable mutable environments.
> >
> > * guix/scripts/environment.scm (launch-environment launch-
> >
> > environment/fork
> >
> > launch-environment/container guix-environment*): Add #:set-profile?
> >
> > parameter set GUIX_PROFILE
> >
> > when --profile option is used.
> >
> > * guix/scripts/environment.scm (guix-environment*): Profile could
> >
> > point to a
> >
> > profile directory instead of a store directory.
>
> This one not so much. You already have GUIX_ENVIRONMENT set, which
>
> should be enough for lookup purposes. If it's about manipulating PATH
>
> and other environment variables, I think it'd better to do this
>
> manually – you could even spawn a lightweight shell on your own by
>
> simply doing the following:
>
> $ $SHELL
>
> $ GUIX_PROFILE=/path/to/profile
>
> $ source $GUIX_PROFILE/etc/profile
>
> If you want mutable environments, that's the easiest and imho best way
>
> of achieving such a thing. Guix is functional by design and we should
>
> not compromise on that.
>
> As far as using $GUIX_PROFILE together with `guix shell' is concerned, if anything is even holding it back, it's that we don't respect GUIX_PROFILE in the Scheme code that adds those variables. Which to be fair might be for the best, as GUIX_PROFILE is not guaranteed to correspond to this invocation of` guix shell' when people are careless.
>
> That's not to say that support for this couldn't be added, but at the
>
> very least we'd have to be more careful about it.
>
> Cheers

Hi Charles,

Am Sonntag, dem 13.03.2022 um 23:38 +0000 schrieb Charles:
> Thanks for taking a look Liliana. I'm glad you like guix shell
> detecting a local profile.
> 
> > $ $SHELL
> > $ GUIX_PROFILE=/path/to/profile
> > $ source $GUIX_PROFILE/etc/profile
> 
> This is basically what I was going for. I just find this useful to be
> built in. When I think "guix shell", I think: "make an environment
> where I have everything I need". If I want to adjust what I need, why
> do I need to restart everything from scratch. If I did not know about
> guix shell and did all my development using ~/.guix-profile, this
> functionallity would already be available.
Yes, but it'd also be broken af.  You are focusing on a very particular
use case and ignoring all others.  What if instead of just "a package"
I wanted to add or remove a package that defines a search path, e.g.
Emacs?  In the adding case, EMACSLOADPATH would not get defined and in
the removing case it would still linger.  What you describe as desired
functionality is not at all well-defined.  Using `guix shell --profile'
on the other hand has the same semantics as the other `guix shell'
commands, and is thus imo less confusing once you start to think about
it.

Cheers

> What if instead of just "a package" I wanted to add or remove a package that defines a search path

As with ~/.guix-profile, you will have to . "$GUIX_PROFILE/etc/profile" just like the hint says. For removing, they linger, that is exactly what the default profile does too.

This comes down to us disagreeing which is more important & useful: guix shell semantics vs profile semantics. guix shell --profile could go either way.

Another way to look at it is that I this sequence of 3 commands very useful (so might others). They clearly have something to do with guix profiles, and they spawn a new shell. Thus I thought guix shell --profile was the perfect place for them. You claim they do not belong there. Is there a better place you would suggest?

Charles via Guix-patches via schreef op zo 13-03-2022 om 18:21 [+0000]:
> sample useage:
> $ cd project
> $ guix package --manifest=manifest.scm --profile=.guix-profile
> $ guix shell # --profile=.guix-profile is implicit
> 
> do some stuff. realize that you want to bring in another package.
> 
> $ guix install --profile=.guix-profile additional-package
> 
> Then additional package is instantly available. This is especially useful to lisp programmers because, currently, bringing in an additional library involves restarting shell, lisp process, reloading source files, regenerating process state.

If this is added, then this needs to be documented in the manual,
preferably with some examples on how to use it.

Greetings,
Maxime.

Charles via Guix-patches via schreef op zo 13-03-2022 om 18:21 [+0000]:
> sample useage:
> $ cd project
> $ guix package --manifest=manifest.scm --profile=.guix-profile
> $ guix shell # --profile=.guix-profile is implicit

Alternative suggestion, which IMHO fits the non-persistent naturre of
"guix shell" better:

$ cd project
$ guix shell --allow-temporary-modifications # fails on read-only file systems
$ echo $GUIX_ENVIRONMENT
# a temporary (mutable) profile was created
> $HOME/.cache/.../profile
# install things in the $GUIX_ENVIRONMENT profile
$ guix install foo bar ... 

As noted by lilyp, this won't work in all situations due to search path
issues.  These could be resolved by always setting a few search paths
(PATH, EMACSLOADPATH, GUILE_LOAD_PATH, INFOPATH, ...) (even in not currently
present in the profile!), at cost of having to make them ‘magical’ in some
sense, which ludo does not seem to like IIUC.  (This could also be done
for ~/.guix-profile)

(I don't like it much either, but it seems a very convenient solution that
does not seem to have any concrete downsides and IMHO it seems much better
than telling the user to re-source the profile.)

Greetings,
Maxime.

Thanks for taking a look Maxime.

I actually like Your alternate suggestion better (though the name --allow-temporary-modifications is a bit long). I'm not sure how to implement it though.

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Charles via Guix-patches via schreef op zo 13-03-2022 om 18:21 [+0000]:
>> sample useage:
>> $ cd project
>> $ guix package --manifest=manifest.scm --profile=.guix-profile
>> $ guix shell # --profile=.guix-profile is implicit
>
> Alternative suggestion, which IMHO fits the non-persistent naturre of
> "guix shell" better:
>
> $ cd project
> $ guix shell --allow-temporary-modifications # fails on read-only file systems

Or ‘--transient’.

> $ echo $GUIX_ENVIRONMENT
> # a temporary (mutable) profile was created
>> $HOME/.cache/.../profile
> # install things in the $GUIX_ENVIRONMENT profile
> $ guix install foo bar ... 
>
> As noted by lilyp, this won't work in all situations due to search path
> issues.  These could be resolved by always setting a few search paths
> (PATH, EMACSLOADPATH, GUILE_LOAD_PATH, INFOPATH, ...) (even in not currently
> present in the profile!), at cost of having to make them ‘magical’ in some
> sense, which ludo does not seem to like IIUC.  (This could also be done
> for ~/.guix-profile)

I like this approach.

Charles’ proposal reminds me of ‘module’, a venerable “environment
management” tool widely used in high-performance computing (HPC):

  http://modules.sourceforge.net/

That command allows for an incremental style like Charles proposes:

  module load gcc      # adds GCC to $PATH
  module load libgc    # adds libgc to $C_INCLUDE_PATH
  module unload libgc  # removes libgc from $C_INCLUDE_PATH

For this to work though, ‘module’ is actually a shell function: that
lets it adjust environment variables here and now.

Anyway, I digress…

I thought about this other approach:

  1. ‘guix shell’ always builds a new profile, as it already does.

  2. When ‘GUIX_ENVIRONMENT’ is defined, it computes search paths as a
     combination of the new profile and that pointed to by
     $GUIX_ENVIRONMENT.

  3. It spawns a new shell, like it already does.

Thus, instead of doing upfront:

  guix shell a b c

One could do:

  $ guix shell a
  [env]$ guix shell b
  [env]$ guix shell c

It is more accurate but less flexible than what you propose though,
because only child processes of the nested shells would see the
“changes”.  So maybe not a good idea, after all.

Hmm, need more thought…

Thanks,
Ludo’.

Hi,

On Mon, 14 Mar 2022 at 23:19, Ludovic Courtès <ludo@gnu.org> wrote:

> That command allows for an incremental style like Charles proposes:
>
>   module load gcc      # adds GCC to $PATH
>   module load libgc    # adds libgc to $C_INCLUDE_PATH
>   module unload libgc  # removes libgc from $C_INCLUDE_PATH
>
> For this to work though, ‘module’ is actually a shell function: that
> lets it adjust environment variables here and now.

While I understand the need of Charles's proposal, especially in
development or exploration, I am convinced that this "incremental
style" is the root of many unreproducible computational environments.
The "incremental style" is a quick and dirty approach for creating a
computational environment and because the discipline is wrong then it
lead more than often to hard-to-reproduce computational environment,
therefore Guix should not try to mimick, IMHO.  I agree with Liliana's
words: «Guix is functional by design and we should not compromise on
that.».  To me, it is as programmers being strongly used to imperative
style complaining about the paradigm shift of the functional style --
or vice-versa... I digress. :-)

To mitigate because we have to work and sadly need to have things done...

> One could do:
>
>   $ guix shell a
>   [env]$ guix shell b
>   [env]$ guix shell c

..somehow, it would ease if when requiring the package 'b', the
environment is "aware" of the search paths of the environment required
by the package 'a'.  For instance,

--8<---------------cut here---------------start------------->8---
$ guix shell man-db
[env]$ tree $MANPATH
/gnu/store/m3i9pga6rqfg342sajzww4gl8w51q6sz-profile/share/man
├── index.db ->
/gnu/store/pdh5nz4qkg5q243q5rgxkk1hnmaf1plq-manual-database/share/man/index.db
├── it -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/it
├── man1 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man1
├── man5 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man5
└── man8 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man8

4 directories, 1 file

[env]$ guix shell coreutils
[env]$ tree $MANPATH
/gnu/store/m3i9pga6rqfg342sajzww4gl8w51q6sz-profile/share/man
├── index.db ->
/gnu/store/pdh5nz4qkg5q243q5rgxkk1hnmaf1plq-manual-database/share/man/index.db
├── it -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/it
├── man1 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man1
├── man5 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man5
└── man8 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man8
--8<---------------cut here---------------end--------------->8---

Here, 'coreutils' is not "aware" of the parent 'man-db' and thus
MANPATH does not contains the expected 114 files.

--8<---------------cut here---------------start------------->8---
     $ guix shell coreutils man-db
[env]$ tree $MANPATH
[...]
4 directories, 114 files
--8<---------------cut here---------------end--------------->8---

Using an option, say '--transient', a new profile is created each time
but what this new profile contains would depends of the parent
profile.  Last, the manipulation of a "temporary" profile is done via
"guix shell" thus this "transient" profile should be manipulated via
"guix shell" and not "guix package" (or aliases as "guix install").
However, what could appear still annoying is the stack of environment:

  guix shell a
  guix shell b
  guix shell c

then remove the package 'a' from the environment requires to exit 2
environments and then re-create them.  Need more thoughts. :-)


Cheers,
simon

Hello simon; thank you for your input.

> The "incremental style" is a quick and dirty approach for creating a computational environment

If profiles are used, it can be exported to a manifest.

> Guix is functional by design and we should not compromise on that.

I want to reiterate that I am not proposing anything less FP than profiles.

Nested shells do not fit my use case because a long running process started in shell "a" does not become aware of packages installed on shell "b".

> thus this "transient" profile should be manipulated via "guix shell" and not "guix package"

I chose "guix install" for my proposal because it already has the functionality of adding a new generation. I do not think it would be good to change the semantics of nested shells.

I failed to mention earlier that I like the name "--transient"