| Message ID | 20220122104757.1837-1-ludo@gnu.org |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [bug#53439] doc: Document search paths. | expand |
| Context | Check | Description |
|---|---|---|
| cbaines/comparison | success | View comparision |
| cbaines/git branch | success | View Git branch |
| cbaines/applying patch | success | View Laminar job |
| cbaines/issue | success | View issue |
Ludovic Courtès schreef op za 22-01-2022 om 11:47 [+0100]: > +What this @code{native-search-paths} field says is that, when the > +@code{python} package is used, the @env{GUIX_PYTHONPATH} environment > +variable must be defined to include all the > +@file{lib/python/3.9/site-packages} sub-directories encountered in its > +environment. (The @code{native-} bit means that, if we are in a > +cross-compilation environment, only native inputs may be added to the > +search path.) In the NumPy example above, the profile where This bit about inputs/native-inputs seems a little imprecise: (package (name "foobarify") [...] (inputs guile) (propagated-inputs ;; TODO: would be nice to unpropagate (list guile-foo guile-bar guile-ify)) ;; "foobarify" allows running arbitrary Guile scripts, ;; so allow the user to use arbitrary Guile libraries ;; from the environment in their scripts at will. (native-search-paths [GUILE_LOAD_PATH, GUILE_LOAD_COMPILED_PATH, ...])) $ [ guix cross-compile a virtual image with foobarify in the system profile] IIUC, guile-foo, guile-bar and guile-ify will be included in the GUILE_LOAD_PATH/GUILE_LOAD_COMPILED_PATH of the system profile, even though we are cross-compiling and they are 'inputs' and not 'native- inputs' of 'foobarify' ... Greetings, Maxime.
Ludovic Courtès schreef op za 22-01-2022 om 11:47 [+0100]:
> +``belongs'' to Python, not NumPy: Python actually reads the value of
AFAIK Texinfo supports ‘curly’ quotes; there are a few instances of
‘ and ’ in doc/guix.texi. There does not appear to be any need for
transliteration.
Greetings,
Maxime.
Ludovic Courtès schreef op za 22-01-2022 om 11:47 [+0100]: > * doc/guix.texi (package Reference): Link to "Search Paths". > (Build Phases): Mention 'set-paths' phase. > (Search Paths): New node. > --- > doc/guix.texi | 184 +++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 183 insertions(+), 1 deletion(-) > > Hi! > > Here's a long overdue piece of documentation. > > Thoughts? > > Ludo'. Aside from a remark about 'native-' vs. non-native it LGTM. Greetings, Maxime.
Hi Maxime, Maxime Devos <maximedevos@telenet.be> skribis: > Ludovic Courtès schreef op za 22-01-2022 om 11:47 [+0100]: >> +What this @code{native-search-paths} field says is that, when the >> +@code{python} package is used, the @env{GUIX_PYTHONPATH} environment >> +variable must be defined to include all the >> +@file{lib/python/3.9/site-packages} sub-directories encountered in its >> +environment. (The @code{native-} bit means that, if we are in a >> +cross-compilation environment, only native inputs may be added to the >> +search path.) In the NumPy example above, the profile where > > This bit about inputs/native-inputs seems a little imprecise: > > (package > (name "foobarify") > [...] > (inputs guile) > (propagated-inputs > ;; TODO: would be nice to unpropagate > (list guile-foo guile-bar guile-ify)) > ;; "foobarify" allows running arbitrary Guile scripts, > ;; so allow the user to use arbitrary Guile libraries > ;; from the environment in their scripts at will. > (native-search-paths > [GUILE_LOAD_PATH, GUILE_LOAD_COMPILED_PATH, ...])) > > $ [ guix cross-compile a virtual image with foobarify in the system > profile] > > IIUC, guile-foo, guile-bar and guile-ify will be included in the > GUILE_LOAD_PATH/GUILE_LOAD_COMPILED_PATH of the system profile, even > though we are cross-compiling and they are 'inputs' and not 'native- > inputs' of 'foobarify' ... I think the key bit in the parenthetical sentence you refer to is “if we are in a cross-compilation environment”. If we are *not* cross-compiling, then both inputs and native inputs are included in the search path. Is that what you mean? I think we need a section about cross-compilation though, and that will allow us to clarify these things. I didn’t want to muddy the main message in cross-compilation considerations here. WDYT? Thanks for reviewing! Ludo’.
Hi Ludovic, Ludovic Courtès <ludo@gnu.org> writes: > * doc/guix.texi (package Reference): Link to "Search Paths". > (Build Phases): Mention 'set-paths' phase. > (Search Paths): New node. > --- > doc/guix.texi | 184 +++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 183 insertions(+), 1 deletion(-) [...] > +@item @code{file-type} (default: @code{'directory}) > +The type of file being matched. > + > +In the libxml2 example above, we would match regular files. Here I am left wondering what other valid values (other than 'regular or 'directory) may be? I remember trying to find such answer, and it was not obvious (I can't remember where the answer lies in the Guile manual as I type this!). > +@item @code{file-pattern} (default: @code{#f}) > +When true, this is a regular expression specifying files to be > matched I'd replace "When true", which I find confusing, by 'Optional' or similar. > +@emph{within} the sub-directories specified by the @code{files} field. > + > +Again, the libxml2 example shows a situation where this is needed. > +@end table > +@end deftp > + > +How do you turn search path specifications on one hand and a bunch of > +directories on the other hand in a set of environment variable > +definitions? That's the job of @code{evaluate-search-paths}. > + > +@deffn {Scheme Procedure} evaluate-search-paths @var{search-paths} @ > + @var{directories} [@var{getenv}] > +Evaluate @var{search-paths}, a list of search-path specifications, for > +@var{directories}, a list of directory names, and return a list of > +specification/value pairs. Use @var{getenv} to determine the current > +settings and report only settings not already effective. > +@end deffn > + > +The @code{(guix profiles)} provides a higher-level helper procedure, > +@code{load-profile}, that sets the environment variables of a profile. > + > @node The Store > @section The Store This LGTM, taking the comments from Maxime into account. Thank you! Maxim
Hi, Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis: > Ludovic Courtès <ludo@gnu.org> writes: > >> * doc/guix.texi (package Reference): Link to "Search Paths". >> (Build Phases): Mention 'set-paths' phase. >> (Search Paths): New node. >> --- >> doc/guix.texi | 184 +++++++++++++++++++++++++++++++++++++++++++++++++- >> 1 file changed, 183 insertions(+), 1 deletion(-) > > [...] > >> +@item @code{file-type} (default: @code{'directory}) >> +The type of file being matched. >> + >> +In the libxml2 example above, we would match regular files. > > Here I am left wondering what other valid values (other than 'regular or > 'directory) may be? I remember trying to find such answer, and it was > not obvious (I can't remember where the answer lies in the Guile manual > as I type this!). Oh right, these are the symbols returned by ‘stat:kind’. I’ll clarify that. >> +@item @code{file-pattern} (default: @code{#f}) >> +When true, this is a regular expression specifying files to be >> matched > > I'd replace "When true", which I find confusing, by 'Optional' or > similar. “When true” is to be taken literally: if it has truth value, in the Scheme sense. But “Unless @code{#f}” might be clearer? (“Optional” sounds confusing to me because there has to be a value, default or not.) Thanks for taking a look! Ludo’.
Pushed as 3878cf96183a27d63d374378d3044fd091231d0c with a couple of tweaks that you suggested, though now I realize I forgot the “when true” bit. Ludo’.
> Pushed as 3878cf96183a27d63d374378d3044fd091231d0c with a couple of > tweaks that you suggested, though now I realize I forgot the “when true” > bit. Pushed a followup as a00dff3ac113722a709dbe97a727777b3739a5c1. Ludo’.
Hi Ludovic, Ludovic Courtès <ludo@gnu.org> writes: > Hi, > > Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis: > >> Ludovic Courtès <ludo@gnu.org> writes: >> >>> * doc/guix.texi (package Reference): Link to "Search Paths". >>> (Build Phases): Mention 'set-paths' phase. >>> (Search Paths): New node. >>> --- >>> doc/guix.texi | 184 +++++++++++++++++++++++++++++++++++++++++++++++++- >>> 1 file changed, 183 insertions(+), 1 deletion(-) >> >> [...] >> >>> +@item @code{file-type} (default: @code{'directory}) >>> +The type of file being matched. >>> + >>> +In the libxml2 example above, we would match regular files. >> >> Here I am left wondering what other valid values (other than 'regular or >> 'directory) may be? I remember trying to find such answer, and it was >> not obvious (I can't remember where the answer lies in the Guile manual >> as I type this!). > > Oh right, these are the symbols returned by ‘stat:kind’. I’ll clarify > that. > >>> +@item @code{file-pattern} (default: @code{#f}) >>> +When true, this is a regular expression specifying files to be >>> matched >> >> I'd replace "When true", which I find confusing, by 'Optional' or >> similar. > > “When true” is to be taken literally: if it has truth value, in the > Scheme sense. But “Unless @code{#f}” might be clearer? > > (“Optional” sounds confusing to me because there has to be a value, > default or not.) From the user point of view, specifying it is really optional though. Yours is more correct but perhaps confusing to those not knowing a string is truthy in Guile (and I'd argue it takes attention away from what is important here). What I had on mind was: "An optional regular expression to specify which files should be matched, based on their base name." or similar. (optional because if you don't specify it defaults to #f, which means "no added behavior"). I've used that approach when describing optional fields of service configurations in the past. My 2 cents, and either way I'm fine with it :-) Thanks, Maxim
Hello, Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis: > Ludovic Courtès <ludo@gnu.org> writes: [...] >> “When true” is to be taken literally: if it has truth value, in the >> Scheme sense. But “Unless @code{#f}” might be clearer? >> >> (“Optional” sounds confusing to me because there has to be a value, >> default or not.) > > From the user point of view, specifying it is really optional though. Yes, that’s what the “(default: @code{#f})” bit conveys. > Yours is more correct but perhaps confusing to those not knowing a > string is truthy in Guile (and I'd argue it takes attention away from > what is important here). > > What I had on mind was: > > "An optional regular expression to specify which files should be > matched, based on their base name." or similar. > > (optional because if you don't specify it defaults to #f, which means > "no added behavior"). > > I've used that approach when describing optional fields of service > configurations in the past. OK. The way I see it, that it’s optional is already implied by the fact it has a default value. Regarding the truth value, I ended up in a00dff3ac113722a709dbe97a727777b3739a5c1 with hopefully clearer wording: @item @code{file-pattern} (default: @code{#f}) This must be either @code{#f} or a regular expression specifying files to be matched @emph{within} the sub-directories specified by the @code{files} field. I guess we’re nitpicking :-) but I like to have, on one hand, text in natural language with examples that conveys ideas in an informal way, and on the other hand, reference material (@deftp, @deffn, etc.) that’s rigorous. Ludo’.
Hello! [...] > Regarding the truth value, I ended up in > a00dff3ac113722a709dbe97a727777b3739a5c1 with hopefully clearer wording: > > @item @code{file-pattern} (default: @code{#f}) > This must be either @code{#f} or a regular expression specifying > files to be matched @emph{within} the sub-directories specified by the > @code{files} field. That's explicit and avoid the issue altogether, I like it! > I guess we’re nitpicking :-) but I like to have, on one hand, text in > natural language with examples that conveys ideas in an informal way, > and on the other hand, reference material (@deftp, @deffn, etc.) that’s > rigorous. I see; that makes sense. Thanks for taking the time to adjust the final result! Maxim
diff --git a/doc/guix.texi b/doc/guix.texi index 912a8e3c5a..15d887576d 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -279,6 +279,7 @@ Programming Interface * Build Systems:: Specifying how packages are built. * Build Phases:: Phases of the build process of a package. * Build Utilities:: Helpers for your package definitions and more. +* Search Paths:: Declaring search path environment variables. * The Store:: Manipulating the package store. * Derivations:: Low-level interface to package derivations. * The Store Monad:: Purely functional interface to the store. @@ -6859,6 +6860,7 @@ package definitions. * Build Systems:: Specifying how packages are built. * Build Phases:: Phases of the build process of a package. * Build Utilities:: Helpers for your package definitions and more. +* Search Paths:: Declaring search path environment variables. * The Store:: Manipulating the package store. * Derivations:: Low-level interface to package derivations. * The Store Monad:: Purely functional interface to the store. @@ -7239,7 +7241,8 @@ Outputs}, for typical uses of additional outputs. @item @code{native-search-paths} (default: @code{'()}) @itemx @code{search-paths} (default: @code{'()}) A list of @code{search-path-specification} objects describing -search-path environment variables honored by the package. +search-path environment variables honored by the package. @xref{Search +Paths}, for more on search path specifications. @item @code{replacement} (default: @code{#f}) This must be either @code{#f} or a package object that will be used as a @@ -8900,6 +8903,10 @@ standard list of phases. For @code{gnu-build-system}, the main build phases are the following: @table @code +@item set-paths +Define search path environment variables for all the input packages, +including @env{PATH} (@pxref{Search Paths}). + @item unpack Unpack the source tarball, and change the current directory to the extracted source tree. If the source is actually a directory, copy it @@ -9393,6 +9400,181 @@ executable files to be installed: @c TODO: Add more examples. +@node Search Paths +@section Search Paths + +@cindex search path +Many programs and libraries look for input data in a @dfn{search path}, +a list of directories: shells like Bash look for executables in the +command search path, a C compiler looks for @file{.h} files in its +header search path, and the Python interpreter looks for @file{.py} +files in its search path, the spell checker has a search path for +dictionaries, and so on. + +Search paths can usually be defined or overridden @i{via} environment +variables (@pxref{Environment Variables,,, libc, The GNU C Library +Reference Manual}). For example, the search paths mentioned above can +be changed by defining the @env{PATH}, @env{C_INCLUDE_PATH}, +@env{PYTHONPATH} (or @env{GUIX_PYTHONPATH}), and @env{DICPATH} +environment variables---you know, all these something-PATH variables +that you need to get right or things ``won't be found''. + +You may have noticed from the command line that Guix ``knows'' which +search path environment variables should be defined, and how. When you +install packages in your default profile, the file +@file{~/.guix-profile/etc/profile} is created, which you can ``source'' +from the shell to set those variables. Likewise, if you ask +@command{guix shell} to create an environment containing Python and +NumPy, a Python library, and if you pass it the @option{--search-paths} +option, it will tell you about @env{PATH} and @env{GUIX_PYTHONPATH} +(@pxref{Invoking guix shell}): + +@example +$ guix shell python python-numpy --pure --search-paths +export PATH="/gnu/store/@dots{}-profile/bin" +export GUIX_PYTHONPATH="/gnu/store/@dots{}-profile/lib/python3.9/site-packages" +@end example + +When you omit @option{--search-paths}, it defines these environment +variables right away, such that Python can readily find NumPy: + +@example +$ guix shell python python-numpy -- python3 +Python 3.9.6 (default, Jan 1 1970, 00:00:01) +[GCC 10.3.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import numpy +>>> numpy.version.version +'1.20.3' +@end example + +For this to work, the definition of the @code{python} package +@emph{declares} the search path it cares about and its associated +environment variable, @env{GUIX_PYTHONPATH}. It looks like this: + +@lisp +(package + (name "python") + (version "3.9.9") + ;; some fields omitted... + (native-search-paths + (list (search-path-specification + (variable "GUIX_PYTHONPATH") + (files (list "lib/python/3.9/site-packages")))))) +@end lisp + +What this @code{native-search-paths} field says is that, when the +@code{python} package is used, the @env{GUIX_PYTHONPATH} environment +variable must be defined to include all the +@file{lib/python/3.9/site-packages} sub-directories encountered in its +environment. (The @code{native-} bit means that, if we are in a +cross-compilation environment, only native inputs may be added to the +search path.) In the NumPy example above, the profile where +@code{python} appears contains exactly one such sub-directory, and +@env{GUIX_PYTHONPATH} is set to that. When there are several +@file{lib/python/3.9/site-packages}---this is the case in package build +environments---they are all added to @env{GUIX_PYTHONPATH}, separated by +colons (@code{:}). + +@quotation Note +Notice that @env{GUIX_PYTHONPATH} is specified as part of the definition +of the @code{python} package, and @emph{not} as part of that of +@code{python-numpy}. This is because this environment variable +``belongs'' to Python, not NumPy: Python actually reads the value of +that variable and honors it. + +Corollary: if you create a profile that does not contain @code{python}, +@code{GUIX_PYTHONPATH} will @emph{not} be defined, even if it contains +packages that provide @file{.py} files: + +@example +$ guix shell python-numpy --search-paths --pure +export PATH="/gnu/store/@dots{}-profile/bin" +@end example + +This makes a lot of sense if we look at this profile in isolation: no +software in this profile would read @env{GUIX_PYTHONPATH}. +@end quotation + +Of course, there are many variations on that theme: some packages honor +more than one search path, some use separators other than colon, some +accumulate several directories in their search path, and so on. A more +complex example is the search path of libxml2: the value of the +@env{XML_CATALOG_FILES} environment variable is space-separated, it must +contain a list of @file{catalog.xml} files (not directories), which are +to be found in @file{xml} sub-directories---nothing less. The search +path specification looks like this: + +@lisp +(define libxml2 + (package + (name "libxml2") + ;; some fields omitted + (native-search-paths + (list (search-path-specification + (variable "XML_CATALOG_FILES") + (separator " ") + (files '("xml")) + (file-pattern "^catalog\\.xml$") + (file-type 'regular)))))) +@end lisp + +Worry not, search path specifications are usually not this tricky. + +The @code{(guix search-paths)} module defines the data type of search +path specifications and a number of helper procedures. Below is the +reference of search path specifications. + +@deftp {Data Type} search-path-specification +The data type for search path specifications. + +@table @asis +@item @code{variable} +The name of the environment variable for this search path (a string). + +@item @code{files} +The list of sub-directories (strings) that should be added to the search +path. + +@item @code{separator} (default: @code{":"}) +The string used to separate search path components. + +As a special case, a @code{separator} value of @code{#f} specifies a +``single-component search path''---in other words, a search path that +cannot contain more than one element. This is useful in some cases, +such as the @code{SSL_CERT_DIR} variable (honored by OpenSSL, cURL, and +a few other packages) or the @code{ASPELL_DICT_DIR} variable (honored by +the GNU Aspell spell checker), both of which must point to a single +directory. + +@item @code{file-type} (default: @code{'directory}) +The type of file being matched. + +In the libxml2 example above, we would match regular files. + +@item @code{file-pattern} (default: @code{#f}) +When true, this is a regular expression specifying files to be matched +@emph{within} the sub-directories specified by the @code{files} field. + +Again, the libxml2 example shows a situation where this is needed. +@end table +@end deftp + +How do you turn search path specifications on one hand and a bunch of +directories on the other hand in a set of environment variable +definitions? That's the job of @code{evaluate-search-paths}. + +@deffn {Scheme Procedure} evaluate-search-paths @var{search-paths} @ + @var{directories} [@var{getenv}] +Evaluate @var{search-paths}, a list of search-path specifications, for +@var{directories}, a list of directory names, and return a list of +specification/value pairs. Use @var{getenv} to determine the current +settings and report only settings not already effective. +@end deffn + +The @code{(guix profiles)} provides a higher-level helper procedure, +@code{load-profile}, that sets the environment variables of a profile. + @node The Store @section The Store