[bug#62264,core-updates,0/6] Add `guix index` subcommand

Hello,

This introduces a new `guix index` subcommand (initiated by
civodul). This is in charge of indexing guix packages to ease user
lookups. There are 2 indexation methods:
- `manifest`: fast but less efficient, expected to be used by local
  users
- `store`: slow but more efficient, expected to be used on a guix build
  machine. Another service could be in charge of exposing a search
  package service to ease remote search of packages.

** Impact (positive)

tl;dr This is an equivalent of `nix-index/nix-locate` or `apt-file
search` cli we can find respectively in Nix or Debian's toolbox
(unbeknownst of their implementation).

This should help users transitioning from other distributions. Usually,
people will want to look up for files or packages they are already
using. They'd expect an easy way to find those back in guix. And we
currently cannot do so easily. We can ask on #guix-devel irc (as someone
mentions to me there) though that's not scalable nor really practical
for non-irc users.

** How did I test/run it?

I've been running both the:
- `guix index` subcommand to index packages (with both methods in
  separated or not dbs).
- `guix index search FILE` to actually search for packages present in
  the db

I've inspected time and again the db through the `sqlite3` program after
multiple runs too to ensure data were consistent. Data are consistent.

** Build?

The makefile routine will build that subcommand too.

** Development details

The backend technology is sqlite3. The command is in charge of migrating
the data model (if any). That data model migration process is
transparent for users. The data model is currently at version 3 because
it underwent some migrations during its development process
already. It's kept for dogfooding and examples reasons.

The --version flag mentions the default backend locations and data model
version:
--8<---------------cut here---------------end--------------->8---
$ guix index --version
Extension local cache database:
- path: /home/tony/.cache/guix/index/db.sqlite
- version: 3

guix index (GNU Guix) 1.4.0.3874-372b2
Copyright (C) 2023 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ git log | head -1
commit 372b2b9660b8293eebd6280bb46a4ec07d4192a7
--8<---------------cut here---------------end--------------->8---

** Current caveat ("help needed on test")

The code works but...

I can't make the integration test i wrote work. Hence my delay to send
the overall patch (as the subcommand has been ready for a while). As
it's been way too long already, I've posted it in the hope someone
interested enough sees what's missing. Thanks in advance.

What's happening during the test is that somehow the execution of the
`guix index` subcommand triggers an exit code > 0 (error). Which,
expectedly, stops the test immediately. I don't get why the subcommand
exits that way and I don't see how to dig further unfortunately. Any
clues would be very much appreciated.

** Use

The subcommand is documented exhaustively.

--8<---------------cut here---------------end--------------->8---
$ guix index --help
Usage: guix index [OPTIONS...] [search FILE...]
Without argument, indexes (package, file) relationships from the machine.
This allows indexation with 2 methods, out of the local:

- manifests: This is the fastest implementation with the caveat of indexing
less packages. That'd be typically the use case of user local indexation.

- store: This is slowest implementation. It discusses with the store
daemon. That'd be typically the use case of building the largest db in one of
the build farm node.

With 'search FILE', search for packages installing FILE.

Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default.
See --db-path for customization.

The valid values for OPTIONS are:

  -h, --help          Display this help and exit
  -V, --version       Display version information and exit
  --db-path=DIR       Change default location of the cache db

  --method=METH       Change default indexation method. By default it uses the
                      local "manifests" (faster). It can also uses the local
                      "store" (slower, typically on the farm build ci).
The valid values for ARGS are:

  search FILE     Search for packages installing the FILE (from cache db)

  <EMPTY>         Without any argument, it index packages. This fills in the
                  db cache using whatever indexation method is defined.
Report bugs to: bug-guix@gnu.org.
GNU Guix home page: <https://www.gnu.org/software/guix/>
General help using Guix and GNU software: <https://guix.gnu.org/en/help/>
--8<---------------cut here---------------end--------------->8---

Example of an indexation (fast method by default):

--8<---------------cut here---------------end--------------->8---
$ guix index
Registering 133 packages  ▕█████▎                                             ▏
--8<---------------cut here---------------end--------------->8---

Example of a search:

--8<---------------cut here---------------end--------------->8---
$ guix index search sqlite3
sqlite@3.37.0        /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3
sqlite@3.36.0        /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3
--8<---------------cut here---------------end--------------->8---

Note: inspired from zimoun's way of quoting (/me like those, thx ;)

** Development process (optional read)

Note that this started as a subcommand (again from civodul's bootstrap
code).  Then as a good proposal from zimoun, this got simplified into a
guix extension (awesome work zimoun ;). Fwiw, developing it as an
extension felt way simpler for dev and run. Versus, having to setup my
machines to develop with guix. It continued as a guix subcommand in the
end as civodul proposed it that way instead.

Sent from my MUA as git-send-email would not work...
--8<---------------cut here---------------end--------------->8---

$ git send-email outgoing/0000-cover-letter.patch -a \
      --to=guix-patches@gnu.org
outgoing/0000-cover-letter.patch
Can't exec "sh": No such file or directory at
/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real
line 231.
the editor exited uncleanly, aborting everything at
/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real
line 252.
--8<---------------cut here---------------end--------------->8---

Hoping this finds you well nonetheless.

Cheers,
--
tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------
gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8


===File ~/repo/public/guix/guix/outgoing/0000-cover-letter.patch===
From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Sat, 18 Mar 2023 16:27:22 +0100
Subject: [PATCH core-updates 0/6] Add `guix index` subcommand
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

*** BLURB HERE ****

Antoine R. Dumont (@ardumont) (5):
  scripts-index: Transform `guix index` extension into a Guix script
  scripts-index: Store outputs alongside packages
  Makefile.am: Reference new script to compile
  Bootstrap tests for guix index subcommand
  Allow gcroot function to exceptionally ignore error

Ludovic Courtès (1):
  index: Add initial implementation from civodul

 Makefile.am            |   2 +
 guix/scripts/home.scm  |   2 +-
 guix/scripts/index.scm | 595 +++++++++++++++++++++++++++++++++++++++++
 guix/store/roots.scm   |  10 +-
 tests/guix-index.sh    |  73 +++++
 tests/store-roots.scm  |   7 +-
 6 files changed, 686 insertions(+), 3 deletions(-)
 create mode 100644 guix/scripts/index.scm
 create mode 100755 tests/guix-index.sh


base-commit: 962277fd4313f20c0e0333effbd88352c0a7d461
  

Hi Antoine,

"Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
skribis:

> Example of an indexation (fast method by default):
>
> $ guix index
> Registering 133 packages  ▕█████▎                                             ▏
>
>
> Example of a search:
>
> $ guix index search sqlite3
> sqlite@3.37.0        /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3
> sqlite@3.36.0        /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3

This is really nice!  I finally got around to looking at the patches.
The final version will be a single patch that adds code, tests, and
documentation.

I squashed the ‘guix index’ patches you posted, followed up with
individual changes on top of that (those will have to be squashed
eventually), and pushed the result in the ‘wip-guix-index’ branch.
Lemme know what you think!

Good news is “make check TESTS=tests/guix-index.sh” passes.  I disabled
tests for the ‘store’ method by default because it’s too expensive (it
would take several minutes with my SSD).

There were other things I wanted to fiddle with but I ran out of time
:-) so I’ll take another look later.

The main missing bit is a new node in the manual documenting it.

Perhaps one feature that would be nice to have before we publish is
pruning of database entries, so it doesn’t grow endlessly, but I’m not
sure how to do that (based on whether a Directory entry actually
exists?).

Thanks!

Ludo’.
  

Hello!

Ludovic Courtès <ludo@gnu.org> skribis:

> I squashed the ‘guix index’ patches you posted, followed up with
> individual changes on top of that (those will have to be squashed
> eventually), and pushed the result in the ‘wip-guix-index’ branch.
> Lemme know what you think!

I’ve pushed additional changes.  There are a few more things I’d like to
do before we can merge it:

  - [ ] 'guix index' renamed to 'guix locate'
  - [ ] 'search' action removed; option added to force reindexing
  - [ ] warning printed when database is too old
  - [ ] start anew when database is very old
  - [ ] documentation written
  - [ ] '--help' output made more concise

We can think about fetching databases in a second iteration.

Ludo’.
  

Hi,

On dim., 21 mai 2023 at 23:45, Ludovic Courtès <ludo@gnu.org> wrote:

>   - [ ] 'guix index' renamed to 'guix locate'

Well, I have some patches that rename to “guix file”. :-)

Well, ’guix locale’ appears to me also fine. 

>   - [ ] 'search' action removed; option added to force reindexing

I have an unpolished stuff that do:

    guix file --index=TYPE
    guix file foo bar

Somehow, “guix file <terms>” the default action (= search).  And then
some action with --action (as index or else).

>   - [ ] warning printed when database is too old
>   - [ ] start anew when database is very old
>   - [ ] documentation written

I have started… but I am not very far. :-)

>   - [ ] '--help' output made more concise

Maybe also remove some “debug” code, no?


Cheers,
simon
  

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On dim., 21 mai 2023 at 23:45, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>   - [ ] 'guix index' renamed to 'guix locate'
>
> Well, I have some patches that rename to “guix file”. :-)
>
> Well, ’guix locale’ appears to me also fine. 

“locate” sounds better to me, notably because it’s a verb.

>>   - [ ] 'search' action removed; option added to force reindexing
>
> I have an unpolished stuff that do:
>
>     guix file --index=TYPE
>     guix file foo bar
>
> Somehow, “guix file <terms>” the default action (= search).  And then
> some action with --action (as index or else).
>
>>   - [ ] warning printed when database is too old
>>   - [ ] start anew when database is very old
>>   - [ ] documentation written
>
> I have started… but I am not very far. :-)

Could you share what you have?

(I didn’t expect feedback so I thought I’d end up hack on the stuff
head-down.  Now we need to synchronize.  :-))

>>   - [ ] '--help' output made more concise
>
> Maybe also remove some “debug” code, no?

Done already.

Thanks!

Ludo’.
  

Hi Ludo,

On mer., 24 mai 2023 at 16:40, Ludovic Courtès <ludo@gnu.org> wrote:

>> Well, I have some patches that rename to “guix file”. :-)
>>
>> Well, ’guix locale’ appears to me also fine.

I was meaning ’locate’ not ’locale’, sorry for the typo.

> “locate” sounds better to me, notably because it’s a verb.

Yeah, ’locate’ is better.  Especially when one knows about the good ol’
locate program. ;-)


>>>   - [ ] 'search' action removed; option added to force reindexing
>>
>> I have an unpolished stuff that do:
>>
>>     guix file --index=TYPE
>>     guix file foo bar
>>
>> Somehow, “guix file <terms>” the default action (= search).  And then
>> some action with --action (as index or else).
>>
>>>   - [ ] warning printed when database is too old
>>>   - [ ] start anew when database is very old
>>>   - [ ] documentation written
>>
>> I have started… but I am not very far. :-)
>
> Could you share what you have?

I will.  Do you prefer a patch?  Or that I push to the branch?


> (I didn’t expect feedback so I thought I’d end up hack on the stuff
> head-down.  Now we need to synchronize.  :-))

There is much to synchronize because I did few. :-)

My concerns were:

 1. about the name “guix index” and now it’s fixed by “guix locate” ;-)

 and 2. about the CLI; the default “guix locate <term>” should be the
 search action.  The update action should be triggered with an option as
 “-i/--index=”.


Cheers,
simon
  

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> I will.  Do you prefer a patch?  Or that I push to the branch?

Patch, please!

> My concerns were:
>
>  1. about the name “guix index” and now it’s fixed by “guix locate” ;-)
>
>  and 2. about the CLI; the default “guix locate <term>” should be the
>  search action.  The update action should be triggered with an option as
>  “-i/--index=”.

Ah yes, agreed!

Ludo’.
  

Ludovic Courtès <ludo@gnu.org> skribis:

> I’ve pushed additional changes.  There are a few more things I’d like to
> do before we can merge it:
>
>   - [ ] 'guix index' renamed to 'guix locate'
>   - [ ] 'search' action removed; option added to force reindexing
>   - [ ] warning printed when database is too old
>   - [ ] start anew when database is very old
>   - [ ] documentation written
>   - [ ] '--help' output made more concise

Hi!  I’m pretty much done with the items above (I’ve pushed
‘wip-guix-index’ again).  I’ll take another look and send the final
patch here.

Feedback welcome!

Ludo’.
  

Hello!

Here is the “camera-ready” version of the new ‘guix locate’ command
(formerly ‘guix index’) that Antoine and myself have worked on.
I think it’s ready to go.

This version relies only on local knowledge, which means you can
only find packages already in your store.  In a future iteration,
we’ll add an optional mechanism to download the database from
elsewhere.

Thoughts?

Ludo’.

Ludovic Courtès (3):
  store: Tolerate non-existent GC root directories.
  Add 'guix locate'.
  DRAFT news: Add entry for 'guix locate'.

 Makefile.am             |   2 +
 doc/guix.texi           | 118 ++++++++
 etc/news.scm            |  17 ++
 guix/scripts/locate.scm | 657 ++++++++++++++++++++++++++++++++++++++++
 guix/store/roots.scm    |  12 +-
 po/guix/POTFILES.in     |   1 +
 tests/guix-locate.sh    |  72 +++++
 tests/store-roots.scm   |  18 +-
 8 files changed, 892 insertions(+), 5 deletions(-)
 create mode 100644 guix/scripts/locate.scm
 create mode 100755 tests/guix-locate.sh


base-commit: e8f9fb3e03ea8fee0e13f13706a6b16414f74a7b