| Message ID | 87v8iy59j1.fsf@gmail.com |
|---|---|
| Headers | show |
| Series | Add `guix index` subcommand | expand |
Hi Antoine, "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com> skribis: > Example of an indexation (fast method by default): > > $ guix index > Registering 133 packages ▕█████▎ ▏ > > > Example of a search: > > $ guix index search sqlite3 > sqlite@3.37.0 /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3 > sqlite@3.36.0 /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3 This is really nice! I finally got around to looking at the patches. The final version will be a single patch that adds code, tests, and documentation. I squashed the ‘guix index’ patches you posted, followed up with individual changes on top of that (those will have to be squashed eventually), and pushed the result in the ‘wip-guix-index’ branch. Lemme know what you think! Good news is “make check TESTS=tests/guix-index.sh” passes. I disabled tests for the ‘store’ method by default because it’s too expensive (it would take several minutes with my SSD). There were other things I wanted to fiddle with but I ran out of time :-) so I’ll take another look later. The main missing bit is a new node in the manual documenting it. Perhaps one feature that would be nice to have before we publish is pruning of database entries, so it doesn’t grow endlessly, but I’m not sure how to do that (based on whether a Directory entry actually exists?). Thanks! Ludo’.
Hello! Ludovic Courtès <ludo@gnu.org> skribis: > I squashed the ‘guix index’ patches you posted, followed up with > individual changes on top of that (those will have to be squashed > eventually), and pushed the result in the ‘wip-guix-index’ branch. > Lemme know what you think! I’ve pushed additional changes. There are a few more things I’d like to do before we can merge it: - [ ] 'guix index' renamed to 'guix locate' - [ ] 'search' action removed; option added to force reindexing - [ ] warning printed when database is too old - [ ] start anew when database is very old - [ ] documentation written - [ ] '--help' output made more concise We can think about fetching databases in a second iteration. Ludo’.
Hi, On dim., 21 mai 2023 at 23:45, Ludovic Courtès <ludo@gnu.org> wrote: > - [ ] 'guix index' renamed to 'guix locate' Well, I have some patches that rename to “guix file”. :-) Well, ’guix locale’ appears to me also fine. > - [ ] 'search' action removed; option added to force reindexing I have an unpolished stuff that do: guix file --index=TYPE guix file foo bar Somehow, “guix file <terms>” the default action (= search). And then some action with --action (as index or else). > - [ ] warning printed when database is too old > - [ ] start anew when database is very old > - [ ] documentation written I have started… but I am not very far. :-) > - [ ] '--help' output made more concise Maybe also remove some “debug” code, no? Cheers, simon
Hi, Simon Tournier <zimon.toutoune@gmail.com> skribis: > On dim., 21 mai 2023 at 23:45, Ludovic Courtès <ludo@gnu.org> wrote: > >> - [ ] 'guix index' renamed to 'guix locate' > > Well, I have some patches that rename to “guix file”. :-) > > Well, ’guix locale’ appears to me also fine. “locate” sounds better to me, notably because it’s a verb. >> - [ ] 'search' action removed; option added to force reindexing > > I have an unpolished stuff that do: > > guix file --index=TYPE > guix file foo bar > > Somehow, “guix file <terms>” the default action (= search). And then > some action with --action (as index or else). > >> - [ ] warning printed when database is too old >> - [ ] start anew when database is very old >> - [ ] documentation written > > I have started… but I am not very far. :-) Could you share what you have? (I didn’t expect feedback so I thought I’d end up hack on the stuff head-down. Now we need to synchronize. :-)) >> - [ ] '--help' output made more concise > > Maybe also remove some “debug” code, no? Done already. Thanks! Ludo’.
Hi Ludo, On mer., 24 mai 2023 at 16:40, Ludovic Courtès <ludo@gnu.org> wrote: >> Well, I have some patches that rename to “guix file”. :-) >> >> Well, ’guix locale’ appears to me also fine. I was meaning ’locate’ not ’locale’, sorry for the typo. > “locate” sounds better to me, notably because it’s a verb. Yeah, ’locate’ is better. Especially when one knows about the good ol’ locate program. ;-) >>> - [ ] 'search' action removed; option added to force reindexing >> >> I have an unpolished stuff that do: >> >> guix file --index=TYPE >> guix file foo bar >> >> Somehow, “guix file <terms>” the default action (= search). And then >> some action with --action (as index or else). >> >>> - [ ] warning printed when database is too old >>> - [ ] start anew when database is very old >>> - [ ] documentation written >> >> I have started… but I am not very far. :-) > > Could you share what you have? I will. Do you prefer a patch? Or that I push to the branch? > (I didn’t expect feedback so I thought I’d end up hack on the stuff > head-down. Now we need to synchronize. :-)) There is much to synchronize because I did few. :-) My concerns were: 1. about the name “guix index” and now it’s fixed by “guix locate” ;-) and 2. about the CLI; the default “guix locate <term>” should be the search action. The update action should be triggered with an option as “-i/--index=”. Cheers, simon
Hi, Simon Tournier <zimon.toutoune@gmail.com> skribis: > I will. Do you prefer a patch? Or that I push to the branch? Patch, please! > My concerns were: > > 1. about the name “guix index” and now it’s fixed by “guix locate” ;-) > > and 2. about the CLI; the default “guix locate <term>” should be the > search action. The update action should be triggered with an option as > “-i/--index=”. Ah yes, agreed! Ludo’.
Ludovic Courtès <ludo@gnu.org> skribis: > I’ve pushed additional changes. There are a few more things I’d like to > do before we can merge it: > > - [ ] 'guix index' renamed to 'guix locate' > - [ ] 'search' action removed; option added to force reindexing > - [ ] warning printed when database is too old > - [ ] start anew when database is very old > - [ ] documentation written > - [ ] '--help' output made more concise Hi! I’m pretty much done with the items above (I’ve pushed ‘wip-guix-index’ again). I’ll take another look and send the final patch here. Feedback welcome! Ludo’.
Hello, This introduces a new `guix index` subcommand (initiated by civodul). This is in charge of indexing guix packages to ease user lookups. There are 2 indexation methods: - `manifest`: fast but less efficient, expected to be used by local users - `store`: slow but more efficient, expected to be used on a guix build machine. Another service could be in charge of exposing a search package service to ease remote search of packages. ** Impact (positive) tl;dr This is an equivalent of `nix-index/nix-locate` or `apt-file search` cli we can find respectively in Nix or Debian's toolbox (unbeknownst of their implementation). This should help users transitioning from other distributions. Usually, people will want to look up for files or packages they are already using. They'd expect an easy way to find those back in guix. And we currently cannot do so easily. We can ask on #guix-devel irc (as someone mentions to me there) though that's not scalable nor really practical for non-irc users. ** How did I test/run it? I've been running both the: - `guix index` subcommand to index packages (with both methods in separated or not dbs). - `guix index search FILE` to actually search for packages present in the db I've inspected time and again the db through the `sqlite3` program after multiple runs too to ensure data were consistent. Data are consistent. ** Build? The makefile routine will build that subcommand too. ** Development details The backend technology is sqlite3. The command is in charge of migrating the data model (if any). That data model migration process is transparent for users. The data model is currently at version 3 because it underwent some migrations during its development process already. It's kept for dogfooding and examples reasons. The --version flag mentions the default backend locations and data model version: --8<---------------cut here---------------end--------------->8--- $ guix index --version Extension local cache database: - path: /home/tony/.cache/guix/index/db.sqlite - version: 3 guix index (GNU Guix) 1.4.0.3874-372b2 Copyright (C) 2023 the Guix authors License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. $ git log | head -1 commit 372b2b9660b8293eebd6280bb46a4ec07d4192a7 --8<---------------cut here---------------end--------------->8--- ** Current caveat ("help needed on test") The code works but... I can't make the integration test i wrote work. Hence my delay to send the overall patch (as the subcommand has been ready for a while). As it's been way too long already, I've posted it in the hope someone interested enough sees what's missing. Thanks in advance. What's happening during the test is that somehow the execution of the `guix index` subcommand triggers an exit code > 0 (error). Which, expectedly, stops the test immediately. I don't get why the subcommand exits that way and I don't see how to dig further unfortunately. Any clues would be very much appreciated. ** Use The subcommand is documented exhaustively. --8<---------------cut here---------------end--------------->8--- $ guix index --help Usage: guix index [OPTIONS...] [search FILE...] Without argument, indexes (package, file) relationships from the machine. This allows indexation with 2 methods, out of the local: - manifests: This is the fastest implementation with the caveat of indexing less packages. That'd be typically the use case of user local indexation. - store: This is slowest implementation. It discusses with the store daemon. That'd be typically the use case of building the largest db in one of the build farm node. With 'search FILE', search for packages installing FILE. Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default. See --db-path for customization. The valid values for OPTIONS are: -h, --help Display this help and exit -V, --version Display version information and exit --db-path=DIR Change default location of the cache db --method=METH Change default indexation method. By default it uses the local "manifests" (faster). It can also uses the local "store" (slower, typically on the farm build ci). The valid values for ARGS are: search FILE Search for packages installing the FILE (from cache db) <EMPTY> Without any argument, it index packages. This fills in the db cache using whatever indexation method is defined. Report bugs to: bug-guix@gnu.org. GNU Guix home page: <https://www.gnu.org/software/guix/> General help using Guix and GNU software: <https://guix.gnu.org/en/help/> --8<---------------cut here---------------end--------------->8--- Example of an indexation (fast method by default): --8<---------------cut here---------------end--------------->8--- $ guix index Registering 133 packages ▕█████▎ ▏ --8<---------------cut here---------------end--------------->8--- Example of a search: --8<---------------cut here---------------end--------------->8--- $ guix index search sqlite3 sqlite@3.37.0 /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3 sqlite@3.36.0 /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3 --8<---------------cut here---------------end--------------->8--- Note: inspired from zimoun's way of quoting (/me like those, thx ;) ** Development process (optional read) Note that this started as a subcommand (again from civodul's bootstrap code). Then as a good proposal from zimoun, this got simplified into a guix extension (awesome work zimoun ;). Fwiw, developing it as an extension felt way simpler for dev and run. Versus, having to setup my machines to develop with guix. It continued as a guix subcommand in the end as civodul proposed it that way instead. Sent from my MUA as git-send-email would not work... --8<---------------cut here---------------end--------------->8--- $ git send-email outgoing/0000-cover-letter.patch -a \ --to=guix-patches@gnu.org outgoing/0000-cover-letter.patch Can't exec "sh": No such file or directory at /gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real line 231. the editor exited uncleanly, aborting everything at /gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real line 252. --8<---------------cut here---------------end--------------->8--- Hoping this finds you well nonetheless. Cheers, -- tony / Antoine R. Dumont (@ardumont) ----------------------------------------------------------------- gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8 ===File ~/repo/public/guix/guix/outgoing/0000-cover-letter.patch=== From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001 From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com> Date: Sat, 18 Mar 2023 16:27:22 +0100 Subject: [PATCH core-updates 0/6] Add `guix index` subcommand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit *** BLURB HERE **** Antoine R. Dumont (@ardumont) (5): scripts-index: Transform `guix index` extension into a Guix script scripts-index: Store outputs alongside packages Makefile.am: Reference new script to compile Bootstrap tests for guix index subcommand Allow gcroot function to exceptionally ignore error Ludovic Courtès (1): index: Add initial implementation from civodul Makefile.am | 2 + guix/scripts/home.scm | 2 +- guix/scripts/index.scm | 595 +++++++++++++++++++++++++++++++++++++++++ guix/store/roots.scm | 10 +- tests/guix-index.sh | 73 +++++ tests/store-roots.scm | 7 +- 6 files changed, 686 insertions(+), 3 deletions(-) create mode 100644 guix/scripts/index.scm create mode 100755 tests/guix-index.sh base-commit: 962277fd4313f20c0e0333effbd88352c0a7d461