mbox

[bug#39258,v3,0/3] Package metadata cache for guix search

Message ID 20200327162654.18785-1-arunisaac@systemreboot.net
Headers show

Message

Arun Isaac March 27, 2020, 4:26 p.m. UTC 
Hi everyone,

This is v3 of my attempt to make guix search faster. In this version, I have
abandoned use of xapian. Instead I build a cache of the metadata of all
packages in a profile hook. Then, I use that cache to search and display
search results. This way, package guile modules are not loaded during guix
search.

Speedup is around 2x. Both measurements below are with a warm cache.

--8<---------------cut here---------------start------------->8---
$ time guix search inkscape

real	0m1.722s
user	0m1.776s
sys	0m0.097s
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ time /tmp/test/bin/guix search inkscape

real	0m0.749s
user	0m0.770s
sys	0m0.020s
--8<---------------cut here---------------end--------------->8---

This patchset does not affect the search API nor does it improve the relevance
of search results. If there is interest in this approach, I'll complete this
patchset properly. But, in the long run, I do think we should aim to get
xapian or the like for guix search. WDYT?

Unfortunately, generate-package-metadata-cache takes 43 seconds to build the
cache on my relatively slow computer. Performance should be better on other
people's machines.

Meanwhile, it would still be useful if someone built patchset v2 on their
machine and reported the time it took to build the xapian index.

* How to test this patchset

Apply patches and build as usual. Do a guix pull into a temporary profile.

$ ./pre-inst-env guix pull --url=$PWD --branch=the-name-of-the-branch-you-applied-patches-to -p /tmp/test

Then, run guix search from the built profile

$ /tmp/test/bin/guix search inkscape

Thanks!

Arun Isaac (3):
  guix: Generate package metadata cache.
  guix: Search package metadata cache.
  guix: Use package metadata cache for package search.

 gnu/packages.scm         |  88 +++++++++++++++++++++++++-
 guix/channels.scm        |  34 +++++++++-
 guix/packages.scm        |  32 ++++++++++
 guix/scripts/package.scm |   5 +-
 guix/ui.scm              | 132 ++++++++++++++++++++++++++++++++++++---
 5 files changed, 277 insertions(+), 14 deletions(-)

Comments

Ludovic Courtès April 5, 2020, 2:08 p.m. UTC | #1 
Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> skribis:

> This is v3 of my attempt to make guix search faster. In this version, I have
> abandoned use of xapian. Instead I build a cache of the metadata of all
> packages in a profile hook. Then, I use that cache to search and display
> search results. This way, package guile modules are not loaded during guix
> search.
>
> Speedup is around 2x. Both measurements below are with a warm cache.

Sorry for the delay!  Just to say that I like the approach, and I’ll
take a closer look once the release is out…

Thank you!

Ludo’.
Ludovic Courtès April 24, 2020, 9:05 p.m. UTC | #2 
Hi,

Arun Isaac <arunisaac@systemreboot.net> skribis:

> Speedup is around 2x. Both measurements below are with a warm cache.
>
> $ time guix search inkscape
>
> real	0m1.722s
> user	0m1.776s
> sys	0m0.097s
>
> $ time /tmp/test/bin/guix search inkscape
>
> real	0m0.749s
> user	0m0.770s
> sys	0m0.020s
>
> This patchset does not affect the search API nor does it improve the relevance
> of search results. If there is interest in this approach, I'll complete this
> patchset properly. But, in the long run, I do think we should aim to get
> xapian or the like for guix search. WDYT?
>
> Unfortunately, generate-package-metadata-cache takes 43 seconds to build the
> cache on my relatively slow computer. Performance should be better on other
> people's machines.

43 seconds is a lot.  How do these 43 seconds compare to current ‘guix
search’ on your computer?  (Both do roughly the same thing.)

Thanks,
Ludo’.