| Message ID | 20210715073328.212123-1-zimon.toutoune@gmail.com |
|---|---|
| Headers | show |
| Series | DRAFT "guix search" performances | expand |
Hi! zimoun <zimon.toutoune@gmail.com> skribis: > This is an attempt to improve the performance of "guix search". It is still > half baked but it allows to discuss further the idea about expanding the > current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-) Thanks for resuming this discussion. :-) > From my understanding, the issue that 'package-relevance' accepts a 'package' > (and then all the chain until displaying) and 'fold-avaibale-packages' does > not return a package. Well, I do not know; especially where to put something > similiar to 'read-package-from'. Yeah that’s annoying. Perhaps we need <proto-package> or <package-metadata>. With some trickery we could have record type inheritance or something, maybe. Dunno. It would be good if we could arrange so that ‘fold-available-packages’ doesn’t allocate anything though. > Let compare only for cold cache and time this cache building (Guix 7db8fd6): > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) > > real 0m28,848s > user 0m1,481s > sys 0m0,252s > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache)) > > real 0m40,279s > user 0m1,582s > sys 0m0,232s > > It seems longer but compared to the time of "guix pull" completion, it seems > acceptable. Both the initial timing and the target are waaay too much. :-/ On my i7 laptop I have: --8<---------------cut here---------------start------------->8--- $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t.cache")' real 0m20.738s user 0m44.413s sys 0m0.341s --8<---------------cut here---------------end--------------->8--- It’s CPU-bound; we should probably start by optimizing that. In Guile 3.0.7 there was a change that improved this noticeably: https://git.savannah.gnu.org/cgit/guile.git/commit/?id=05614f792bfabbc33798863edd0bb67c744e9299 We should prolly look for similar optimization opportunities in the assembler… > Let compare for some queries: > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix search game | recsel -C -P name | wc -l > 371 > > real 0m7,561s > user 0m3,525s > sys 0m0,391s I think you should run: time guix search game > /dev/null otherwise Bash’s built-in ‘time’ shows the wall-clock time of the whole pipeline, and the processing time of ‘recsel’ is probably not negligible here. [...] > Last, two minor remarks about previous comments. > > 1. Ludo commented: > > > Therefore the cache '/lib/guix/package.cache' contains more > > information. > > This breaks the binary interface, so we’ll have to analyze the impact of > such a change and devise a strategy. > > <http://issues.guix.gnu.org/39258#93> > > and after some checking, this should be fine, IIUC. The '--news' is ok > because of '#:allow-other-keys'. And other parts are modified accordingly. > Guix revision N creates a cache that Guix revision N+1 will read but it should > not be an issue; see 'inferior-available-packages'. > > 2. And Ludo wrote: > > I realize the other cache also has that problem, but it would be nice to add a > version tag to the cache. Basically emit something like: > > (package-metadata-cache (version 0) VECTOR …) > > instead of just: > > (VECTOR …) > > <http://issues.guix.gnu.org/39258#93> > > which is, after discussions, not necessary. Versioning does not make sense > here because the cache is read by the Guix which generates it. Therefore, > specify a version is extraneous here. I confirm! :-) Thanks, Ludo’.
Hi Ludo, On Fri, 23 Jul 2021 at 17:43, Ludovic Courtès <ludo@gnu.org> wrote: > > From my understanding, the issue that 'package-relevance' accepts a 'package' > > (and then all the chain until displaying) and 'fold-avaibale-packages' does > > not return a package. Well, I do not know; especially where to put something > > similiar to 'read-package-from'. > > Yeah that’s annoying. Perhaps we need <proto-package> or > <package-metadata>. With some trickery we could have record type > inheritance or something, maybe. Dunno. > > It would be good if we could arrange so that ‘fold-available-packages’ > doesn’t allocate anything though. It does not allocate, the allocation is done by 'find-packages-by-description'. Well, I think v4/v3 [0] is the direction to follow. Therefore, I would revisit this version and try to address two of Ludo's comments [1] and the other ones in v3. BTW, I have not investigated from where the slowness comes: - allocation - garbage collection - '(module-ref (resolve-interface module) symbol)' because I have been a bit disappointed by the performance of this v6. 0: <http://issues.guix.gnu.org/39258#89> 1: <http://issues.guix.gnu.org/39258#93> > > Let compare only for cold cache and time this cache building (Guix 7db8fd6): > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) > > > > real 0m28,848s > > user 0m1,481s > > sys 0m0,252s > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache)) > > > > real 0m40,279s > > user 0m1,582s > > sys 0m0,232s > > > > It seems longer but compared to the time of "guix pull" completion, it seems > > acceptable. > > Both the initial timing and the target are waaay too much. :-/ Yes, but that's another story. :-) We cannot fix all in the same time, IMHO. Here the point is to speedup "guix search" and to accept to pay a little more at "guix pull" time; then we could optimize the cache generation. Considering the overall time of "guix pull", this extra time appears to me acceptable---if "guix search" is faster. :-) > On my i7 laptop I have: > > --8<---------------cut here---------------start------------->8--- > $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t.cache")' > > real 0m20.738s > user 0m44.413s > sys 0m0.341s > --8<---------------cut here---------------end--------------->8--- > > It’s CPU-bound; we should probably start by optimizing that. > > In Guile 3.0.7 there was a change that improved this noticeably: > > https://git.savannah.gnu.org/cgit/guile.git/commit/?id=05614f792bfabbc33798863edd0bb67c744e9299 > > We should prolly look for similar optimization opportunities in the > assembler… Yes, but this kind of optimization will not provide a faster "guix search" but a faster "guix pull". --8<---------------cut here---------------start------------->8--- $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t1.cache")' real 0m15,728s user 0m23,940s sys 0m0,826s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(load-compiled "/tmp/t1.cache/lib/guix/package.cache")' real 0m1,026s user 0m0,258s sys 0m0,051s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t2.cache")' real 0m35,570s user 3m12,951s sys 0m3,807s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(load-compiled "/tmp/t2.cache/lib/guix/package.cache")' real 0m1,055s user 0m0,283s sys 0m0,055s --8<---------------cut here---------------end--------------->8--- (And if we speak about performance, raw loading the cache takes ~1s but "guix package -A >/dev/null" takes ~8s. It is a big gap for parsing the CLI and sorting. Worse, "guix --version >/dev/null" takes ~2s on cold cache. We should probably start by reduce this overhead.) > > Let compare for some queries: > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix search game | recsel -C -P name | wc -l > > 371 > > > > real 0m7,561s > > user 0m3,525s > > sys 0m0,391s > > I think you should run: > > time guix search game > /dev/null > > otherwise Bash’s built-in ‘time’ shows the wall-clock time of the whole > pipeline, and the processing time of ‘recsel’ is probably not negligible > here. First, I am confused... If the formatter (recsel) is not negligible, then it should be dropped and replaced by an internal (fast) formatter. Well, I mean that as an end-user I am interested by the time of the whole pipeline. Second, on my machine the time is somehow negligible*. ;-) On cold cache, 10 runs using the pipe or using the redirection, keeping the max and the min for each: --8<---------------cut here---------------start------------->8--- real 0m9,598s user 0m3,961s sys 0m0,415s real 0m8,744s user 0m3,772s sys 0m0,431s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- real 0m8,755s user 0m3,869s sys 0m0,540s real 0m8,390s user 0m3,767s sys 0m0,416s --8<---------------cut here---------------end--------------->8--- *negligible: better said, it does not give the bad impression. Even if it is roughly 5% of difference. Cheers, simon
Hi, This is an attempt to improve the performance of "guix search". It is still half baked but it allows to discuss further the idea about expanding the current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-) Let start by what needs to be improved: the part when cache is not authoritative. It is slower than the current approach because the package is read twice, i.e., the module is indeed loaded twice, once by 'fold-available-packages' via 'fold-module-public-variables*' and then again by 'find-packages-by-description' via 'read-package-from'. The issue is to have a common interface for both cases (cache and no-cache). More thoughts are required. ;-) Then, using the cache is slower than expected. Therefore, something is maybe twisted -- quick implementation before holidays ;-) -- with the use of 'fold-avaibale-packages' as proposed by Ludo [1]. Note that instead another 'fold-packages' (say 'fold-packages*') using the new cache should be used. As it is done with v4 and the performances were as expected: <http://issues.guix.gnu.org/39258#89> 1: <http://issues.guix.gnu.org/39258#93> From my understanding, the issue that 'package-relevance' accepts a 'package' (and then all the chain until displaying) and 'fold-avaibale-packages' does not return a package. Well, I do not know; especially where to put something similiar to 'read-package-from'. To test, after applying the patches, the command is: ./pre-inst-env guix pull --allow-downgrades --disable-authentication \ --url=$(pwd) --branch=search-v6 -p /tmp/new Let compare only for cold cache and time this cache building (Guix 7db8fd6): sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) real 0m28,848s user 0m1,481s sys 0m0,252s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache)) real 0m40,279s user 0m1,582s sys 0m0,232s It seems longer but compared to the time of "guix pull" completion, it seems acceptable. However, maybe it could become an issue when running a lot of "guix time-machine"... Well, hard trade-off. ;-) Let compare for some queries: --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search game | recsel -C -P name | wc -l 371 real 0m7,561s user 0m3,525s sys 0m0,391s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search game | recsel -C -P name | wc -l 371 real 0m9,814s user 0m3,240s sys 0m0,363s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search strategy game | recsel -C -P name | wc -l 16 real 0m8,565s user 0m2,803s sys 0m0,430s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search strategy game | recsel -C -P name | wc -l 16 real 0m9,679s user 0m2,370s sys 0m0,334s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search strategy game caesar | recsel -C -P name | wc -l 0 real 0m8,307s user 0m2,388s sys 0m0,366s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search strategy game caesar | recsel -C -P name | wc -l 0 real 0m3,626s user 0m0,948s sys 0m0,101s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search game strategy the | recsel -C -P name | wc -l 4 real 0m8,776s user 0m2,903s sys 0m0,454s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search game strategy the | recsel -C -P name | wc -l 4 real 0m9,495s user 0m2,546s sys 0m0,313s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search the game strategy | recsel -C -P name | wc -l 4 real 0m8,502s user 0m2,534s sys 0m0,388s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search the game strategy | recsel -C -P name | wc -l 4 real 0m9,508s user 0m2,254s sys 0m0,363s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search crypto library | recsel -C -P name | grep libb2 libb2 real 0m8,744s user 0m2,875s sys 0m0,374s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search crypto library | recsel -C -P name | grep libb2 libb2 real 0m9,229s user 0m2,448s sys 0m0,397s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search cuirass integration | recsel -C -P name | wc -l 1 real 0m8,132s user 0m2,343s sys 0m0,407s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search cuirass integration | recsel -C -P name | wc -l 1 real 0m8,940s user 0m2,036s sys 0m0,369s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search cuirass | recsel -C -P name | wc -l 2 real 0m8,240s user 0m2,461s sys 0m0,367s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search cuirass | recsel -C -P name | wc -l 2 real 0m8,863s user 0m2,019s sys 0m0,377s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search cuirass integration foo | recsel -C -P name | wc -l 0 real 0m8,258s user 0m2,418s sys 0m0,521s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search cuirass integration foo | recsel -C -P name | wc -l 0 real 0m3,358s user 0m0,867s sys 0m0,139s --8<---------------cut here---------------end--------------->8--- This last example suggests that 'read-package-from' is the slowdown. (On a side note, maybe I am doing wrong, but there is no improvement by the recent introduction of 'cut' for multi-terms as the query "the game strategy" and "game strategy the". Another story. :-)) When cache is not authoritative, it is worse, as expected: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search -L /tmp/my-pkgs cuirass integration foo | recsel -C -P name | wc -l 0 real 0m12,503s user 0m7,807s sys 0m0,529s and note that currently the performances of "guix search" is the same for both cases (authoritative and not authoritative); i.e., see previous timing. Last, two minor remarks about previous comments. 1. Ludo commented: > Therefore the cache '/lib/guix/package.cache' contains more > information. This breaks the binary interface, so we’ll have to analyze the impact of such a change and devise a strategy. <http://issues.guix.gnu.org/39258#93> and after some checking, this should be fine, IIUC. The '--news' is ok because of '#:allow-other-keys'. And other parts are modified accordingly. Guix revision N creates a cache that Guix revision N+1 will read but it should not be an issue; see 'inferior-available-packages'. 2. And Ludo wrote: I realize the other cache also has that problem, but it would be nice to add a version tag to the cache. Basically emit something like: (package-metadata-cache (version 0) VECTOR …) instead of just: (VECTOR …) <http://issues.guix.gnu.org/39258#93> which is, after discussions, not necessary. Versioning does not make sense here because the cache is read by the Guix which generates it. Therefore, specify a version is extraneous here. Comments are welcome for this work-in-progress. ;-) Cheers, simon zimoun (2): DRAFT packages: Add fields to packages cache. DRAFT scripts: package: Use cache in 'find-packages-by-description'. gnu/packages.scm | 52 +++++++++++++++++++++++++++------------- guix/scripts/package.scm | 46 +++++++++++++++++++++++++---------- 2 files changed, 70 insertions(+), 28 deletions(-) base-commit: 4196087f3d6fc254db5b4c47658e5679c835516f