mbox series

[bug#34060,00/10] Add a cache for package lookups

Message ID 20190113154532.29606-1-ludo@gnu.org
Headers show
Series Add a cache for package lookups | expand

Message

Ludovic Courtès Jan. 13, 2019, 3:45 p.m. UTC 
Hello!

This patch adjust ‘guix pull’ & co. such that Guix comes with a
cache to speed up package lookups.  The end result is that:

  guix build coreutils

becomes equivalent to:

  guix build -e '(@ (gnu packages base) coreutils)'

That means we have fewer files to open (only the closure of
(gnu packages base) instead of all the (gnu packages …) modules),
and thus less I/O and a smaller memory footprint.  Without cache,
we have:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build coreutils -nd
/gnu/store/ddlgwpishzb8985gwg5fdrydvllg254a-coreutils-8.30.drv
Garbage collection statistics:
  heap size:        46.18 MiB
  allocated:        102.57 MiB
  GC times:         14
  time spent in GC: 0.25 seconds (25% of user time)
1.01user 0.06system 0:00.96elapsed 112%CPU (0avgtext+0avgdata 158916maxresident)k
0inputs+0outputs (0major+14602minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

With the cache, we get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./foo/bin/guix build coreutils -nd
/gnu/store/ddlgwpishzb8985gwg5fdrydvllg254a-coreutils-8.30.drv
Garbage collection statistics:
  heap size:        36.52 MiB
  allocated:        87.30 MiB
  GC times:         14
  time spent in GC: 0.24 seconds (27% of user time)
0.89user 0.06system 0:00.84elapsed 112%CPU (0avgtext+0avgdata 124168maxresident)k
0inputs+0outputs (0major+11423minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

As a bonus, we take advantage of the cache to speed up ‘guix
package -A’ (which now runs in ~0.5s instead of ~1.7s) and
‘guix edit’.

The cache is populated by packages of all the channels specified
during ‘guix pull’.

The cache is *not* used at all when GUIX_PACKAGE_PATH is set or
a ‘-L’ flag is passed, or ./pre-inst-env is used.

Feedback welcome!

Ludo’.

Ludovic Courtès (10):
  profiling: Add a "gc" profiling component.
  guix package: Avoid 'find-newest-available-packages'.
  packages: Remove 'find-newest-available-packages'.
  inferior: Add 'gexp->derivation-in-inferior'.
  discovery: Add 'fold-module-public-variables*'.
  pull: Build profile with 'channel-instances->derivation'.
  channels: Compute a package cache and use it.
  edit: Use 'specification->location' to read information from the
    cache.
  guix package: '--list-available' can use data from the cache.
  status: Distinguish 'package-cache' profile hook.

 gnu/packages.scm         | 253 ++++++++++++++++++++++++++++++++++-----
 guix/channels.scm        |  32 ++++-
 guix/discovery.scm       |  28 ++++-
 guix/inferior.scm        |  26 ++++
 guix/profiling.scm       |  25 +++-
 guix/scripts/edit.scm    |  29 ++---
 guix/scripts/package.scm | 137 +++++++++++----------
 guix/scripts/pull.scm    |   4 +-
 guix/status.scm          |   2 +
 tests/packages.scm       |  77 ++++++++++--
 10 files changed, 491 insertions(+), 122 deletions(-)

Comments

Ludovic Courtès Jan. 15, 2019, 7:26 p.m. UTC | #1 
Pushed!

  b9da4b931d status: Distinguish 'package-cache' profile hook.
  0ea939fb79 guix package: '--list-available' can use data from the cache.
  ee8099f5b6 edit: Use 'specification->location' to read information from the cache.
  5fbdc9a5aa channels: Compute a package cache and use it.
  1d90e9d7c9 discovery: Add 'fold-module-public-variables*'.
  ae92782240 inferior: Add 'gexp->derivation-in-inferior'.
  e2a903c807 packages: Remove 'find-newest-available-packages'.
  465a0d65ae guix package: Avoid 'find-newest-available-packages'.
  461d6c2eff profiling: Add a "gc" profiling component.

Ludo’.