[bug#45018,0/6] Process and connection reuse for substitutions

Hi Guix!

The attached patches optimize substitute downloads by: (1) keeping
a single ‘guix substitute’ process for all the substitutes (instead
of respawning a new one each time a store item is substituted), and
(2) reusing connections to substitute servers in between substitutions.

(Note that ‘guix publish’ does not keep connections alive, but
on ci.guix.gnu.org we run nginx in front of ‘guix publish’, and nginx
keeps them alive.)

Anecdotally, the effect is a 10% improvement on the wall-clock time
of ‘guix build --sources=transitive vim’ on my laptop, from 29s to 26s.
Of course it cannot be much better since the rest of the time is spent
actually retrieving bits over the network.

Overall the impact depends on a number of factors.  My laptop has
an SSD and I have fiber-to-the-home (FFTH) with low latency: forking
and initiating a TLS connection to ci.guix.gnu.org are both quite cheap,
so I probably don’t benefit that much.  The impact may be more acute
on a low-end device.

In the ‘guix build --sources=transitive vim’ case, there are a few
large tarballs and several small ones.  The gain is in the lack of
a pause time in between small tarballs.  Also this case is quite
advantageous: there’s no decompression and no unpacking happening.

Anyway, I think it’s a welcome improvement, especially when
downloading lots of stuff such as during the initial system installation.
Plus, there are more deletions than insertions.  :-)

Thoughts?

Ludo’.

Ludovic Courtès (6):
  daemon: 'Agent' constructor takes a list of environment variables.
  daemon: Use 'Agent' to spawn 'guix substitute --query'.
  daemon: Factorize substituter agent spawning.
  daemon: Run 'guix substitute --substitute' as an agent.
  substitute: Cache and reuse connections while substituting.
  daemon: Raise an error if substituter doesn't send the expected hash.

 guix/http-client.scm        |  12 +-
 guix/progress.scm           |   8 +-
 guix/scripts/substitute.scm | 215 +++++++++++++++++++++++-------------
 nix/libstore/build.cc       | 166 ++++++++++++++--------------
 nix/libstore/local-store.cc | 170 +++++++---------------------
 nix/libstore/local-store.hh |  25 +----
 nix/libutil/util.cc         |   6 +-
 nix/libutil/util.hh         |   7 +-
 tests/substitute.scm        |  98 +++++++++-------
 9 files changed, 350 insertions(+), 357 deletions(-)

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Guix!
>
> The attached patches optimize substitute downloads by: (1) keeping
> a single ‘guix substitute’ process for all the substitutes (instead
> of respawning a new one each time a store item is substituted), and
> (2) reusing connections to substitute servers in between substitutions.
> ...
> Anyway, I think it’s a welcome improvement, especially when
> downloading lots of stuff such as during the initial system installation.
> Plus, there are more deletions than insertions.  :-)
>
> Thoughts?

I think this will definitely help my unstable internet connections and
slow hdd, thank you!

Ludovic Courtès <ludo@gnu.org> skribis:

> The attached patches optimize substitute downloads by: (1) keeping
> a single ‘guix substitute’ process for all the substitutes (instead
> of respawning a new one each time a store item is substituted), and
> (2) reusing connections to substitute servers in between substitutions.

As an illustration of what happens at the process level, here’s the perf
timechart in the current situation:
Here’s the timeline once we’re using an agent:
Blue are periods where the process is running, and grey where it’s
idle—e.g., waiting for data.  After each substitution completion,
guix-daemon becomes busy for a little while (computing the hash of the
store item, adding it to the database).  The startup activity of each
‘guix’ process is visible and in the second case it happens only once
for ‘guix substitute’.

Ludo’.

Hey Ludo,

> Blue are periods where the process is running, and grey where it’s
> idle—e.g., waiting for data.  After each substitution completion,
> guix-daemon becomes busy for a little while (computing the hash of the
> store item, adding it to the database).  The startup activity of each
> ‘guix’ process is visible and in the second case it happens only once
> for ‘guix substitute’.

That's a great improvement! Besides a few remarks, it looks good to me!

Thanks,

Mathieu

Hello!

This update fixes issues that Chris uncovered during testing and
addresses comments Mathieu made.  Changes since v1:

  1. The daemon, in ‘SubstitutionGoal::handleChildOutput’, properly
     deal with the case where ‘data’ contains several lines, such as a
     “sha256:…” line and a “success” line.

  2. ‘call-with-cached-connection’ catches exceptions that may be
     raised by (web response) when reusing a stale socket.

  3. The connection cache is now an alist instead of a hash table and
     care is taken to evict old entries once it has reached
     ‘%max-cached-connections’.

  4. Fixed typo in comment in (guix scripts substitute).

I’ll go ahead with this version soonish if there are no objections.

Ludo’.

Ludovic Courtès (6):
  daemon: 'Agent' constructor takes a list of environment variables.
  daemon: Use 'Agent' to spawn 'guix substitute --query'.
  daemon: Factorize substituter agent spawning.
  daemon: Run 'guix substitute --substitute' as an agent.
  substitute: Cache and reuse connections while substituting.
  daemon: Raise an error if substituter doesn't send the expected hash.

 guix/http-client.scm        |  12 +-
 guix/progress.scm           |   8 +-
 guix/scripts/substitute.scm | 243 ++++++++++++++++++++++++------------
 nix/libstore/build.cc       | 173 +++++++++++++------------
 nix/libstore/local-store.cc | 170 ++++++-------------------
 nix/libstore/local-store.hh |  25 +---
 nix/libutil/util.cc         |   6 +-
 nix/libutil/util.hh         |   7 +-
 tests/substitute.scm        |  98 +++++++++------
 9 files changed, 381 insertions(+), 361 deletions(-)