Message ID | 20201215093830.10322-1-ludo@gnu.org |
---|---|
Headers | show |
Series | Pipeline substitute integrity check, deduplication, and canonicalization | expand |
Ludovic Courtès <ludo@gnu.org> skribis: > I tested with substitutes that contain many files: > > guix build pipewire@0.2 ffmpeg ungoogled-chromium vim-full \ > emacs-no-x emacs-no-x-toolkit > > On my laptop with an SSD, the wall-clock time is almost unchanged > when fetching lzip substitutes. You can see that the throughput > displayed while downloading is slightly lower than before, which > is consistent because lzip downloads are CPU-bound¹, but this is > compensated by the lack of processing time between substitutes. > With gzip substitutes, I see a 10% speedup on the wall-clock time > on my laptop. Picture! First the timechart with the current daemon (gzip substitutes, downloading from the LAN): Notice how guix-daemon is busy in between substitute downloads: that’s the time it takes to compute the nar hash of the store item, reset its timestamps, and deduplicate its files. Now the same operation after with this patch series: This time guix-daemon remains idle all along whereas ‘guix substitute’ is almost 100% busy. There’s no pause time between substitutes. Ludo’.
Hey Ludo, Ludovic Courtès <ludo@gnu.org> writes: > Ludovic Courtès <ludo@gnu.org> skribis: > >> I tested with substitutes that contain many files: >> >> guix build pipewire@0.2 ffmpeg ungoogled-chromium vim-full \ >> emacs-no-x emacs-no-x-toolkit >> >> On my laptop with an SSD, the wall-clock time is almost unchanged >> when fetching lzip substitutes. You can see that the throughput >> displayed while downloading is slightly lower than before, which >> is consistent because lzip downloads are CPU-bound¹, but this is >> compensated by the lack of processing time between substitutes. >> With gzip substitutes, I see a 10% speedup on the wall-clock time >> on my laptop. > > Picture! First the timechart with the current daemon (gzip > substitutes, downloading from the LAN): > > > > > Notice how guix-daemon is busy in between substitute downloads: that’s > the time it takes to compute the nar hash of the store item, reset its > timestamps, and deduplicate its files. > > Now the same operation after with this patch series: > > > > > This time guix-daemon remains idle all along whereas ‘guix substitute’ > is almost 100% busy. There’s no pause time between substitutes. > > Ludo’. Very nice! Thanks for this work! I can't wait to try it. Maxim
Ludovic Courtès <ludo@gnu.org> skribis: > tests: Check the build trace for hash mismatches on substitutes. > daemon: Let 'guix substitute' perform hash checks. > tests: Check the mtime and permissions of substituted items. > daemon: Do not reset timestamps and permissions on substituted items. > tests: Make sure substituted items are deduplicated. > daemon: Delegate deduplication to 'guix substitute'. Pushed as c7c7f068c15e419aaf5ef616516aa5ad4e55c2fa. Ludo’.