From patchwork Sun Mar 14 14:38:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 27695 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4876927BC52; Sun, 14 Mar 2021 14:39:16 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 1E20D27BC54 for ; Sun, 14 Mar 2021 14:39:13 +0000 (GMT) Received: from localhost ([::1]:49108 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLRtg-0002PO-6L for patchwork@mira.cbaines.net; Sun, 14 Mar 2021 10:39:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49204) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lLRtY-0002PC-1K for guix-patches@gnu.org; Sun, 14 Mar 2021 10:39:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:50753) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lLRtW-0002qo-Je for guix-patches@gnu.org; Sun, 14 Mar 2021 10:39:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lLRtW-0006kV-I1 for guix-patches@gnu.org; Sun, 14 Mar 2021 10:39:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47137] [PATCH] Adaptive substitute decompression selection Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 14 Mar 2021 14:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 47137 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47137@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161573273425921 (code B ref -1); Sun, 14 Mar 2021 14:39:02 +0000 Received: (at submit) by debbugs.gnu.org; 14 Mar 2021 14:38:54 +0000 Received: from localhost ([127.0.0.1]:34066 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLRtO-0006jy-05 for submit@debbugs.gnu.org; Sun, 14 Mar 2021 10:38:54 -0400 Received: from lists.gnu.org ([209.51.188.17]:50714) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLRtL-0006jp-Vq for submit@debbugs.gnu.org; Sun, 14 Mar 2021 10:38:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49178) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lLRtK-0002OU-To for guix-patches@gnu.org; Sun, 14 Mar 2021 10:38:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52429) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLRtK-0002kp-4p for guix-patches@gnu.org; Sun, 14 Mar 2021 10:38:50 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39814 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLRtJ-0001se-GJ for guix-patches@gnu.org; Sun, 14 Mar 2021 10:38:49 -0400 From: Ludovic =?utf-8?q?Court=C3=A8s?= X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 24 =?utf-8?q?Vent=C3=B4se?= an 229 de la =?utf-8?q?R?= =?utf-8?q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 14 Mar 2021 15:38:47 +0100 Message-ID: <87wnu9ls08.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi! The patch below is a followup to the thread started in December: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html It provides a naïve but apparently good enough way for ‘guix substitute’ to choose the compression method that yields the best speed given the CPU and current networking conditions. On a recent x86_64 laptop with fast networking, using ci.guix.gnu.org, the effect so far is to choose gzip substitutes, which indeed provides slightly faster substitute installation. When ci.guix provides zstd substitutes, the speedup will be higher. I have yet to check that it sticks to lzip when bandwidth is low. Thoughts? Thanks, Ludo’. From 3f95a1ac04c5e178a7fedfc2d03c07bcb1075ead Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= Date: Sun, 14 Mar 2021 15:05:30 +0100 Subject: [PATCH] substitute: Choose compression method based on past CPU usage. This stems from the observation that substitute download can be CPU-bound when high-speed networks are in use: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html * guix/narinfo.scm (decompresses-faster?): New procedure. (narinfo-best-uri): Add #:fast-decompression?. * guix/scripts/substitute.scm (%prefer-fast-decompression?): New variable. (call-with-cpu-usage-monitoring): New procedure. (with-cpu-usage-monitoring): New macro. (display-narinfo-data, process-substitution): Pass #:fast-decompression? to 'narinfo-best-uri'. (process-substitution): Wrap 'restore-file' call in 'with-cpu-usage-monitoring'. Set '%prefer-fast-decompression?'. --- guix/narinfo.scm | 27 ++++++++++++++++--- guix/scripts/substitute.scm | 53 ++++++++++++++++++++++++++++++++----- 2 files changed, 69 insertions(+), 11 deletions(-) diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 2d06124017..72e0f75fda 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2014 Nikita Karetnikov ;;; Copyright © 2018 Kyle Meyer ;;; @@ -297,9 +297,21 @@ this is a rough approximation." (_ (or (string=? compression2 "none") (string=? compression2 "gzip"))))) -(define (narinfo-best-uri narinfo) +(define (decompresses-faster? compression1 compression2) + "Return true if COMPRESSION1 generally has a higher decompression throughput +than COMPRESSION2." + (match compression1 + ("none" #t) + ("zstd" #t) + ("gzip" (string=? compression2 "lzip")) + (_ #f))) + +(define* (narinfo-best-uri narinfo #:key fast-decompression?) "Select the \"best\" URI to download NARINFO's nar, and return three values: -the URI, its compression method (a string), and the compressed file size." +the URI, its compression method (a string), and the compressed file size. +When FAST-DECOMPRESSION? is true, prefer substitutes with faster +decompression (typically zstd) rather than substitutes with a higher +compression ratio (typically lzip)." (define choices (filter (match-lambda ((uri compression file-size) @@ -321,6 +333,13 @@ the URI, its compression method (a string), and the compressed file size." (compresses-better? compression1 compression2)))) (_ #f))) ;we can't tell - (match (sort choices file-size) (narinfo-references narinfo)) - (let-values (((uri compression file-size) (narinfo-best-uri narinfo))) + (let-values (((uri compression file-size) + (narinfo-best-uri narinfo + #:fast-decompression? + %prefer-fast-decompression?))) (format #t "~a\n~a\n" (or file-size 0) (or (narinfo-size narinfo) 0)))) @@ -438,7 +462,9 @@ the current output port." store-item)) (let-values (((uri compression file-size) - (narinfo-best-uri narinfo))) + (narinfo-best-uri narinfo + #:fast-decompression? + %prefer-fast-decompression?))) (unless print-build-trace? (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) @@ -476,11 +502,24 @@ the current output port." ((hashed get-hash) (open-hash-input-port algorithm input))) ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file hashed destination - #:dump-file (if (and destination-in-store? - deduplicate?) - dump-file/deduplicate* - dump-file)) + (define cpu-usage + (with-cpu-usage-monitoring + (restore-file hashed destination + #:dump-file (if (and destination-in-store? + deduplicate?) + dump-file/deduplicate* + dump-file)))) + + ;; Create a hysteresis: depending on CPU usage, favor compression + ;; methods with faster decompression (like ztsd) or methods with better + ;; compression ratios (like lzip). This stems from the observation that + ;; substitution can be CPU-bound when high-speed networks are used: + ;; . + (when (> cpu-usage .8) + (set! %prefer-fast-decompression? #t)) + (when (< cpu-usage .4) + (set! %prefer-fast-decompression? #f)) + (close-port hashed) (close-port input) -- 2.30.2