From patchwork Tue Jun 4 08:06:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 65165 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6CF8C27BBEA; Tue, 4 Jun 2024 09:08:15 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id CAC0727BBE2 for ; Tue, 4 Jun 2024 09:08:14 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sEPD2-0002zl-E8; Tue, 04 Jun 2024 04:07:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sEPD0-0002yt-Tu for guix-patches@gnu.org; Tue, 04 Jun 2024 04:07:54 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sEPD0-0008VC-LU; Tue, 04 Jun 2024 04:07:54 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sEPD8-0001k1-Fu; Tue, 04 Jun 2024 04:08:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#71357] [PATCH] substitute: =?utf-8?b?RG9u4oCZdA==?= keep cache entries more than a few days. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 04 Jun 2024 08:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 71357 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 71357@debbugs.gnu.org Cc: Ludovic =?utf-8?q?Court=C3=A8s?= , Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.17174884346611 (code B ref -1); Tue, 04 Jun 2024 08:08:02 +0000 Received: (at submit) by debbugs.gnu.org; 4 Jun 2024 08:07:14 +0000 Received: from localhost ([127.0.0.1]:56263 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sEPCL-0001iY-AY for submit@debbugs.gnu.org; Tue, 04 Jun 2024 04:07:13 -0400 Received: from lists.gnu.org ([209.51.188.17]:37164) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sEPCE-0001iL-T1 for submit@debbugs.gnu.org; Tue, 04 Jun 2024 04:07:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sEPC1-0002BZ-46 for guix-patches@gnu.org; Tue, 04 Jun 2024 04:06:53 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sEPC0-000763-3X; Tue, 04 Jun 2024 04:06:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=kzkV6ntkiRw1hVp98I6HpFwnHnvpBCMy3V3u/BA3qeg=; b=drlQfBRP6rW1oK esfi+dzBaz/KFbdKC7VxORI6ZR02sjmC32mGqNsalv87CW7KrP16zs5Sugb7gQYAPHVIxTJ8freLK a2AstZVY2lez61OBEtNRLMNCz1lrUmillO9R8YxDXhuVDeHQgUIKSRjVWyalhZtezmDUQSThGMYnA h16yxo+R5VyXT/aQt8dB8/CTUJoitL61PvqTye1CTZ4aKHWCZ12+YcgksTFRXqNwzPgxlrRydeL5V pSxM7y2LvMTCXtd1VmIT1OpRfCsR1hYsErmrZF/nN9UHckr24GaRR1w6nHM0igvexU1AjdgxM22TU 8nl3VX3yimTnHscM4aqw==; From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Tue, 4 Jun 2024 10:06:43 +0200 Message-ID: <40bd5a0087d54531aabe2d0ce8d0cd2a6bd7e547.1717487258.git.ludo@gnu.org> X-Mailer: git-send-email 2.45.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Experience has shown that keeping too many entries increases disk usage and, more importantly, leads to long delays when cleaning up the cache, measured in minutes on slow or busy HDDs with hundreds of thousands of cache entries, as is common on build machines. In those cases, the cost of the cache outweighs its benefit. * guix/scripts/substitute.scm (%narinfo-expired-cache-entry-removal-delay): Reduce to 5 days. (cached-narinfo-expiration-time)[max-ttl]: Reduce to 2 days. Change-Id: Iab212f572ee9041be61716423a3c014f93fe81ed --- guix/scripts/substitute.scm | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) Hello, Chris mentioned it before and I experienced it the hard way on bayfront: https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00177.html A big narinfo cache is a significant performance hit on spinning HDDs when the time comes to remove expired entries. This change makes the cache more ephemeral (2 to 5 days). I still think some caching is needed: one will often run several Guix commands in a day that will query the same narinfos and will only download/build a small subset (keep in mind that that ‘substitution-oracle’, used by ‘derivation-build-plan’, query narinfos for the closure of the requested derivations, minus those already valid); it would be wasteful and inefficient to download them over and over again. I’d like to have metrics to estimate that, but I don’t. Thoughts? Ludo’. base-commit: 85ac164c41fc4c93d3cb2a5d3321c63598c2855f diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index a7ad56dbcd5..8bcbca5e7aa 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -77,7 +77,7 @@ (define-module (guix scripts substitute) (define %narinfo-expired-cache-entry-removal-delay ;; How often we want to remove files corresponding to expired cache entries. - (* 7 24 3600)) + (* 5 24 3600)) (define (warn-about-missing-authentication) (warning (G_ "authentication and authorization of substitutes \ @@ -169,8 +169,9 @@ (define (cached-narinfo-expiration-time file) "Return the expiration time for FILE, which is a cached narinfo." (define max-ttl ;; Upper bound on the TTL used to avoid keeping around cached narinfos for - ;; too long, which makes the cache bigger and more expensive to traverse. - (* 2 30 24 60 60)) ;2 months + ;; too long, which makes the cache bigger and more expensive to traverse + ;; when deleting old entries. + (* 2 24 60 60)) (catch 'system-error (lambda ()