From patchwork Sun Jan 13 15:45:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 727 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3D03616A42; Sun, 13 Jan 2019 15:46:11 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 635CF169F7 for ; Sun, 13 Jan 2019 15:46:09 +0000 (GMT) Received: from localhost ([127.0.0.1]:41149 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gihxg-0004nj-AW for patchwork@mira.cbaines.net; Sun, 13 Jan 2019 10:46:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gihxa-0004mb-Rp for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gihxZ-0003c6-V3 for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:58703) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gihxZ-0003c0-Qk for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gihxZ-0008Ht-OD for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#34060] [PATCH 00/10] Add a cache for package lookups Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 13 Jan 2019 15:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 34060 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 34060@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.154739435631843 (code B ref -1); Sun, 13 Jan 2019 15:46:01 +0000 Received: (at submit) by debbugs.gnu.org; 13 Jan 2019 15:45:56 +0000 Received: from localhost ([127.0.0.1]:57984 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gihxT-0008HW-S1 for submit@debbugs.gnu.org; Sun, 13 Jan 2019 10:45:56 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44763) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gihxS-0008HK-2B for submit@debbugs.gnu.org; Sun, 13 Jan 2019 10:45:54 -0500 Received: from lists.gnu.org ([209.51.188.17]:54780) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gihxM-0003VC-TR for submit@debbugs.gnu.org; Sun, 13 Jan 2019 10:45:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54962) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gihxL-0004ft-Tf for guix-patches@gnu.org; Sun, 13 Jan 2019 10:45:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gihxL-0003Ti-2A for guix-patches@gnu.org; Sun, 13 Jan 2019 10:45:47 -0500 Received: from hera.aquilenet.fr ([2a0c:e300::1]:52546) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gihxK-0003SA-OL; Sun, 13 Jan 2019 10:45:47 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 5F7B417CF; Sun, 13 Jan 2019 16:45:43 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AoNdamOWzn8M; Sun, 13 Jan 2019 16:45:42 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 6FF921410; Sun, 13 Jan 2019 16:45:42 +0100 (CET) From: Ludovic =?utf-8?q?Court=C3=A8s?= Date: Sun, 13 Jan 2019 16:45:32 +0100 Message-Id: <20190113154532.29606-1-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hello! This patch adjust ‘guix pull’ & co. such that Guix comes with a cache to speed up package lookups. The end result is that: guix build coreutils becomes equivalent to: guix build -e '(@ (gnu packages base) coreutils)' That means we have fewer files to open (only the closure of (gnu packages base) instead of all the (gnu packages …) modules), and thus less I/O and a smaller memory footprint. Without cache, we have: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=gc time ./pre-inst-env guix build coreutils -nd /gnu/store/ddlgwpishzb8985gwg5fdrydvllg254a-coreutils-8.30.drv Garbage collection statistics: heap size: 46.18 MiB allocated: 102.57 MiB GC times: 14 time spent in GC: 0.25 seconds (25% of user time) 1.01user 0.06system 0:00.96elapsed 112%CPU (0avgtext+0avgdata 158916maxresident)k 0inputs+0outputs (0major+14602minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- With the cache, we get: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=gc time ./foo/bin/guix build coreutils -nd /gnu/store/ddlgwpishzb8985gwg5fdrydvllg254a-coreutils-8.30.drv Garbage collection statistics: heap size: 36.52 MiB allocated: 87.30 MiB GC times: 14 time spent in GC: 0.24 seconds (27% of user time) 0.89user 0.06system 0:00.84elapsed 112%CPU (0avgtext+0avgdata 124168maxresident)k 0inputs+0outputs (0major+11423minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- As a bonus, we take advantage of the cache to speed up ‘guix package -A’ (which now runs in ~0.5s instead of ~1.7s) and ‘guix edit’. The cache is populated by packages of all the channels specified during ‘guix pull’. The cache is *not* used at all when GUIX_PACKAGE_PATH is set or a ‘-L’ flag is passed, or ./pre-inst-env is used. Feedback welcome! Ludo’. Ludovic Courtès (10): profiling: Add a "gc" profiling component. guix package: Avoid 'find-newest-available-packages'. packages: Remove 'find-newest-available-packages'. inferior: Add 'gexp->derivation-in-inferior'. discovery: Add 'fold-module-public-variables*'. pull: Build profile with 'channel-instances->derivation'. channels: Compute a package cache and use it. edit: Use 'specification->location' to read information from the cache. guix package: '--list-available' can use data from the cache. status: Distinguish 'package-cache' profile hook. gnu/packages.scm | 253 ++++++++++++++++++++++++++++++++++----- guix/channels.scm | 32 ++++- guix/discovery.scm | 28 ++++- guix/inferior.scm | 26 ++++ guix/profiling.scm | 25 +++- guix/scripts/edit.scm | 29 ++--- guix/scripts/package.scm | 137 +++++++++++---------- guix/scripts/pull.scm | 4 +- guix/status.scm | 2 + tests/packages.scm | 77 ++++++++++-- 10 files changed, 491 insertions(+), 122 deletions(-)