From patchwork Fri Mar 27 16:26:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arun Isaac X-Patchwork-Id: 20907 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6EE5D27BBEB; Fri, 27 Mar 2020 16:28:12 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 354EA27BBE4 for ; Fri, 27 Mar 2020 16:28:12 +0000 (GMT) Received: from localhost ([::1]:44148 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jHrq7-0003Rp-P6 for patchwork@mira.cbaines.net; Fri, 27 Mar 2020 12:28:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42327) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jHrq0-0003P5-MQ for guix-patches@gnu.org; Fri, 27 Mar 2020 12:28:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jHrpy-000750-I4 for guix-patches@gnu.org; Fri, 27 Mar 2020 12:28:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:55548) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jHrpy-00074Z-F7 for guix-patches@gnu.org; Fri, 27 Mar 2020 12:28:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jHrpy-0000Wp-BM for guix-patches@gnu.org; Fri, 27 Mar 2020 12:28:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search References: In-Reply-To: Resent-From: Arun Isaac Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 27 Mar 2020 16:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39258 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: 39258@debbugs.gnu.org Cc: Arun Isaac , mail@ambrevar.xyz, ludo@gnu.org, zimon.toutoune@gmail.com Received: via spool by 39258-submit@debbugs.gnu.org id=B39258.15853264401934 (code B ref 39258); Fri, 27 Mar 2020 16:28:02 +0000 Received: (at 39258) by debbugs.gnu.org; 27 Mar 2020 16:27:20 +0000 Received: from localhost ([127.0.0.1]:33281 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jHrpH-0000V6-JY for submit@debbugs.gnu.org; Fri, 27 Mar 2020 12:27:19 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:36902) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jHrpD-0000Uj-4D for 39258@debbugs.gnu.org; Fri, 27 Mar 2020 12:27:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=systemreboot.net; s=default; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=B64pW/Bb79FmNyJmie0BJLwHNVHZT3dbkR8Ek8IwJWw=; b=KcTZRQ0sRRvJ/tKV6hy86zQCuQ SNseCuBp11SoFCSoIj32BkMYBLHauf92+MtrxuBq/BOmKoyGSkImDHB8iXyaNss9JnWsQIxDxCHEQ +1eDJW6ZXKl23pO8bF5J3IjD9hw6NrWyoS+ja3n6YshaxRLR1n1EklLC1vrDwfrUE+60=; Received: from [192.168.2.1] (helo=steel.lan) by systemreboot.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1jHrp4-0044U6-FN; Fri, 27 Mar 2020 21:57:06 +0530 From: Arun Isaac Date: Fri, 27 Mar 2020 21:56:51 +0530 Message-Id: <20200327162654.18785-1-arunisaac@systemreboot.net> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi everyone, This is v3 of my attempt to make guix search faster. In this version, I have abandoned use of xapian. Instead I build a cache of the metadata of all packages in a profile hook. Then, I use that cache to search and display search results. This way, package guile modules are not loaded during guix search. Speedup is around 2x. Both measurements below are with a warm cache. --8<---------------cut here---------------start------------->8--- $ time guix search inkscape real 0m1.722s user 0m1.776s sys 0m0.097s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ time /tmp/test/bin/guix search inkscape real 0m0.749s user 0m0.770s sys 0m0.020s --8<---------------cut here---------------end--------------->8--- This patchset does not affect the search API nor does it improve the relevance of search results. If there is interest in this approach, I'll complete this patchset properly. But, in the long run, I do think we should aim to get xapian or the like for guix search. WDYT? Unfortunately, generate-package-metadata-cache takes 43 seconds to build the cache on my relatively slow computer. Performance should be better on other people's machines. Meanwhile, it would still be useful if someone built patchset v2 on their machine and reported the time it took to build the xapian index. * How to test this patchset Apply patches and build as usual. Do a guix pull into a temporary profile. $ ./pre-inst-env guix pull --url=$PWD --branch=the-name-of-the-branch-you-applied-patches-to -p /tmp/test Then, run guix search from the built profile $ /tmp/test/bin/guix search inkscape Thanks! Arun Isaac (3): guix: Generate package metadata cache. guix: Search package metadata cache. guix: Use package metadata cache for package search. gnu/packages.scm | 88 +++++++++++++++++++++++++- guix/channels.scm | 34 +++++++++- guix/packages.scm | 32 ++++++++++ guix/scripts/package.scm | 5 +- guix/ui.scm | 132 ++++++++++++++++++++++++++++++++++++--- 5 files changed, 277 insertions(+), 14 deletions(-)