From patchwork Tue Sep 7 15:34:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: M X-Patchwork-Id: 32695 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id D495627BBE3; Tue, 7 Sep 2021 16:35:27 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, FREEMAIL_FROM,LOTS_OF_MONEY,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,T_DKIM_INVALID,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E010027BBE1 for ; Tue, 7 Sep 2021 16:35:26 +0100 (BST) Received: from localhost ([::1]:34344 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNd89-0001T5-SO for patchwork@mira.cbaines.net; Tue, 07 Sep 2021 11:35:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47766) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNd7n-0001RR-75 for guix-patches@gnu.org; Tue, 07 Sep 2021 11:35:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45857) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNd7m-0007tR-GC for guix-patches@gnu.org; Tue, 07 Sep 2021 11:35:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mNd7m-0000wL-BY for guix-patches@gnu.org; Tue, 07 Sep 2021 11:35:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50456] Optimise bytevector->nix-base32-string and bytevector->base16-string. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 07 Sep 2021 15:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 50456 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: 50456@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16310288933583 (code B ref -1); Tue, 07 Sep 2021 15:35:02 +0000 Received: (at submit) by debbugs.gnu.org; 7 Sep 2021 15:34:53 +0000 Received: from localhost ([127.0.0.1]:57402 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNd7c-0000vg-Jg for submit@debbugs.gnu.org; Tue, 07 Sep 2021 11:34:53 -0400 Received: from lists.gnu.org ([209.51.188.17]:37578) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNd7Z-0000vX-UM for submit@debbugs.gnu.org; Tue, 07 Sep 2021 11:34:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47716) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNd7X-0001LK-Bv for guix-patches@gnu.org; Tue, 07 Sep 2021 11:34:49 -0400 Received: from michel.telenet-ops.be ([2a02:1800:110:4::f00:18]:54396) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNd7T-0007V5-WF for guix-patches@gnu.org; Tue, 07 Sep 2021 11:34:47 -0400 Received: from butterfly.local ([188.188.3.227]) by michel.telenet-ops.be with bizsmtp id r3ad2500K4tskic063aeyc; Tue, 07 Sep 2021 17:34:38 +0200 Message-ID: <7831fcdd8b8aab99cc95ba904076014b4c3cb6d2.camel@telenet.be> From: Maxime Devos Date: Tue, 07 Sep 2021 17:34:28 +0200 User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1631028878; bh=W44FlGgv4GCh8GmH2khGm2mCRpX3UPW/VMeE/4kDzLs=; h=Subject:From:To:Date; b=oh+WLei3DZiBTf8HjkSIdvRoP9OsfrBvblI4PCNWiP1JC0oVqR0sD402T/EMF9VwR Ume6bf2tUgBH0Tz1VkPC4Dmb2HlIPzr8hVXDkp70b+ujyzZwDgXluM+4RcAi5usOH2 NXV9E6WZk1l45b+WtB0gdf4S3lmxyrvPhZRvJenheozfoK3H/C0zNnOtr14ehL1n8C Ov1BLAbUYyhJ8i/e6KMUSE6vdLujUbLRFQs4cDbpGnOTOeCi97Sff7D/BHhnC2p3OA 8RZp1IKfRus7bg6yD1/N6KyIItXL6PWC37e7kwteHmhfGpzZHiL3vqHM4Rd4FiIBbZ 8WXLQfrnZAaSw== Received-SPF: pass client-ip=2a02:1800:110:4::f00:18; envelope-from=maximedevos@telenet.be; helo=michel.telenet-ops.be X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, LOTS_OF_MONEY=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi guix, The two atached patches optimise bytevector->nix-base32-string and bytevector->base16-string, making them about 20% and two times faster respectively, by reducing allocations. They are called from 'output-path', 'fixed-output-path' and 'store-path' in (guix store). Unfortunately, this does not decrease timings to a noticable degree, but it does decrease the allocated memory by 2.3% (*), and it does not seem to increase timings. (See perf-numbers.txt.) (*) GUIX_PROFILING=gc guix build -d pigx --no-grafts Greetings, Maxime. while true; do time GUIX_PROFILING=gc ./the-unoptimised-baseN-guix/bin/guix build -d pigx --no-grafts; done # First run removed /gnu/store/fq6x8d2vcm6sbjkimg7g8kcgb4c5xv1b-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 325.20 MiB GC times: 18 time spent in GC: 3.69 seconds (24% of user time) real 0m15,066s user 0m15,149s sys 0m0,709s /gnu/store/fq6x8d2vcm6sbjkimg7g8kcgb4c5xv1b-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 325.19 MiB GC times: 18 time spent in GC: 3.70 seconds (24% of user time) real 0m15,924s user 0m15,695s sys 0m0,836s /gnu/store/fq6x8d2vcm6sbjkimg7g8kcgb4c5xv1b-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 325.20 MiB GC times: 18 time spent in GC: 3.66 seconds (24% of user time) real 0m15,369s user 0m15,339s sys 0m0,704s /gnu/store/fq6x8d2vcm6sbjkimg7g8kcgb4c5xv1b-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 325.20 MiB GC times: 18 time spent in GC: 3.69 seconds (25% of user time) real 0m14,889s user 0m15,066s sys 0m0,679s Summary. (define (avg . r) (/ (apply + r) (length r))) (define (stddev . r) (* (/ (length r) (- (length r) 1)) (sqrt (apply avg (map (lambda (x) (expt (- x (apply avg r)) 2)) r))))) (define %time/gc '(3.69 3.70 3.66 3.69)) (values (apply avg %time/gc) (apply stddev %time/gc)) $7 = 3.685 $8 = 0.01999999999999997 (define %real '(15.066 15.924 15.369 14.889)) (define %user '(15.149 15.695 15.339 15.066)) (define %sys '(0.709 0.836 0.704 0.679)) (values (apply avg %real) (apply stddev %real)) $1 = 15.312000000000001 $2 = 0.5237633053202561 (values (apply avg %user) (apply stddev %user)) $3 = 15.31225 $4 = 0.32283655995634153 (values (apply avg %sys) (apply stddev %sys)) $5 = 0.732 $6 = 0.08148074073737369 while true; do time GUIX_PROFILING=gc ./the-optimised-baseN-guix/bin/guix build -d pigx --no-grafts; done /gnu/store/jfjfg7dnis7v6947a0rncxdn3y1nz0ad-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 317.83 MiB GC times: 18 time spent in GC: 3.71 seconds (22% of user time) real 0m17,646s user 0m16,539s sys 0m0,705s /gnu/store/jfjfg7dnis7v6947a0rncxdn3y1nz0ad-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 317.83 MiB GC times: 18 time spent in GC: 3.63 seconds (22% of user time) real 0m18,733s user 0m16,698s sys 0m0,691s /gnu/store/jfjfg7dnis7v6947a0rncxdn3y1nz0ad-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 317.82 MiB GC times: 18 time spent in GC: 3.72 seconds (24% of user time) real 0m15,429s user 0m15,448s sys 0m0,696s /gnu/store/jfjfg7dnis7v6947a0rncxdn3y1nz0ad-pigx-0.0.3.drv Garbage collection statistics: heap size: 93.85 MiB allocated: 317.82 MiB GC times: 18 time spent in GC: 3.70 seconds (24% of user time) real 0m15,292s user 0m15,315s sys 0m0,635s (define %time/gc '(3.71 3.63 3.72 3.70)) (define %time/real '(17.646 18.733 15.429 15.292)) (define %time/user '(16.539 16.698 15.448 15.315)) (define %time/sys '(0.705 0.691 0.696 0.635)) (values (apply avg %time/gc) (apply stddev %time/gc)) $17 = 3.6900000000000004 $18 = 0.04714045207910329 (values (apply avg %time/real) (apply stddev %time/real)) $19 = 16.775000000000002 $20 = 1.9554380015172506 (values (apply avg %time/user) (apply stddev %time/user)) $21 = 16.0 $22 = 0.8304360300468671 (values (apply avg %time/sys) (apply stddev %time/sys)) $23 = 0.6817499999999999 $24 = 0.03660449274186007 Tests show neither a decrease nor an increase in timings. Now looking at the allocation count: The heap size before: heap size: 93.85 MiB allocated: 325.20 MiB The heap size after: heap size: 93.85 MiB allocated: 317.82 MiB A small improvement (-2.3%). From dfd9b7557e31823320fcbd7abed77de295b7dce1 Mon Sep 17 00:00:00 2001 From: Maxime Devos Date: Mon, 6 Sep 2021 00:46:17 +0200 Subject: [PATCH 2/2] base16: Reduce GC pressure in bytevector->base16-string. This makes bytevector->base16-string two times faster. * guix/base16.scm (bytevector->base16-string): Use utf8->string and iteration instead of string-concatenate and named let. --- guix/base16.scm | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/guix/base16.scm b/guix/base16.scm index 6c15a9f588..9ac964dff0 100644 --- a/guix/base16.scm +++ b/guix/base16.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2014, 2017 Ludovic Courtès +;;; Copyright © 2021 Maxime Devos ;;; ;;; This file is part of GNU Guix. ;;; @@ -32,27 +33,28 @@ (define (bytevector->base16-string bv) "Return the hexadecimal representation of BV's contents." - (define len - (bytevector-length bv)) - - (let-syntax ((base16-chars (lambda (s) - (syntax-case s () - (_ - (let ((v (list->vector - (unfold (cut > <> 255) - (lambda (n) - (format #f "~2,'0x" n)) - 1+ - 0)))) - v)))))) - (define chars base16-chars) - (let loop ((i len) - (r '())) - (if (zero? i) - (string-concatenate r) - (let ((i (- i 1))) - (loop i - (cons (vector-ref chars (bytevector-u8-ref bv i)) r))))))) + (define len (bytevector-length bv)) + (define utf8 (make-bytevector (* len 2))) + (let-syntax ((base16-octet-pairs + (lambda (s) + (syntax-case s () + (_ + (string->utf8 + (string-concatenate + (unfold (cut > <> 255) + (lambda (n) + (format #f "~2,'0x" n)) + 1+ + 0)))))))) + (define octet-pairs base16-octet-pairs) + (let loop ((i 0)) + (when (< i len) + (bytevector-u16-native-set! + utf8 (* 2 i) + (bytevector-u16-native-ref octet-pairs + (* 2 (bytevector-u8-ref bv i)))) + (loop (+ i 1)))) + (utf8->string utf8))) (define base16-string->bytevector (let ((chars->value (fold (lambda (i r) -- 2.33.0