From patchwork Tue Jan 9 23:10:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 58712 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3FA7E27BBE9; Tue, 9 Jan 2024 23:11:14 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id BAABE27BBE2 for ; Tue, 9 Jan 2024 23:11:13 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rNLFM-0007e5-8v; Tue, 09 Jan 2024 18:11:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rNLFI-0007bs-D7 for guix-patches@gnu.org; Tue, 09 Jan 2024 18:10:57 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rNLFH-0007lL-5O for guix-patches@gnu.org; Tue, 09 Jan 2024 18:10:56 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rNLFN-0006TL-W3 for guix-patches@gnu.org; Tue, 09 Jan 2024 18:11:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#68266] [PATCH 1/7] gnu: Memozise make-ld-wrapper results. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 09 Jan 2024 23:11:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 68266 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: 68266@debbugs.gnu.org Received: via spool by 68266-submit@debbugs.gnu.org id=B68266.170484182424831 (code B ref 68266); Tue, 09 Jan 2024 23:11:01 +0000 Received: (at 68266) by debbugs.gnu.org; 9 Jan 2024 23:10:24 +0000 Received: from localhost ([127.0.0.1]:41254 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rNLEl-0006SR-QP for submit@debbugs.gnu.org; Tue, 09 Jan 2024 18:10:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:37558) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rNLEk-0006SD-83 for 68266@debbugs.gnu.org; Tue, 09 Jan 2024 18:10:22 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rNLEX-0007Pv-KA; Tue, 09 Jan 2024 18:10:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=CldhrM5j3uJvBBy8LZ4aN5RSUGHm8zBlvU0kfDe8qjo=; b=TDGRJxbPqrxASc2w9Uql 6VnSimOw7zW2ObokeHOu6KpfFAb8I1tuVqdcKKvdrZ2UYJ0OEVTpol94kR/sCFxnisDqxbg08k5wk pS9twliHkK2UC3EDzsEzn2/UHPUeXNNtv/1K/Py1uLMdutyFQPzsb926GwH14cigCND4PQpgFjk0u izF7M86JFLkubU7yH6GMVcAnhf14HQWIRXtPt2S7G1PNydbj4tzvXL5QQSiZItJF+AdUWgEKgeBM3 +69JGS4A3A4VJzEVLKvHYyzcpFhB7vfP2Sn+PdzwFsi8BCfQ5sXxmDwSiRGLZaIWhbv19osRAtJoy rGc8+hEn53U3Hw==; From: Ludovic =?utf-8?q?Court=C3=A8s?= In-Reply-To: <878r4zr6a0.fsf@cbaines.net> (Christopher Baines's message of "Mon, 08 Jan 2024 19:01:58 +0000") References: <87plyfrb2x.fsf@cbaines.net> <13f83a0db585e81572240e35dfef473aa73fe996.1704472849.git.mail@cbaines.net> <87v8833fl7.fsf@gnu.org> <878r4zr6a0.fsf@cbaines.net> Date: Wed, 10 Jan 2024 00:10:04 +0100 Message-ID: <87plyaw1bn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Hi, Christopher Baines skribis: > I believe the reason packages from make-ld-wrapper were showing up > multiple times in the cache for me is linked to it's use in the > cross-base module, as part of the cross-gcc procedure. > > A later commit does change cross-gcc to return a single package record > for some given arguments, so that probably resolves the biggest misuse > of make-ld-wrapper. Oh, I see. Before resorting to memoization (which doesn’t come for free), we can for instance make sure we do not create new package records from inputs fields (since they are thunked, a fresh record would be created each time the field is accessed). Here’s an example: This particular one doesn’t have a huge impact on simple cases but it might pay off for bigger graphs: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=object-cache ./pre-inst-env guix build hello --target=aarch64-linux-gnu --no-grafts -d /gnu/store/i8x1gg73x3fvxn4lrrgbcb97sz4qq1yx-hello-2.12.1.drv Object Cache: fresh caches: 20 lookups: 5514 hits: 5008 (90.8%) cache size: 505 entries $ # before: $ GUIX_PROFILING=object-cache ./pre-inst-env guix build hello --target=aarch64-linux-gnu --no-grafts -d /gnu/store/kdnh9bwg874aq8bvvw9y2vkl7z94icqa-hello-2.12.1.drv Object Cache: fresh caches: 20 lookups: 5406 hits: 4907 (90.8%) cache size: 498 entries --8<---------------cut here---------------end--------------->8--- > I think there's other cases (in the llvm and mold modules) where it > looks like it's called multiple times with the same arguments, so maybe > that's an argument for having memoization around make-ld-wrapper even > though it's not needed for all uses. On guix-devel, Efraim mentioned cargo-build-system packages. Is that related? Thanks for looking into this! Ludo’. diff --git a/gnu/packages/cross-base.scm b/gnu/packages/cross-base.scm index 6ee7b315d8..ca3b41a4c7 100644 --- a/gnu/packages/cross-base.scm +++ b/gnu/packages/cross-base.scm @@ -306,6 +306,11 @@ (define* (cross-gcc target XGCC as the base compiler. Use XBINUTILS as the associated cross-Binutils. If LIBC is false, then build a GCC that does not target a libc; otherwise, target that libc." + (define ld-wrapper + (make-ld-wrapper (string-append "ld-wrapper-" target) + #:target (const target) + #:binutils xbinutils)) + (package (inherit xgcc) (name (string-append "gcc-cross-" @@ -313,8 +318,7 @@ (define* (cross-gcc target target)) (source (origin - (inherit - (package-source xgcc)) + (inherit (package-source xgcc)) (patches (append (origin-patches (package-source xgcc)) @@ -353,10 +357,7 @@ (define* (cross-gcc target ,@(cross-gcc-arguments target xgcc libc))) (native-inputs - `(("ld-wrapper-cross" ,(make-ld-wrapper - (string-append "ld-wrapper-" target) - #:target (const target) - #:binutils xbinutils)) + `(("ld-wrapper-cross" ,ld-wrapper) ("binutils-cross" ,xbinutils) ,@(let ((inputs (append (package-inputs xgcc) From patchwork Fri Jan 12 14:13:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ludovic_Court=C3=A8s?= X-Patchwork-Id: 58820 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B2ED927BBE9; Fri, 12 Jan 2024 14:14:43 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5615C27BBE2 for ; Fri, 12 Jan 2024 14:14:40 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rOIIS-0005v3-3z; Fri, 12 Jan 2024 09:14:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOIIP-0005uQ-Bx for guix-patches@gnu.org; Fri, 12 Jan 2024 09:14:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rOIIO-0003OW-UL for guix-patches@gnu.org; Fri, 12 Jan 2024 09:14:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rOIIM-00032J-Sr for guix-patches@gnu.org; Fri, 12 Jan 2024 09:14:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#68266] [PATCH 7/7] packages: rust: Memoize make-rust-sysroot results. Resent-From: Ludovic =?utf-8?q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 12 Jan 2024 14:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 68266 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: Efraim Flashner , 68266@debbugs.gnu.org Received: via spool by 68266-submit@debbugs.gnu.org id=B68266.170506880411593 (code B ref 68266); Fri, 12 Jan 2024 14:14:02 +0000 Received: (at 68266) by debbugs.gnu.org; 12 Jan 2024 14:13:24 +0000 Received: from localhost ([127.0.0.1]:35248 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOIHj-00030u-PV for submit@debbugs.gnu.org; Fri, 12 Jan 2024 09:13:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42824) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOIHg-00030I-Mw for 68266@debbugs.gnu.org; Fri, 12 Jan 2024 09:13:21 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOIHc-00038Q-7u; Fri, 12 Jan 2024 09:13:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=SxuXzKc5P8PsutD3J3SVD96Wo9cxqDPf4RTO1OOEG6U=; b=SAFb1+cTeU8K+Ylp2gRP OEjFOg+Urz1fhe2KUK48YOt4Yu6bTyGNUm4+kY54pwtwNBovQta+uWvKXPMD0FM9ahDNZqnHL8/pp hVzUFRloMAybd40Xutbh4cfKkOFGxySBMbChzsmVjVKskA+IqbkLdKidjmaJ/BdTEsYZ1J/Ak8fpz 2NwIb30X/orFyrbU0hj+xB0IDtG7W3sVDhJbLzJfpa8dFNuL9vKyUR9uc0MKrERmolfp9RfWq6ZP5 L5bcXzfYUuvL+5bSuVvCedZVLeYbSX1sld9QAIz9T9CnaLUbY7OlArOW8a1ArduggGXTC1quioBwa Z6YOtZ6DYbS/FQ==; From: Ludovic =?utf-8?q?Court=C3=A8s?= In-Reply-To: (Christopher Baines's message of "Fri, 5 Jan 2024 16:40:49 +0000") References: <13f83a0db585e81572240e35dfef473aa73fe996.1704472849.git.mail@cbaines.net> Date: Fri, 12 Jan 2024 15:13:12 +0100 Message-ID: <878r4uk5c7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Hi, Christopher Baines skribis: > To ensure that it just returns a single package record for some given > arguments, as this helps to avoid poor performance of the store connection > object cache. > > * gnu/packages/rust.scm (make-rust-sysroot): Move code to > make-rust-sysroot/implementation. > (make-rust-sysroot/implementation): New variable. > > Change-Id: Ibb30c7398328c87c032bb8828635a34ada935167 [...] > (define*-public (make-rust-sysroot target) > - (let ((base-rust rust)) > + (make-rust-sysroot/implementation target rust)) > + > +(define make-rust-sysroot/implementation > + (mlambda (target base-rust) > (package > (inherit base-rust) > (name (string-append "rust-sysroot-for-" target)) We should avoid using ‘mlambda’ (without ‘q’) with packages as it leads to deep object comparisons. That’s why for packages we typically always have one-argument (mlambdaq (package) …). But since ‘base-rust’ wasn’t a parameter before, let’s keep it simple (‘diff --ignore-space-change’): It should help when cross-compiling several ‘cargo-build-system’ packages: --8<---------------cut here---------------start------------->8--- $ time GUIX_PROFILING=object-cache guix build --no-grafts greetd wlgreet du-dust circtools --target=aarch64-linux-gnu -d # before /gnu/store/mivzv83wryv9gp5bjncg5m1831dx2xwr-circtools-1.0.0.drv /gnu/store/4xf7kh9mi0vpvs8m1ak4x8w1rpsdpv6z-du-dust-0.8.6.drv /gnu/store/ayk54gvlbc1qam6irzf9kaig56dhzni0-wlgreet-0.4.1.drv /gnu/store/r601i40cii9ic5w1k4hy5c2yngfayh64-greetd-0.9.0.drv Object Cache: fresh caches: 22 lookups: 40435 hits: 36821 (91.1%) cache size: 3613 entries real 0m2.964s user 0m2.925s sys 0m0.174s $ time GUIX_PROFILING=object-cache ./pre-inst-env guix build --no-grafts greetd wlgreet du-dust circtools --target=aarch64-linux-gnu -d # after /gnu/store/1wsldmvigjb8w2gk418npbnfznlb0ck1-circtools-1.0.0.drv /gnu/store/b5c73fawjdvkgy431qxz9l6l9y9a9lhz-du-dust-0.8.6.drv /gnu/store/zwc7qzsbzf62dgbbzy74lki4hsr406bw-wlgreet-0.4.1.drv /gnu/store/vjdd23hc82701afb132z1ajcqa7hfd74-greetd-0.9.0.drv Object Cache: fresh caches: 22 lookups: 37942 hits: 34523 (91.0%) cache size: 3418 entries real 0m2.980s user 0m3.300s sys 0m0.160s --8<---------------cut here---------------end--------------->8--- (Here we removed ~2.5k nodes from the cache.) WDYT? Ludo’. diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm index 2b86cf3c8b..983a3f0f41 100644 --- a/gnu/packages/rust.scm +++ b/gnu/packages/rust.scm @@ -62,6 +62,7 @@ (define-module (gnu packages rust) #:use-module (guix download) #:use-module (guix git-download) #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix memoization) #:use-module (guix packages) #:use-module ((guix build utils) #:select (alist-replace)) #:use-module (guix utils) @@ -1060,7 +1061,8 @@ (define-public rust `("procps" ,procps) (package-native-inputs base-rust)))))) -(define*-public (make-rust-sysroot target) +(define-public make-rust-sysroot + (mlambda (target) (let ((base-rust rust)) (package (inherit base-rust) @@ -1228,7 +1230,7 @@ (define*-public (make-rust-sysroot target) (cross-binutils target))))) (properties `((hidden? . #t) - ,(package-properties base-rust)))))) + ,(package-properties base-rust))))))) (define-public rust-analyzer (package