From patchwork Tue Dec 17 14:18:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Prikler X-Patchwork-Id: 16547 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8504E1792D; Tue, 17 Dec 2019 14:19:16 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_DKIM_INVALID, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTP id 36A7B1792B for ; Tue, 17 Dec 2019 14:19:16 +0000 (GMT) Received: from localhost ([::1]:41110 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihDgx-0000OG-HH for patchwork@mira.cbaines.net; Tue, 17 Dec 2019 09:19:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53997) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihDgp-0000LI-LW for guix-patches@gnu.org; Tue, 17 Dec 2019 09:19:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ihDgk-0001Fk-6F for guix-patches@gnu.org; Tue, 17 Dec 2019 09:19:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34902) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ihDgk-0001Er-35 for guix-patches@gnu.org; Tue, 17 Dec 2019 09:19:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ihDgj-0003nl-U0 for guix-patches@gnu.org; Tue, 17 Dec 2019 09:19:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#38649] [PATCH] Parallelize `guix package` Resent-From: Leo Prikler Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 17 Dec 2019 14:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 38649 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 38649@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.157659232614584 (code B ref -1); Tue, 17 Dec 2019 14:19:01 +0000 Received: (at submit) by debbugs.gnu.org; 17 Dec 2019 14:18:46 +0000 Received: from localhost ([127.0.0.1]:40875 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDgU-0003nA-GQ for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:18:46 -0500 Received: from lists.gnu.org ([209.51.188.17]:46184) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDgT-0003n3-56 for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:18:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53851) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihDgR-0008L2-4W for guix-patches@gnu.org; Tue, 17 Dec 2019 09:18:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ihDgP-00080h-7B for guix-patches@gnu.org; Tue, 17 Dec 2019 09:18:42 -0500 Received: from mailrelay.tugraz.at ([129.27.2.202]:37029) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ihDgO-0007hW-HQ for guix-patches@gnu.org; Tue, 17 Dec 2019 09:18:41 -0500 Received: from nijino.local (213-240-64-42.hdsl.highway.telekom.at [213.240.64.42]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 47cgGz5fPWz3wYB for ; Tue, 17 Dec 2019 15:18:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1576592315; bh=/rn7p/S+N8X15E0RfwSAPCkzZjwUMtV5cWFUk5UwMU0=; h=Subject:From:To:Date; b=lBlXXp/+k+TBodNm/xg9QERqepvyERRebB5E6NAN5lnOBSGsYmEXud1KJa6N0Gf1e O/D8vhpsU0JceSdAD0wH69izKc5CaGZxc2A9PJ5zxHCA96Ou9e3O0itsweieEdRzzT AwGznt3JXq6pAYoDvo/EVQhj4V625QG5O7AoOri4= Message-ID: From: Leo Prikler Date: Tue, 17 Dec 2019 15:18:44 +0100 User-Agent: Evolution 3.32.4 MIME-Version: 1.0 X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Scanned-By: MIMEDefang 2.74 on 129.27.10.117 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches Hi Guix! Yesterday I had an interesting conversation on IRC about the behaviour of multiple `guix package` processes running in parallel. Specifically, when two transactions target the same profile (usually /var/guix/profiles/per-user/$USER/guix-profile) at the same time, one of them will fail to claim the lock and abort. 0001 makes it so that the process waits for the lock. 0002 makes it so that packages specified via -i can be built in parallel. Regards, Leo From 336692df15e77f9d90619d0fe60e864c4d2fb37a Mon Sep 17 00:00:00 2001 From: Leo Prikler Date: Tue, 17 Dec 2019 14:04:12 +0100 Subject: [PATCH 2/2] guix: Build to be installed packages in parallel. * guix/scripts/package.scm (options->buildable): New procedure. (process-actions): Build packages before acquiring profile lock. --- guix/scripts/package.scm | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm index 202a6d6470..1278d1a65f 100644 --- a/guix/scripts/package.scm +++ b/guix/scripts/package.scm @@ -587,6 +587,19 @@ the resulting manifest entry." (package->manifest-entry package output #:properties (provenance-properties package))) +(define (options->buildable opts) + (filter-map (match-lambda + (('install . (? package? p)) + p) + (('install . (? string? spec)) + (if (store-path? spec) + #f ;; assume already interned + (specification->package spec))) + (('install . obj) + (leave (G_ "cannot build non-package object: ~s~%") + obj)) + (_ #f)) + opts)) (define (options->installable opts manifest transaction) "Given MANIFEST, the current manifest, and OPTS, the result of 'args-fold', @@ -861,8 +874,15 @@ processed, #f otherwise." (package-version item) (manifest-entry-version entry)))))) + ;; First, process installations, as these can be handled in parallel. + (unless dry-run? + (let* ((drv (map (compose (lambda (drv) (drv store)) package->derivation) + (options->buildable opts)))) + (show-what-to-build store drv + #:use-substitutes? substitutes?) + (build-derivations store drv))) - ;; First, acquire a lock on the profile, to ensure only one guix process + ;; Now, acquire a lock on the profile, to ensure only one guix process ;; is modifying it at a time. (format #t "Waiting for lock on ~a...~%" profile) (with-profile-lock profile -- 2.24.1