From patchwork Sun Apr 21 09:42:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 26289 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0FB4B27BBEB; Sun, 21 Apr 2024 10:44:08 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7C06227BBE2 for ; Sun, 21 Apr 2024 10:44:07 +0100 (BST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ryTjq-0005Ss-Ig; Sun, 21 Apr 2024 05:43:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ryTjj-0005PE-EP for guix-patches@gnu.org; Sun, 21 Apr 2024 05:43:51 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ryTji-0002lV-G6; Sun, 21 Apr 2024 05:43:50 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ryTjv-0006TR-DT; Sun, 21 Apr 2024 05:44:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#70494] [PATCH 01/23] store: database: Register derivation outputs. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Sun, 21 Apr 2024 09:44:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70494 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 70494@debbugs.gnu.org Cc: Christopher Baines , Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 70494-submit@debbugs.gnu.org id=B70494.171369260124404 (code B ref 70494); Sun, 21 Apr 2024 09:44:03 +0000 Received: (at 70494) by debbugs.gnu.org; 21 Apr 2024 09:43:21 +0000 Received: from localhost ([127.0.0.1]:41756 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ryTjD-0006LA-0T for submit@debbugs.gnu.org; Sun, 21 Apr 2024 05:43:20 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:51273) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ryTiw-0006HM-CR for 70494@debbugs.gnu.org; Sun, 21 Apr 2024 05:43:04 -0400 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id 1EA3427BBE2; Sun, 21 Apr 2024 10:42:46 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id e25e3982; Sun, 21 Apr 2024 09:42:45 +0000 (UTC) From: Christopher Baines Date: Sun, 21 Apr 2024 10:42:19 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <87bk632h36.fsf@cbaines.net> References: <87bk632h36.fsf@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Caleb Ristvedt * guix/store/database.scm (register-derivation-outputs, registered-derivation-outputs): New procedures (register-valid-path): Call register-derivation-outputs for derivations. Co-authored-by: Christopher Baines Change-Id: Id958709f36f24ee1c9c375807e8146a9d1cc4259 --- guix/store/database.scm | 49 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) base-commit: 92af4ea17f70207fbbf2513f677f3171d4eafd41 diff --git a/guix/store/database.scm b/guix/store/database.scm index a847f9d2f0..6a9acc2aef 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -22,6 +22,9 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config) + #:use-module (guix serialization) + #:use-module (guix store) + #:use-module (guix derivations) #:use-module (guix store deduplication) #:use-module (guix base16) #:use-module (guix progress) @@ -44,7 +47,9 @@ (define-module (guix store database) valid-path-id register-valid-path + register-derivation-outputs register-items + registered-derivation-outputs %epoch reset-timestamps vacuum-database)) @@ -206,6 +211,26 @@ (define-inlinable (assert-integer proc in-range? key number) "Integer ~A out of range: ~S" (list key number) (list number)))) +(define (register-derivation-outputs db drv) + "Register all output paths of DRV as being produced by it (note that +this doesn't mean 'already produced by it', but rather just 'associated with +it')." + (let ((stmt (sqlite-prepare + db + " +INSERT OR REPLACE INTO DerivationOutputs (drv, id, path) +SELECT id, :outid, :outpath FROM ValidPaths WHERE path = :drvpath;" + #:cache? #t))) + (for-each (match-lambda + ((outid . ($ path)) + (sqlite-bind-arguments stmt + #:drvpath (derivation-file-name + drv) + #:outid outid + #:outpath path) + (sqlite-step-and-reset stmt))) + (derivation-outputs drv)))) + (define (add-references db referrer references) "REFERRER is the id of the referring store item, REFERENCES is a list ids of items referred to." @@ -284,6 +309,11 @@ (define* (register-valid-path db #:key path (references '()) (sqlite-step-and-reset stmt) (last-insert-row-id db))))) + (when (derivation-path? path) + (register-derivation-outputs db + (read-derivation-from-file + path))) + ;; Call 'path-id' on each of REFERENCES. This ensures we get a ;; "non-NULL constraint" failure if one of REFERENCES is unregistered. (add-references db id @@ -331,6 +361,25 @@ (define %epoch ;; When it all began. (make-time time-utc 0 1)) +(define (registered-derivation-outputs db drv) + "Get the list of (id, output-path) pairs registered for DRV." + (let ((stmt (sqlite-prepare + db + " +SELECT id, path +FROM DerivationOutputs +WHERE drv in (SELECT id from ValidPaths where path = :drv)" + #:cache? #t))) + (sqlite-bind-arguments stmt #:drv drv) + (let ((result (sqlite-fold (lambda (current prev) + (match current + (#(id path) + (cons (cons id path) + prev)))) + '() stmt))) + (sqlite-reset stmt) + result))) + (define* (register-items db items #:key prefix (registration-time (timestamp))