From patchwork Tue Feb 20 19:39:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60812 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 9510027BBEA; Tue, 20 Feb 2024 19:41:22 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6BE6D27BBE2 for ; Tue, 20 Feb 2024 19:41:21 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcVzL-00036x-SK; Tue, 20 Feb 2024 14:41:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcVys-0002kd-Em for guix-patches@gnu.org; Tue, 20 Feb 2024 14:40:43 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcVys-0000oz-6M; Tue, 20 Feb 2024 14:40:42 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcVzB-0002Up-Nm; Tue, 20 Feb 2024 14:41:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69292] [PATCH 1/6] store: database: Remove call-with-savepoint and associated code. References: <87h6i3lyph.fsf@cbaines.net> In-Reply-To: <87h6i3lyph.fsf@cbaines.net> Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:41:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69292 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69292@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69292-submit@debbugs.gnu.org id=B69292.17084580049454 (code B ref 69292); Tue, 20 Feb 2024 19:41:01 +0000 Received: (at 69292) by debbugs.gnu.org; 20 Feb 2024 19:40:04 +0000 Received: from localhost ([127.0.0.1]:46619 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyE-0002S8-Np for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:04 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43146) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyC-0002RR-5F for 69292@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:01 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id A98CC27BBE2 for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 1616ca35 for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:39:01 +0000 Message-ID: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches While care does need to be taken with making updates or inserts to the ValidPaths table, I think that trying to ensure this within update-or-insert is the wrong approach. Instead, when working with the store database, only one connection should be used to make changes to the database and those changes should happen in transactions that ideally begin immediately. This reverts commit 37545de4a3bf59611c184b31506fe9a16abe4c8b. * .dir-locals.el (scheme-mode): Remove entries for call-with-savepoint and call-with-retrying-savepoint. * guix/store/database.scm (call-with-savepoint, call-with-retrying-savepoint): Remove procedures. (update-or-insert): Remove use of call-with-savepoint. Change-Id: I2f986e8623d8235a90c40d5f219c1292c1ab157b --- .dir-locals.el | 2 -- guix/store/database.scm | 75 +++++++---------------------------------- 2 files changed, 13 insertions(+), 64 deletions(-) base-commit: f4af19b037826cad90bbcfe400ad864f028cc7d8 diff --git a/.dir-locals.el b/.dir-locals.el index d18e6ba760..f135eb69a5 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -133,8 +133,6 @@ (eval . (put 'call-with-transaction 'scheme-indent-function 1)) (eval . (put 'with-statement 'scheme-indent-function 3)) (eval . (put 'call-with-retrying-transaction 'scheme-indent-function 1)) - (eval . (put 'call-with-savepoint 'scheme-indent-function 1)) - (eval . (put 'call-with-retrying-savepoint 'scheme-indent-function 1)) (eval . (put 'call-with-container 'scheme-indent-function 1)) (eval . (put 'container-excursion 'scheme-indent-function 1)) diff --git a/guix/store/database.scm b/guix/store/database.scm index 2968f13492..3093fd816a 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -151,39 +151,11 @@ (define* (call-with-transaction db proc #:key restartable?) (false-if-exception (exec "rollback;")) (apply throw args)))) -(define* (call-with-savepoint db proc - #:optional (savepoint-name "SomeSavepoint")) - "Call PROC after creating a savepoint named SAVEPOINT-NAME. If PROC exits -abnormally, rollback to that savepoint. In all cases, remove the savepoint -prior to returning." - (define (exec sql) - (with-statement db sql stmt - (sqlite-fold cons '() stmt))) - - (dynamic-wind - (lambda () - (exec (string-append "SAVEPOINT " savepoint-name ";"))) - (lambda () - (catch #t - proc - (lambda args - (exec (string-append "ROLLBACK TO " savepoint-name ";")) - (apply throw args)))) - (lambda () - (exec (string-append "RELEASE " savepoint-name ";"))))) - (define* (call-with-retrying-transaction db proc #:key restartable?) (call-with-SQLITE_BUSY-retrying (lambda () (call-with-transaction db proc #:restartable? restartable?)))) -(define* (call-with-retrying-savepoint db proc - #:optional (savepoint-name - "SomeSavepoint")) - (call-with-SQLITE_BUSY-retrying - (lambda () - (call-with-savepoint db proc savepoint-name)))) - (define %default-database-file ;; Default location of the store database. (string-append %store-database-directory "/db.sqlite")) @@ -261,40 +233,19 @@ (define* (update-or-insert db #:key path deriver hash nar-size time) (assert-integer "update-or-insert" positive? #:nar-size nar-size) (assert-integer "update-or-insert" (cut >= <> 0) #:time time) - ;; It's important that querying the path-id and the insert/update operation - ;; take place in the same transaction, as otherwise some other - ;; process/thread/fiber could register the same path between when we check - ;; whether it's already registered and when we register it, resulting in - ;; duplicate paths (which, due to a 'unique' constraint, would cause an - ;; exception to be thrown). With the default journaling mode this will - ;; prevent writes from occurring during that sensitive time, but with WAL - ;; mode it will instead arrange to return SQLITE_BUSY when a write occurs - ;; between the start of a read transaction and its upgrading to a write - ;; transaction (see https://sqlite.org/rescode.html#busy_snapshot). - ;; Experimentally, it seems this SQLITE_BUSY will ignore a busy_timeout and - ;; immediately return (makes sense, since waiting won't change anything). - - ;; Note that when that kind of SQLITE_BUSY error is returned, it will keep - ;; being returned every time we try to upgrade the same outermost - ;; transaction to a write transaction. So when retrying, we have to restart - ;; the *outermost* write transaction. We can't inherently tell whether - ;; we're the outermost write transaction, so we leave the retry-handling to - ;; the caller. - (call-with-savepoint db - (lambda () - (let ((id (path-id db path))) - (if id - (with-statement db update-sql stmt - (sqlite-bind-arguments stmt #:id id - #:deriver deriver - #:hash hash #:size nar-size #:time time) - (sqlite-fold cons '() stmt)) - (with-statement db insert-sql stmt - (sqlite-bind-arguments stmt - #:path path #:deriver deriver - #:hash hash #:size nar-size #:time time) - (sqlite-fold cons '() stmt))) - (last-insert-row-id db))))) + (let ((id (path-id db path))) + (if id + (with-statement db update-sql stmt + (sqlite-bind-arguments stmt #:id id + #:deriver deriver + #:hash hash #:size nar-size #:time time) + (sqlite-fold cons '() stmt)) + (with-statement db insert-sql stmt + (sqlite-bind-arguments stmt + #:path path #:deriver deriver + #:hash hash #:size nar-size #:time time) + (sqlite-fold cons '() stmt))) + (last-insert-row-id db))) (define add-reference-sql "INSERT OR REPLACE INTO Refs (referrer, reference) VALUES (:referrer, :reference);") From patchwork Tue Feb 20 19:39:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60809 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 1AA7527BBE2; Tue, 20 Feb 2024 19:40:19 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id D9A1D27BBE9 for ; Tue, 20 Feb 2024 19:40:17 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcVy3-0002NF-96; Tue, 20 Feb 2024 14:39:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcVxw-0002Lj-HU for guix-patches@gnu.org; Tue, 20 Feb 2024 14:39:44 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcVxw-0000Pj-0L; Tue, 20 Feb 2024 14:39:44 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcVyG-0002SY-As; Tue, 20 Feb 2024 14:40:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69292] [PATCH 2/6] store: database: Remove with-statement and associated code. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:40:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69292 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69292@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69292-submit@debbugs.gnu.org id=B69292.17084580039435 (code B ref 69292); Tue, 20 Feb 2024 19:40:04 +0000 Received: (at 69292) by debbugs.gnu.org; 20 Feb 2024 19:40:03 +0000 Received: from localhost ([127.0.0.1]:46615 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyD-0002Ru-Pg for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:02 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyC-0002RS-5C for 69292@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:00 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id AF90427BBE9 for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id e23dd3da for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:39:02 +0000 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> References: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches I think using dynamic-wind to finalize all statements is the wrong approach. Firstly it would be good to allow reseting statements rather than finalizing them. Then for the problem of handling errors, the approach I've settled on in the build coordinator is to close the database connection, since that'll trigger guile-sqlite3 to finalize all the cached statements. This reverts commit 5d6e2255286e591def122ec2f4a3cbda497fea21. * .dir-locals.el (scheme-mode): Remove with-statement. * guix/store/database.scm (call-with-statement): Remove procedure. (with-statement): Remove syntax rule. (call-with-transaction, last-insert-row-id, path-id, update-or-insert, add-references): Don't use with-statement. Change-Id: I2fd976b3f12ec8105cc56350933a953cf53647e8 --- .dir-locals.el | 1 - guix/store/database.scm | 62 ++++++++++++++++++----------------------- 2 files changed, 27 insertions(+), 36 deletions(-) diff --git a/.dir-locals.el b/.dir-locals.el index f135eb69a5..2d1a03c313 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -131,7 +131,6 @@ (eval . (put 'with-database 'scheme-indent-function 2)) (eval . (put 'call-with-database 'scheme-indent-function 1)) (eval . (put 'call-with-transaction 'scheme-indent-function 1)) - (eval . (put 'with-statement 'scheme-indent-function 3)) (eval . (put 'call-with-retrying-transaction 'scheme-indent-function 1)) (eval . (put 'call-with-container 'scheme-indent-function 1)) diff --git a/guix/store/database.scm b/guix/store/database.scm index 3093fd816a..de72b79860 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -130,25 +130,22 @@ (define* (call-with-transaction db proc #:key restartable?) the transaction, otherwise commit the transaction after it finishes. RESTARTABLE? may be set to a non-#f value when it is safe to run PROC multiple times. This may reduce contention for the database somewhat." - (define (exec sql) - (with-statement db sql stmt - (sqlite-fold cons '() stmt))) ;; We might use begin immediate here so that if we need to retry, we figure ;; that out immediately rather than because some SQLITE_BUSY exception gets ;; thrown partway through PROC - in which case the part already executed ;; (which may contain side-effects!) might have to be executed again for ;; every retry. - (exec (if restartable? "begin;" "begin immediate;")) + (sqlite-exec db (if restartable? "begin;" "begin immediate;")) (catch #t (lambda () (let-values ((result (proc))) - (exec "commit;") + (sqlite-exec db "commit;") (apply values result))) (lambda args ;; The roll back may or may not have occurred automatically when the ;; error was generated. If it has occurred, this does nothing but signal ;; an error. If it hasn't occurred, this needs to be done. - (false-if-exception (exec "rollback;")) + (false-if-exception (sqlite-exec db "rollback;")) (apply throw args)))) (define* (call-with-retrying-transaction db proc #:key restartable?) @@ -170,26 +167,14 @@ (define-syntax with-database ((_ file db exp ...) (call-with-database file (lambda (db) exp ...))))) -(define (call-with-statement db sql proc) - (let ((stmt (sqlite-prepare db sql #:cache? #t))) - (dynamic-wind - (const #t) - (lambda () - (proc stmt)) - (lambda () - (sqlite-finalize stmt))))) - -(define-syntax-rule (with-statement db sql stmt exp ...) - "Run EXP... with STMT bound to a prepared statement corresponding to the sql -string SQL for DB." - (call-with-statement db sql - (lambda (stmt) exp ...))) - (define (last-insert-row-id db) ;; XXX: (sqlite3) currently lacks bindings for 'sqlite3_last_insert_rowid'. ;; Work around that. - (with-statement db "SELECT last_insert_rowid();" stmt - (match (sqlite-fold cons '() stmt) + (let* ((stmt (sqlite-prepare db "SELECT last_insert_rowid();" + #:cache? #t)) + (result (sqlite-fold cons '() stmt))) + (sqlite-finalize stmt) + (match result ((#(id)) id) (_ #f)))) @@ -199,11 +184,13 @@ (define path-id-sql (define* (path-id db path) "If PATH exists in the 'ValidPaths' table, return its numerical identifier. Otherwise, return #f." - (with-statement db path-id-sql stmt + (let ((stmt (sqlite-prepare db path-id-sql #:cache? #t))) (sqlite-bind-arguments stmt #:path path) - (match (sqlite-fold cons '() stmt) - ((#(id) . _) id) - (_ #f)))) + (let ((result (sqlite-fold cons '() stmt))) + (sqlite-finalize stmt) + (match result + ((#(id) . _) id) + (_ #f))))) (define update-sql "UPDATE ValidPaths SET hash = :hash, registrationTime = :time, deriver = @@ -235,17 +222,20 @@ (define* (update-or-insert db #:key path deriver hash nar-size time) (let ((id (path-id db path))) (if id - (with-statement db update-sql stmt + (let ((stmt (sqlite-prepare db update-sql #:cache? #t))) (sqlite-bind-arguments stmt #:id id #:deriver deriver #:hash hash #:size nar-size #:time time) - (sqlite-fold cons '() stmt)) - (with-statement db insert-sql stmt + (sqlite-fold cons '() stmt) + (sqlite-finalize stmt) + (last-insert-row-id db)) + (let ((stmt (sqlite-prepare db insert-sql #:cache? #t))) (sqlite-bind-arguments stmt #:path path #:deriver deriver #:hash hash #:size nar-size #:time time) - (sqlite-fold cons '() stmt))) - (last-insert-row-id db))) + (sqlite-fold cons '() stmt) ;execute it + (sqlite-finalize stmt) + (last-insert-row-id db))))) (define add-reference-sql "INSERT OR REPLACE INTO Refs (referrer, reference) VALUES (:referrer, :reference);") @@ -253,13 +243,15 @@ (define add-reference-sql (define (add-references db referrer references) "REFERRER is the id of the referring store item, REFERENCES is a list ids of items referred to." - (with-statement db add-reference-sql stmt + (let ((stmt (sqlite-prepare db add-reference-sql #:cache? #t))) (for-each (lambda (reference) (sqlite-reset stmt) (sqlite-bind-arguments stmt #:referrer referrer #:reference reference) - (sqlite-fold cons '() stmt)) - references))) + (sqlite-fold cons '() stmt) ;execute it + (last-insert-row-id db)) + references) + (sqlite-finalize stmt))) (define (timestamp) "Return a timestamp, either the current time of SOURCE_DATE_EPOCH." From patchwork Tue Feb 20 19:39:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60811 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id B064227BBE2; Tue, 20 Feb 2024 19:41:13 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 60CF727BBE9 for ; Tue, 20 Feb 2024 19:41:12 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcVyy-0002sf-1a; Tue, 20 Feb 2024 14:40:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcVys-0002kZ-AV for guix-patches@gnu.org; Tue, 20 Feb 2024 14:40:43 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcVyq-0000ob-Nn; Tue, 20 Feb 2024 14:40:41 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcVzC-0002V0-3F; Tue, 20 Feb 2024 14:41:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69292] [PATCH 3/6] store: database: Inline SQL to where it's used. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:41:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69292 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69292@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69292-submit@debbugs.gnu.org id=B69292.17084580109501 (code B ref 69292); Tue, 20 Feb 2024 19:41:02 +0000 Received: (at 69292) by debbugs.gnu.org; 20 Feb 2024 19:40:10 +0000 Received: from localhost ([127.0.0.1]:46638 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyM-0002TA-5C for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:10 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43150) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyC-0002RT-6e for 69292@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:01 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id B53DD27BBEA for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id c38561ed for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:39:03 +0000 Message-ID: <5abcabc4cb65ea63db7c4f046700116e4882b287.1708457946.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> References: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches This makes the code easier to read, as you don't have to keep jumping between the two places. * guix/store/database.scm (path-id-sql, update-sql, insert-sql, add-reference-sql): Remove variables. (path-id, update-or-insert, add-references): Include SQL. Change-Id: I53b4ab973be8d0cd10a0f35ba25972f1c9680353 --- guix/store/database.scm | 45 ++++++++++++++++++++++++----------------- 1 file changed, 27 insertions(+), 18 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index de72b79860..7e3a2873ce 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -178,13 +178,14 @@ (define (last-insert-row-id db) ((#(id)) id) (_ #f)))) -(define path-id-sql - "SELECT id FROM ValidPaths WHERE path = :path") - (define* (path-id db path) "If PATH exists in the 'ValidPaths' table, return its numerical identifier. Otherwise, return #f." - (let ((stmt (sqlite-prepare db path-id-sql #:cache? #t))) + (let ((stmt (sqlite-prepare + db + " +SELECT id FROM ValidPaths WHERE path = :path" + #:cache? #t))) (sqlite-bind-arguments stmt #:path path) (let ((result (sqlite-fold cons '() stmt))) (sqlite-finalize stmt) @@ -192,14 +193,6 @@ (define* (path-id db path) ((#(id) . _) id) (_ #f))))) -(define update-sql - "UPDATE ValidPaths SET hash = :hash, registrationTime = :time, deriver = -:deriver, narSize = :size WHERE id = :id") - -(define insert-sql - "INSERT INTO ValidPaths (path, hash, registrationTime, deriver, narSize) -VALUES (:path, :hash, :time, :deriver, :size)") - (define-inlinable (assert-integer proc in-range? key number) (unless (integer? number) (throw 'wrong-type-arg proc @@ -222,14 +215,28 @@ (define* (update-or-insert db #:key path deriver hash nar-size time) (let ((id (path-id db path))) (if id - (let ((stmt (sqlite-prepare db update-sql #:cache? #t))) + (let ((stmt (sqlite-prepare + db + " +UPDATE ValidPaths +SET hash = :hash, + registrationTime = :time, + deriver = :deriver, + narSize = :size +WHERE id = :id" + #:cache? #t))) (sqlite-bind-arguments stmt #:id id #:deriver deriver #:hash hash #:size nar-size #:time time) (sqlite-fold cons '() stmt) (sqlite-finalize stmt) (last-insert-row-id db)) - (let ((stmt (sqlite-prepare db insert-sql #:cache? #t))) + (let ((stmt (sqlite-prepare + db + " +INSERT INTO ValidPaths (path, hash, registrationTime, deriver, narSize) +VALUES (:path, :hash, :time, :deriver, :size)" + #:cache? #t))) (sqlite-bind-arguments stmt #:path path #:deriver deriver #:hash hash #:size nar-size #:time time) @@ -237,13 +244,15 @@ (define* (update-or-insert db #:key path deriver hash nar-size time) (sqlite-finalize stmt) (last-insert-row-id db))))) -(define add-reference-sql - "INSERT OR REPLACE INTO Refs (referrer, reference) VALUES (:referrer, :reference);") - (define (add-references db referrer references) "REFERRER is the id of the referring store item, REFERENCES is a list ids of items referred to." - (let ((stmt (sqlite-prepare db add-reference-sql #:cache? #t))) + (let ((stmt (sqlite-prepare + db + " +INSERT OR REPLACE INTO Refs (referrer, reference) +VALUES (:referrer, :reference)" + #:cache? #t))) (for-each (lambda (reference) (sqlite-reset stmt) (sqlite-bind-arguments stmt #:referrer referrer From patchwork Tue Feb 20 19:39:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60807 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4158727BBEA; Tue, 20 Feb 2024 19:40:02 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 55C9D27BBE2 for ; Tue, 20 Feb 2024 19:40:01 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcVy0-0002MJ-MJ; Tue, 20 Feb 2024 14:39:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcVxw-0002Ll-Ha for guix-patches@gnu.org; Tue, 20 Feb 2024 14:39:44 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcVxv-0000Pb-Mh; Tue, 20 Feb 2024 14:39:43 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcVyE-0002Rz-9a; Tue, 20 Feb 2024 14:40:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69292] [PATCH 4/6] store: database: Stop finalizing prepared statements. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:40:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69292 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69292@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69292-submit@debbugs.gnu.org id=B69292.17084579749354 (code B ref 69292); Tue, 20 Feb 2024 19:40:02 +0000 Received: (at 69292) by debbugs.gnu.org; 20 Feb 2024 19:39:34 +0000 Received: from localhost ([127.0.0.1]:46608 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVxl-0002Qn-OD for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:39:34 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43142) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVxi-0002QW-6i for 69292@debbugs.gnu.org; Tue, 20 Feb 2024 14:39:32 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id BAD1627BBEB for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6e2d8eda for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:39:04 +0000 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> References: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches Especially since we're asking for these to be cached. Management of prepared statements isn't trivial, since you don't want to keep them forever as this can lead to poor query performance, but I don't think that finalizing them immediately is the right solution. Change-Id: I61706b4d09d771835bb8f074b8f6a6ee871f5e2d * guix/store/database.scm (sqlite-step-and-reset): New procedure. (last-insert-row, path-id, update-or-insert, add-references): Don't finalize prepared statements. Change-Id: I2a2c6deb43935d67df9e43000a5105343d72b3e6 --- guix/store/database.scm | 40 ++++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index 7e3a2873ce..8d8b7346e0 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -167,16 +167,19 @@ (define-syntax with-database ((_ file db exp ...) (call-with-database file (lambda (db) exp ...))))) +(define (sqlite-step-and-reset statement) + (let ((val (sqlite-step statement))) + (sqlite-reset statement) + val)) + (define (last-insert-row-id db) ;; XXX: (sqlite3) currently lacks bindings for 'sqlite3_last_insert_rowid'. ;; Work around that. - (let* ((stmt (sqlite-prepare db "SELECT last_insert_rowid();" - #:cache? #t)) - (result (sqlite-fold cons '() stmt))) - (sqlite-finalize stmt) - (match result - ((#(id)) id) - (_ #f)))) + (let ((stmt (sqlite-prepare db + "SELECT last_insert_rowid();" + #:cache? #t))) + (vector-ref (sqlite-step-and-reset stmt) + 0))) (define* (path-id db path) "If PATH exists in the 'ValidPaths' table, return its numerical @@ -187,11 +190,9 @@ (define* (path-id db path) SELECT id FROM ValidPaths WHERE path = :path" #:cache? #t))) (sqlite-bind-arguments stmt #:path path) - (let ((result (sqlite-fold cons '() stmt))) - (sqlite-finalize stmt) - (match result - ((#(id) . _) id) - (_ #f))))) + (match (sqlite-step-and-reset stmt) + (#(id) id) + (#f #f)))) (define-inlinable (assert-integer proc in-range? key number) (unless (integer? number) @@ -228,9 +229,8 @@ (define* (update-or-insert db #:key path deriver hash nar-size time) (sqlite-bind-arguments stmt #:id id #:deriver deriver #:hash hash #:size nar-size #:time time) - (sqlite-fold cons '() stmt) - (sqlite-finalize stmt) - (last-insert-row-id db)) + (sqlite-step-and-reset stmt) + id) (let ((stmt (sqlite-prepare db " @@ -240,8 +240,7 @@ (define* (update-or-insert db #:key path deriver hash nar-size time) (sqlite-bind-arguments stmt #:path path #:deriver deriver #:hash hash #:size nar-size #:time time) - (sqlite-fold cons '() stmt) ;execute it - (sqlite-finalize stmt) + (sqlite-step-and-reset stmt) (last-insert-row-id db))))) (define (add-references db referrer references) @@ -254,13 +253,10 @@ (define (add-references db referrer references) VALUES (:referrer, :reference)" #:cache? #t))) (for-each (lambda (reference) - (sqlite-reset stmt) (sqlite-bind-arguments stmt #:referrer referrer #:reference reference) - (sqlite-fold cons '() stmt) ;execute it - (last-insert-row-id db)) - references) - (sqlite-finalize stmt))) + (sqlite-step-and-reset stmt)) + references))) (define (timestamp) "Return a timestamp, either the current time of SOURCE_DATE_EPOCH." From patchwork Tue Feb 20 19:39:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60808 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E989C27BBEC; Tue, 20 Feb 2024 19:40:18 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id D5E8527BBE2 for ; Tue, 20 Feb 2024 19:40:17 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcVy2-0002Mu-2V; Tue, 20 Feb 2024 14:39:50 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcVxw-0002Lk-H4 for guix-patches@gnu.org; Tue, 20 Feb 2024 14:39:44 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcVxv-0000Pa-Ld; Tue, 20 Feb 2024 14:39:43 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcVyF-0002SH-8A; Tue, 20 Feb 2024 14:40:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69292] [PATCH 5/6] store: database: Refactor sqlite-register. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:40:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69292 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69292@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69292-submit@debbugs.gnu.org id=B69292.17084579779365 (code B ref 69292); Tue, 20 Feb 2024 19:40:03 +0000 Received: (at 69292) by debbugs.gnu.org; 20 Feb 2024 19:39:37 +0000 Received: from localhost ([127.0.0.1]:46610 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVxp-0002Qy-5P for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:39:37 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43144) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVxi-0002QX-6o for 69292@debbugs.gnu.org; Tue, 20 Feb 2024 14:39:33 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id C06C227BBEC for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 150cb772 for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:39:05 +0000 Message-ID: <11b24242dd71349c09486341ed0d01b55f5f8f9a.1708457946.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> References: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches The update-or-insert procedure name was unhelpfully generic, and these changes should improve the code readability. * guix/store/database.scm (update-or-insert): Remove procedure and inline functionality in to sqlite-register. Change-Id: Ifab0cdb7972d095460cc1f79b8b2f0e9b958059c --- guix/store/database.scm | 132 ++++++++++++++++++++-------------------- 1 file changed, 66 insertions(+), 66 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index 8d8b7346e0..0b570eabcd 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -194,54 +194,15 @@ (define* (path-id db path) (#(id) id) (#f #f)))) -(define-inlinable (assert-integer proc in-range? key number) - (unless (integer? number) - (throw 'wrong-type-arg proc - "Wrong type argument ~A: ~S" (list key number) - (list number))) - (unless (in-range? number) - (throw 'out-of-range proc - "Integer ~A out of range: ~S" (list key number) - (list number)))) - -(define* (update-or-insert db #:key path deriver hash nar-size time) - "The classic update-if-exists and insert-if-doesn't feature that sqlite -doesn't exactly have... they've got something close, but it involves deleting -and re-inserting instead of updating, which causes problems with foreign keys, -of course. Returns the row id of the row that was modified or inserted." - - ;; Make sure NAR-SIZE is valid. - (assert-integer "update-or-insert" positive? #:nar-size nar-size) - (assert-integer "update-or-insert" (cut >= <> 0) #:time time) - - (let ((id (path-id db path))) - (if id - (let ((stmt (sqlite-prepare - db - " -UPDATE ValidPaths -SET hash = :hash, - registrationTime = :time, - deriver = :deriver, - narSize = :size -WHERE id = :id" - #:cache? #t))) - (sqlite-bind-arguments stmt #:id id - #:deriver deriver - #:hash hash #:size nar-size #:time time) - (sqlite-step-and-reset stmt) - id) - (let ((stmt (sqlite-prepare - db - " -INSERT INTO ValidPaths (path, hash, registrationTime, deriver, narSize) -VALUES (:path, :hash, :time, :deriver, :size)" - #:cache? #t))) - (sqlite-bind-arguments stmt - #:path path #:deriver deriver - #:hash hash #:size nar-size #:time time) - (sqlite-step-and-reset stmt) - (last-insert-row-id db))))) +(define (timestamp) + "Return a timestamp, either the current time of SOURCE_DATE_EPOCH." + (match (getenv "SOURCE_DATE_EPOCH") + (#f + (current-time time-utc)) + ((= string->number seconds) + (if seconds + (make-time time-utc 0 seconds) + (current-time time-utc))))) (define (add-references db referrer references) "REFERRER is the id of the referring store item, REFERENCES is a list @@ -258,15 +219,15 @@ (define (add-references db referrer references) (sqlite-step-and-reset stmt)) references))) -(define (timestamp) - "Return a timestamp, either the current time of SOURCE_DATE_EPOCH." - (match (getenv "SOURCE_DATE_EPOCH") - (#f - (current-time time-utc)) - ((= string->number seconds) - (if seconds - (make-time time-utc 0 seconds) - (current-time time-utc))))) +(define-inlinable (assert-integer proc in-range? key number) + (unless (integer? number) + (throw 'wrong-type-arg proc + "Wrong type argument ~A: ~S" (list key number) + (list number))) + (unless (in-range? number) + (throw 'out-of-range proc + "Integer ~A out of range: ~S" (list key number) + (list number)))) (define* (sqlite-register db #:key path (references '()) deriver hash nar-size @@ -279,15 +240,54 @@ (define* (sqlite-register db #:key path (references '()) the database or #f, meaning \"right now\". Every store item in REFERENCES must already be registered." - (let ((id (update-or-insert db #:path path - #:deriver deriver - #:hash hash - #:nar-size nar-size - #:time (time-second time)))) - ;; Call 'path-id' on each of REFERENCES. This ensures we get a - ;; "non-NULL constraint" failure if one of REFERENCES is unregistered. - (add-references db id - (map (cut path-id db <>) references)))) + + (define registration-time + (time-second time)) + + ;; Make sure NAR-SIZE is valid. + (assert-integer "sqlite-register" positive? #:nar-size nar-size) + (assert-integer "sqlite-register" (cut >= <> 0) #:time registration-time) + + (define id + (let ((existing-id (path-id db path))) + (if existing-id + (let ((stmt (sqlite-prepare + db + " +UPDATE ValidPaths +SET hash = :hash, + registrationTime = :time, + deriver = :deriver, + narSize = :size +WHERE id = :id" + #:cache? #t))) + (sqlite-bind-arguments stmt + #:id existing-id + #:deriver deriver + #:hash hash + #:size nar-size + #:time registration-time) + (sqlite-step-and-reset stmt) + existing-id) + (let ((stmt (sqlite-prepare + db + " +INSERT INTO ValidPaths (path, hash, registrationTime, deriver, narSize) +VALUES (:path, :hash, :time, :deriver, :size)" + #:cache? #t))) + (sqlite-bind-arguments stmt + #:path path + #:deriver deriver + #:hash hash + #:size nar-size + #:time registration-time) + (sqlite-step-and-reset stmt) + (last-insert-row-id db))))) + + ;; Call 'path-id' on each of REFERENCES. This ensures we get a + ;; "non-NULL constraint" failure if one of REFERENCES is unregistered. + (add-references db id + (map (cut path-id db <>) references))) ;;; From patchwork Tue Feb 20 19:39:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Baines X-Patchwork-Id: 60810 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E619527BBE2; Tue, 20 Feb 2024 19:41:07 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id BCC4F27BBE9 for ; Tue, 20 Feb 2024 19:41:06 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcVz0-0002ts-5G; Tue, 20 Feb 2024 14:40:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcVys-0002kY-9z for guix-patches@gnu.org; Tue, 20 Feb 2024 14:40:43 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcVyq-0000oa-NP; Tue, 20 Feb 2024 14:40:41 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rcVzC-0002V9-HD; Tue, 20 Feb 2024 14:41:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69292] [PATCH 6/6] store: database: Rename a couple of procedures. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 20 Feb 2024 19:41:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69292 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69292@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?utf-8?q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 69292-submit@debbugs.gnu.org id=B69292.17084580119508 (code B ref 69292); Tue, 20 Feb 2024 19:41:02 +0000 Received: (at 69292) by debbugs.gnu.org; 20 Feb 2024 19:40:11 +0000 Received: from localhost ([127.0.0.1]:46640 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyM-0002TC-IB for submit@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:11 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43152) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rcVyE-0002S4-Lf for 69292@debbugs.gnu.org; Tue, 20 Feb 2024 14:40:04 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id C60D227BBED for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 9b5e01bd for <69292@debbugs.gnu.org>; Tue, 20 Feb 2024 19:39:07 +0000 (UTC) From: Christopher Baines Date: Tue, 20 Feb 2024 19:39:06 +0000 Message-ID: <12f872394c84039fa553e50381d040de24f24d3e.1708457946.git.mail@cbaines.net> X-Mailer: git-send-email 2.41.0 In-Reply-To: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> References: <4b6a268daab5e0b307dff2229d551a47c9fe1ebc.1708457946.git.mail@cbaines.net> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches These names should be more descriptive. * guix/store/database.scm (path-id): Rename to select-valid-path-id. (sqlite-register): Rename to register-valid-path. (register-items): Update accordingly. Change-Id: I6d4a14d4cde9d71ab34d6ffdbfbfde51b2c0e1db --- guix/store/database.scm | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index 0b570eabcd..0190696ad5 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -40,8 +40,10 @@ (define-module (guix store database) store-database-file call-with-database with-database - path-id - sqlite-register + + select-valid-path-id + + register-valid-path register-items %epoch reset-timestamps @@ -181,9 +183,9 @@ (define (last-insert-row-id db) (vector-ref (sqlite-step-and-reset stmt) 0))) -(define* (path-id db path) - "If PATH exists in the 'ValidPaths' table, return its numerical -identifier. Otherwise, return #f." +(define (select-valid-path-id db path) + "If PATH exists in the 'ValidPaths' table, return its numerical identifier. +Otherwise, return #f." (let ((stmt (sqlite-prepare db " @@ -229,9 +231,9 @@ (define-inlinable (assert-integer proc in-range? key number) "Integer ~A out of range: ~S" (list key number) (list number)))) -(define* (sqlite-register db #:key path (references '()) - deriver hash nar-size - (time (timestamp))) +(define* (register-valid-path db #:key path (references '()) + deriver hash nar-size + (time (timestamp))) "Registers this stuff in DB. PATH is the store item to register and REFERENCES is the list of store items PATH refers to; DERIVER is the '.drv' that produced PATH, HASH is the base16-encoded Nix sha256 hash of @@ -249,7 +251,7 @@ (define* (sqlite-register db #:key path (references '()) (assert-integer "sqlite-register" (cut >= <> 0) #:time registration-time) (define id - (let ((existing-id (path-id db path))) + (let ((existing-id (select-valid-path-id db path))) (if existing-id (let ((stmt (sqlite-prepare db @@ -287,7 +289,8 @@ (define* (sqlite-register db #:key path (references '()) ;; Call 'path-id' on each of REFERENCES. This ensures we get a ;; "non-NULL constraint" failure if one of REFERENCES is unregistered. (add-references db id - (map (cut path-id db <>) references))) + (map (cut select-valid-path-id db <>) references))) + ;;; @@ -364,18 +367,18 @@ (define* (register-items db items ;; When TO-REGISTER is already registered, skip it. This makes a ;; significant differences when 'register-closures' is called ;; consecutively for overlapping closures such as 'system' and 'bootcfg'. - (unless (path-id db to-register) + (unless (select-valid-path-id db to-register) (let-values (((hash nar-size) (nar-sha256 real-file-name))) (call-with-retrying-transaction db (lambda () - (sqlite-register db #:path to-register - #:references (store-info-references item) - #:deriver (store-info-deriver item) - #:hash (string-append - "sha256:" - (bytevector->base16-string hash)) - #:nar-size nar-size - #:time registration-time)))))) + (register-valid-path db #:path to-register + #:references (store-info-references item) + #:deriver (store-info-deriver item) + #:hash (string-append + "sha256:" + (bytevector->base16-string hash)) + #:nar-size nar-size + #:time registration-time)))))) (let* ((prefix (format #f "registering ~a items" (length items))) (progress (progress-reporter/bar (length items)