From patchwork Mon Feb 24 14:38:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39115 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id D30A127BBE9; Mon, 24 Feb 2025 14:40:42 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 90B7F27BBE2 for ; Mon, 24 Feb 2025 14:40:41 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZcu-0006TJ-PA; Mon, 24 Feb 2025 09:40:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006Q1-Ma for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZco-0006hX-NV for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:From:To:In-Reply-To:References:Subject; bh=GZ9Vt/+3exncLsJkVeAUAzaPucqP2ol5s5FhXbuMpgM=; b=jR4YcYXBNmpvrYI3vtbPmLz4KiKWPhPKaJ49pT+4zYxHcvYxnr+dAbTJ7qGtzBVC6FfUQRVARuca6N0eSSUmArMwpNc6u3M06q0S2WDrf1r8tJMT1BIweT2Qk+kRPVT+YhptEaf+k6ThBRuJTs6ogVUqgkfUGhT/Rs+VEIw8/HH6+Rkgw3BuBKv6rQZ23jEkkNCuZa97q2pSfxQjXqJhftJDqK+WS+J5sPqvCd0XrZkoosHEjgnjfsnfIinADJcf7XJ7ORAJUcHbIzBxRpolgmiUU8U4lb7VduiazwWBL1ZlrzXS8gVeL9b9KA2aQo9PLcSQo88IAI5+Ds0vjbE/WA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZco-000136-Hx for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 1/7] base: =?utf-8?b?4oCYcmVtb3RlLWJ1?= =?utf-8?b?aWxkZXLigJk=?= listens for notifications from =?utf-8?b?4oCYY3Vp?= =?utf-8?b?cmFzcyByZW1vdGUtc2VydmVy4oCZLg==?= References: <20250221153551.22658-1-romain.garbage@inria.fr> In-Reply-To: <20250221153551.22658-1-romain.garbage@inria.fr> Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079573882 (code B ref 76474); Mon, 24 Feb 2025 14:40:02 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:17 +0000 Received: from localhost ([127.0.0.1]:39164 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZc3-00010W-VR for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:16 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZbv-0000zE-Mv for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=GZ9Vt/+3exncLsJkVeAUAzaPucqP2ol5s5FhXbuMpgM=; b=sbOBmxJ4VNV1xBiFvyWSqJZIGF7uxHWA00ywqX+MlS76tD824tqXywq6 hcjk6W6V2LRiGpgK2p546AVG10PTa9t4zO9ofI7cayZQdlkqYW0HIdBL9 2AL0cTnAeLmhygz8D2BB7BC21nuo1r/mH5gMVZw/7KB2MNAu47KCTUUje 8=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907805" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:38:59 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:06 +0100 Message-ID: <20250224143851.5789-1-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Ludovic Courtès * src/cuirass/base.scm (%remote-server-socket-file-name): New variable. (remote-builder-listener): New procedure. (remote-builder): Add ‘socket’ parameter. Call ‘remote-builder-listener’. Handle ‘build-status-change’ messages. (spawn-remote-builder): Add call to ‘open-unix-listening-socket’ and pass it to ‘remote-builder’. * src/cuirass/scripts/remote-server.scm (%notification-socket-pool): New variable. (send-build-status-change-notification, update-build-status): New procedures. (run-fetch, serve-build-requests): Call ‘update-build-status’ instead of ‘db-update-build-status!’ and ‘set-build-successful!’. (open-build-notification-socket): New procedure. (cuirass-remote-server): Use it and parameterize ‘%notification-socket-pool’. * tests/remote.scm (notification-server): New variable. (terminate-process): New procedure. (stop-worker, stop-server): Use it. (start-notification-server, stop-notification-server): New procedures. ("remote-server"): Call ‘start-notification-server’. ("clean-up"): Call ‘stop-notification-server’. --- src/cuirass/base.scm | 53 +++++++++++++-- src/cuirass/scripts/remote-server.scm | 96 ++++++++++++++++++--------- tests/remote.scm | 44 ++++++++++-- 3 files changed, 151 insertions(+), 42 deletions(-) base-commit: b6c2e340b24c4ea9701eb687669a091027bd361e diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 27eb89a..631e81f 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -90,6 +90,7 @@ ;; Parameters. %bridge-socket-file-name open-bridge-connection + %remote-server-socket-file-name %package-cachedir %fallback?)) @@ -166,6 +167,8 @@ (string-append (%cuirass-state-directory) "/run")))) (define %bridge-socket-file-name + ;; Socket 'cuirass register' listens to, allowing 'cuirass web' to connect + ;; to it. (make-parameter (string-append (%cuirass-run-state-directory) "/cuirass/bridge"))) @@ -178,6 +181,12 @@ socket." (connect sock AF_UNIX file) sock)) +(define %remote-server-socket-file-name + ;; Socket 'cuirass register' listens to, allowing 'cuirass remote-server' to + ;; connect to it. + (make-parameter (string-append (%cuirass-run-state-directory) + "/cuirass/remote-builds"))) + ;;; ;;; Read parameters. @@ -456,22 +465,56 @@ by handing them to the local build daemon." (spawn-fiber (local-builder channel)) channel)) -(define (remote-builder channel) +(define (remote-builder-listener socket channel) + "Spawn a server that accepts connections on SOCKET and forwards messages it +reads to CHANNEL." + (define (serve-client client) + (let loop () + (match (read client) + ((? eof-object?) + (log-info "terminating remote server client connection on EOF")) + (message + (put-message channel message) + (loop))))) + + (lambda () + (let loop () + (match (accept socket (logior SOCK_NONBLOCK SOCK_CLOEXEC)) + ((connection . peer) + (spawn-fiber (lambda () + (log-info "remote builder accepted connection: ~s" + connection) + (serve-client connection))) + (loop)))))) + +(define (remote-builder channel socket) + "Spawn a remote builder that accepts messages on CHANNEL and receives +notifications from 'cuirass remote-server' over SOCKET." (lambda () (log-info "builds will be delegated to 'cuirass remote-server'") + (spawn-fiber (remote-builder-listener socket channel)) + (let loop () (match (get-message channel) (`(build ,derivations) ;; Currently there's nothing to do here: 'cuirass remote-server' ;; periodically calls 'db-get-pending-build'. ;; TODO: Push notifications to 'remote-server' instead. - (log-info "~a pending derivation builds" (length derivations)))) + (log-info "~a pending derivation builds" (length derivations))) + (`(build-status-change ,derivation ,status) + ;; TODO: Handle database operations, notifications, etc. from here. + (log-info "status of '~a' changed to ~a" derivation status))) (loop)))) (define (spawn-remote-builder) - "Spawn a build actor that performs builds using \"remote workers\"." - (let ((channel (make-channel))) - (spawn-fiber (remote-builder channel)) + "Spawn a build actor that performs builds using \"remote workers\". Return +once ready to listen for incoming connections from 'cuirass remote-server'." + (log-info "listening for 'cuirass remote-server' notifications on '~a'" + (%remote-server-socket-file-name)) + (let ((channel (make-channel)) + (socket (open-unix-listening-socket + (%remote-server-socket-file-name)))) + (spawn-fiber (remote-builder channel socket)) channel)) diff --git a/src/cuirass/scripts/remote-server.scm b/src/cuirass/scripts/remote-server.scm index 90a3db7..ae45232 100644 --- a/src/cuirass/scripts/remote-server.scm +++ b/src/cuirass/scripts/remote-server.scm @@ -1,6 +1,6 @@ ;;; remote-server.scm -- Remote build server. ;;; Copyright © 2020, 2021 Mathieu Othacehe -;;; Copyright © 2023, 2024 Ludovic Courtès +;;; Copyright © 2023-2025 Ludovic Courtès ;;; ;;; This file is part of Cuirass. ;;; @@ -20,7 +20,8 @@ (define-module (cuirass scripts remote-server) #:autoload (cuirass base) (read-parameters set-build-successful! - spawn-build-maintainer) + spawn-build-maintainer + %remote-server-socket-file-name) #:use-module (cuirass config) #:use-module (cuirass database) #:use-module (cuirass logging) @@ -88,6 +89,10 @@ (define %publish-port (make-parameter #f)) +(define %notification-socket-pool + ;; Pool of sockets connected to 'cuirass register'. + (make-parameter #f)) + (define service-name "Cuirass remote server") @@ -288,6 +293,22 @@ signing key for URL is not authorized." #t)))) outputs)))) +(define (send-build-status-change-notification drv status) + "Send a notification to 'cuirass register' that the status of DRV changed to +STATUS." + (with-resource-from-pool (%notification-socket-pool) sock + (write `(build-status-change ,drv ,status) sock) + (newline sock))) + +(define* (update-build-status drv status + #:key log-file) + "Change the status of DRV to STATUS, both in the database and by sending a +notification to 'cuirass register'." + (if (= (build-status succeeded) status) + (set-build-successful! drv) + (db-update-build-status! drv status #:log-file log-file)) + (send-build-status-change-notification drv status)) + (define* (run-fetch message) "Read MESSAGE and download the corresponding build outputs. If %CACHE-DIRECTORY is set, download the matching NAR and NARINFO files in this @@ -311,15 +332,15 @@ directory." (add-to-store drv outputs url)) (begin (log-info "build succeeded: '~a'" drv) - (set-build-successful! drv)) + (update-build-status drv (build-status succeeded))) (begin (log-error "failed to retrieve output of \ successful build '~a'; rescheduling" drv) - (db-update-build-status! drv (build-status scheduled)))))) + (update-build-status drv (build-status scheduled)))))) (('build-failed ('drv drv) ('url url) _ ...) (log-info "build failed: '~a'" drv) - (db-update-build-status! drv (build-status failed))))) + (update-build-status drv (build-status failed))))) (define (fetch-worker channel max-parallel-downloads) (define queue-size @@ -529,7 +550,7 @@ Use WORKER-DIRECTORY to maintain the list of active workers." ('dependency dependency) _ ...) (log-info "build failed: dependency '~a' of '~a'" dependency drv) - (db-update-build-status! drv (build-status failed-dependency)) + (update-build-status drv (build-status failed-dependency)) (put-message build-maintainer 'failed-dependency) ;mark 'failed-dependency' builds (let ((parent (db-get-build drv))) (when parent @@ -567,7 +588,7 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (build-id build) derivation (build-system build))) (db-update-build-worker! derivation name) - (db-update-build-status! derivation (build-status submitted)) + (update-build-status derivation (build-status submitted)) (catch 'zmq-error (lambda () (reply-worker @@ -583,7 +604,7 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (worker-name worker) (worker-address worker) message) - (db-update-build-status! derivation + (update-build-status derivation (build-status scheduled))))) (begin (when worker @@ -605,8 +626,8 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (worker-name worker) drv)) (db-update-build-worker! drv name) - (db-update-build-status! drv (build-status started) - #:log-file log-file))) + (update-build-status drv (build-status started) + #:log-file log-file))) (`(build-rejected (drv ,drv) (worker ,name)) ;; Worker rejected the build, which might be either because the ;; derivation is unavailable or because of a transient error. In @@ -619,11 +640,11 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (begin (log-warning "~a: build rejected: ~a; rescheduling" name drv) - (db-update-build-status! drv (build-status scheduled))) + (update-build-status drv (build-status scheduled))) (begin (log-warning "~a: build rejected: ~a; canceling" name drv) - (db-update-build-status! drv (build-status canceled))))) + (update-build-status drv (build-status canceled))))) (_ (log-warning "ignoring unrecognized message: ~s" command))))) @@ -699,6 +720,15 @@ exiting." (terminate-helper-processes) (primitive-exit 1)))) +(define (open-build-notification-socket) + "Return a socket connected to the 'cuirass register' process, used to send +status updates." + (let ((sock (socket AF_UNIX + (logior SOCK_STREAM SOCK_CLOEXEC SOCK_NONBLOCK) + 0))) + (connect sock AF_UNIX (%remote-server-socket-file-name)) + sock)) + (define (cuirass-remote-server args) (signal-handler) (with-error-handling @@ -793,24 +823,30 @@ exiting." (run-fibers (lambda () - (with-database - (receive-logs log-port (%cache-directory)) - (spawn-notification-fiber) - (spawn-build-log-cleaner (assoc-ref opts 'build-log-expiry)) + (parameterize ((%notification-socket-pool + (make-resource-pool + (map (lambda (i) + (open-build-notification-socket)) + (iota 8)) + 'notification-socket))) + (with-database + (receive-logs log-port (%cache-directory)) + (spawn-notification-fiber) + (spawn-build-log-cleaner (assoc-ref opts 'build-log-expiry)) - (let ((fetch-worker (spawn-fetch-worker)) - (worker-directory (spawn-worker-directory))) - (catch 'zmq-error - (lambda () - (serve-build-requests backend-port - fetch-worker - worker-directory - #:build-maintainer - (spawn-build-maintainer))) - (lambda (key errno message . _) - (log-error (G_ "ZeroMQ error in build server: ~a") - message) - (terminate-helper-processes) - (primitive-exit 1)))))) + (let ((fetch-worker (spawn-fetch-worker)) + (worker-directory (spawn-worker-directory))) + (catch 'zmq-error + (lambda () + (serve-build-requests backend-port + fetch-worker + worker-directory + #:build-maintainer + (spawn-build-maintainer))) + (lambda (key errno message . _) + (log-error (G_ "ZeroMQ error in build server: ~a") + message) + (terminate-helper-processes) + (primitive-exit 1))))))) #:hz 0 #:parallelism (min 8 (current-processor-count))))))) diff --git a/tests/remote.scm b/tests/remote.scm index 797c92e..3fa9299 100644 --- a/tests/remote.scm +++ b/tests/remote.scm @@ -29,6 +29,7 @@ symbol)))) (cuirass specification) ((cuirass remote) #:select (worker-systems)) + ((cuirass base) #:select (spawn-remote-builder)) (gnu packages base) (guix build utils) (guix channels) @@ -39,6 +40,7 @@ ((guix store) #:hide (build)) ((guix utils) #:select (%current-system)) (tests common) + (fibers) (squee) (simple-zmq) ((zlib) #:select (call-with-gzip-input-port)) @@ -57,6 +59,9 @@ (define worker (make-parameter #f)) +(define notification-server + (make-parameter #f)) + (define (start-worker) (setenv "REQUEST_PERIOD" "1") (setenv "CUIRASS_LOGGING_LEVEL" "debug") @@ -67,10 +72,12 @@ "--public-key=tests/signing-key.pub") #:search-path? #t))) +(define (terminate-process pid) + (kill pid SIGINT) + (waitpid pid)) + (define (stop-worker) - (let ((worker (worker))) - (kill worker SIGINT) - (waitpid worker))) + (terminate-process (worker))) (define (start-server) (mkdir-p "tests/cache") @@ -85,9 +92,30 @@ #:search-path? #t))) (define (stop-server) - (let ((server (server))) - (kill server SIGINT) - (waitpid server))) + (terminate-process (server))) + +(define (start-notification-server) + ;; Spawn the notification server that normally runs as part of 'cuirass + ;; register', and which 'cuirass remote-server' connects to. Do so in a + ;; separate process because 'run-fibers' installs suspendable ports, which + ;; this process may not be able to deal with. + (notification-server + (spawn "guile" + (list "guile" "-c" + (object->string + '(begin + (use-modules (cuirass base) (fibers)) + + (setvbuf (current-output-port) 'none) + (setvbuf (current-error-port) 'none) + (run-fibers + (lambda () + (spawn-remote-builder) + (sleep 120)) ;wait + #:drain? #t))))))) + +(define (stop-notification-server) + (terminate-process (notification-server))) (define* (dummy-drv #:optional sleep #:key (name "foo") dependency message) (let ((dependency (and=> dependency read-derivation-from-file))) @@ -185,6 +213,7 @@ (test-assert "remote-server" (begin + (start-notification-server) (start-server) #t)) @@ -384,4 +413,5 @@ Failing dependency ~s.\n" (test-assert "clean-up" (begin (stop-worker) - (stop-server)))) + (stop-server) + (stop-notification-server)))) From patchwork Mon Feb 24 14:38:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39114 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 3ED7A27BBEA; Mon, 24 Feb 2025 14:40:38 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 9DD8427BBE2 for ; Mon, 24 Feb 2025 14:40:37 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZcs-0006QT-2G; Mon, 24 Feb 2025 09:40:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZco-0006OW-F3 for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:02 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZco-0006gm-5H for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=w6UjAkouCcw3gN6AWlaFIu9pL/FGYAezoRSZprU3cj8=; b=D6m1zgV4VI8sKOA+nuSUT9IWL4Uoz+FtYgUz1GuxZWGnZdY/MVNUGY+M2eZ0iVmzRzxIIcT0LGLr6GxVhqd7d0VeLYfBL4P9Ahipndq29uVXD/1/pjoBUsSd1/KzLrD96sLywzSbjCPWESXeEapgu7VzO/i1BZObmn1xrnmDzFpZeqXt+vcZa7wD6Wb5jS9GZPdxyhYO7ppKQTxGc2lgEhZl1zyGSFo1MH6u92SpyRgun0G9H7LnHq0i2QtmsjAUPMa9tA3et+/6UT7X3Bq8DkD2gcWshER27ZWn65mJ3p88h1BdcCSoEJtoceHKj/1jarv/NNiRKfnUbfmHh1W9cA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZcn-00012t-TD for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 2/7] utils: Add ring buffer implementation. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079493840 (code B ref 76474); Mon, 24 Feb 2025 14:40:01 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:09 +0000 Received: from localhost ([127.0.0.1]:39157 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZbx-0000zn-7U for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:09 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZbu-0000zE-BS for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=w6UjAkouCcw3gN6AWlaFIu9pL/FGYAezoRSZprU3cj8=; b=cOfFgwVEEq3hsL+N6CR2MQ1aDcljc9y4LVoHXGKVKBJsVOOyV0jqdAE1 ikMEwv1lgbAo8Bkp0Pr5UjFQqeornXA6Zv8OX+P/bU1uMnXKmQ42pugJb drez5lVyK/2O3dXofnRE1Eimztc3pLg7kR7GLnvvamRgMLKPXYmEFIwKb 0=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907807" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:39:00 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:07 +0100 Message-ID: <20250224143851.5789-2-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224143851.5789-1-romain.garbage@inria.fr> References: <20250224143851.5789-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/utils.scm (): New record type. (ring-buffer, ring-buffer-insert, ring-buffer->list): New variables. * tests/utils.scm: Add tests. --- src/cuirass/utils.scm | 67 ++++++++++++++++++++++++++++++++++++++++++- tests/utils.scm | 17 +++++++++++ 2 files changed, 83 insertions(+), 1 deletion(-) diff --git a/src/cuirass/utils.scm b/src/cuirass/utils.scm index c406a84..b06f451 100644 --- a/src/cuirass/utils.scm +++ b/src/cuirass/utils.scm @@ -28,6 +28,7 @@ #:use-module (rnrs bytevectors) #:use-module (system foreign) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) #:use-module (srfi srfi-19) #:use-module (srfi srfi-71) #:autoload (guix build utils) (mkdir-p) @@ -60,7 +61,15 @@ open-unix-listening-socket atomic-box-fetch-and-increment! - atomic-box-fetch-and-decrement!)) + atomic-box-fetch-and-decrement! + + ring-buffer + ring-buffer-head + ring-buffer-head-length + ring-buffer-limit + ring-buffer-rear + ring-buffer-insert + ring-buffer->list)) (define-exception-type &cuirass-assertion-failure &assertion-failure make-cuirass-assertion-failure @@ -347,3 +356,59 @@ and store the result inside the BOX." (define (atomic-box-fetch-and-decrement! box) "Atomically decrement the value of the integer stored inside the given BOX." (atomic-box-fetch-and-update! box 1-)) + +;;; +;;; Ring buffer implementation. Copied from GNU Shepherd. +;;; + +;; Helper function needed by ring-buffer->list. +(define (at-most max-length lst) + "If @var{lst} is shorter than @var{max-length}, return it and the empty list; +otherwise return its @var{max-length} first elements and its tail." + (let loop ((len 0) + (lst lst) + (result '())) + (match lst + (() + (values (reverse result) '())) + ((head . tail) + (if (>= len max-length) + (values (reverse result) lst) + (loop (+ 1 len) tail (cons head result))))))) + +;; The poor developer's persistent "ring buffer": it holds between N and 2N +;; elements, but has O(1) insertion. +(define-record-type + (%ring-buffer limit front-length front rear) + ring-buffer? + (limit ring-buffer-limit) + (front-length ring-buffer-front-length) + (front ring-buffer-front) + (rear ring-buffer-rear)) + +(define (ring-buffer size) + "Return an ring buffer that can hold @var{size} elements." + (%ring-buffer size 0 '() '())) + +(define-inlinable (ring-buffer-insert element buffer) + "Insert @var{element} to the front of @var{buffer}. If @var{buffer} is +already full, its oldest element is removed." + (match buffer + (($ limit front-length front rear) + (if (< front-length limit) + (let ((front-length (+ 1 front-length))) + (%ring-buffer limit front-length + (cons element front) + (if (= limit front-length) + '() + rear))) + (%ring-buffer limit 1 + (list element) front))))) + +(define (ring-buffer->list buffer) + "Convert @var{buffer} into a list." + (match buffer + (($ limit front-length front rear) + (if (= limit front-length) + front + (append front (at-most (- limit front-length) rear)))))) diff --git a/tests/utils.scm b/tests/utils.scm index e1ac1b8..4a3b48f 100644 --- a/tests/utils.scm +++ b/tests/utils.scm @@ -83,4 +83,21 @@ (const 42))))) #:to 'value)) +(test-equal "ring-buffer->list 1 element" + '(el) + (let ((buffer (ring-buffer 5))) + (ring-buffer->list (ring-buffer-insert 'el buffer)))) + +(test-equal "ring-buffer->list empty buffer" + '() + (ring-buffer->list (ring-buffer 5))) + +(test-equal "ring-buffer->list full" + '(9 8 7 6 5) + (ring-buffer->list (fold + (lambda (e r) + (pk 'r (ring-buffer-insert e r))) + (ring-buffer 5) + (iota 10)))) + (test-end) From patchwork Mon Feb 24 14:38:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39116 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 6C86327BBEA; Mon, 24 Feb 2025 14:40:50 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B9AF027BBE2 for ; Mon, 24 Feb 2025 14:40:49 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZcz-0006Ws-2S; Mon, 24 Feb 2025 09:40:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006QO-Te for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZcp-0006lu-BK for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=uzkvuqLHyHAZz9+lIY+UkXlkd7Ju6e2swNiwQ8sGj2Q=; b=u7rAkchzZlMRrlrZbeHYPZmxzwXrmDQWJzG+BHAc0c07cefpsAaUE8XLca8VfD+bHzRrVCru6WQ9P3YkHnPsJVm5JnLhdYviioaQ8kU4+tsOwNYgaX9p+vt57GqlEb+kds0kehgJ2dHrj3dzEMwOYb9EbbgXTISsq/3jL/ypODuBxZyAjAtbkLejymDOFEXTp1gVQQKFRyNl/ktKMm1aDmF/5aZR2XGu2uRCoNLO6jv68rnb6XXnP8YkP9ZVJ/HaEIc1Mbndw/rk3eUHsZAvdnbcg3UfYFcqWCu8Us55F1b/PewciRCYzcvcx1ukuQbG1mDHM1wFILb/Lx2ou2BWAw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZcp-00013Q-4I for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 3/7] base: Add event-log agent. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079573890 (code B ref 76474); Mon, 24 Feb 2025 14:40:03 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:17 +0000 Received: from localhost ([127.0.0.1]:39166 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZc4-00010Z-Tw for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:17 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZbx-0000zE-Di for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uzkvuqLHyHAZz9+lIY+UkXlkd7Ju6e2swNiwQ8sGj2Q=; b=dwL6bT/FqUMVM7HnjtV0QaT871kzN5Eu3i4lVijPbRA5e/ACF8knoWN5 UnEusofQswfWqjDvgnRg1G8lZFXq+R+dXp5pzwwOTRSMpO9uvUMjlgECg 7OCt/rScszQrhyRolhzCW154MwYdSFcyILO/Qea32mEjxoKMOnmak2Kj5 M=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907808" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:39:00 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:08 +0100 Message-ID: <20250224143851.5789-3-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224143851.5789-1-romain.garbage@inria.fr> References: <20250224143851.5789-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (event-log-service, spawn-event-log-service): New variables. * tests/base.scm: Add tests for the event-log-service agent. --- src/cuirass/base.scm | 71 +++++++++++++++++++++++++++++++++++++++++++- tests/base.scm | 46 ++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+), 1 deletion(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 631e81f..fa3481f 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -73,6 +73,7 @@ spawn-remote-builder spawn-channel-update-service spawn-jobset-evaluator + spawn-event-log-service spawn-jobset-registry spawn-gc-root-cleaner spawn-build-maintainer @@ -113,7 +114,11 @@ ;;; 'guix-daemon' process, while the remote build delegates builds to ;;; 'cuirass remote-server'. ;;; -;;; - Each jobset as an associated "monitor"; it requests channel updates, +;;; - The "event-log" agent receives events from the different agents. It +;;; also dispatches these events to the agents which subscribed to event +;;; notification. +;;; +;;; - Each jobset has an associated "monitor"; it requests channel updates, ;;; evaluations, and builds to the actors above. It also receives requests ;;; such as evaluation triggers that can come, for example, from the ;;; /jobset/NAME/hook/evaluate HTTP endpoint. @@ -770,6 +775,70 @@ concurrently; it sends derivation build requests to BUILDER." max-parallel-evaluations)) channel)) +;;; +;;; Logging events +;;; + +(define event-log-buffer-size + (make-parameter 1000)) + +(define (event-log-service channel) + "Keep events received on CHANNEL in a circular buffer, keeping track of +notification subscriptions." + (lambda () + (define events (ring-buffer (event-log-buffer-size))) + + (let loop ((events events) + (subscribers '())) + (match (get-message channel) + (`(subscribe ,channel) + (loop events (cons channel subscribers))) + (`(unsubscribe ,channel) + (loop events (delq channel subscribers))) + (`(new-event ,event) + ;; Events are stored as a list in the format + ;; (event-type timestamp rest-of-data) + (let* ((data (match event + ((event-type . rest) + (append `(,event-type ,(current-time time-utc)) + rest)))) + (events (ring-buffer-insert data events))) + (match event + ;; This is what is received from the builders. This code aims to + ;; rebuild the original build object related to the derivation + ;; (when it exists), so it can be used by other agents. + (`(derivation-built ,derivation ,status) + (spawn-fiber + (lambda () + (let ((build (db-get-build derivation))) + (when build + (put-message channel + `(new-event (build-status-changed ,build)))))))) + (_ #t)) + ;; For now, every new event is sent to all subscribers. + (for-each (lambda (reply) + (put-message reply data)) + subscribers) + (loop events subscribers))) + (`(recent-events ,reply) + (put-message reply (ring-buffer->list events)) + (loop events subscribers)) + ;; Catchall for malformed messages. + (message + (log-error "malformed message sent to the event-log-service: ~s" + message) + (loop events subscribers)))))) + +(define (spawn-event-log-service) + "Spawn an actor responsible for centralizing events." + (let ((channel (make-channel))) + (spawn-fiber (event-log-service channel)) + channel)) + +;;; +;;; Monitoring jobsets +;;; + (define %jobset-trigger-rate-window ;; Window (seconds) over which the jobset trigger rate is computed. (* 5 60)) ;5 minutes diff --git a/tests/base.scm b/tests/base.scm index 1cfbecb..e3709ee 100644 --- a/tests/base.scm +++ b/tests/base.scm @@ -17,6 +17,10 @@ ;;; along with Cuirass. If not, see . (use-modules (cuirass base) + (fibers) + (fibers channels) + (ice-9 match) + (srfi srfi-1) (srfi srfi-64)) (test-begin "base") @@ -25,4 +29,46 @@ 'wrong-type-arg (%package-cachedir #f)) +(test-equal "event-log recent-events" + ;; Elements are in reverse order since they come from a ring-buffer. + '((test-event "Third message") + (test-event "Second message") + (test-event "First message")) + (run-fibers + (lambda () + (let ((event-log (spawn-event-log-service)) + (reply-channel (make-channel))) + (put-message event-log '(new-event (test-event "First message"))) + (put-message event-log '(new-event (test-event "Second message"))) + (put-message event-log '(new-event (test-event "Third message"))) + (put-message event-log `(recent-events ,reply-channel)) + (filter-map (match-lambda + ((type timestamp value) + ;; Drop timestamp. + (list type value)) + (_ #f)) + (get-message reply-channel)))))) + +(test-equal "event-log recent-events (empty buffer)" + '() + (run-fibers + (lambda () + (let ((event-log (spawn-event-log-service)) + (reply-channel (make-channel))) + (put-message event-log `(recent-events ,reply-channel)) + (get-message reply-channel))))) + +(test-equal "event-log subscribe" + '(test-event "test-value") + (run-fibers + (lambda () + (let ((event-log (spawn-event-log-service)) + (reply (make-channel))) + (put-message event-log `(subscribe ,reply)) + (put-message event-log '(new-event (test-event "test-value"))) + (match (get-message reply) + ((type timestamp value) + ;; Drop timestamp. + (list type value))))))) + (test-end) From patchwork Mon Feb 24 14:38:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39118 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8AA2E27BBE9; Mon, 24 Feb 2025 14:41:04 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7D2CD27BBE2 for ; Mon, 24 Feb 2025 14:41:02 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZcu-0006TB-LZ; Mon, 24 Feb 2025 09:40:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006QQ-UY for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZcp-0006r6-Te for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=00WZ/i2HqcJp6qPO+qRrox25u5AjxV03QDlLNXpueH8=; b=g87w0E2kLeKK6KuEVd9teE9e0fB5jGk8JELVi5sXT4FH/uS7dbZLJIV0RhcdD23NzNRj5ZgAE6a4NrrxrkeWtct6VXbhOTEcfa+aue2Rh3hLHPzUkqNvJA2OidMLTE5nDPe8qR+A6VrXOwA4VNIsb0jtJ/GZmuDfcYX2UWKylHzlDGenCdoLzoH1HrbEuzM/zDo7Yd0EiRm7SQdEOvqY3KqPO6WKKmXkakkr1jUbc376oOtA+/DPdTbKzD8l00HsbhfBoqscy08aJB8AW8GDajdwBnNnMZbZayGIJiIkD6QDVgYnfGWgaGCtNGlAgV85nDLIVxL4RVJ2fKKM8TPtew==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZcp-00013Y-LZ for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 4/7] register: Add support for event logging in builders. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079583898 (code B ref 76474); Mon, 24 Feb 2025 14:40:03 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:18 +0000 Received: from localhost ([127.0.0.1]:39168 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZc5-00010h-Lu for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:18 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZby-0000zE-J0 for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=00WZ/i2HqcJp6qPO+qRrox25u5AjxV03QDlLNXpueH8=; b=prYZURtHRa7c3lvHSZK1qyqoBMOKCwzvW2+rXm8+3KOS4NFsaS7SzkMX ljlP2u6y8yfKnAT0uIEdlm1WKrcc+i2NcxzCIFWF90pimr39bOqjwlRld zbFbPpMNrYhVqu5YbGeHU4QTKjE/s0qiauFX28IStGhqhreUEJHZHvBfr w=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907810" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:39:00 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:09 +0100 Message-ID: <20250224143851.5789-4-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224143851.5789-1-romain.garbage@inria.fr> References: <20250224143851.5789-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (local-builder, spawn-local-builder, remote-builder, spawn-remote-builder): Add support for logging events to the corresponding agent through Fibers. * src/cuirass/scripts/register.scm (cuirass-register): Add event-log-service agent and use it in builders. * tests/remote.scm (start-notification-server): Add event-log-service agent. --- src/cuirass/base.scm | 29 ++++++++++++++++++++--------- src/cuirass/scripts/register.scm | 5 +++-- tests/remote.scm | 7 +++++-- 3 files changed, 28 insertions(+), 13 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index fa3481f..efa3dc8 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -439,7 +439,7 @@ OUTPUTS, a list of records." (checksum "")))))) ;TODO: Implement it. outputs)) -(define (local-builder channel) +(define (local-builder channel event-log) (lambda () (log-info "builds will be made via the local build daemon") (let loop () @@ -460,14 +460,22 @@ OUTPUTS, a list of records." (fail (- (length derivations) success))) (log-info "outputs:\n~a" (string-join outs "\n")) + ;; Log the same kind of information as remote-builder. + (for-each (lambda (build) + (put-message event-log + `(new-event (derivation-built + ,(build-derivation build) + ,(build-current-status build))))) + results) results)))))) (loop)))) -(define (spawn-local-builder) +(define (spawn-local-builder event-log) "Spawn a build actor that executes the derivation build requests it receives -by handing them to the local build daemon." +by handing them to the local build daemon. The build actor sends the build +status to the event log service through EVENT-LOG, a Fibers channel." (let ((channel (make-channel))) - (spawn-fiber (local-builder channel)) + (spawn-fiber (local-builder channel event-log)) channel)) (define (remote-builder-listener socket channel) @@ -492,9 +500,10 @@ reads to CHANNEL." (serve-client connection))) (loop)))))) -(define (remote-builder channel socket) +(define (remote-builder channel socket event-log) "Spawn a remote builder that accepts messages on CHANNEL and receives -notifications from 'cuirass remote-server' over SOCKET." +notifications from 'cuirass remote-server' over SOCKET. It also sends the log +events to the event log service using EVENT-LOG, a Fibers channel." (lambda () (log-info "builds will be delegated to 'cuirass remote-server'") (spawn-fiber (remote-builder-listener socket channel)) @@ -508,10 +517,12 @@ notifications from 'cuirass remote-server' over SOCKET." (log-info "~a pending derivation builds" (length derivations))) (`(build-status-change ,derivation ,status) ;; TODO: Handle database operations, notifications, etc. from here. - (log-info "status of '~a' changed to ~a" derivation status))) + (log-info "status of '~a' changed to ~a" derivation status) + (put-message event-log + `(new-event (derivation-built ,derivation ,status))))) (loop)))) -(define (spawn-remote-builder) +(define (spawn-remote-builder event-log) "Spawn a build actor that performs builds using \"remote workers\". Return once ready to listen for incoming connections from 'cuirass remote-server'." (log-info "listening for 'cuirass remote-server' notifications on '~a'" @@ -519,7 +530,7 @@ once ready to listen for incoming connections from 'cuirass remote-server'." (let ((channel (make-channel)) (socket (open-unix-listening-socket (%remote-server-socket-file-name)))) - (spawn-fiber (remote-builder channel socket)) + (spawn-fiber (remote-builder channel socket event-log)) channel)) diff --git a/src/cuirass/scripts/register.scm b/src/cuirass/scripts/register.scm index e32fe52..1698a8c 100644 --- a/src/cuirass/scripts/register.scm +++ b/src/cuirass/scripts/register.scm @@ -271,9 +271,10 @@ (if one-shot? (leave (G_ "'--one-shot' is currently unimplemented~%")) (let* ((exit-channel (make-channel)) + (event-log-service (spawn-event-log-service)) (builder (if (option-ref opts 'build-remote #f) - (spawn-remote-builder) - (spawn-local-builder))) + (spawn-remote-builder event-log-service) + (spawn-local-builder event-log-service))) (evaluator (spawn-jobset-evaluator #:max-parallel-evaluations max-parallel-evaluations diff --git a/tests/remote.scm b/tests/remote.scm index 3fa9299..e7856ba 100644 --- a/tests/remote.scm +++ b/tests/remote.scm @@ -29,7 +29,8 @@ symbol)))) (cuirass specification) ((cuirass remote) #:select (worker-systems)) - ((cuirass base) #:select (spawn-remote-builder)) + ((cuirass base) #:select (spawn-remote-builder + spawn-event-log-service)) (gnu packages base) (guix build utils) (guix channels) @@ -110,7 +111,9 @@ (setvbuf (current-error-port) 'none) (run-fibers (lambda () - (spawn-remote-builder) + (spawn-remote-builder + ;; A channel for event logs must be set up. + (spawn-event-log-service)) (sleep 120)) ;wait #:drain? #t))))))) From patchwork Mon Feb 24 14:38:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39113 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 0ABF227BBEA; Mon, 24 Feb 2025 14:40:37 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 82C9727BBE9 for ; Mon, 24 Feb 2025 14:40:36 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZcu-0006TQ-TN; Mon, 24 Feb 2025 09:40:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006QN-Tf for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZcq-0006sS-HZ for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=iSs2K3p3TnntSkPqQkAPP1SbdCN73rnfBdXh/kjsqLc=; b=eIGZZt3YG4Y8K5+8M6SM60P1B8Xq93ZmlM/DyaZV1vV4esG5I9lmyjHpDEOl8l6xRwJ2xfkfMmyFGjPaMIBYDmHCQaMyw2hDNrRjNxfmgK11d1owIYYc78qDofRhRKDPG725PM3eFkNSHB6RTQ6lqQEcsqKrYLROL/PNrB0vAGHqesW9vnJWbCY+Y4LQTKVZdNpjb7R5ExjrUuFCnnnl4u/FOOtD8BywwGVmo81h2079Hn9S/FrU2LOtCSLtYsgCLCPyezzxM+kFQcBt/FQgzgvyYdmgBRFvBspOiKAoi/vDYdYuS4CmVlfk2iLzTANCCyK7F+djEmyD3l/Ez5TcCQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZcq-00013x-8Q for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 5/7] register: Add support for event logging in the jobset evaluator. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079583905 (code B ref 76474); Mon, 24 Feb 2025 14:40:04 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:18 +0000 Received: from localhost ([127.0.0.1]:39170 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZc6-00010p-7k for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:18 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZbz-0000zE-Ik for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iSs2K3p3TnntSkPqQkAPP1SbdCN73rnfBdXh/kjsqLc=; b=TOysvfTNA0AaYzs04u7X5sub9+QeTDTKYZPC5lf6mV7wfSlvIGduoEob NtLAWmaEuxFimSezU1VWFVKZSTgz5xyDc36wXaoUr5+AkFbbcA6S7A2Un UzIFvNzvspIimAi0wdw06bxssh9DojgeTRnI8jojNen/gfLOxGqs3OQoL s=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907812" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:39:00 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:10 +0100 Message-ID: <20250224143851.5789-5-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224143851.5789-1-romain.garbage@inria.fr> References: <20250224143851.5789-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (spawn-jobset-evaluator, jobset-evaluator): Add support for event logging. * src/cuirass/scripts/register.scm (cuirass-register): Add support for event logging in jobset evaluator. --- src/cuirass/base.scm | 17 ++++++++++++++--- src/cuirass/scripts/register.scm | 3 ++- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index efa3dc8..41e0d5f 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -699,10 +699,11 @@ Return a list of jobs that are associated to EVAL-ID." (name (specification-name spec)) (id eval-id))))))))) -(define (start-evaluation spec instances timestamp) +(define (start-evaluation spec instances timestamp event-log) "Start an evaluation of SPEC using the given channel INSTANCES. Return #f if nothing has changed (and thus no new evaluation was created), otherwise return -the ID of the new evaluation." +the ID of the new evaluation. EVENT-LOG is a Fibers channel used to return the +evaluation ID before the evaluation is started." (let* ((channels (map channel-instance-channel instances)) (new-spec (specification (inherit spec) @@ -726,6 +727,11 @@ the ID of the new evaluation." ;; dependencies that are not declared in the initial specification ;; channels. Update the given SPEC to take them into account. (db-add-or-update-specification new-spec) + ;; Since evaluate blocks until the end of evaluation, an + ;; evaluation-started event is sent just before starting the + ;; evaluation. + (put-message event-log + `(new-event (evaluation-started ,eval-id ,spec))) (evaluate spec eval-id) (db-set-evaluation-time eval-id) @@ -734,6 +740,7 @@ the ID of the new evaluation." (define* (jobset-evaluator channel #:key builder + event-log (max-parallel-evaluations (current-processor-count))) (define pool @@ -752,13 +759,15 @@ the ID of the new evaluation." (with-resource-from-pool pool token (log-info "evaluating '~a' with token #~a" (specification-name spec) token) - (start-evaluation spec instances timestamp))) + (start-evaluation spec instances timestamp event-log))) (when eval-id (let* ((builds (db-get-builds `((evaluation . ,eval-id)))) (derivations (map build-derivation builds))) (log-info "evaluation ~a of jobset '~a' registered ~a builds" eval-id (specification-name spec) (length builds)) + (put-message event-log + `(new-event (evaluation-completed ,eval-id ,spec))) (db-set-evaluation-status eval-id (evaluation-status succeeded)) @@ -774,6 +783,7 @@ the ID of the new evaluation." (define* (spawn-jobset-evaluator #:key builder + event-log (max-parallel-evaluations (current-processor-count))) "Spawn the actor responsible for evaluating jobsets for a given spec and set @@ -782,6 +792,7 @@ concurrently; it sends derivation build requests to BUILDER." (let ((channel (make-channel))) (spawn-fiber (jobset-evaluator channel #:builder builder + #:event-log event-log #:max-parallel-evaluations max-parallel-evaluations)) channel)) diff --git a/src/cuirass/scripts/register.scm b/src/cuirass/scripts/register.scm index 1698a8c..e977458 100644 --- a/src/cuirass/scripts/register.scm +++ b/src/cuirass/scripts/register.scm @@ -278,7 +278,8 @@ (evaluator (spawn-jobset-evaluator #:max-parallel-evaluations max-parallel-evaluations - #:builder builder)) + #:builder builder + #:event-log event-log-service)) (update-service (spawn-channel-update-service))) (clear-build-queue) From patchwork Mon Feb 24 14:38:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39112 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 5262827BBEA; Mon, 24 Feb 2025 14:40:36 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id F096327BBE2 for ; Mon, 24 Feb 2025 14:40:35 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZct-0006ST-Jq; Mon, 24 Feb 2025 09:40:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006QR-VI for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006sY-9T for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=DJVUmYKs0dbN81AwLTbNd/fBnvCyD7PMnZ7hnHmvU94=; b=aSITEtK7yTnHTFjWR9SNHbqln+wY4BAELmupoGKoGevtXrcV2rjy8n5Te0mUlAcBFyOymyo5IIoGINTxPAo7fuJcDf04M8UtpYIdljSQSUIKUGWjkVCp0o4iti829jnp/+/T2daTmUwM3wMgP2MCQzWL9Muvy2tVxt5ZcgTHcaYwqSK12DzwGDtqYgnmzQhuT6KlyQaNaRgpBnQHnB9Yp255QpWszVgjZhqCp0aEw5tEL5nzOMZMuZdPRoJJNMXDoqX1YTKhfzTWayNbLpauS/n6ZmfdJHALwssK3IRHm2psfFi7O0FQRfHeN4LoYFARQUVr7c+osc+oC7bfkEghnw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZcr-00014p-2u for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 6/7] register: Add support for event logging in the jobset registry. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:05 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079593913 (code B ref 76474); Mon, 24 Feb 2025 14:40:05 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:19 +0000 Received: from localhost ([127.0.0.1]:39172 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZc6-00010w-Ne for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:19 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZc0-0000zE-Gq for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DJVUmYKs0dbN81AwLTbNd/fBnvCyD7PMnZ7hnHmvU94=; b=ecvPe0OrBcJ83MtwCWoePVaXHfgqvp7VvJN39zPGDOIpZpMbtgrW1oP6 welITjFbQ0WMC+H8AL2xG252y5RHKksVTad0HqrgguDPbLbYTn6l4InAa urjLu81p4+tqzlc/mvZyj8uAJgL8Rh4Vr7dSC5jT5a/7E1HmReUJ94/sr 4=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907814" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:39:00 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:11 +0100 Message-ID: <20250224143851.5789-6-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224143851.5789-1-romain.garbage@inria.fr> References: <20250224143851.5789-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (jobset-registry, spawn-jobset-registry): Add support for event logging. * src/cuirass/scripts/register.scm (cuirass-register): Add log event support to the jobset registry. --- src/cuirass/base.scm | 11 +++++++++-- src/cuirass/scripts/register.scm | 1 + 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 41e0d5f..3ef6c39 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -1003,7 +1003,8 @@ POLLING-PERIOD seconds." (define* (jobset-registry channel #:key (polling-period 60) - update-service evaluator) + update-service evaluator + event-log) (lambda () (spawn-fiber (lambda () @@ -1027,6 +1028,8 @@ POLLING-PERIOD seconds." (#f (log-error "cannot update non-existent spec '~s'" name)) ((_ . monitor) + (put-message event-log + `(new-event (jobset-updated ,spec))) (put-message monitor `(update-spec ,spec))))) (loop registry)) (`(register ,spec) @@ -1042,6 +1045,8 @@ POLLING-PERIOD seconds." #:polling-period period)) (name (specification-name spec))) (log-info "registering new jobset '~a'" name) + (put-message event-log + `(new-event (jobset-registered ,spec))) (loop (vhash-consq name monitor registry)))) ((_ . monitor) @@ -1050,13 +1055,15 @@ POLLING-PERIOD seconds." (loop registry)))))))) (define* (spawn-jobset-registry #:key (polling-period 60) - update-service evaluator) + update-service evaluator + event-log) "Spawn a jobset registry. In turn, the registry creates a new jobset monitoring actor for each 'register' message it receives." (let ((channel (make-channel))) (spawn-fiber (jobset-registry channel #:update-service update-service #:evaluator evaluator + #:event-log event-log #:polling-period polling-period)) channel)) diff --git a/src/cuirass/scripts/register.scm b/src/cuirass/scripts/register.scm index e977458..38db9a5 100644 --- a/src/cuirass/scripts/register.scm +++ b/src/cuirass/scripts/register.scm @@ -302,6 +302,7 @@ (let ((registry (spawn-jobset-registry #:update-service update-service #:evaluator evaluator + #:event-log event-log-service #:polling-period interval))) ;; Spawn the bridge through which other 'cuirass' ;; processes, such as 'cuirass web', may talk to the From patchwork Mon Feb 24 14:38:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 39117 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id D6FB127BBE9; Mon, 24 Feb 2025 14:40:53 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7BF3D27BBE2 for ; Mon, 24 Feb 2025 14:40:53 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tmZd0-0006Xd-L1; Mon, 24 Feb 2025 09:40:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tmZcs-0006Qt-4C for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:06 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tmZcr-0006sz-Qt for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=RGBWntLgnIxa/ngd9F6Xzh+O4y5rHjBY7Tp9eN2ETYE=; b=i6E1P6KtAbH6qqcB8/9TuSaCDK4PLXRRLifaDqktBXC0j2YV3LPUhpXRHr+wkH9M1xpDgwglvCzstV+iNMcC13+bFKLni8Yt7ckSc3X2pVrc6nyMdSwAFXffE6jGUYFqrtP/yIuxl/HbHOZZGdlO6mpENtInABkm2DgsJmUDY2uJHcagRTuAvNAYnaScSRHsk+W34uYJD6iXymt6+cV0zGcICH2KrX9XtVYhmLdBQwcLWNDLYOpjaRH9kyyyy3iOqBmhYaaNB19wf1jVMIgAKZcfXOfatkirEs6lqbqqkUSlO/JUqihloDlBBmTj+RDpwuZE2ce1iGEaPiAgmEgxmQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tmZcr-00015P-LD for guix-patches@gnu.org; Mon, 24 Feb 2025 09:40:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass v3 7/7] base: Add support for event logging in jobset-monitor. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 24 Feb 2025 14:40:05 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17404079593919 (code B ref 76474); Mon, 24 Feb 2025 14:40:05 +0000 Received: (at 76474) by debbugs.gnu.org; 24 Feb 2025 14:39:19 +0000 Received: from localhost ([127.0.0.1]:39174 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tmZc7-000114-4M for submit@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:19 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:10205) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tmZc1-0000zE-EB for 76474@debbugs.gnu.org; Mon, 24 Feb 2025 09:39:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RGBWntLgnIxa/ngd9F6Xzh+O4y5rHjBY7Tp9eN2ETYE=; b=X6wKiQi/tdn9Q2TsmNvW7wzu9KNcsH9q7ZLzEtyPdDmvjOdDyRWlcp1s tMwVdaQG2LiTwioZBf24gw8yVu10EGhPPofP34Nca5/F0B20j4xkI8ZWp TolFcMjW+IPkHYWoA6Erkoq+Zj9EcMp/irWGJj6GNo7tE+FLGaNkOWQeg I=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,311,1732575600"; d="scan'208";a="109907816" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.221]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2025 15:39:00 +0100 From: Romain GARBAGE Date: Mon, 24 Feb 2025 15:38:12 +0100 Message-ID: <20250224143851.5789-7-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250224143851.5789-1-romain.garbage@inria.fr> References: <20250224143851.5789-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (jobset-monitor, spawn-jobset-monitor): Add support for event logging. --- src/cuirass/base.scm | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 3ef6c39..c3a0fb6 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -871,7 +871,7 @@ notification subscriptions." (define* (jobset-monitor channel spec #:key (polling-period 60) - update-service evaluator) + update-service evaluator event-log) (define name (specification-name spec)) (lambda () @@ -984,7 +984,8 @@ notification subscriptions." (define* (spawn-jobset-monitor spec #:key (polling-period 60) - update-service evaluator) + update-service evaluator + event-log) "Spawn an actor responsible for monitoring the jobset corresponding to SPEC, a record, and return it. The actor will send messages to UPDATE-SERVICE anytime it needs Guix channels to be updated, at most every @@ -993,7 +994,8 @@ POLLING-PERIOD seconds." (spawn-fiber (jobset-monitor channel spec #:update-service update-service #:evaluator evaluator - #:polling-period polling-period)) + #:polling-period polling-period + #:event-log event-log)) channel)) @@ -1042,7 +1044,8 @@ POLLING-PERIOD seconds." #:update-service update-service #:evaluator evaluator - #:polling-period period)) + #:polling-period period + #:event-log event-log)) (name (specification-name spec))) (log-info "registering new jobset '~a'" name) (put-message event-log