From patchwork Fri Feb 21 15:40:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 38918 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 83BF227BBEC; Fri, 21 Feb 2025 15:42:47 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 7158F27BBE2 for ; Fri, 21 Feb 2025 15:42:46 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tlVAE-0000ub-PH; Fri, 21 Feb 2025 10:42:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tlVAC-0000uJ-Nm for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:04 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tlVAC-0002Rn-Eo for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:From:To:In-Reply-To:References:Subject; bh=GZ9Vt/+3exncLsJkVeAUAzaPucqP2ol5s5FhXbuMpgM=; b=UC0mtvdP4P7cUnWtoqR0ONBGo4YThQCnpNDCQ8DxXsor9D7DsFMC4lRHK9iF2tWDA9n4N9yPewnO+g1VqEXJRH1KiCl7VlPBd1p453JbB8VSJDbsnmwzg6aPTF2MBvufh56rs/qudFC87Es6q+gv9fIvpfqlo6er/fX6kNYvCE88lCsT9EM4yCprJl5I6mb8lE9IPcp3ezhou+NWoRAVbuVA1xRAHgO2scLvxC/OAGYwKaf5Os4wDBGomz3jJGfKqgSifHtodpIgTQEja/I+Lz0E77Asjbt2/xA74AwA6UCj+birHIdzxuWtIa/KJ0ZWHkBdsmlgyWLLy8rzYZvIQQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tlVAB-0001qi-U6 for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass 1/6] base: =?utf-8?b?4oCYcmVtb3RlLWJ1?= =?utf-8?b?aWxkZXLigJk=?= listens for notifications from =?utf-8?b?4oCYY3Vp?= =?utf-8?b?cmFzcyByZW1vdGUtc2VydmVy4oCZLg==?= References: <20250221153551.22658-1-romain.garbage@inria.fr> In-Reply-To: <20250221153551.22658-1-romain.garbage@inria.fr> Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 21 Feb 2025 15:42:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Ludovic =?utf-8?q?Court=C3=A8s?= Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17401524886758 (code B ref 76474); Fri, 21 Feb 2025 15:42:03 +0000 Received: (at 76474) by debbugs.gnu.org; 21 Feb 2025 15:41:28 +0000 Received: from localhost ([127.0.0.1]:33088 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tlV9Y-0001kT-KJ for submit@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:27 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:63151) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tlV9V-0001j2-BH for 76474@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=GZ9Vt/+3exncLsJkVeAUAzaPucqP2ol5s5FhXbuMpgM=; b=vsV+zYKBn/Fc/wVdegbuJ1pSE2Gm2QvD2sspqkT0BImoxRKKSNT02P8a Bagk+QX/o/kG360IdBQJFV3qBjR8BKdJr9kxzYOCLHYh5ucObHLusklEn TROyjPb00DH5UYdj/UYSSF3BExyqfFqFk2YRnloUf8XFqUWuqR/ysMh7D o=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,305,1732575600"; d="scan'208";a="109734821" Received: from 91-160-179-8.subs.proxad.net (HELO localhost.localdomain) ([91.160.179.8]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2025 16:41:14 +0100 From: Romain GARBAGE Date: Fri, 21 Feb 2025 16:40:10 +0100 Message-ID: <20250221154108.16375-1-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches From: Ludovic Courtès * src/cuirass/base.scm (%remote-server-socket-file-name): New variable. (remote-builder-listener): New procedure. (remote-builder): Add ‘socket’ parameter. Call ‘remote-builder-listener’. Handle ‘build-status-change’ messages. (spawn-remote-builder): Add call to ‘open-unix-listening-socket’ and pass it to ‘remote-builder’. * src/cuirass/scripts/remote-server.scm (%notification-socket-pool): New variable. (send-build-status-change-notification, update-build-status): New procedures. (run-fetch, serve-build-requests): Call ‘update-build-status’ instead of ‘db-update-build-status!’ and ‘set-build-successful!’. (open-build-notification-socket): New procedure. (cuirass-remote-server): Use it and parameterize ‘%notification-socket-pool’. * tests/remote.scm (notification-server): New variable. (terminate-process): New procedure. (stop-worker, stop-server): Use it. (start-notification-server, stop-notification-server): New procedures. ("remote-server"): Call ‘start-notification-server’. ("clean-up"): Call ‘stop-notification-server’. --- src/cuirass/base.scm | 53 +++++++++++++-- src/cuirass/scripts/remote-server.scm | 96 ++++++++++++++++++--------- tests/remote.scm | 44 ++++++++++-- 3 files changed, 151 insertions(+), 42 deletions(-) base-commit: b6c2e340b24c4ea9701eb687669a091027bd361e diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 27eb89a..631e81f 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -90,6 +90,7 @@ ;; Parameters. %bridge-socket-file-name open-bridge-connection + %remote-server-socket-file-name %package-cachedir %fallback?)) @@ -166,6 +167,8 @@ (string-append (%cuirass-state-directory) "/run")))) (define %bridge-socket-file-name + ;; Socket 'cuirass register' listens to, allowing 'cuirass web' to connect + ;; to it. (make-parameter (string-append (%cuirass-run-state-directory) "/cuirass/bridge"))) @@ -178,6 +181,12 @@ socket." (connect sock AF_UNIX file) sock)) +(define %remote-server-socket-file-name + ;; Socket 'cuirass register' listens to, allowing 'cuirass remote-server' to + ;; connect to it. + (make-parameter (string-append (%cuirass-run-state-directory) + "/cuirass/remote-builds"))) + ;;; ;;; Read parameters. @@ -456,22 +465,56 @@ by handing them to the local build daemon." (spawn-fiber (local-builder channel)) channel)) -(define (remote-builder channel) +(define (remote-builder-listener socket channel) + "Spawn a server that accepts connections on SOCKET and forwards messages it +reads to CHANNEL." + (define (serve-client client) + (let loop () + (match (read client) + ((? eof-object?) + (log-info "terminating remote server client connection on EOF")) + (message + (put-message channel message) + (loop))))) + + (lambda () + (let loop () + (match (accept socket (logior SOCK_NONBLOCK SOCK_CLOEXEC)) + ((connection . peer) + (spawn-fiber (lambda () + (log-info "remote builder accepted connection: ~s" + connection) + (serve-client connection))) + (loop)))))) + +(define (remote-builder channel socket) + "Spawn a remote builder that accepts messages on CHANNEL and receives +notifications from 'cuirass remote-server' over SOCKET." (lambda () (log-info "builds will be delegated to 'cuirass remote-server'") + (spawn-fiber (remote-builder-listener socket channel)) + (let loop () (match (get-message channel) (`(build ,derivations) ;; Currently there's nothing to do here: 'cuirass remote-server' ;; periodically calls 'db-get-pending-build'. ;; TODO: Push notifications to 'remote-server' instead. - (log-info "~a pending derivation builds" (length derivations)))) + (log-info "~a pending derivation builds" (length derivations))) + (`(build-status-change ,derivation ,status) + ;; TODO: Handle database operations, notifications, etc. from here. + (log-info "status of '~a' changed to ~a" derivation status))) (loop)))) (define (spawn-remote-builder) - "Spawn a build actor that performs builds using \"remote workers\"." - (let ((channel (make-channel))) - (spawn-fiber (remote-builder channel)) + "Spawn a build actor that performs builds using \"remote workers\". Return +once ready to listen for incoming connections from 'cuirass remote-server'." + (log-info "listening for 'cuirass remote-server' notifications on '~a'" + (%remote-server-socket-file-name)) + (let ((channel (make-channel)) + (socket (open-unix-listening-socket + (%remote-server-socket-file-name)))) + (spawn-fiber (remote-builder channel socket)) channel)) diff --git a/src/cuirass/scripts/remote-server.scm b/src/cuirass/scripts/remote-server.scm index 90a3db7..ae45232 100644 --- a/src/cuirass/scripts/remote-server.scm +++ b/src/cuirass/scripts/remote-server.scm @@ -1,6 +1,6 @@ ;;; remote-server.scm -- Remote build server. ;;; Copyright © 2020, 2021 Mathieu Othacehe -;;; Copyright © 2023, 2024 Ludovic Courtès +;;; Copyright © 2023-2025 Ludovic Courtès ;;; ;;; This file is part of Cuirass. ;;; @@ -20,7 +20,8 @@ (define-module (cuirass scripts remote-server) #:autoload (cuirass base) (read-parameters set-build-successful! - spawn-build-maintainer) + spawn-build-maintainer + %remote-server-socket-file-name) #:use-module (cuirass config) #:use-module (cuirass database) #:use-module (cuirass logging) @@ -88,6 +89,10 @@ (define %publish-port (make-parameter #f)) +(define %notification-socket-pool + ;; Pool of sockets connected to 'cuirass register'. + (make-parameter #f)) + (define service-name "Cuirass remote server") @@ -288,6 +293,22 @@ signing key for URL is not authorized." #t)))) outputs)))) +(define (send-build-status-change-notification drv status) + "Send a notification to 'cuirass register' that the status of DRV changed to +STATUS." + (with-resource-from-pool (%notification-socket-pool) sock + (write `(build-status-change ,drv ,status) sock) + (newline sock))) + +(define* (update-build-status drv status + #:key log-file) + "Change the status of DRV to STATUS, both in the database and by sending a +notification to 'cuirass register'." + (if (= (build-status succeeded) status) + (set-build-successful! drv) + (db-update-build-status! drv status #:log-file log-file)) + (send-build-status-change-notification drv status)) + (define* (run-fetch message) "Read MESSAGE and download the corresponding build outputs. If %CACHE-DIRECTORY is set, download the matching NAR and NARINFO files in this @@ -311,15 +332,15 @@ directory." (add-to-store drv outputs url)) (begin (log-info "build succeeded: '~a'" drv) - (set-build-successful! drv)) + (update-build-status drv (build-status succeeded))) (begin (log-error "failed to retrieve output of \ successful build '~a'; rescheduling" drv) - (db-update-build-status! drv (build-status scheduled)))))) + (update-build-status drv (build-status scheduled)))))) (('build-failed ('drv drv) ('url url) _ ...) (log-info "build failed: '~a'" drv) - (db-update-build-status! drv (build-status failed))))) + (update-build-status drv (build-status failed))))) (define (fetch-worker channel max-parallel-downloads) (define queue-size @@ -529,7 +550,7 @@ Use WORKER-DIRECTORY to maintain the list of active workers." ('dependency dependency) _ ...) (log-info "build failed: dependency '~a' of '~a'" dependency drv) - (db-update-build-status! drv (build-status failed-dependency)) + (update-build-status drv (build-status failed-dependency)) (put-message build-maintainer 'failed-dependency) ;mark 'failed-dependency' builds (let ((parent (db-get-build drv))) (when parent @@ -567,7 +588,7 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (build-id build) derivation (build-system build))) (db-update-build-worker! derivation name) - (db-update-build-status! derivation (build-status submitted)) + (update-build-status derivation (build-status submitted)) (catch 'zmq-error (lambda () (reply-worker @@ -583,7 +604,7 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (worker-name worker) (worker-address worker) message) - (db-update-build-status! derivation + (update-build-status derivation (build-status scheduled))))) (begin (when worker @@ -605,8 +626,8 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (worker-name worker) drv)) (db-update-build-worker! drv name) - (db-update-build-status! drv (build-status started) - #:log-file log-file))) + (update-build-status drv (build-status started) + #:log-file log-file))) (`(build-rejected (drv ,drv) (worker ,name)) ;; Worker rejected the build, which might be either because the ;; derivation is unavailable or because of a transient error. In @@ -619,11 +640,11 @@ Use WORKER-DIRECTORY to maintain the list of active workers." (begin (log-warning "~a: build rejected: ~a; rescheduling" name drv) - (db-update-build-status! drv (build-status scheduled))) + (update-build-status drv (build-status scheduled))) (begin (log-warning "~a: build rejected: ~a; canceling" name drv) - (db-update-build-status! drv (build-status canceled))))) + (update-build-status drv (build-status canceled))))) (_ (log-warning "ignoring unrecognized message: ~s" command))))) @@ -699,6 +720,15 @@ exiting." (terminate-helper-processes) (primitive-exit 1)))) +(define (open-build-notification-socket) + "Return a socket connected to the 'cuirass register' process, used to send +status updates." + (let ((sock (socket AF_UNIX + (logior SOCK_STREAM SOCK_CLOEXEC SOCK_NONBLOCK) + 0))) + (connect sock AF_UNIX (%remote-server-socket-file-name)) + sock)) + (define (cuirass-remote-server args) (signal-handler) (with-error-handling @@ -793,24 +823,30 @@ exiting." (run-fibers (lambda () - (with-database - (receive-logs log-port (%cache-directory)) - (spawn-notification-fiber) - (spawn-build-log-cleaner (assoc-ref opts 'build-log-expiry)) + (parameterize ((%notification-socket-pool + (make-resource-pool + (map (lambda (i) + (open-build-notification-socket)) + (iota 8)) + 'notification-socket))) + (with-database + (receive-logs log-port (%cache-directory)) + (spawn-notification-fiber) + (spawn-build-log-cleaner (assoc-ref opts 'build-log-expiry)) - (let ((fetch-worker (spawn-fetch-worker)) - (worker-directory (spawn-worker-directory))) - (catch 'zmq-error - (lambda () - (serve-build-requests backend-port - fetch-worker - worker-directory - #:build-maintainer - (spawn-build-maintainer))) - (lambda (key errno message . _) - (log-error (G_ "ZeroMQ error in build server: ~a") - message) - (terminate-helper-processes) - (primitive-exit 1)))))) + (let ((fetch-worker (spawn-fetch-worker)) + (worker-directory (spawn-worker-directory))) + (catch 'zmq-error + (lambda () + (serve-build-requests backend-port + fetch-worker + worker-directory + #:build-maintainer + (spawn-build-maintainer))) + (lambda (key errno message . _) + (log-error (G_ "ZeroMQ error in build server: ~a") + message) + (terminate-helper-processes) + (primitive-exit 1))))))) #:hz 0 #:parallelism (min 8 (current-processor-count))))))) diff --git a/tests/remote.scm b/tests/remote.scm index 797c92e..3fa9299 100644 --- a/tests/remote.scm +++ b/tests/remote.scm @@ -29,6 +29,7 @@ symbol)))) (cuirass specification) ((cuirass remote) #:select (worker-systems)) + ((cuirass base) #:select (spawn-remote-builder)) (gnu packages base) (guix build utils) (guix channels) @@ -39,6 +40,7 @@ ((guix store) #:hide (build)) ((guix utils) #:select (%current-system)) (tests common) + (fibers) (squee) (simple-zmq) ((zlib) #:select (call-with-gzip-input-port)) @@ -57,6 +59,9 @@ (define worker (make-parameter #f)) +(define notification-server + (make-parameter #f)) + (define (start-worker) (setenv "REQUEST_PERIOD" "1") (setenv "CUIRASS_LOGGING_LEVEL" "debug") @@ -67,10 +72,12 @@ "--public-key=tests/signing-key.pub") #:search-path? #t))) +(define (terminate-process pid) + (kill pid SIGINT) + (waitpid pid)) + (define (stop-worker) - (let ((worker (worker))) - (kill worker SIGINT) - (waitpid worker))) + (terminate-process (worker))) (define (start-server) (mkdir-p "tests/cache") @@ -85,9 +92,30 @@ #:search-path? #t))) (define (stop-server) - (let ((server (server))) - (kill server SIGINT) - (waitpid server))) + (terminate-process (server))) + +(define (start-notification-server) + ;; Spawn the notification server that normally runs as part of 'cuirass + ;; register', and which 'cuirass remote-server' connects to. Do so in a + ;; separate process because 'run-fibers' installs suspendable ports, which + ;; this process may not be able to deal with. + (notification-server + (spawn "guile" + (list "guile" "-c" + (object->string + '(begin + (use-modules (cuirass base) (fibers)) + + (setvbuf (current-output-port) 'none) + (setvbuf (current-error-port) 'none) + (run-fibers + (lambda () + (spawn-remote-builder) + (sleep 120)) ;wait + #:drain? #t))))))) + +(define (stop-notification-server) + (terminate-process (notification-server))) (define* (dummy-drv #:optional sleep #:key (name "foo") dependency message) (let ((dependency (and=> dependency read-derivation-from-file))) @@ -185,6 +213,7 @@ (test-assert "remote-server" (begin + (start-notification-server) (start-server) #t)) @@ -384,4 +413,5 @@ Failing dependency ~s.\n" (test-assert "clean-up" (begin (stop-worker) - (stop-server)))) + (stop-server) + (stop-notification-server)))) From patchwork Fri Feb 21 15:40:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 38916 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 62F3E27BBEA; Fri, 21 Feb 2025 15:42:34 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id C567527BBE2 for ; Fri, 21 Feb 2025 15:42:33 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tlVAF-0000uf-0g; Fri, 21 Feb 2025 10:42:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tlVAD-0000uS-Fq for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tlVAD-0002Rz-6x for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=w6UjAkouCcw3gN6AWlaFIu9pL/FGYAezoRSZprU3cj8=; b=U8EC4UKXxwT9xJTtOTq+j6NnqQ/O0+1PhJhYSkAMyCr+somqfptA8W1uQBxOcNCtY0Esv7AjwSxSIzhisSKd30he7rjNrQgi1x/bMdb4e9bhxZh2UUk2C4AnarQTOqeRYZxwLNtoU6lNlkCrvoBPxHWPpJ/uR3TgRiyY809bCsL2SB94d1WFzuA5Y0ZcfZF5dhVOjoTMJC+qFlBCIstYA0W0LiOwajjU2DoGfw3vek8w8DiShobXctN91QFoLg5fAbRgakXiBO1fAsaIG+HRICSeM0zO39GlJYujuuAFNPxAIrGsbQ9IIQF+pwPcTamfaFUPdyHlAl7/2ien1b6Kjg==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tlVAD-0001qz-0y for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:05 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass 2/6] utils: Add ring buffer implementation. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 21 Feb 2025 15:42:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17401524976860 (code B ref 76474); Fri, 21 Feb 2025 15:42:04 +0000 Received: (at 76474) by debbugs.gnu.org; 21 Feb 2025 15:41:37 +0000 Received: from localhost ([127.0.0.1]:33097 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tlV9k-0001mU-KD for submit@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:37 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:63151) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tlV9X-0001j2-7b for 76474@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=w6UjAkouCcw3gN6AWlaFIu9pL/FGYAezoRSZprU3cj8=; b=mIvtSEsVE8w7FSN0GXiS242UfMrVd4eYOz/Ah5xntk9kdIUf+ePlNLTi qh58usHT+9JslvP6u60h4sHOyDjUL9Th/sVEfr6LJaP5pniSuHyq2sSlC Tffxv0mOvyLeTZWpBL8IlsLTfVgCN/IEQhsdJmzuRGY8vC0BfMznQ9KJx M=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,305,1732575600"; d="scan'208";a="109734822" Received: from 91-160-179-8.subs.proxad.net (HELO localhost.localdomain) ([91.160.179.8]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2025 16:41:14 +0100 From: Romain GARBAGE Date: Fri, 21 Feb 2025 16:40:11 +0100 Message-ID: <20250221154108.16375-2-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250221154108.16375-1-romain.garbage@inria.fr> References: <20250221154108.16375-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/utils.scm (): New record type. (ring-buffer, ring-buffer-insert, ring-buffer->list): New variables. * tests/utils.scm: Add tests. --- src/cuirass/utils.scm | 67 ++++++++++++++++++++++++++++++++++++++++++- tests/utils.scm | 17 +++++++++++ 2 files changed, 83 insertions(+), 1 deletion(-) diff --git a/src/cuirass/utils.scm b/src/cuirass/utils.scm index c406a84..b06f451 100644 --- a/src/cuirass/utils.scm +++ b/src/cuirass/utils.scm @@ -28,6 +28,7 @@ #:use-module (rnrs bytevectors) #:use-module (system foreign) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) #:use-module (srfi srfi-19) #:use-module (srfi srfi-71) #:autoload (guix build utils) (mkdir-p) @@ -60,7 +61,15 @@ open-unix-listening-socket atomic-box-fetch-and-increment! - atomic-box-fetch-and-decrement!)) + atomic-box-fetch-and-decrement! + + ring-buffer + ring-buffer-head + ring-buffer-head-length + ring-buffer-limit + ring-buffer-rear + ring-buffer-insert + ring-buffer->list)) (define-exception-type &cuirass-assertion-failure &assertion-failure make-cuirass-assertion-failure @@ -347,3 +356,59 @@ and store the result inside the BOX." (define (atomic-box-fetch-and-decrement! box) "Atomically decrement the value of the integer stored inside the given BOX." (atomic-box-fetch-and-update! box 1-)) + +;;; +;;; Ring buffer implementation. Copied from GNU Shepherd. +;;; + +;; Helper function needed by ring-buffer->list. +(define (at-most max-length lst) + "If @var{lst} is shorter than @var{max-length}, return it and the empty list; +otherwise return its @var{max-length} first elements and its tail." + (let loop ((len 0) + (lst lst) + (result '())) + (match lst + (() + (values (reverse result) '())) + ((head . tail) + (if (>= len max-length) + (values (reverse result) lst) + (loop (+ 1 len) tail (cons head result))))))) + +;; The poor developer's persistent "ring buffer": it holds between N and 2N +;; elements, but has O(1) insertion. +(define-record-type + (%ring-buffer limit front-length front rear) + ring-buffer? + (limit ring-buffer-limit) + (front-length ring-buffer-front-length) + (front ring-buffer-front) + (rear ring-buffer-rear)) + +(define (ring-buffer size) + "Return an ring buffer that can hold @var{size} elements." + (%ring-buffer size 0 '() '())) + +(define-inlinable (ring-buffer-insert element buffer) + "Insert @var{element} to the front of @var{buffer}. If @var{buffer} is +already full, its oldest element is removed." + (match buffer + (($ limit front-length front rear) + (if (< front-length limit) + (let ((front-length (+ 1 front-length))) + (%ring-buffer limit front-length + (cons element front) + (if (= limit front-length) + '() + rear))) + (%ring-buffer limit 1 + (list element) front))))) + +(define (ring-buffer->list buffer) + "Convert @var{buffer} into a list." + (match buffer + (($ limit front-length front rear) + (if (= limit front-length) + front + (append front (at-most (- limit front-length) rear)))))) diff --git a/tests/utils.scm b/tests/utils.scm index e1ac1b8..4a3b48f 100644 --- a/tests/utils.scm +++ b/tests/utils.scm @@ -83,4 +83,21 @@ (const 42))))) #:to 'value)) +(test-equal "ring-buffer->list 1 element" + '(el) + (let ((buffer (ring-buffer 5))) + (ring-buffer->list (ring-buffer-insert 'el buffer)))) + +(test-equal "ring-buffer->list empty buffer" + '() + (ring-buffer->list (ring-buffer 5))) + +(test-equal "ring-buffer->list full" + '(9 8 7 6 5) + (ring-buffer->list (fold + (lambda (e r) + (pk 'r (ring-buffer-insert e r))) + (ring-buffer 5) + (iota 10)))) + (test-end) From patchwork Fri Feb 21 15:40:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 38917 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 15DEE27BBEA; Fri, 21 Feb 2025 15:42:44 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 6B8E827BBE2 for ; Fri, 21 Feb 2025 15:42:43 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tlVAJ-0000wa-1y; Fri, 21 Feb 2025 10:42:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tlVAE-0000ue-Tv for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:06 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tlVAE-0002SO-KD for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=bqEXtnqyP8yK88lMSZmlYhLfwRI4iA5B0fqLwk4QutY=; b=Xz7GezC/CT+c0oR6n1lh6yb0HENaS2MLkeXYKNgwXLvHjN4NDx4j/dzEbfxSYRfqWqj3Ujmog0JkxORbS6cjv34+a7ypUBYx/tjezCCeTEcfMtt+kH3pa/Uy/QDszN8NsdshyRCKOo7bI2USe4UDsKkHtVx8ANBXtzUWHbrCAMJoRloe1zfLkzbTBETY5T0RZUYPXE57cM0Y4JgBXvz9sotLLvaeeMHfp+KAHkMmQPfSjaAFEXmQ5SE3Xstz2dlRX92n211/aN8+5JJ/tUMGxfQP4JOqTCP6AFQ70NUvQDeNqJQ+vSUJK2HAXLX9wHnpTlFPTl3oWiBovlPtE69g1Q==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tlVAE-0001rF-1h for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:06 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass 3/6] base: Add event-log agent. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 21 Feb 2025 15:42:05 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17401524996888 (code B ref 76474); Fri, 21 Feb 2025 15:42:05 +0000 Received: (at 76474) by debbugs.gnu.org; 21 Feb 2025 15:41:39 +0000 Received: from localhost ([127.0.0.1]:33099 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tlV9l-0001mb-Cu for submit@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:39 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:63151) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tlV9Y-0001j2-FH for 76474@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bqEXtnqyP8yK88lMSZmlYhLfwRI4iA5B0fqLwk4QutY=; b=FP/QMV5quBvghIm/HOKBGL7aMGdZtQJv8wDglT2sh/cK/fpeC690JGcG onW2hghrEC2r6VS/mmx6Rh8xjQXSzbqRwpttvqoBM4gmcuPKTXYmW294H 3g+W17AZfouyIg35zvTsMMPN8flvcxzOb05cW3O+RdnyO50dYCsQP/Z8l E=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,305,1732575600"; d="scan'208";a="109734824" Received: from 91-160-179-8.subs.proxad.net (HELO localhost.localdomain) ([91.160.179.8]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2025 16:41:15 +0100 From: Romain GARBAGE Date: Fri, 21 Feb 2025 16:40:12 +0100 Message-ID: <20250221154108.16375-3-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250221154108.16375-1-romain.garbage@inria.fr> References: <20250221154108.16375-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (event-log-service, spawn-event-log-service): New variables. * tests/base.scm: Add tests for the event-log-service agent. --- src/cuirass/base.scm | 59 +++++++++++++++++++++++++++++++++++++++++++- tests/base.scm | 46 ++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 1 deletion(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 631e81f..688c033 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -73,6 +73,7 @@ spawn-remote-builder spawn-channel-update-service spawn-jobset-evaluator + spawn-event-log-service spawn-jobset-registry spawn-gc-root-cleaner spawn-build-maintainer @@ -113,7 +114,11 @@ ;;; 'guix-daemon' process, while the remote build delegates builds to ;;; 'cuirass remote-server'. ;;; -;;; - Each jobset as an associated "monitor"; it requests channel updates, +;;; - The "event-log" agent receives events from the different agents. It +;;; also dispatches these events to the agents which subscribed to event +;;; notification. +;;; +;;; - Each jobset has an associated "monitor"; it requests channel updates, ;;; evaluations, and builds to the actors above. It also receives requests ;;; such as evaluation triggers that can come, for example, from the ;;; /jobset/NAME/hook/evaluate HTTP endpoint. @@ -770,6 +775,58 @@ concurrently; it sends derivation build requests to BUILDER." max-parallel-evaluations)) channel)) +;;; +;;; Logging events +;;; + +(define event-log-buffer-size + (make-parameter 1000)) + +(define (event-log-service channel) + "Keep events received on CHANNEL in a circular buffer, keeping track of +notification subscriptions." + (lambda () + (define events (ring-buffer (event-log-buffer-size))) + + (let loop ((events events) + (subscribers '())) + (match (get-message channel) + (`(subscribe ,channel) + (loop events (cons channel subscribers))) + (`(unsubscribe ,channel) + (loop events (delq channel subscribers))) + (`(new-event ,event) + ;; Events are stored as a list in the format + ;; (event-type timestamp rest-of-data) + (let* ((data (match event + ((event-type . rest) + (append `(,event-type ,(current-time time-utc)) + rest)))) + (events (ring-buffer-insert data events))) + ;; For now, every new event is sent to all subscribers. + (for-each (lambda (reply) + (put-message reply data)) + subscribers) + (loop events subscribers))) + (`(recent-events ,reply) + (put-message reply (ring-buffer->list events)) + (loop events subscribers)) + ;; Catchall for malformed messages. + (message + (log-error "malformed message sent to the event-log-service: ~s" + message) + (loop events subscribers)))))) + +(define (spawn-event-log-service) + "Spawn an actor responsible for centralizing events." + (let ((channel (make-channel))) + (spawn-fiber (event-log-service channel)) + channel)) + +;;; +;;; Monitoring jobsets +;;; + (define %jobset-trigger-rate-window ;; Window (seconds) over which the jobset trigger rate is computed. (* 5 60)) ;5 minutes diff --git a/tests/base.scm b/tests/base.scm index 1cfbecb..e3709ee 100644 --- a/tests/base.scm +++ b/tests/base.scm @@ -17,6 +17,10 @@ ;;; along with Cuirass. If not, see . (use-modules (cuirass base) + (fibers) + (fibers channels) + (ice-9 match) + (srfi srfi-1) (srfi srfi-64)) (test-begin "base") @@ -25,4 +29,46 @@ 'wrong-type-arg (%package-cachedir #f)) +(test-equal "event-log recent-events" + ;; Elements are in reverse order since they come from a ring-buffer. + '((test-event "Third message") + (test-event "Second message") + (test-event "First message")) + (run-fibers + (lambda () + (let ((event-log (spawn-event-log-service)) + (reply-channel (make-channel))) + (put-message event-log '(new-event (test-event "First message"))) + (put-message event-log '(new-event (test-event "Second message"))) + (put-message event-log '(new-event (test-event "Third message"))) + (put-message event-log `(recent-events ,reply-channel)) + (filter-map (match-lambda + ((type timestamp value) + ;; Drop timestamp. + (list type value)) + (_ #f)) + (get-message reply-channel)))))) + +(test-equal "event-log recent-events (empty buffer)" + '() + (run-fibers + (lambda () + (let ((event-log (spawn-event-log-service)) + (reply-channel (make-channel))) + (put-message event-log `(recent-events ,reply-channel)) + (get-message reply-channel))))) + +(test-equal "event-log subscribe" + '(test-event "test-value") + (run-fibers + (lambda () + (let ((event-log (spawn-event-log-service)) + (reply (make-channel))) + (put-message event-log `(subscribe ,reply)) + (put-message event-log '(new-event (test-event "test-value"))) + (match (get-message reply) + ((type timestamp value) + ;; Drop timestamp. + (list type value))))))) + (test-end) From patchwork Fri Feb 21 15:40:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 38915 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id E96A627BBEA; Fri, 21 Feb 2025 15:42:16 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 5791027BBE2 for ; Fri, 21 Feb 2025 15:42:16 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tlVAG-0000w0-VK; Fri, 21 Feb 2025 10:42:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tlVAF-0000uw-FL for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:07 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tlVAF-0002Se-6u for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=/jolzagQNEMfRgBJEO4DsW67bmudEF/tObtibrNMyDg=; b=BKT8c0/hn00YSQLzV+bRf+Q+uKj6rDR9ikaAL2gM9JZ3oI/QbXrkXnXR3hIu/y6TocmFbHur3mGqxyXJOjaxeqERG9NjnGbb2KNlSavsZEqVtH6y/Tcps/M/upxQdqgkJ1CreUnwtx+emlva8J09NdYxAaMzkN/jpUqgspKYDEDRz5DLT2T3ln4u74kSzyUMsqHPZteLYJPFGz2rCMQEv3qLZP6iM3vR//26EafkcZI4dK+clUD5ShjhjrZeLpy60r+X4K3h+WmyhUoYDGwii9m/3w8mr0XVuf4uYKasvaot/EMfy8TIZVU8T16oD5TTrO53igYGEgI/nsF3vNs+wg==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tlVAF-0001rT-0i for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:07 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass 4/6] register: Add support for event logging in builders. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 21 Feb 2025 15:42:06 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17401525076959 (code B ref 76474); Fri, 21 Feb 2025 15:42:06 +0000 Received: (at 76474) by debbugs.gnu.org; 21 Feb 2025 15:41:47 +0000 Received: from localhost ([127.0.0.1]:33104 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tlV9u-0001o4-6t for submit@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:47 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:63151) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tlV9a-0001j2-BA for 76474@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/jolzagQNEMfRgBJEO4DsW67bmudEF/tObtibrNMyDg=; b=LnIqIRywWzfcJ1ovx2njztba27VxowjhbQ+p3mtpwe0alwk69zsB9WXo 0BQZnvE9LMJW2RyliqufDU1OGqtXiSrLAxrwDu+fBCvAp7I5iSPQWx28+ Zhs9FD/IWCzrWvCgp2UF6eDjLetqpnXD6CFl2okTNNE8mRb1iWVK+cXBd M=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,305,1732575600"; d="scan'208";a="109734827" Received: from 91-160-179-8.subs.proxad.net (HELO localhost.localdomain) ([91.160.179.8]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2025 16:41:15 +0100 From: Romain GARBAGE Date: Fri, 21 Feb 2025 16:40:13 +0100 Message-ID: <20250221154108.16375-4-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250221154108.16375-1-romain.garbage@inria.fr> References: <20250221154108.16375-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (local-builder, spawn-local-builder, remote-builder, spawn-remote-builder): Add support for logging events to the corresponfing agent through Fibers. * src/cuirass/scripts/register.scm (cuirass-register): Add event-log-service agent and use it in builders. * tests/remote.scm (start-notification-server): Add event-log-service agent. --- src/cuirass/base.scm | 29 ++++++++++++++++++++--------- src/cuirass/scripts/register.scm | 5 +++-- tests/remote.scm | 7 +++++-- 3 files changed, 28 insertions(+), 13 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 688c033..8ea52d8 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -439,7 +439,7 @@ OUTPUTS, a list of records." (checksum "")))))) ;TODO: Implement it. outputs)) -(define (local-builder channel) +(define (local-builder channel event-log) (lambda () (log-info "builds will be made via the local build daemon") (let loop () @@ -460,14 +460,22 @@ OUTPUTS, a list of records." (fail (- (length derivations) success))) (log-info "outputs:\n~a" (string-join outs "\n")) + ;; Log the same kind of information as remote-builder. + (for-each (lambda (result) + (put-message event-log + `(new-event (build-status-change + ,(build-derivation result) + ,(build-current-status result))))) + results) results)))))) (loop)))) -(define (spawn-local-builder) +(define (spawn-local-builder event-log) "Spawn a build actor that executes the derivation build requests it receives -by handing them to the local build daemon." +by handing them to the local build daemon. The build actor sends the build +status to the event log service through EVENT-LOG, a Fibers channel." (let ((channel (make-channel))) - (spawn-fiber (local-builder channel)) + (spawn-fiber (local-builder channel event-log)) channel)) (define (remote-builder-listener socket channel) @@ -492,9 +500,10 @@ reads to CHANNEL." (serve-client connection))) (loop)))))) -(define (remote-builder channel socket) +(define (remote-builder channel socket event-log) "Spawn a remote builder that accepts messages on CHANNEL and receives -notifications from 'cuirass remote-server' over SOCKET." +notifications from 'cuirass remote-server' over SOCKET. It also sends the log +events to the event log service using EVENT-LOG, a Fibers channel." (lambda () (log-info "builds will be delegated to 'cuirass remote-server'") (spawn-fiber (remote-builder-listener socket channel)) @@ -508,10 +517,12 @@ notifications from 'cuirass remote-server' over SOCKET." (log-info "~a pending derivation builds" (length derivations))) (`(build-status-change ,derivation ,status) ;; TODO: Handle database operations, notifications, etc. from here. - (log-info "status of '~a' changed to ~a" derivation status))) + (log-info "status of '~a' changed to ~a" derivation status) + (put-message event-log + `(new-event (build-status-change ,derivation ,status))))) (loop)))) -(define (spawn-remote-builder) +(define (spawn-remote-builder event-log) "Spawn a build actor that performs builds using \"remote workers\". Return once ready to listen for incoming connections from 'cuirass remote-server'." (log-info "listening for 'cuirass remote-server' notifications on '~a'" @@ -519,7 +530,7 @@ once ready to listen for incoming connections from 'cuirass remote-server'." (let ((channel (make-channel)) (socket (open-unix-listening-socket (%remote-server-socket-file-name)))) - (spawn-fiber (remote-builder channel socket)) + (spawn-fiber (remote-builder channel socket event-log)) channel)) diff --git a/src/cuirass/scripts/register.scm b/src/cuirass/scripts/register.scm index e32fe52..1698a8c 100644 --- a/src/cuirass/scripts/register.scm +++ b/src/cuirass/scripts/register.scm @@ -271,9 +271,10 @@ (if one-shot? (leave (G_ "'--one-shot' is currently unimplemented~%")) (let* ((exit-channel (make-channel)) + (event-log-service (spawn-event-log-service)) (builder (if (option-ref opts 'build-remote #f) - (spawn-remote-builder) - (spawn-local-builder))) + (spawn-remote-builder event-log-service) + (spawn-local-builder event-log-service))) (evaluator (spawn-jobset-evaluator #:max-parallel-evaluations max-parallel-evaluations diff --git a/tests/remote.scm b/tests/remote.scm index 3fa9299..e7856ba 100644 --- a/tests/remote.scm +++ b/tests/remote.scm @@ -29,7 +29,8 @@ symbol)))) (cuirass specification) ((cuirass remote) #:select (worker-systems)) - ((cuirass base) #:select (spawn-remote-builder)) + ((cuirass base) #:select (spawn-remote-builder + spawn-event-log-service)) (gnu packages base) (guix build utils) (guix channels) @@ -110,7 +111,9 @@ (setvbuf (current-error-port) 'none) (run-fibers (lambda () - (spawn-remote-builder) + (spawn-remote-builder + ;; A channel for event logs must be set up. + (spawn-event-log-service)) (sleep 120)) ;wait #:drain? #t))))))) From patchwork Fri Feb 21 15:40:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 38920 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 2146627BBE9; Fri, 21 Feb 2025 15:42:54 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id B737427BBE2 for ; Fri, 21 Feb 2025 15:42:53 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tlVAJ-0000wn-Bl; Fri, 21 Feb 2025 10:42:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tlVAH-0000w7-Ni for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:09 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tlVAH-0002T2-F2 for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=CTeL1YhvYVGtf/wUIkMyocKpY+fpbBVG9KWfzKfLv0Q=; b=BoUiPfEeVuTPKWA+a2Z/eX+LcFDJD9o9hYucB+gyFRjrTZ5lNNdR1FotwdMiH0xvbF1aOc5GNVdgFPi8jXJ5XiFIouZH5HkwyBcrBO1j1S32i8FUHgLVZ0ocYHUGiV7z4I0+pilELD0rpy20Tx7kaerDsXLcG7j2bUhGQarWXBAF//JJfscgtWOxE6E3UhrPN97g63xAbKYWz9dOYATCmOtxdOhutLUtYuVMgXX4m+R7ft9Y8lGcZn2sCWkYOWuIXXmOyXDHxPsPSPIDQD/An+kYnzWaBh4I01DRg2p4U9PwS7Ojeg82vI4cCa6gZBHxqQKeNn4B3HJSYkkIQe4dKg==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tlVAH-0001s0-7n for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:09 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass 5/6] register: Add support for event logging in the jobset evaluator. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 21 Feb 2025 15:42:09 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17401525217077 (code B ref 76474); Fri, 21 Feb 2025 15:42:09 +0000 Received: (at 76474) by debbugs.gnu.org; 21 Feb 2025 15:42:01 +0000 Received: from localhost ([127.0.0.1]:33114 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tlVA7-0001pj-7t for submit@debbugs.gnu.org; Fri, 21 Feb 2025 10:42:00 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:63151) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tlV9c-0001j2-Qe for 76474@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CTeL1YhvYVGtf/wUIkMyocKpY+fpbBVG9KWfzKfLv0Q=; b=mu0Jvr40oWFkhP1WwmUP6h590SQsczLLYyyuL9+25v+rBuABWg5N9Wbn rNAlurod8ofZB2PXTAGmkcUxJOI7a2n1PtNpBNyUEM3y1tWAlgdDfG1D/ Krq1JXihdxh94y2tl+zAuaNzON5FZSoIYwjv1IboT/LJUB6GXTfDAU3Sq c=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,305,1732575600"; d="scan'208";a="109734830" Received: from 91-160-179-8.subs.proxad.net (HELO localhost.localdomain) ([91.160.179.8]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2025 16:41:15 +0100 From: Romain GARBAGE Date: Fri, 21 Feb 2025 16:40:14 +0100 Message-ID: <20250221154108.16375-5-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250221154108.16375-1-romain.garbage@inria.fr> References: <20250221154108.16375-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (spawn-jobset-evaluator, jobset-evaluator): Add support for event logging. * src/cuirass/scripts/register.scm (cuirass-register): Add support for event logging in jobset evaluator. --- src/cuirass/base.scm | 16 +++++++++++++--- src/cuirass/scripts/register.scm | 3 ++- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 8ea52d8..01af377 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -699,10 +699,11 @@ Return a list of jobs that are associated to EVAL-ID." (name (specification-name spec)) (id eval-id))))))))) -(define (start-evaluation spec instances timestamp) +(define (start-evaluation spec instances timestamp event-log) "Start an evaluation of SPEC using the given channel INSTANCES. Return #f if nothing has changed (and thus no new evaluation was created), otherwise return -the ID of the new evaluation." +the ID of the new evaluation. EVENT-LOG is a Fibers channel used to return the +evaluation ID before the evaluation is started." (let* ((channels (map channel-instance-channel instances)) (new-spec (specification (inherit spec) @@ -726,6 +727,10 @@ the ID of the new evaluation." ;; dependencies that are not declared in the initial specification ;; channels. Update the given SPEC to take them into account. (db-add-or-update-specification new-spec) + ;; Since evaluate blocks until the end of evaluation, an + ;; eval-started event is sent just before starting the evaluation. + (put-message event-log + `(new-event (eval-started ,eval-id ,spec))) (evaluate spec eval-id) (db-set-evaluation-time eval-id) @@ -734,6 +739,7 @@ the ID of the new evaluation." (define* (jobset-evaluator channel #:key builder + event-log (max-parallel-evaluations (current-processor-count))) (define pool @@ -752,13 +758,15 @@ the ID of the new evaluation." (with-resource-from-pool pool token (log-info "evaluating '~a' with token #~a" (specification-name spec) token) - (start-evaluation spec instances timestamp))) + (start-evaluation spec instances timestamp event-log))) (when eval-id (let* ((builds (db-get-builds `((evaluation . ,eval-id)))) (derivations (map build-derivation builds))) (log-info "evaluation ~a of jobset '~a' registered ~a builds" eval-id (specification-name spec) (length builds)) + (put-message event-log + `(new-event (eval-finished ,eval-id ,spec))) (db-set-evaluation-status eval-id (evaluation-status succeeded)) @@ -774,6 +782,7 @@ the ID of the new evaluation." (define* (spawn-jobset-evaluator #:key builder + event-log (max-parallel-evaluations (current-processor-count))) "Spawn the actor responsible for evaluating jobsets for a given spec and set @@ -782,6 +791,7 @@ concurrently; it sends derivation build requests to BUILDER." (let ((channel (make-channel))) (spawn-fiber (jobset-evaluator channel #:builder builder + #:event-log event-log #:max-parallel-evaluations max-parallel-evaluations)) channel)) diff --git a/src/cuirass/scripts/register.scm b/src/cuirass/scripts/register.scm index 1698a8c..e977458 100644 --- a/src/cuirass/scripts/register.scm +++ b/src/cuirass/scripts/register.scm @@ -278,7 +278,8 @@ (evaluator (spawn-jobset-evaluator #:max-parallel-evaluations max-parallel-evaluations - #:builder builder)) + #:builder builder + #:event-log event-log-service)) (update-service (spawn-channel-update-service))) (clear-build-queue) From patchwork Fri Feb 21 15:40:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain GARBAGE X-Patchwork-Id: 38919 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 46C4427BBE9; Fri, 21 Feb 2025 15:42:48 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE, SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id 02B0B27BBEA for ; Fri, 21 Feb 2025 15:42:47 +0000 (GMT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tlVAJ-0000wX-0v; Fri, 21 Feb 2025 10:42:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tlVAG-0000vW-DU for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:08 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tlVAG-0002Sq-47 for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=9YIzF4eOJV5pJ1LynZos0EbDT6PfrwbyM0UznZentw4=; b=XvQjOxF/atSoyt3uhvrRVj+yZh9wi2w9n+TYt0nsCEiEi0PGNlvk6Fo0uaG4koDHKvF7LtEV8oGHXOC1lG39BVLegr9TRwZ0aIkx+Q2Oz9P3ZIMMqhEtILiAuWrKjqDaiXItgAocSAnBGim6V4FFbyvAyCytMqy7Wjxm5/MTJU6c4aFSQEOB407j8ROqbh1UuxJVd75gIuiThajtNJqGq8Vt7OMYwXKS2rU+r4dou7V/IgKH0dfoEzoj4dYEv8Maup+WxqgbbECBOyI4aSmTsNZPxSzqMf1A7VcVsrTl7NurQl08qHe/onUkP94dw8fCegeEOCnhaxShPAW4qCfzEw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tlVAF-0001rh-TQ for guix-patches@gnu.org; Fri, 21 Feb 2025 10:42:07 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#76474] [PATCH Cuirass 6/6] register: Add support for event logging in the jobset registry. Resent-From: Romain GARBAGE Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 21 Feb 2025 15:42:07 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 76474 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 76474@debbugs.gnu.org Cc: ludovic.courtes@inria.fr, Romain GARBAGE Received: via spool by 76474-submit@debbugs.gnu.org id=B76474.17401525197056 (code B ref 76474); Fri, 21 Feb 2025 15:42:07 +0000 Received: (at 76474) by debbugs.gnu.org; 21 Feb 2025 15:41:59 +0000 Received: from localhost ([127.0.0.1]:33112 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tlVA6-0001pa-Bb for submit@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:59 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:63157) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tlV9c-0001k6-Vh for 76474@debbugs.gnu.org; Fri, 21 Feb 2025 10:41:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9YIzF4eOJV5pJ1LynZos0EbDT6PfrwbyM0UznZentw4=; b=VhQjYwt7j6TtyyvgHFtm+n26PxgmxF3ZZxFjQDrr4SHC9VgOIPbziWS+ jg2YxWBhP7lpSWPfEPbYdA5KmFmvJu5xXE9J4WMRuWG6a3+WXZiZv/Xs0 O/ZjxyunMBq0P2XEXB0TzI/oydijIaPhshHVGRa8auJUHwvWPai6MjPdA Q=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.13,305,1732575600"; d="scan'208";a="109734832" Received: from 91-160-179-8.subs.proxad.net (HELO localhost.localdomain) ([91.160.179.8]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2025 16:41:15 +0100 From: Romain GARBAGE Date: Fri, 21 Feb 2025 16:40:15 +0100 Message-ID: <20250221154108.16375-6-romain.garbage@inria.fr> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250221154108.16375-1-romain.garbage@inria.fr> References: <20250221154108.16375-1-romain.garbage@inria.fr> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org X-getmail-retrieved-from-mailbox: Patches * src/cuirass/base.scm (jobset-registry, spawn-jobset-registry): Add support for event logging. * src/cuirass/scripts/register.scm (cuirass-register): Add log event support to the jobset registry. --- src/cuirass/base.scm | 11 +++++++++-- src/cuirass/scripts/register.scm | 1 + 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 01af377..6c67645 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -990,7 +990,8 @@ POLLING-PERIOD seconds." (define* (jobset-registry channel #:key (polling-period 60) - update-service evaluator) + update-service evaluator + event-log) (lambda () (spawn-fiber (lambda () @@ -1014,6 +1015,8 @@ POLLING-PERIOD seconds." (#f (log-error "cannot update non-existent spec '~s'" name)) ((_ . monitor) + (put-message event-log + `(new-event (jobset-updated ,spec))) (put-message monitor `(update-spec ,spec))))) (loop registry)) (`(register ,spec) @@ -1029,6 +1032,8 @@ POLLING-PERIOD seconds." #:polling-period period)) (name (specification-name spec))) (log-info "registering new jobset '~a'" name) + (put-message event-log + `(new-event (jobset-registered ,spec))) (loop (vhash-consq name monitor registry)))) ((_ . monitor) @@ -1037,13 +1042,15 @@ POLLING-PERIOD seconds." (loop registry)))))))) (define* (spawn-jobset-registry #:key (polling-period 60) - update-service evaluator) + update-service evaluator + event-log) "Spawn a jobset registry. In turn, the registry creates a new jobset monitoring actor for each 'register' message it receives." (let ((channel (make-channel))) (spawn-fiber (jobset-registry channel #:update-service update-service #:evaluator evaluator + #:event-log event-log #:polling-period polling-period)) channel)) diff --git a/src/cuirass/scripts/register.scm b/src/cuirass/scripts/register.scm index e977458..38db9a5 100644 --- a/src/cuirass/scripts/register.scm +++ b/src/cuirass/scripts/register.scm @@ -302,6 +302,7 @@ (let ((registry (spawn-jobset-registry #:update-service update-service #:evaluator evaluator + #:event-log event-log-service #:polling-period interval))) ;; Spawn the bridge through which other 'cuirass' ;; processes, such as 'cuirass web', may talk to the