From patchwork Wed Apr 21 12:21:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Othacehe X-Patchwork-Id: 28721 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 4ADE627BC79; Wed, 21 Apr 2021 13:22:16 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id EE0B627BC7B for ; Wed, 21 Apr 2021 13:22:10 +0100 (BST) Received: from localhost ([::1]:41408 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZBru-0002Xq-5Z for patchwork@mira.cbaines.net; Wed, 21 Apr 2021 08:22:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48628) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZBrm-0002X3-Ng for guix-patches@gnu.org; Wed, 21 Apr 2021 08:22:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45576) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lZBrm-00079q-GI for guix-patches@gnu.org; Wed, 21 Apr 2021 08:22:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lZBrm-0005a5-A5 for guix-patches@gnu.org; Wed, 21 Apr 2021 08:22:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47929] [PATCH 1/5] ci: Add manifest support to channel-with-substitutes-available. References: <20210421121610.2045-1-othacehe@gnu.org> In-Reply-To: <20210421121610.2045-1-othacehe@gnu.org> Resent-From: Mathieu Othacehe Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 21 Apr 2021 12:22:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47929 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47929@debbugs.gnu.org Cc: Mathieu Othacehe Received: via spool by 47929-submit@debbugs.gnu.org id=B47929.161900768921373 (code B ref 47929); Wed, 21 Apr 2021 12:22:02 +0000 Received: (at 47929) by debbugs.gnu.org; 21 Apr 2021 12:21:29 +0000 Received: from localhost ([127.0.0.1]:57111 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZBrE-0005YX-VN for submit@debbugs.gnu.org; Wed, 21 Apr 2021 08:21:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35874) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZBrD-0005YB-0x for 47929@debbugs.gnu.org; Wed, 21 Apr 2021 08:21:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:54763) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZBr7-0006lD-SI for 47929@debbugs.gnu.org; Wed, 21 Apr 2021 08:21:21 -0400 Received: from [2a01:e0a:19b:d9a0:9576:8bbf:4795:82ee] (port=38154 helo=localhost.localdomain) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lZBr6-0003pa-P3; Wed, 21 Apr 2021 08:21:21 -0400 From: Mathieu Othacehe Date: Wed, 21 Apr 2021 14:21:04 +0200 Message-Id: <20210421122108.2344-1-othacehe@gnu.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * guix/ci.scm (%default-guix-specification, %default-package-specification): New variables. (, ): New records. (job, job-history, sort-history-by-coverage, channel-commit, package->job-name, manifest->jobs): New procedures. (find-latest-commit-with-substitutes): Rename it into ... (latest-checkouts-with-substitutes): ... this new procedure. (channel-with-substitutes-available): Add an optional manifest argument and honor it. * doc/guix.texi (Channels with Substitutes): Update it. --- doc/guix.texi | 31 ++++++-- guix/ci.scm | 205 ++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 207 insertions(+), 29 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index b9019d5550..c39bbdb3d5 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5201,11 +5201,32 @@ server at @url{https://ci.guix.gnu.org}. "https://ci.guix.gnu.org")) @end lisp -Note that this does not mean that all the packages that you will -install after running @command{guix pull} will have available -substitutes. It only ensures that @command{guix pull} will not try to -compile package definitions. This is particularly useful when using -machines with limited resources. +It is also possible to ask @command{guix pull} to use the latest commit +with the maximal number of available substitutes for a given manifest +this way: + +@lisp +(use-modules (guix ci)) + +(list (channel-with-substitutes-available + %default-guix-channel + "https://ci.guix.gnu.org" + "/path/to/manifest)) +@end lisp + +or this way: + +@lisp +(use-modules (guix ci)) + +(list (channel-with-substitutes-available + %default-guix-channel + "https://ci.guix.gnu.org" + (specifications->manifest + '("git" "emacs-minimal")))) +@end lisp + +This is particularly useful when using machines with limited resources. @node Creating a Channel @section Creating a Channel diff --git a/guix/ci.scm b/guix/ci.scm index c70e5bb9e6..780e90ef32 100644 --- a/guix/ci.scm +++ b/guix/ci.scm @@ -18,10 +18,16 @@ ;;; along with GNU Guix. If not, see . (define-module (guix ci) + #:use-module (gnu packages) + #:use-module (guix channels) #:use-module (guix http-client) + #:use-module (guix packages) + #:use-module (guix profiles) + #:use-module (guix ui) #:use-module (guix utils) #:use-module (json) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:use-module (ice-9 match) #:use-module (guix i18n) #:use-module (guix diagnostics) @@ -58,6 +64,7 @@ latest-evaluations evaluations-for-commit + manifest->jobs channel-with-substitutes-available)) ;;; Commentary: @@ -67,6 +74,14 @@ ;;; ;;; Code: +;; The name of the CI specification building the 'guix-modular' package. +(define %default-guix-specification + (make-parameter "guix")) + +;; The default name of the CI specification building all the packages. +(define %default-package-specification + (make-parameter "master")) + (define-json-mapping make-build-product build-product? json->build-product @@ -109,6 +124,24 @@ (map json->checkout (vector->list checkouts))))) +(define-json-mapping make-job job? + json->job + (name job-name) ;string + (build job-build) ;integer + (status job-status)) ;integer + +(define-json-mapping make-history history? + json->history + (evaluation history-evaluation) ;integer + (checkouts history-checkouts "checkouts" ;* + (lambda (checkouts) + (map json->checkout + (vector->list checkouts)))) + (jobs history-jobs "jobs" + (lambda (jobs) + (map json->job + (vector->list jobs))))) + (define %query-limit ;; Max number of builds requested in queries. 1000) @@ -172,34 +205,158 @@ as one of their inputs." (evaluation-checkouts evaluation))) (latest-evaluations url limit))) -(define (find-latest-commit-with-substitutes url) - "Return the latest commit with available substitutes for the Guix package -definitions at URL. Return false if no commit were found." - (let* ((job-name (string-append "guix." (%current-system))) - (build (match (latest-builds url 1 - #:job job-name - #:status 0) ;success - ((build) build) - (_ #f))) - (evaluation (and build - (evaluation url (build-evaluation build)))) - (commit (and evaluation - (match (evaluation-checkouts evaluation) - ((checkout) - (checkout-commit checkout)))))) - commit)) - -(define (channel-with-substitutes-available chan url) +(define* (job url name #:key evaluation) + "Return the job which name is NAME for the given EVALUATION, from the CI +server at URL." + (map json->job + (vector->list + (json->scm + (http-fetch + (format #f "~a/api/jobs?evaluation=~a&names=~a" + url evaluation name)))))) + +(define* (jobs-history url jobs + #:key + (specification "master") + (limit 20)) + "Return the job history for the SPECIFICATION jobs which names are part of +the JOBS list, from the CI server at URL. Limit the history to the latest +LIMIT evaluations. " + (let ((names (string-join jobs ","))) + (map json->history + (vector->list + (json->scm + (http-fetch + (format #f "~a/api/jobs/history?spec=~a&names=~a&nr=~a" + url specification names (number->string limit)))))))) + +(define (sort-history-by-coverage history) + "Sort and return the given evaluation HISTORY list by descending successful +jobs count. This means that the first element of the list will be the +evaluation with the higher successful jobs count." + (let ((coverage + (map (cut fold + (lambda (status prev) + (if (eq? status 0) ;successful + (1+ prev) + prev)) + 0 <>) + (map (compose + (cut map job-status <>) history-jobs) + history)))) + (map (match-lambda + ((cov . hist) hist)) + (sort (map cons coverage history) + (match-lambda* + (((c1 . h1) (c2 . h2)) + (> c1 c2))))))) + +(define (channel-commit checkouts channel) + "Return the CHANNEL commit from CHECKOUTS." + (any (lambda (checkout) + (and (string=? (checkout-channel checkout) channel) + (checkout-commit checkout))) + checkouts)) + +(define (package->job-name package) + "Return the CI job name for the given PACKAGE name." + (string-append package "." (%current-system))) + +(define (manifest->jobs manifest) + "Return the list of job names that are part of the given MANIFEST." + (define (load-manifest file) + (let ((user-module (make-user-module '((guix profiles) (gnu))))) + (load* file user-module))) + + (let* ((manifest (cond + ((string? manifest) + (load-manifest manifest)) + ((manifest? manifest) + manifest) + (else #f))) + (packages (delete-duplicates + (map manifest-entry-item + (manifest-transitive-entries manifest)) + eq?))) + (map (lambda (package) + (package->job-name (package-name package))) + packages))) + +(define* (latest-checkouts-with-substitutes url jobs) + "Return a list of latest checkouts, sorted by descending substitutes +coverage of the given JOBS list on the CI server at URL. Only evaluations for +which the Guix package is built are considered. + +If JOBS is false, return a list of latest checkouts for which the Guix package +is built. Return false if no checkouts were found." + (define guix-history + (filter (lambda (hist) + (let ((jobs (history-jobs hist))) + (match jobs + ((job) + (eq? (job-status job) 0)) + (else #f)))) + (jobs-history url (list (package->job-name "guix")) + #:specification + (%default-guix-specification)))) + + (define (guix-commit checkouts) + (let ((name (symbol->string + (channel-name %default-guix-channel)))) + (channel-commit checkouts name))) + + (define (guix-package-available? hist) + (any (lambda (guix-hist) + (string=? (guix-commit + (history-checkouts hist)) + (guix-commit + (history-checkouts guix-hist))) + hist) + guix-history)) + + (define (first-checkout checkouts) + (match checkouts + ((checkouts _ ...) + checkouts) + (() #f))) + + (if jobs + (let* ((jobs-history + (sort-history-by-coverage + (jobs-history url jobs + #:specification + (%default-package-specification)))) + (checkouts + (map history-checkouts + (filter-map guix-package-available? + jobs-history)))) + (first-checkout checkouts)) + (first-checkout + (map history-checkouts guix-history)))) + +(define* (channel-with-substitutes-available chan url + #:optional manifest) "Return a channel inheriting from CHAN but which commit field is set to the latest commit with available substitutes for the Guix package definitions at -URL. The current system is taken into account. +URL. If the MANIFEST argument is passed, return the latest commit with the +maximal substitutes coverage of MANIFEST. MANIFEST can be an absolute path as +a string, or a record. The current system is taken into account. If no commit with available substitutes were found, the commit field is set to false and a warning message is printed." - (let ((commit (find-latest-commit-with-substitutes url))) - (unless commit + (let* ((jobs (and manifest + (manifest->jobs manifest))) + (checkouts + (latest-checkouts-with-substitutes url jobs))) + (unless checkouts (warning (G_ "could not find available substitutes at ~a~%") url)) - (channel - (inherit chan) - (commit commit)))) + (let* ((name (channel-name chan)) + (name-str (if (symbol? name) + (symbol->string name) + name)) + (commit (and checkouts + (channel-commit checkouts name-str)))) + (channel + (inherit chan) + (commit commit)))))