From patchwork Mon Jan 18 06:24:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 26453 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id 8C51227BC16; Mon, 18 Jan 2021 06:27:09 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id E9EE227BC15 for ; Mon, 18 Jan 2021 06:27:07 +0000 (GMT) Received: from localhost ([::1]:53626 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l1O0J-0007B0-3p for patchwork@mira.cbaines.net; Mon, 18 Jan 2021 01:27:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59658) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0006n0-I0 for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34615) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0001Cz-B1 for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l1NzH-0003tW-7Z for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45948] [PATCH 4/5] build: test-driver.scm: Add test cases filtering options. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 18 Jan 2021 06:26:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45948 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45948@debbugs.gnu.org Received: via spool by 45948-submit@debbugs.gnu.org id=B45948.161095113314894 (code B ref 45948); Mon, 18 Jan 2021 06:26:03 +0000 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:33 +0000 Received: from localhost ([127.0.0.1]:46151 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nym-0003s8-Pv for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:33 -0500 Received: from mail-qv1-f43.google.com ([209.85.219.43]:46115) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyj-0003rQ-HL for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:30 -0500 Received: by mail-qv1-f43.google.com with SMTP id a1so7027051qvd.13 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=gQWH98SyRFPA7QVmOdsO2pB0Vhy2uUM4zEg8CJPOCY4HVYXO03GAlHdft3rq0JiDlP tmQcyO7nepja6gtox7VqvQDgV7p3uGTXvC0ncyZIhucwHwTnQX4kdmTjJNaC6pT3wTlm 3bCpvQHgwfXclSOfFVeuD+e0w/8wFV9/ya0cf/SWSb3BrqLJCnSPR7l4pt2WJXnwBbys hg1H5rW+beX7YB8E4juxkG1b85AlQSF1HFZa6Z7VnA9hT0MC0LP3ZwzkmNfzepwrzava 6jK2mwhuzjcrzJgpwkeq6udgVPgYxuRLnvjOpz9jpgXxblsl9DMXQWNpE2O3a7VgZ3Rs stNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=aQ8aSb93+SExdCdJSRF4s+yb0exEf0nNkGdavKGhPiJ94b5yAszKm3QPWkgY6KX9bn SXsD3Cdadlz0x6EiAMjJg4MX2vRpmg91NSo/EyfI/EB4wYgqc4JNg+8MXcvQvNY9wFNa t5Z/4KlXpI4Phu1Gem15LfHUdVGXOtIRcdoEkdiY0H9dH71vLALXYIdDTuEyedVMJfwn 57o9ffoAIYp0ujRQqxWnN0EVimt1jEZkfeD513uKxNuhbLz+o38qraN5UJTRa/QAdzca eOFjqx5qPvlePnNGaFa1luczqh3+6hRs3AA8AdtaG/knVJYlQZD3q5XPaDatgzcvItdc SuSA== X-Gm-Message-State: AOAM533yXfOUAWh5TQHNcVldFv3ik/4yiqM9B9Z7PQnmIqqEmgR/D2TK 5utcMUtDwoqHEtMelYw12xu01BUHXXk= X-Google-Smtp-Source: ABdhPJyv1oC/q+bCdriupaaNGoaeOfGZQwyaxDAH+YmZCvEjI88wS3tz2rZoMuPkJo47BaNdPiIsfQ== X-Received: by 2002:a05:6214:1583:: with SMTP id m3mr22847389qvw.48.1610951123645; Sun, 17 Jan 2021 22:25:23 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:23 -0800 (PST) From: Maxim Cournoyer Date: Mon, 18 Jan 2021 01:24:59 -0500 Message-Id: <20210118062501.27022-4-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * build-aux/test-driver.scm (show-help): Add help text for the new --select and --exclude options. (%options): Add the new select and exclude options. (test-runner-gnu): Pass them to the test runner. Update doc. (test-match-name*, test-match-name*/negated, %test-match-all): New variables. (main): Compute the test specifier based on the values of the new options and apply it to the current test runner when running the test file. * doc/guix.texi (Running the Test Suite): Document the new options. --- build-aux/test-driver.scm | 66 ++++++++++++++++++++++++++++++++------- doc/guix.texi | 12 +++++++ 2 files changed, 66 insertions(+), 12 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 767a9fb25d..3ad6b0c93f 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -27,6 +27,8 @@ (use-modules (ice-9 getopt-long) (ice-9 pretty-print) + (ice-9 regex) + (srfi srfi-1) (srfi srfi-26) (srfi srfi-64)) @@ -34,14 +36,19 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] + [--select=REGEXP] [--exclude=REGEXP] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] -The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) +The '--test-name', '--log-file' and '--trs-file' options are mandatory. +The '--select' and '--exclude' options allow selecting or excluding individual +test cases via a regexp, respectively.\n")) (define %options '((test-name (value #t) (required? #t)) (log-file (value #t)) (trs-file (value #t)) + (select (value #t)) + (exclude (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -76,14 +83,22 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) "") ;no color result))) + +;;; +;;; SRFI 64 custom test runner. +;;; + (define* (test-runner-gnu test-name #:key color? brief? (out-port (current-output-port)) - (trs-port (%make-void-port "w"))) + (trs-port (%make-void-port "w")) + select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the file name of the current the test. COLOR? specifies whether to use colors, and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged." +void port, which means no TRS output is logged. SELECT and EXCLUDE may take a +regular expression to select or exclude individual test cases based on their +names." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the @@ -148,6 +163,24 @@ void port, which means no TRS output is logged." (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) runner)) + +;;; +;;; SRFI 64 test specifiers. +;;; +(define (test-match-name* regexp) + "Return a test specifier that matches a test name against REGEXP." + (lambda (runner) + (string-match regexp (test-runner-test-name runner)))) + +(define (test-match-name*/negated regexp) + "Return a negated test specifier version of test-match-name*." + (lambda (runner) + (not (string-match regexp (test-runner-test-name runner))))) + +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list +;;; of test specifiers computed at run time. +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) + ;;; ;;; Entry point. @@ -158,15 +191,22 @@ void port, which means no TRS output is logged." (option (cut option-ref opts <> <>))) (cond ((option 'help #f) (show-help)) - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) + ((option 'version #f) (format #t "test-driver.scm ~A" 'script-version)) (else - (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) - (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) - (out (duplicate-port (current-output-port) "wl")) - (test-name (option 'test-name #f)) - (color-tests (if (assoc 'color-tests opts) - (option->boolean opts 'color-tests) - #t))) + (let* ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) + (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) + (out (duplicate-port (current-output-port) "wl")) + (test-name (option 'test-name #f)) + (select (option 'select #f)) + (exclude (option 'exclude #f)) + (test-specifiers (filter-map + identity + (list (and=> select test-match-name*) + (and=> exclude test-match-name*/negated)))) + (test-specifier (apply %test-match-all test-specifiers)) + (color-tests (if (assoc 'color-tests opts) + (option->boolean opts 'color-tests) + #t))) (when log (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) @@ -176,7 +216,9 @@ void port, which means no TRS output is logged." #:color? color-tests #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) - (load-from-path test-name)) + (test-apply test-specifier + (lambda _ + (load-from-path test-name)))) (and=> log close-port) (and=> trs close-port) (close-port out)))) diff --git a/doc/guix.texi b/doc/guix.texi index 194bb3a314..d59d2806cb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -915,6 +915,18 @@ the @code{SCM_LOG_DRIVER_FLAGS} makefile variable as in this example: make check TESTS="tests/base64.scm" SCM_LOG_DRIVER_FLAGS="--brief=no" @end example +The underlying SRFI 64 custom Automake test driver used for the 'check' +test suite (located at @file{build-aux/test-driver.scm}) also allows +selecting which test cases to run at a finer level, via its +@option{--select} and @option{--exclude} options. Here's an example, to +run all the test cases from the @file{tests/packages.scm} test file +whose names start with ``transaction-upgrade-entry'': + +@example +export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" +make check TESTS="tests/packages.scm" +@end example + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in From patchwork Mon Jan 18 06:25:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Cournoyer X-Patchwork-Id: 26454 Return-Path: X-Original-To: patchwork@mira.cbaines.net Delivered-To: patchwork@mira.cbaines.net Received: by mira.cbaines.net (Postfix, from userid 113) id C9C3427BC15; Mon, 18 Jan 2021 06:27:09 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mira.cbaines.net (Postfix) with ESMTPS id DADCF27BC14 for ; Mon, 18 Jan 2021 06:27:07 +0000 (GMT) Received: from localhost ([::1]:53642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l1O0J-0007BL-3E for patchwork@mira.cbaines.net; Mon, 18 Jan 2021 01:27:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59660) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0006nL-Q9 for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34616) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l1NzI-0001D3-IK for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l1NzI-0003ts-EW for guix-patches@gnu.org; Mon, 18 Jan 2021 01:26:04 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45948] [PATCH 5/5] build: test-driver.scm: Add a new '--errors-only' option. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 18 Jan 2021 06:26:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45948 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45948@debbugs.gnu.org Received: via spool by 45948-submit@debbugs.gnu.org id=B45948.161095113814920 (code B ref 45948); Mon, 18 Jan 2021 06:26:04 +0000 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:38 +0000 Received: from localhost ([127.0.0.1]:46157 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nys-0003sZ-6f for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:38 -0500 Received: from mail-qt1-f180.google.com ([209.85.160.180]:46492) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyl-0003rU-Qb for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:32 -0500 Received: by mail-qt1-f180.google.com with SMTP id h19so10557405qtq.13 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dwh4qyXoSLILAiAytmhHLl4/osgRShSj7MzucfUFLug=; b=A6YjNY87nRt7JcWI3+h9+5Geh2aleRymUYlutfnzhDJE9g2bsaT355beurlkC2FBqa rft88rQcSohRVR8FOZ9cReljm1KUEZK1lXGGZEJ2raM2dFhABAwGQ74zAIgHZffC6ebN KV2+fI0ACs3lpXlxUNOWbuMLU4CRUFnAYJH9tLGgkpGEHOgeJFb79xAKvo3Z2HLoJVpP 5d5+mnniGO8ghzV5cMIopf9whQTlxVVYvyZD05FY+p8K+LKMdNcAYysB5sSZ6oBJOfEW V81IrXwvJxig7piEiRQrQ81OHKBlL3sxKAVGOUFEHBbwo3c2TV53v5sniDhoS3kbyDby r6mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dwh4qyXoSLILAiAytmhHLl4/osgRShSj7MzucfUFLug=; b=QihlMiHSlt/+mHeQ23cfXrU4UxQnDbsFED+1aIl5A+lzgiBDcgiJZwh6vP/k6beb33 DuhgnibJIyNNVJg72Gf8/NM00ibWx38mTXGOTyjbMxZvL8uu9c1ox95UcpEiBm9QVJoD GYRqldsdYnkeoQaeY/f5ZLOFmgP9z0T1MO7XkICSq9tbiqbzbZv8jjG48uhAQBcuHlPV Cs+zq9xVLLTGxz1SqVJZes9uHzspZWWiqgdlz0zcd59O5tGEy57QWxzYI5LKlH1IlGVJ tw3xlAW5wZjnvVYUrjuoptD4h8Rwmnh0EUGP7TsSlgc3Zx2l3hILPaT9XChKSq7g0ls0 jvLw== X-Gm-Message-State: AOAM530eSyRVRYZYxDRkg41xcHNmY3CbksmuXN61fFTubIBmwCze9+YS umNR7/sJ6UgyYNX6U/6Mfvk4Kf0tFVI= X-Google-Smtp-Source: ABdhPJw/ERNTEtYOqStqVWIkZZArI9GnxNkVstgu9+PydRdt0BkFet8mJIOToJ3RAohaUvmGBOSh7g== X-Received: by 2002:ac8:7153:: with SMTP id h19mr3371443qtp.118.1610951126025; Sun, 17 Jan 2021 22:25:26 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:25 -0800 (PST) From: Maxim Cournoyer Date: Mon, 18 Jan 2021 01:25:01 -0500 Message-Id: <20210118062501.27022-6-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+patchwork=mira.cbaines.net@gnu.org Sender: "Guix-patches" X-getmail-retrieved-from-mailbox: Patches * build-aux/test-driver.scm (show-help): Add the help text for the new '--errors-only' option. (%options): Add the errors-only option. (test-runner-gnu): Add the errors-only? parameter and update doc. Move the logging of the test data after the test has completed, so a choice can be made whether to keep it or discard it based on the value of the test result. (main): Pass the errors-only? option to the driver. * doc/guix.texi (Running the Test Suite): Document the new option. --- build-aux/test-driver.scm | 73 ++++++++++++++++++++++----------------- doc/guix.texi | 12 +++++++ 2 files changed, 54 insertions(+), 31 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 3ad6b0c93f..3f593abc7b 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -36,12 +36,15 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] - [--select=REGEXP] [--exclude=REGEXP] + [--select=REGEXP] [--exclude=REGEXP] [--errors-only={yes|no}] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] The '--test-name', '--log-file' and '--trs-file' options are mandatory. The '--select' and '--exclude' options allow selecting or excluding individual -test cases via a regexp, respectively.\n")) +test cases via a regexp, respectively. The '--errors-only' option can be set +to \"yes\" to limit the logged test case metadata to only those test cases +that failed. When set to \"yes\", the '--brief' option disables printing the +individual test case result to the console.\n")) (define %options '((test-name (value #t) (required? #t)) @@ -49,6 +52,7 @@ test cases via a regexp, respectively.\n")) (trs-file (value #t)) (select (value #t)) (exclude (value #t)) + (errors-only (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -88,27 +92,26 @@ test cases via a regexp, respectively.\n")) ;;; SRFI 64 custom test runner. ;;; -(define* (test-runner-gnu test-name #:key color? brief? +(define* (test-runner-gnu test-name #:key color? brief? errors-only? (out-port (current-output-port)) (trs-port (%make-void-port "w")) select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the -file name of the current the test. COLOR? specifies whether to use colors, -and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. -OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged. SELECT and EXCLUDE may take a -regular expression to select or exclude individual test cases based on their -names." - - (define (test-on-test-begin-gnu runner) - ;; Procedure called at the start of an individual test case, before the - ;; test expression (and expected value) are evaluated. - (let ((result (cute assq-ref (test-result-alist runner) <>))) - (format #t "test-name: ~A~%" (result 'test-name)) - (format #t "location: ~A~%" - (string-append (result 'source-file) ":" - (number->string (result 'source-line)))) - (test-display "source" (result 'source-form) #:pretty? #t))) +file name of the current the test. COLOR? specifies whether to use colors. +When BRIEF? is true, the individual test cases results are masked and only the +summary is shown. ERRORS-ONLY? reduces the amount of test case metadata +logged to only that of the failed test cases. OUT-PORT and TRS-PORT must be +output ports. OUT-PORT defaults to the current output port, while TRS-PORT +defaults to a void port, which means no TRS output is logged. SELECT and +EXCLUDE may take a regular expression to select or exclude individual test +cases based on their names." + + (define (test-skipped? runner) + (eq? 'skip (test-result-kind runner))) + + (define (test-failed? runner) + (not (or (test-passed? runner) + (test-skipped? runner)))) (define (test-on-test-end-gnu runner) ;; Procedure called at the end of an individual test case, when the result @@ -116,21 +119,29 @@ names." (let* ((results (test-result-alist runner)) (result? (cut assq <> results)) (result (cut assq-ref results <>))) - (unless brief? + (unless (or brief? (and errors-only? (test-skipped? runner))) ;; Display the result of each test case on the console. (format out-port "~A: ~A - ~A~%" (result->string (test-result-kind runner) #:colorize? color?) test-name (test-runner-test-name runner))) - (when (result? 'expected-value) - (test-display "expected-value" (result 'expected-value))) - (when (result? 'expected-error) - (test-display "expected-error" (result 'expected-error) #:pretty? #t)) - (when (result? 'actual-value) - (test-display "actual-value" (result 'actual-value))) - (when (result? 'actual-error) - (test-display "actual-error" (result 'actual-error) #:pretty? #t)) - (format #t "result: ~a~%" (result->string (result 'result-kind))) - (newline) + + (unless (and errors-only? (not (test-failed? runner))) + (format #t "test-name: ~A~%" (result 'test-name)) + (format #t "location: ~A~%" + (string-append (result 'source-file) ":" + (number->string (result 'source-line)))) + (test-display "source" (result 'source-form) #:pretty? #t) + (when (result? 'expected-value) + (test-display "expected-value" (result 'expected-value))) + (when (result? 'expected-error) + (test-display "expected-error" (result 'expected-error) #:pretty? #t)) + (when (result? 'actual-value) + (test-display "actual-value" (result 'actual-value))) + (when (result? 'actual-error) + (test-display "actual-error" (result 'actual-error) #:pretty? #t)) + (format #t "result: ~a~%" (result->string (result 'result-kind))) + (newline)) + (format trs-port ":test-result: ~A ~A~%" (result->string (test-result-kind runner)) (test-runner-test-name runner)))) @@ -157,7 +168,6 @@ names." #f)) (let ((runner (test-runner-null))) - (test-runner-on-test-begin! runner test-on-test-begin-gnu) (test-runner-on-test-end! runner test-on-test-end-gnu) (test-runner-on-group-end! runner test-on-group-end-gnu) (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) @@ -215,6 +225,7 @@ names." (test-runner-gnu test-name #:color? color-tests #:brief? (option->boolean opts 'brief) + #:errors-only? (option->boolean opts 'errors-only) #:out-port out #:trs-port trs) (test-apply test-specifier (lambda _ diff --git a/doc/guix.texi b/doc/guix.texi index d59d2806cb..ac20a541f5 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -927,6 +927,18 @@ export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" make check TESTS="tests/packages.scm" @end example +Those wishing to inspect the results of failed tests directly from the +command line can add the @option{--errors-only=yes} option to the +@code{SCM_LOG_DRIVER_FLAGS} makefile variable and set the @code{VERBOSE} +Automake makefile variable, as in: + +@example +make check SCM_LOG_DRIVER_FLAGS="--brief=no --errors-only=yes" VERBOSE=1 +@end example + +@xref{Parallel Test Harness,,,automake,GNU Automake} for more +information about the Automake Parallel Test Harness. + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in