Message ID | 20210118062501.27022-5-maxim.cournoyer@gmail.com |
---|---|
State | Accepted |
Headers | show |
Series | Improvements to the Automake SRFI 64 test driver. | expand |
Hi! Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis: > * build-aux/test-driver.scm (show-help): Add help text for the new --select > and --exclude options. > (%options): Add the new select and exclude options. > (test-runner-gnu): Pass them to the test runner. Update doc. > (test-match-name*, test-match-name*/negated, %test-match-all): New variables. > (main): Compute the test specifier based on the values of the new options and > apply it to the current test runner when running the test file. > * doc/guix.texi (Running the Test Suite): Document the new options. I never felt the need for this since most individual files run quickly enough (and those that don’t should be optimized…), but it can be useful. > +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list > +;;; of test specifiers computed at run time. > +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) Since this is an internal procedure in Guile that could vanish anytime, I recommend copying it here (it’s just 9 lines). > +The underlying SRFI 64 custom Automake test driver used for the 'check' > +test suite (located at @file{build-aux/test-driver.scm}) also allows Maybe shorten to “The underlying test driver (located at @file{build-aux/test-driver.scm}) also allows”. I didn’t test it, but it otherwise LGTM. Ludo’.
Hello, Ludovic Courtès <ludo@gnu.org> writes: [...] > I never felt the need for this since most individual files run quickly > enough (and those that don’t should be optimized…), but it can be > useful. What triggered it for me was trying to iterate using tests added to the tests/packages.scm test module: --8<---------------cut here---------------start------------->8--- $ time make check TESTS=tests/packages.scm VERBOSE=1 SCM_LOG_DRIVER_FLAGS="--brief=no" time make check TESTS=tests/packages.scm VERBOSE=1 SCM_LOG_DRIVER_FLAGS="--brief=no" [...] PASS: tests/packages.scm - package-patched-vulnerabilities PASS: tests/packages.scm - fold-packages PASS: tests/packages.scm - fold-packages, hidden package [... time passes ...] PASS: tests/packages.scm - fold-available-packages with/without cache [...] PASS: tests/packages.scm - find-package-locations with cache PASS: tests/packages.scm - specification->location ============================================================================ Testsuite summary for GNU Guix UNKNOWN ============================================================================ # TOTAL: 100 # PASS: 100 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ [...] real 0m46.172s user 1m4.885s sys 0m0.376s --8<---------------cut here---------------end--------------->8--- That's on the fastest machine I have access to (on my vintage desktop, it took nearly 4 minutes). The slowest test seems to be 'fold-available-packages with/without cache'. Now with the new select, one can do: --8<---------------cut here---------------start------------->8--- $ time make check TESTS=tests/packages.scm VERBOSE=1 SCM_LOG_DRIVER_FLAGS="--brief=no --select='bag->derivation' --errors-only=yes" [...] PASS: tests/packages.scm - bag->derivation PASS: tests/packages.scm - bag->derivation, cross-compilation ============================================================================ Testsuite summary for GNU Guix UNKNOWN ============================================================================ # TOTAL: 100 # PASS: 2 # SKIP: 98 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ [...] real 0m1.569s user 0m2.382s sys 0m0.154s --8<---------------cut here---------------end--------------->8--- 1.6 s; better than 46 s! We can also check the time the suspected slow test took: $ time make check TESTS=tests/packages.scm SCM_LOG_DRIVER_FLAGS="--select='fold-available-packages with/without cache'" [...] PASS: tests/packages.scm - fold-available-packages with/without cache ============================================================================ Testsuite summary for GNU Guix UNKNOWN ============================================================================ # TOTAL: 100 # PASS: 1 # SKIP: 99 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ [...] real 0m36.627s user 0m45.731s sys 0m0.264s So yes, this is the most expensive test of tests/packages.scm. >> +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list >> +;;; of test specifiers computed at run time. >> +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) > > Since this is an internal procedure in Guile that could vanish anytime, > I recommend copying it here (it’s just 9 lines). Done! >> +The underlying SRFI 64 custom Automake test driver used for the 'check' >> +test suite (located at @file{build-aux/test-driver.scm}) also allows > > Maybe shorten to “The underlying test driver (located at > @file{build-aux/test-driver.scm}) also allows”. I see value in explicitly stating what it is, as it took me some effort to be able to answer that question when I started looking at it (the test driver). I've now pushed this series to master; thank you for the review! Maxim
diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 767a9fb25d..3ad6b0c93f 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -27,6 +27,8 @@ (use-modules (ice-9 getopt-long) (ice-9 pretty-print) + (ice-9 regex) + (srfi srfi-1) (srfi srfi-26) (srfi srfi-64)) @@ -34,14 +36,19 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] + [--select=REGEXP] [--exclude=REGEXP] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] -The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) +The '--test-name', '--log-file' and '--trs-file' options are mandatory. +The '--select' and '--exclude' options allow selecting or excluding individual +test cases via a regexp, respectively.\n")) (define %options '((test-name (value #t) (required? #t)) (log-file (value #t)) (trs-file (value #t)) + (select (value #t)) + (exclude (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -76,14 +83,22 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) "[m") ;no color result))) + +;;; +;;; SRFI 64 custom test runner. +;;; + (define* (test-runner-gnu test-name #:key color? brief? (out-port (current-output-port)) - (trs-port (%make-void-port "w"))) + (trs-port (%make-void-port "w")) + select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the file name of the current the test. COLOR? specifies whether to use colors, and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged." +void port, which means no TRS output is logged. SELECT and EXCLUDE may take a +regular expression to select or exclude individual test cases based on their +names." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the @@ -148,6 +163,24 @@ void port, which means no TRS output is logged." (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) runner)) + +;;; +;;; SRFI 64 test specifiers. +;;; +(define (test-match-name* regexp) + "Return a test specifier that matches a test name against REGEXP." + (lambda (runner) + (string-match regexp (test-runner-test-name runner)))) + +(define (test-match-name*/negated regexp) + "Return a negated test specifier version of test-match-name*." + (lambda (runner) + (not (string-match regexp (test-runner-test-name runner))))) + +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list +;;; of test specifiers computed at run time. +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) + ;;; ;;; Entry point. @@ -158,15 +191,22 @@ void port, which means no TRS output is logged." (option (cut option-ref opts <> <>))) (cond ((option 'help #f) (show-help)) - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) + ((option 'version #f) (format #t "test-driver.scm ~A" 'script-version)) (else - (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) - (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) - (out (duplicate-port (current-output-port) "wl")) - (test-name (option 'test-name #f)) - (color-tests (if (assoc 'color-tests opts) - (option->boolean opts 'color-tests) - #t))) + (let* ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) + (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) + (out (duplicate-port (current-output-port) "wl")) + (test-name (option 'test-name #f)) + (select (option 'select #f)) + (exclude (option 'exclude #f)) + (test-specifiers (filter-map + identity + (list (and=> select test-match-name*) + (and=> exclude test-match-name*/negated)))) + (test-specifier (apply %test-match-all test-specifiers)) + (color-tests (if (assoc 'color-tests opts) + (option->boolean opts 'color-tests) + #t))) (when log (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) @@ -176,7 +216,9 @@ void port, which means no TRS output is logged." #:color? color-tests #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) - (load-from-path test-name)) + (test-apply test-specifier + (lambda _ + (load-from-path test-name)))) (and=> log close-port) (and=> trs close-port) (close-port out)))) diff --git a/doc/guix.texi b/doc/guix.texi index 194bb3a314..d59d2806cb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -915,6 +915,18 @@ the @code{SCM_LOG_DRIVER_FLAGS} makefile variable as in this example: make check TESTS="tests/base64.scm" SCM_LOG_DRIVER_FLAGS="--brief=no" @end example +The underlying SRFI 64 custom Automake test driver used for the 'check' +test suite (located at @file{build-aux/test-driver.scm}) also allows +selecting which test cases to run at a finer level, via its +@option{--select} and @option{--exclude} options. Here's an example, to +run all the test cases from the @file{tests/packages.scm} test file +whose names start with ``transaction-upgrade-entry'': + +@example +export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" +make check TESTS="tests/packages.scm" +@end example + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in