[bug#64711,03/43] gnu: coreutils: Skip hanging and failing test on the Hurd.

* gnu/packages/base.scm (coreutils)[arguments]: When building natively on the
Hurd, remove "test/df/unreadable.sh" and "test-perror2" from #:make-flags's
XFAIL_TESTS as they now pass.  Move failing script tests from XFAIL_TESTS to
'remove-tests' phase.  In phase 'remove-tests' rewrite substitute* to skip
tests instead of removing them.  Add hanging and failing tests.
---
 gnu/packages/base.scm | 47 ++++++++++++++++++++++++++++---------------
 1 file changed, 31 insertions(+), 16 deletions(-)

Hello!

Great work!

A general comment: these changes to test suites were made based on
feedback from non-chrooted builds, whose outcome is influenced by system
state such as running translators.  Until we have chroot builds, I think
we should be cautious and not invest too much time.

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

> * gnu/packages/base.scm (coreutils)[arguments]: When building natively on the
> Hurd, remove "test/df/unreadable.sh" and "test-perror2" from #:make-flags's
> XFAIL_TESTS as they now pass.  Move failing script tests from XFAIL_TESTS to
> 'remove-tests' phase.  In phase 'remove-tests' rewrite substitute* to skip
> tests instead of removing them.  Add hanging and failing tests.

[...]

> +                           '((substitute*
> +                                 ;; These tests hang
> +                                 '("tests/cp/sparse-to-pipe.sh"
> +                                   "tests/split/fail.sh"
> +                                   ;; These tests error
> +                                   "tests/dd/nocache.sh"
> +                                   ;; These tests fail
> +                                   "tests/cp/sparse.sh"
> +                                   "tests/cp/special-f.sh"
> +                                   "tests/dd/bytes.sh"
> +                                   "tests/dd/stats.sh"
> +                                   "tests/ls/dangle.sh"
> +                                   "tests/ls/follow-slink.sh"
> +                                   "tests/ls/hyperlink.sh"
> +                                   "tests/ls/infloop.sh"
> +                                   "tests/ls/inode.sh"
> +                                   "tests/ls/selinux-segfault.sh"
> +                                   "tests/misc/env-S.pl"
> +                                   "tests/misc/factor-parallel.sh"
> +                                   "tests/misc/ls-misc.pl"
> +                                   "tests/misc/nice.sh"
> +                                   "tests/misc/pwd-long.sh"
> +                                   "tests/misc/shred-passes.sh"
> +                                   "tests/misc/stat-slash.sh"
> +                                   "tests/rm/fail-eperm.xpl"
> +                                   "tests/split/filter.sh")
> +                               (("^#!.*" all)
> +                                (string-append all "exit 77;\n")))

For example, I wonder how reliable this long list is.  Some of these
tests may fiddle with /proc, /etc/fstab, and other things not under
control, which means some might suddenly pass for some reason.  So I
wonder if we should just #:tests? #f in this case.

Thoughts?

(With other, higher-level test suites with a couple of failures, it’s
probably still a good idea to skip individual tests instead of the whole
test suite.)

Ludo’.

Ludovic Courtès writes:

Hi!

> Great work!

Thank yout!

> A general comment: these changes to test suites were made based on
> feedback from non-chrooted builds, whose outcome is influenced by system
> state such as running translators.  Until we have chroot builds, I think
> we should be cautious and not invest too much time.

That's a helpful perspective, thanks for sharing.

[snip coreutil's long list of failing patches]

> For example, I wonder how reliable this long list is.  Some of these
> tests may fiddle with /proc, /etc/fstab, and other things not under
> control, which means some might suddenly pass for some reason.  So I
> wonder if we should just #:tests? #f in this case.
>
> Thoughts?

In this case, having invested the time to identify hanging and failing
tests, I would suggent to keep the list; but I'm also fine with using
#:tests? #f when more than, say 15 tests fail?

What I failed to mention, is that I looked into what Debian is doing for
cases like these, and without exception they just skip the test suite
for the Hurd, even for critical packages such as coreutils and python.
>
> (With other, higher-level test suites with a couple of failures, it’s
> probably still a good idea to skip individual tests instead of the whole
> test suite.)

Noted.  Now I'm wondering whether glib should be considered to have
higher level tests.  It would have been nice if I kept some kind of
record on this.  It will be a bit less painful te re-check once CI built
most upto glib, dunno.

Greetings,
Janneke

Ludovic Courtès writes:

Hi!

> Great work!

Thank you!

> A general comment: these changes to test suites were made based on
> feedback from non-chrooted builds, whose outcome is influenced by system
> state such as running translators.  Until we have chroot builds, I think
> we should be cautious and not invest too much time.

That's a helpful perspective, thanks for sharing.

[snip coreutil's long list of failing patches]

> For example, I wonder how reliable this long list is.  Some of these
> tests may fiddle with /proc, /etc/fstab, and other things not under
> control, which means some might suddenly pass for some reason.  So I
> wonder if we should just #:tests? #f in this case.
>
> Thoughts?

In this case, having invested the time to identify hanging and failing
tests, I would suggent to keep the list; but I'm also fine with using
#:tests? #f when more than, say 15 tests fail?

What I failed to mention, is that I looked into what Debian is doing for
cases like these, and without exception they just skip the test suite
for the Hurd, even for critical packages such as coreutils and python.
>
> (With other, higher-level test suites with a couple of failures, it’s
> probably still a good idea to skip individual tests instead of the whole
> test suite.)

Noted.  Now I'm wondering whether glib should be considered to have
higher level tests.  It would have been nice if I kept some kind of
record on this.  It will be a bit less painful te re-check once CI built
most upto glib, dunno.

Greetings,
Janneke