When running multiple sections and hundreds of tests in each config
section, it's hard to see what failed from the summary output
because of the hundreds of tests listed in the "tests run" and
"tests not run" output. Add a "-b" option for brief result summaries
that only output the tests that failed and the summary count of
tests failed.
Signed-Off-By: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
Sometimes it's useful to keep track of why we're excluding a particular
test. Technically, we can include whatever explanation we want in the
exclude file since we just grep for the test names, but properly
supporting comments is a little more robust. This patch makes it so that
the rest of a line after a '#' is ignored.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
After the previous patch moved few lines of code, one seqnum
assignment is now immediately overwritten by another. Remove the
useless one.
Signed-off-by: Jan Tulak <jtulak@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
The code handling "./check foo/123", when the real test is
"foo/123-bar-baz" was moved to the earliest position, so everything
working with the test name or path will know the full name. Thus, no
"123" and "123-bar-baz" mix is possible.
An example of this issue is $testname.notrun file. When _notrun
"foo" was run during ./check foo/$name command, it created
$name.notrun. But few lines later, it wanted $fullname.notrun. So if
you did ./check foo/999, but the file was 999-bar-baz, then you got
comparing outputs (and most likely a fail) instead of a skip.
Another example of this mix is in xfstests output:
./check xfs/999
[...]
xfs/999 0s ... 0s
Ran: xfs/999-test-case
Signed-off-by: Jan Tulak <jtulak@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
results/check.log is inconsistent with what goes to stdout and $tmp.summary. It
passes the ran and failed tests through fmt but not the "Not Run" tests for some
reason, and nobody else passes it through fmt. So fix this to be the same
output as what goes to stdout and $tmp.summary to make for easier post parsing.
Now we'll get
Ran: <single line with all the tests>
Not run: <single line with all the not run tests>
Failures: <single line with all the failed tests>
Signed-off-by: Josef Bacik <jbacik@fb.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
There are no tmpfs specific tests, so tests/tmpfs does not exist.
This commit avoids printing a spurious error message when running
specifying a group of tests (e.g., "check -g quick"):
DEVICE: test:/tmp
MK2FS OPTIONS:
MOUNT OPTIONS: -o block_validity
./check: line 96: tests/tmpfs/group: No such file or directory
FSTYP -- tmpfs
PLATFORM -- Linux/i686 kvm-xfstests 4.5.0-rc2ext4-00002-g6df2762
MKFS_OPTIONS -- test:/scratch
MOUNT_OPTIONS -- -o size=1G test:/scratch /test/scratch
generic/001 [10:31:10][ 5.811742] run fstests generic/001
...
Similar problems have been reported when testing nfs using xfstests.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
When we have a config file with multiple sections, we might want to
exclude certain config sections from running. Rather than specifying
all the section we want to run, add a "-S <section>" option to build
up a list of sections to exclude.
This is useful if a given section config is known to cause a fatal
failure,but you still want to run all the other config sections.
Also add support to the setup program that emits the currently
configured setup for each section in the config file.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Make the setup command section aware so that it is easy to test
whether the section config code is generating the correct
configurations or not.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Adding basic overlayfs support to fstests, it doesn't test anything
overlayfs specific, but runs existing tests on top of overlayfs. It's
following the path from Eric's patchset and Zab's review back in Mar.
A new fstype "overlay" is added, and TEST_DEV/SCRATCH_DEV are required
to be fs paths, and overlayfs is mounted at TEST_DIR/SCRATCH_MNT, so
tests can be run there.
To test overlayfs, setup config as something like the following
TEST_DEV=/mnt/ovl/test
TEST_DIR=/mnt/testarea/test
SCRATCH_DEV=/mnt/ovl/scratch
SCRATCH_MNT=/mnt/testarea/scratch
then run
./check -overlay -g auto
Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Replace every explicit mount/umount of scratch or test devices with
helper functions. This allows the next patch to add in hooks to these
functions in order to set up & tear down overlayfs on every mount/umount
(also adds _test_unmount(), which didn't exist prior)
[Eryu Guan rebased the patch agains latest master and replaced more
mount/umount with helpers]
Signed-off-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eric Sandeen <esandeen@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Add -d debug dump flag to ./check to directly print a test output
to stdout, instead of just saving it into a file and showing a diff
snippet.
Useful e.g. when writing a new test.
Signed-off-by: Jan Tulak <jtulak@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
When running ./check and calling a test with a name, id is enough
to find the test (names added in 03c633bf).
If the full test path is tests/xfs/123-foo-bar, then all these
invocations should work, as long as the given part of the test name
is valid and the three-digits id is here.
./check xfs/123-foo-bar
./check xfs/123-foo
./check xfs/123
Always use full test name in results.
Signed-off-by: Jan Tulak <jtulak@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
In manual:
# ./check --help
...
-g group[,group...] include tests from these groups
-x group[,group...] exclude tests from these groups
...
Above format is not supported, because ',' is not parsed in code.
This patch fixed it.
Before patch:
# ./check -g scrub,replace
Group "scrub,replace" is empty or not defined?
After patch:
# ./check -g scrub,replace
FSTYP -- btrfs
PLATFORM -- Linux/x86_64 kerneldev 4.2.0-rc2_HEAD_c65b99f046843d2455aa231747b5a07a999a9f3d_+
MKFS_OPTIONS -- /dev/vdd
MOUNT_OPTIONS -- /dev/vdd /var/ltf/tester/scratch_mnt
...(right result)...
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
check has trapped 'exit 1' and exit with $status, but check always
returns 0 on error because status never gets updated. This causes
problems while running some tests in a loop until it fails, e.g.
while ./check generic/081; do : ; done
Just set status to 1 before exit, as what we do in the tests.
Also remove an unused $flag while we're at it.
Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
To allow test names to be more descriptive, allow a suffix to be
added to the numeric name of the test. e.g. a test can be named
"tests/generic/001-some-descriptive-name".
Name suffixes are limited to alphanumeric characters and dash - the
name is always prefixed with an unique id for easy identification
of a specific test. Hence we can still use shorthand forms such as
"generic/001" when referring to a test and be clearly understood.
Signed-off-by: Jan Tulak <jtulak@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: David Sterba <dsterba@suse.cz>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Check kernel BUG, WARNING etc. in dmesg log after each test and fail the
test if something is found. dmesg log can be found at result dir.
This check now depends on the logging of running tests in dmesg, so this
check can be done without clearing dmesg buffer nor dumping all dmesg to
a file, which can potentially eat all free space on testing host.
Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
We already log the running test to system logs via "logger"
but viro pointed out that we can use /dev/kmsg to insert it
into dmesg as well. When looking at the serial console that
could be pretty useful.
Thanks to viro for the test -w suggestion too.
[Eryu Guan adds timestamp to dmesg too]
Signed-off-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Currently if _check_test_fs and/or _check_scratch_fs find corruption,
the test itself is still reported as pass, like
[root@hp-dl388eg8-01 xfstests]# ./check xfs/071 xfs/072
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/x86_64 hp-dl388eg8-01 3.18.0-rc7+
MKFS_OPTIONS -- -f -bsize=4096 /dev/sda6
MOUNT_OPTIONS -- -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch
xfs/071 2s
_check_xfs_filesystem: filesystem on /dev/sda6 is inconsistent (r) (see /root/xfstests/results//xfs/071.full)
xfs/072 1s
Ran: xfs/071 xfs/072
Passed all 2 tests
[root@hp-dl388eg8-01 xfstests]# echo $?
0
Usually it's not a problem, but it does confuse scripts that depend on
return value of check. Update check to treat _check_{test,scratch}_fs
failures as test failures too, new test output is like
[root@hp-dl388eg8-01 xfstests]# ./check xfs/071 xfs/072
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/x86_64 hp-dl388eg8-01 3.18.0-rc7+
MKFS_OPTIONS -- -f -bsize=4096 /dev/sda6
MOUNT_OPTIONS -- -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch
xfs/071 2s ... 2s
_check_xfs_filesystem: filesystem on /dev/sda6 is inconsistent (r) (see /root/xfstests/results//xfs/071.full)
xfs/072 1s ... 1s
Ran: xfs/071 xfs/072
Failures: xfs/071
Failed 1 of 2 tests
[root@hp-dl388eg8-01 xfstests]# echo $?
1
Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Pass -cifs argument from the command line to enable cifs testing
for $TEST_DEV. Also mention CIFS and missed UDF in README.
Signed-off-by: Pavel Shilovsky <pshilovsky@samba.org>
Reviewed-by: Steve French <smfrench@gmail.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Currently we're checking file system consistency on TEST_DEV after every
successful test run even though the TEST_DEV might not even be used in
that test.
Fix it by introducing _require_test to for the test ti indicate that
it's about to use TEST_DEV.
Also add _require_test to the new script so that this requirement is a
default.
Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
There are about 198 tests which requires scratch_dev, but does not check
the file system consistency afterwards. Xfstests infrastructure does not
do it automatically, so fix it by running _check_scratch_fs() after
each test that _require_scratch.
Also remove all the _check_scratch_fs() calls that are not actually needed
and will be covered by the check script.
Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Currently the -X option is intended to specify a set of expunging
files which are stored in each test/* subdirectory. As described in
the commit description for 0b1e8abd4, in order to exclude the test
generic/280, the -X option is used as follows:
$ cat tests/generic/3.0-stable-avoid
280
$ sudo ./check -X 3.0-stable-avoid generic/280
However, it is sometimes useful to store the set of expunged tests in
a single file, outside of tests/* subdirectories. This commit enables
the following:
$ cat /root/conf/data_journal.exclude
generic/068
ext4/301
$ sudo ./check -E /root/conf/data_journal.exclude -g auto
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
This commit adds -s option which allows you to specify only certain
sections from the config file to be run. The '-s' argument can be
specified multiple times to allow multiple sections to be run.
The options are still carried between section, that includes the
sections which are not going to be run.
Update README.config-sections as well.
Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner
Omar Sandoval
Jan Tulak
Josef Bacik
Theodore Ts'o
Eryu Guan
Eric Sandeen
Kamalesh Babulal
Zhao Lei
Eric Sandeen
Pavel Shilovsky
Lukas Czerner