The single-process test runner is invoked in a number of different
scenarios, including when multiple test dirs are specified or (afaict)
when lit is used to drive the test suite.
Unfortunately the --test-subdir option did not work with the single
process test runner, breaking an important use case (using lit to run
swift-lldb Linux tests):
Failure URL: https://ci.swift.org/job/swift-PR-Linux/6841
We won't be able to run lldb tests within swift PR testing without
filtering down the set of tests.
This change makes --test-subdir work with the single-process runner.
llvm-svn: 339929
The first issue is about the flaky test rerun logic. This was grouping
tests by subdir and passing them into walk_and_invoke in the incorrect
form. This part can be just deleted as its not needed anymore.
The second problem (which I noticed while investigating the first one)
was that the "-p" switch was not working in multiprocessing mode. This
happened because we were returning None from process_file instead of a
tuple full of empty values for tests that did not match the -p regex.
Both of these would be caught earlier if python was a more strongly
typed language. :/
llvm-svn: 325519
Summary:
Due to in-tree builds, we were parallelizing the tests at the directory
level. Now that the tests are built out-of-tree, we can remove this
limitation and paralelize at file level instead.
This decreases test suite time by about 10% for me, which is not
world-shattering, but it makes the code slightly simpler and will also
allow us to merge tests which were artificially spread over multiple
folders (TestConcurrentEvents...) to work-around this limitation.
To make this work, I've also needed to include the test file name in the
build directory name, as just the test method name is not unique enough
(plenty of tests have a test method called "test" or similar).
While doing this, I've found a couple of tests that are taking waaay longer then
they ought to (TestBreakpointCaseSensitivity -- 90 seconds), which I plan to
look into in the future.
Reviewers: aprantl
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D43335
llvm-svn: 325322
Summary:
This has been broken at least since the new test result framework was
added, which was over a year ago. It looks like nobody has missed it
since.
Removing this makes the gmodules handling code saner, as it already did
not know how to handle the multiple-compilers case.
My motivation for this is libc++ data formatters support on android -- I
am trying make a central way of determining whether libc++ tests can be
run, and without this, I would have to resort to similar hacks as the
gmodules code.
Reviewers: jingham, zturner
Subscribers: danalbert, tfiala, lldb-commits
Differential Revision: https://reviews.llvm.org/D30779
llvm-svn: 297811
This change addresses the corner case bug in the test
infrastructure where a test file times out *outside*
of any running test method. In those cases, the issue
was charged to the file, not to a test method within
the file. When that file is re-run successfully,
none of the test-method-level successes would clear
the file-level issue.
This change fixes that: for all test files that are
getting rerun (whether by being marked flaky or
via the --rerun-all-issues flag), file-level test
issues are searched for in each of those files. Each
file-level issue found in the rerun file list then
gets cleared.
A test of this feature is added to issue_verification,
using the technique there of moving the *.py.park file
to *.py to do an end-to-end validation.
This change also adds a .gitignore entry for pyenv
project-level files and fixes up a few minor pep8
formatting violations in files I touched.
Fixes:
llvm.org/pr27423
llvm-svn: 282990
This is the Linux counterpart to the sampling support I added
on the macOS side.
This change also introduces zip-file compression if the size of
the sample output is greater than 10 KB. The Linux side can be
quite large and the textual content is averaging over a 10x
compression factor on tests that I force to time out. When
compression takes place, the filename becomes:
{session_dir}/{TestFilename.py}-{pid}.sample.zip
This support relies on the linux 'perf' tool. If it isn't
present, the behavior is to ignore pre-kill processing of
the timed out test process.
Note calling the perf tool under the timeout command appears
to nuke the profiled process. This was causing the timeout
kill logic to fail due to the process having disappeared.
I modified the kill logic to catch the case of the process
not existing, and I have it ignore the kill request in that
case. Any other exception is still raised.
Reviewers: labath
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D24890
llvm-svn: 282436
*** to conform to clang-format’s LLVM style. This kind of mass change has
*** two obvious implications:
Firstly, merging this particular commit into a downstream fork may be a huge
effort. Alternatively, it may be worth merging all changes up to this commit,
performing the same reformatting operation locally, and then discarding the
merge for this particular commit. The commands used to accomplish this
reformatting were as follows (with current working directory as the root of
the repository):
find . \( -iname "*.c" -or -iname "*.cpp" -or -iname "*.h" -or -iname "*.mm" \) -exec clang-format -i {} +
find . -iname "*.py" -exec autopep8 --in-place --aggressive --aggressive {} + ;
The version of clang-format used was 3.9.0, and autopep8 was 1.2.4.
Secondly, “blame” style tools will generally point to this commit instead of
a meaningful prior commit. There are alternatives available that will attempt
to look through this change and find the appropriate prior commit. YMMV.
llvm-svn: 280751
This reverts r275782.
The problem with the commit is that it reports an additional "exit (1)" error for every file
containing a failing test, which is far more than I had intended to do. I'll need to come up with
a more fine-grained way of achieving the result.
llvm-svn: 275791
Summary:
We've run into this problem when the test errored out so early (because it could not connect to
the remote device), that the code in D20193 did not catch the error. This resulted in the test
suite reporting success with 0 tests being run.
This patch makes sure that any non-zero exit code from the inferior process gets reported as an
error. Basically I expand the concept of "exceptional exits", which was previously being used for
signals to cover these cases as well.
Reviewers: tfiala, zturner
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D22404
llvm-svn: 275782
This change moves all the test event handling and its related
ResultsFormatter classes out of the packages/Python/lldbsuite/test dir
into a packages/Python/lldbsuite/test_event package. Formatters are
moved into a sub-package under that.
I am limiting the scope of this change to just the motion and a few
minor issues caught by a static Python checker (e.g. removing unused
import statements).
This is a pre-step for adding package-level tests to the test event
system. I also intend to simplify test event results formatter selection
after I make sure this doesn't break anybody.
See:
http://reviews.llvm.org/D19288
Reviewed by:
Pavel Labath
llvm-svn: 266885
Also does the following:
* adopts PEP8 naming convention for OptionalWith class (now
optional_with).
* moves test_runner/lldb_utils.py to lldbsuite/support/optional_with.py.
* packages tests in a subpackage of test_runner per recommendations in
http://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/creation.html
Tests can be run from within pacakges/Python/lldbsuite/test via this
command:
python -m unittest discover test_runner
The primary cleanup this allows is avoiding the need to muck with the
PYTHONPATH variable from within the source files. This also aids some
of the static code checkers as they don't need to run code to determine
the proper python path.
llvm-svn: 266710
The race boiled down to this:
If a test worker queue is able to run the test inferior and
clean up before the dosep.py listener socket is spun up, and
the worker queue is the last one (as would be the case when
there's only one test rerunning in the rerun queue), then
the test suite will exit the main loop before having a chance
to process any test events coming from the test inferior or
the worker queue job control.
I found this race to be far more likely on fast hardware.
Our Linux CI is one such example. While it will show
up primarily during meta test events generated by
a worker thread when a test inferior times out or
exits with an exceptional exit (e.g. seg fault), it only
requires that the OS takes longer to hook up the
listener socket than it takes for the final test inferior
and worker thread to shut down.
See:
http://reviews.llvm.org/D19214
reviewed by:
Pavel Labath
llvm-svn: 266624
Summary:
The '-p' option for dotest.py was ignored in multiprocess mode,
as the -p argument to the inferior would overwrite the -p argument
passed on the command line.
Reviewers: zturner, tfiala
Subscribers: lldb-commits, sas
Differential Revision: http://reviews.llvm.org/D18779
Change by Francis Ricci <fjricci@fb.com>
llvm-svn: 265422
TestHelloWorld seems to be passing now as far as I can tell. TestExitDuringStep is still hanging.
I have marked the relevant tests as flaky, which should handle the timeouts now as well. I'll be
monitoring the buildbots for fallout.
llvm-svn: 258114
I'm getting rid of the expected timeouts. I'll XFAIL/skip any tests that show up as failing after
this (I haven't seen any when running locally, but maybe the buildbot will disagree).
llvm-svn: 256827
I think it was timing out because of the attach deadlocks, which are now fixed. In any case, it
has passed last 200 buildbot runs, so I am enabling it.
llvm-svn: 256748
Vedant Kumar
Pavel Labath
Todd Fiala
Kate Stone
Enrico Granata
Zachary Turner
Stephane Sezer