With this chunking strategy, the runtimes of tests are taken into account, such that each chunk
takes roughly the same amount of time to finish. Tests belonging to the same manifest will not get
split up.
The algorithm works by sorting every manifest from slowest to fastest. Each manifest is popped off
and its tests are added to the fastest chunk to date until no manifests are left. Total runtimes of
the chunks are re-calculated after every addition.
This patch detects when breaking away from the parent job isn't strictly necessary to manage processses in a new job to allow using nested job objects to manage processes and their children. Loss of job object functionality is handled in mozprocess as non-fatal, however mozrunner and others doing things like restarting firefox require it.
A new parameter called 'processStderrLine' is added. When specified, stdout gets processed by the
'processOutputLine' callbacks and stderr is processed by the 'processStderrLine' callbacks. When
not specified, stderr is redirected to stdout which is the same default behaviour.
A side effect of this is that mozprocess now uses three threads to process output. One thread each
for stdout and stderr that reads output lines and stores them in a Queue as fast as possible, this
makes sure there is no blocking in stdout.write(). A third thread executes the callbacks.
A filter is a callable that accepts an iterable of tests and a dictionary of values (e.g mozinfo.info) and returns an iterable of tests. Note filtering can mean modifying tests in addition to removing them. For example, this implements a "timeout-if" tag in the manifest:
from manifestparser import expression
import mozinfo
def timeout_if(tests, values):
for test in tests:
if 'timeout-if' in test:
timeout, condition = test['timeout-if'].split(',', 1)
if expression.parse(condition, **values):
test['timeout'] = timeout
yield test
tests = mp.active_tests(filters=[timeout_if], **mozinfo.info)
--HG--
extra : rebase_source : adead90910811e71e8ea2bb862f2b8e92f2c1bee
A filter is a callable that accepts an iterable of tests and a dictionary of values (e.g mozinfo.info) and returns an iterable of tests. Note filtering can mean modifying tests in addition to removing them. For example, this implements a "timeout-if" tag in the manifest:
from manifestparser import expression
import mozinfo
def timeout_if(tests, values):
for test in tests:
if 'timeout-if' in test:
timeout, condition = test['timeout-if'].split(',', 1)
if expression.parse(condition, **values):
test['timeout'] = timeout
yield test
tests = mp.active_tests(filters=[timeout_if], **mozinfo.info)
--HG--
extra : rebase_source : 7afc5d677717279e477d420899ba839073de2d8f
Simple refactor that moves logic out of manifestparser.py and into cli.py, expression.py and ini.py.
--HG--
extra : rebase_source : dd454973cdb3bcb7ec29dd2e1c0c594e3b3fb817
