This adds a generic android Python module that handles:
* downloading and unpacking Google's Android SDK and NDK bundles;
* using the |android| tool to install additional Android packages;
* printing a mozconfig snippet suitable for mobile/android builds.
--HG--
extra : rebase_source : abaffcee70eacb391f53d622fba4a445277818db
This lays the foundation for adding support for bootstrapping
mobile/android.
* Queries for which application to bootstrap, currently browser or
mobile/android;
* Adds call to install_APPLICATION_packages after
install_system_packages;
* Adds call to suggest_APPLICATION_mozconfig after bootstrapping
everything;
* and splits install_browser_packages out of install_system_packages
throughout (essentially untested, but generally simple).
To implement a new application (b2g?), just add it to the list of
applications and implement install_b2g_packages throughout.
--HG--
extra : rebase_source : 8c9b4485dbe5ceb64436fd73ea0258592a2b818f
Various bits of the test harnesses key off of mozinfo.info.get('asan');
we will need a similar switch for finding out whether this build
supports tsan.
Now that the mozbuild backend knows about FINAL_TARGET, we are able to
install generated xpt files into their final location. This saves us
from copying xpt files into their final location on every build.
Original patch by gps, rebased and comments addressed by Ms2ger
--HG--
extra : transplant_source : %E2%DC%0F%E0%AD%C2%25%A1%B8%A9%FE%B0%8C%60%FF%CB%02G%25%E5
mopack.BaseFile.copy() performs a generic read/write file copy. Windows
has an explicit CopyFile() call that tests have shown to be
significantly faster. Let's use that instead via the magic of ctypes.
Having SOURCES and its close relatives go through VariablePassthru
objects clutters the handling of VariablePassthru in build backends and
makes it less obvious how to handle things that actually get compiled.
Therefore, this patch introduces four new moz.build objects
corresponding to the major variants of SOURCES. It looks like a large
patch, but there's an ample amount of new tests included, which accounts
for about half of the changes.
An upcoming patch will reuse this code. Split it out into its own
function.
--HG--
extra : rebase_source : 2057266a1ed4d86b1ca857712cd279bdfbd32148
extra : histedit_source : 8f2b7ec06e5b72ab08586902c51c3025a53259e6
With this patch, we now have the ability to declare mach sub-commands.
Subsequent patches will implement dispatching for sub-commands.
--HG--
extra : rebase_source : 32a672ff3ba7a1a1c8c27dff3fe657dec08e3569
extra : histedit_source : 2bb2c27ae0e2dab052a6ef93b0a70f488f8cd56e
We will soon have proper sub-commands. Let's fix function names.
--HG--
extra : rebase_source : 1cb0c008ecc0d2159dd799e76adad4892d65f119
extra : histedit_source : 9472b2d2609224b011eca04385cf9d5a49986864
mach conditions are implemented elsewhere. This comment can die.
--HG--
extra : rebase_source : c956d5af7858d997a1f49f3b24bb54e5f8a52240
extra : histedit_source : 76b8155b12ab0ea86e16e807b1c6f691cb8f3640
Documentation was referencing sub commands. The proper terminology is
"command."
--HG--
extra : rebase_source : aa2fc73c6141850109324fd5299491e49ef29e91
extra : histedit_source : da075bea5a36a9de81527d93a9fc751cd6cae9b2
There were a few parts of the docs that were linking to Python types
incorrectly. Let's fix that.
--HG--
extra : rebase_source : 374be660084727eba1ce9181c2abf384f40eeb82
extra : histedit_source : bfa8de9b7a277666d92ebe71cd64ffed96628518
The main mach docs page is a bit long. Let's split it into multiple
articles to increase readability going forward.
--HG--
rename : python/mach/docs/index.rst => python/mach/docs/commands.rst
rename : python/mach/docs/index.rst => python/mach/docs/driver.rst
rename : python/mach/docs/index.rst => python/mach/docs/logging.rst
extra : rebase_source : 484d60327568333fcb0069e1f3444dc6db4322c0
extra : histedit_source : 18d09ac2e2e93565661763b6567f7a46226735f5
Now that mach's docs are part of a proper Sphinx install, link to
Python type docs. While we're here, we also clean up formatting of code
samples.
--HG--
extra : rebase_source : 974337316dc87c04aca79a9e4a0d464490e5cdd4
extra : histedit_source : db87e8e5cb7c9c1969fd89e58c39623862684a6a
mach has restructured text documentation. Let's expose it to the in-tree
Sphinx config so we can increase visibility of the documentation.
--HG--
rename : python/mach/README.rst => python/mach/docs/index.rst
extra : rebase_source : 68a08bebaa5b99ca39e250fa1ed60ca4d7d0af68
extra : histedit_source : 6afda2fece486547192c1c8eba97a2474e3d99c1
Now that defining $DMD is no longer necessary to run DMD, this patch does the
following.
- Removes all the places where we set DMD=1 (test harnesses, etc.)
- Still handles DMD=1, for backwards compatibility.
- Prints "$DMD is undefined" at DMD start-up if appropriate.
- Writes a |null| value for |dmdEnvVar| in the JSON if $DMD is undefined. Bumps
the DMD output version number accordingly.
- Changes a bunch of the test files accordingly, including changing the mode of
script-ignore-alloc-fns.json in order to test a case where $DMD is undefined.
--HG--
extra : rebase_source : eb1ef5722410734ce6d7658465ff6f442ee4ed49
Various os.path attributes are being used in tight loops. Having local
variables prevents extra dictionary lookups.
This appears to shave 10-20ms off of the tests install manifest
processing time.
--HG--
extra : rebase_source : de941f2978cf0b1fd7c4f7401c848b61d406a2c8
extra : amend_source : e33c896856fa559197496b8227e10ab8149d146e
FileCopier.copy() was performing a lot of os.path.normpath() operations.
Profiling revealed that os.path.normpath() was the function with the
most wall time CPU usage when processing the tests manifests. Upon
subsequent examination of the code in question, all the paths being used
were already normalized. So, os.path.normpath() wasn't accomplishing
anything.
This patch results in ~300ms reduction in wall time to process the tests
install manifest on a fully populated page cache. Execution time drops
from ~2.8s to ~2.5s.
Profiling reveals that after this patch os.stat() is the #1 wall time
consumer. However, os.path.{join,dirname,normpath} still account for
~1.5x the wall time of os.stat(). There is still room to optimize
this function.
--HG--
extra : rebase_source : b6f0862baa5168c609499fd95eb3517854bc8cce
extra : amend_source : 7e04c1eb74132bbbe86e721f0f209b19309a7a51
This patch moves profiling mode selection from post-processing (in dmd.py) to
DMD start-up. This will make it easier to add new kinds of profiling, such as
cumulative heap profiling.
Specifically:
- There's a new --mode option. |LiveWithReports| is the default, as it is
currently.
- dmd.py's --ignore-reports option is gone.
- There's a new |mode| field in the JSON output.
- Reports-related operations are now no-ops if DMD isn't in LiveWithReports
mode.
- Diffs are only allowed for output files that have the same mode.
- A new function ResetEverything() replaces the SetSampleBelowSize() and
ClearBlocks(), which were used by the test to change DMD options.
- The tests in SmokeDMD.cpp are split up so they can be run multiple times, in
different modes. The exact combinations of tests and modes has been changed a
bit.
--HG--
rename : memory/replace/dmd/test/full-reports-empty-expected.txt => memory/replace/dmd/test/full-empty-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-heap-empty-expected.txt => memory/replace/dmd/test/full-empty-live-expected.txt
rename : memory/replace/dmd/test/full-heap-sampled-expected.txt => memory/replace/dmd/test/full-sampled-live-expected.txt
rename : memory/replace/dmd/test/full-reports-unsampled1-expected.txt => memory/replace/dmd/test/full-unsampled1-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-heap-unsampled1-expected.txt => memory/replace/dmd/test/full-unsampled1-live-expected.txt
rename : memory/replace/dmd/test/full-reports-unsampled2-expected.txt => memory/replace/dmd/test/full-unsampled2-dark-matter-expected.txt
rename : memory/replace/dmd/test/script-diff-basic-expected.txt => memory/replace/dmd/test/script-diff-live-expected.txt
rename : memory/replace/dmd/test/script-diff1.json => memory/replace/dmd/test/script-diff-live1.json
rename : memory/replace/dmd/test/script-diff2.json => memory/replace/dmd/test/script-diff-live2.json
extra : rebase_source : bf32cc4e0d82aa1a20ceb55e8ea259850b49cc06
People often seek to learn how mach commands work. A common way to do
this is to launch a debugger and step through the code as it is
executing. But this requires someone to first find and modify the mach
command. This involves overhead.
This patch adds a global --debug-command argument to mach. When present,
we launch an interactive debugger right before command dispatch. This
allows people to easily enter a debugger to see what mach commands are
doing, hopefully lowering the barrier to understanding and contributing.
--HG--
extra : rebase_source : 5ebfa7f5fd89f96dac5f7cb035ab520c63b32e55
extra : amend_source : c53c988763c328020b49aa5c6245df685b6f30bc
This is a straight copy from
a878bf0ba0
paired with a tiny change to use the new quote_chars option.
--HG--
extra : rebase_source : 75d604ffafc7062c663bca4242af35546d2c1e3a
buildlist invocations are slow and can occur in parallel since the
underlying program obtains a lock on the modified file.
Moving the XPT-related buildlist invocation from the serial libs tier to
the parallel misc tier decreased my no-op build time on OS X from 43.5s
to 37.0s.
--HG--
extra : rebase_source : 7d274024c401b1ecfbc771424a69eb487808fcbf
When the misc tier was added, only directories with misc-associated
variables from moz.build were traversed. This patch adds a dummy
variable to moz.build whose presence will add the directory to the misc
tier.
This will enable us to aggressively convert existing libs:: rules
to the misc tier.
--HG--
extra : rebase_source : c5477010c7d87fafa512f4fbbf1d2b995e07ffbd
extra : amend_source : a54eee6254ef2d63ef6b247d45e38eda2ab88fef
extra : histedit_source : c0f3352e84ec18ca7b88571cd5f9c2a33ab8eda9
JS module installation performs simple file copying or preprocessing.
There is no reason it can't occur in parallel. Move it to the misc tier.
As part of this, I recognized that TESTING_JS_MODULES was assigned to a
tier. Since these files are managed by an install manifest, they don't
belong to any tier. So the tier is now listed as None.
--HG--
extra : rebase_source : 963f0813e43802c017837ce9d55f8e666decd76a
The build system being what it currently is, there are various cases where one
wants something explicit, rather than the current autodetection.
For instance, one may want to run
make -C $objdir chrome
instead of
make -C $objdir/chrome
that mach build chrome currently invokes.
There are several such usecases that mach's autodetection makes harder, and
it's sometimes awkward when telling people, to debug their issues, to run
make -C objdir something
and hear back that objdir doesn't exist or something along those lines,
because they took "objdir" literally.
There are, sadly, many combinations of linkage in use throughout the tree.
The main differentiator, though, is between program/libraries related to
Gecko or not. Kind of. Some need mozglue, some don't. Some need dependent
linkage, some standalone.
Anyways, these new templates remove the need to manually define the
right dependencies against xpcomglue, nspr, mozalloc and mozglue
in most cases.
Places that build programs and were resetting MOZ_GLUE_PROGRAM_LDFLAGS
or that build libraries and were resetting MOZ_GLUE_LDFLAGS can now
just not use those Gecko-specific templates.
It's not entirely clear passing down all the metadata makes sense. On the
other hand, when creating the template execution sandbox, passing down
exports does assign the value for the exported variable in that execution
context. When that context is merged with the caller sandbox context, the
exported variable is reassigned, even if the value is not modified. Then,
if the caller sandbox itself reassigns the exported variable, it fails
because calling a template already did it once, unexpectedly.
Not passing down exported variables makes the template execution sandbox
never set those exported variables, so that they are not merged back. The
caller sandbox can then properly reassign the exported variable.
The in-tree Sphinx docs have been broken since bug 1041941 because
processing moz.build files outside their context doesn't work.
Specifically, templates aren't loaded (because this information usually
comes from a parent moz.build file). A new execution mode is needed.
I tried to implement a proper execution mode. However, I kept running
into walls. While we should strive for a proper execution mode, this can
be a follow-up, tracked in bug 1058359.
This patch implements extraction of Sphinx variables from ast walking.
It is extremely low-level and definitely a one-off. But it solves the
problem at hand: |mach build-docs| will work after this patch is
applied.
--HG--
extra : rebase_source : abd0a91a3efb24d3adfa19f4cd281ce5fd6d0915
extra : amend_source : c1b4f79224bab55e65a8c2b0f3103475281416c1
The reason to use '+' prefixing was to distinguish between options to the
mach command itself, and options that are passed down to whatever the
command does (like mach run passing down args to the built application).
That makes things unnecessarily awkward, and quite non-standard.
Instead, use standard '-' prefixing, and pass all the unknown arguments
down. If there is overlap between the known arguments and arguments supported
by the underlying tool (like -remote when using mach run), it is possible to
use '--' to mark all following arguments as being targetted at the underlying
tool.
For instance:
mach run -- -remote something
would run
firefox -remote something
while
mach run -remote something
would run
firefox something
As allow_all_arguments is redundant with the presence of a argparse.REMAINDER
CommandArgument, allow_all_arguments is removed. The only mach command with a
argparse.REMAINDER CommandArgument without allow_all_arguments was "mach dmd",
and it did so because it didn't want to use '+' prefixes.
These manifests are special in that they don't package their test files
into the test package. Each test listed in an instrumentation manifest
serves as an identifier rather than a file.
--HG--
extra : rebase_source : 0321528a2dc380e57b824746efbcf61d295204a9
Up to now, DIRS and TEST_DIRS were dumb values. This change makes them
a list of ContextDerivedValues, and handles the fact that some types of
paths are relative to the current source directory and others to the
topsrcdir.
This also makes us one step closer to fixing bug 991983.
After bug 762358 mk_add_options MOZ_MAKE_FLAGS was simply ignored in client.mk
processing. At the same time, mach environment was expecting a list of options
while the mozconfig reader returned a single string, so straighten this up at
the same time.
Having to walk over elements and strings of HierarchicalStringList with
an external recursive function is un-Pythonic and adds unnecessary
obfuscation to several tasks. Add a walk() function to
HierarchicalStringList, modeled on os.walk(), to handle these cases more
directly.
The forward slash appears to be the standard path separator in zip/JAR
files. Accept back slashes when adding paths to a JAR.
--HG--
extra : rebase_source : bd94eab36b347006e65952d99b53dd397e2ca758
extra : amend_source : 2cefd887d8bb5d989fafb398a3464429ac376e2e
The install manifest processor starts with an empty InstallManifest and
uses |= to "concatenate" instances. It became pretty obvious when
developing some patches that add more preprocessed files to install
manifests that the source install manifest dependency was getting
lost during the |= operation. This patch fixes it.
The solution is not ideal performance wise. But slightly worse
performance (only after config.status, however) is better than
clobbers.
A test has been added to ensure this doesn't regress.
--HG--
extra : rebase_source : 848aebbbc935ce2bca2d3fcc85d1df534e734e0d
- parent and var were used for TOOL_DIRS/TEST_TOOL_DIRS, which were removed in
bug 1043802.
- tier metadata was not really used, and tiers are going to go away soon
anyways.
As handled in configure.in, EXTERNAL_SOURCE_DIR is a single directory for
external sources, and more specifically, comm-central. There is no need
to handle it as a list in the moz.build emitter.
At the same time, avoid checking the external sources for every moz.build,
and bind it to a config environment.
As a special exception, we change List's + and += operators to treat
"+ None" and "+= None" as "+ []" and "+= []" respectively.
This is a hack to make moz.build files simpler so they don't have to
perform "is x" checks before appending x.
While I was here, I fixed the implementation of List.__add__ to return a
List instead of list.
--HG--
extra : rebase_source : e61db4be4cf8144b092609d3baf84c372b9cf9a0
extra : amend_source : dfabc00765582764426f7cf74e79354a0bfb0824
This change merges mozilla-central and comm-central into having a single
topobjdir file but retaining two topsrcdirs (and two build systems). This state
is hopefully only the first part of a series of changes that eliminate the
comm-central build system partial clone completely.
--HG--
extra : rebase_source : 02aa2c4551df405d9783ac85cc41fe90b67bf057
extra : amend_source : d3cc677d59603648165bf65afa28413f2c40b2fd
This patch add a line at the end of the build like follows:
ccache (direct) hit rate: 98.7%; (preprocessed) hit rate: 0.5%; miss rate: 0.8%
--HG--
extra : rebase_source : 9c814745516d7b8b0db2056a1fa0df540a5a239a
This patch makes EXTRA_{PP_,}JS_MODULES similar in functionality to
TESTING_JS_MODULES: we indicate the path relative to
$(FINAL_TARGET)/modules with an appropriate hierarchy of paths.
We're going to build on top of INSTALL_TARGETS for the next patch, and
it's easiest to do so if we can look at the 'directories' in the
hierarchy, rather than the individual strings.
I am slightly surprised that we haven't needed this before, but the
motivating idea behind this is wanting to do:
VARIABLE.with.some["non-pythonic-identifier"] += ...
It seems unfortunate that the behavior is completely overridden by
subclasses, but I don't see a good way around that.
Straight assignments to HierarchicalStringList, which we'll change
EXTRA_JS_MODULES to in a later patch, don't work. This change, in
addition to making things work as expected later on, also is more
consistent with existing practice.
While most environments are using the root topobjdir, some can be using a
different path. This happens for comm-central files. For those, the compile
target needs to have paths relative to the root topobjdir instead of relative
to the comm-central topobjdir.
