Bug 1153566 changed the regexp used in that method in such a way that there
has been a big hit in the time spent executing the make backend. On my machine,
with the current code, mach build-backend indicates:
Backend executed in 5.01s
With the change from bug 1153566 reverted:
Backend executed in 2.97s
That's a significant regression for a 4-character change.
But we can actually avoid using regexp in most cases, which can make things faster
than they were. With this change, we get down to:
Backend executed in 2.28s
For reference, making the _check_blacklisted_variables method do nothing at all
ends with:
Backend executed in 2.20s
This just allows a little versatility for consumers such as b2gdroid,
which are Fennec-like but don't have MOZ_APP_NAME=fennec.
I elected to pass appname as a parameter rather than modify the
existing distdir because I expect to want to differentiate, in some
way, the output AAR files based on the underlying name. That is, in
future we might generate geckoview-fennec-VERSION.aar and
geckolibs-b2gdroid-VERSION.aar, or stuff the name into the Ivy version
information, or...
This also fixes a typo in one of the JarFinder instantiations.
Move Python code into an xpidl subdirectory, and include a setup.py to allow
inclusion from pip install or requirements files. Change build directory
variables appropriately.
I considered three ways to do this:
* one, as a Python script executed with $(shell);
* two, as a Python script that writes an include file for the preprocessor;
* three, as a function exposed to the moz.build sandbox.
I rejected two because it's both tied to the preprocessor, and awkward
to make handle the dependency on the buildid (in a file) and
additional build defines (in config.status).
I rejected three because I know of no precedent for this approach, and
it hides the dependency on the buildid.
One doesn't handle failures in the script gracefully, but neither did
the existing approach. This patch is at least testable.
DONTBUILD NPOTB
This was just an oversight during the initial landing, leading to two
copies of artifact libraries being appended to the same destination
file.
This implements a new "scan" mode for DMD that records the address
and contents of every live unsampled block in the DMD log. This
enables the low-level analysis of references from one block to
another, which can help leak investigations.
bf34d16b6ab2 added absolute_import to this file. When changed, "import
mozinfo" stopped picking up mozbuild.mozinfo and started importing
mozinfo instead.
Use relative imports to force mozbuild.mozinfo to be picked up.
This import brought in significant parts of the mozbuild package. Moving
it to a deferred import reduces the total number of Python modules
imported during mach dispatch by 43.
This removes ambiguity as to which modules are being imported, making
import slightly faster as Python doesn't need to test so many
directories for file presence.
All files should already be using absolute imports because mach command
modules aren't imported to the package they belong to: they instead
belong to the "mach" package. So relative imports shouldn't have been
used.
DONTBUILD ON A CLOSED TREE: Android-only and the build changes are cosmetic.
Very much a first cut, but I'd like to get some Fennec early adopters testing.
This adds:
* |mach artifact install| to fetch and install Fennec binaries;
* |mach artifact last| to print details about what was last installed;
* |mach artifact {print,clear}-caches|, for debugging.
This code is exposed as a new mozbuild.artifacts Python but it's not
particularly general. My intention was to get things out of the mach command
more than produce a general artifact fetching API. We can leave that bike
shed to Bug 1124378.
I've been testing this with --disable-compile-environment and it works well
locally, although there's no reason a knowledgeable developer couldn't use
this in conjunction with a fully-built tree. (I don't know when such a
situation would arise, but I know of no technical impediment.)
We want this logic to live next to where metadata types are defined so
downstream consumers of Files-based metadata don't have to reinvent the
wheel.
The work in this commit will be used to enable auto filing bugs during
pushes to MozReview.
`mach help <command>` currently only displays a brief description of the
command along with its arguments. Sometimes more detailed help text is
needed.
With this commit, the docstrings of mach command handlers will appear in
the output of `mach help <command>` if they are defined.
I've implemented basic docstrings for the three flavors of mach commands
(normal command, main subcommand, subcommand) to demonstate things work.
My hope is others will start to fill in docstrings once this feature
lands so the output for `mach help` can serve as a better learning guide
for new contributors.
The hglib Mercurial finder was nice. But it is somewhat slow, as it
involves a separate Mercurial process.
This commit introduces a native Mercurial finder that speaks directly
to a Mercurial repository instance. It is significantly faster.
The new code is isolated to its own file because it imports Mercurial
code, which is GPL.
This commit adds support for specifying a Mercurial revision with `mach
file-info`. Aside from being a potentially useful feature, it proves
that MercurialRevisionFinder works with BuildReader.
The moz.build reader uses absolute paths when referencing moz.build
files. *Finder classes expect paths to be relative to a base. When we
switched the reader to use *Finder instances for I/O, we simply provided
a default FileFinder based at the filesystem root. This was quick and
easy. Things worked.
Unfortunately, that solution isn't ideal because not all *Finder
instances can accept absolute paths like that. The
MercurialRevisionFinder is one of them.
Changing the moz.build reader to talk in terms of relative paths is a
lot of work. While this would be ideal, it is significantly easier to
defer the problem until later and hack up MercurialRevisionFinder to
accept absolute paths. This commit does exactly that.
Bug 1171069 has been filed to track converting BuildReader to relative
paths.
Now that moz.build files use finders for I/O, we can start reading
moz.build data from other sources.
Version control is essentially a filesystem. We implement a finder
that speaks to Mercurial to obtain file data. It is able to obtain
file data from a specific revision in the repository.
We use the hglib package (which uses the Mercurial command server) for
speaking with Mercurial. This adds overhead compared to consuming the
raw Mercurial APIs. However, it also avoids GPL side-effects of
importing Mercurial's Python modules.
Testing shows that performance is good but not great. A follow-up
commit will introduce a GPL licensed Mercurial finder. For now, get
the base functionality in place.
Sometimes moz.build data may not come from the local filesystem. To
support defining moz.build data in other backing stores, we switch
moz.build reading to use mozpack's *Finder classes for I/O.
The use of a FileFinder bound to / is sub-optimal. We should ideally
be creating a finder bound to topsrcdir. However, doing this would
require refactoring a lot of path handling in the reader, as that code
makes many assumptions that paths are absolute. This would be excellent
follow-up work.
While I was here, I cleaned up some linter warnings for unused imports.
On my machine, this commit results in a slight slowdown of moz.build
reading. Before, `mach build-backend` was consistently reporting 1.99s
for reading 2,572 moz.build files. After, it is consistently reporting
2.07s, an increase of 4%. This is likely due to a) function call
overhead b) the cost of instantiating a few thousand File instances
c) FileFinder performing an os.path.exists() before returning File
instances. I believe the regression is tolerable.
Passing raw file handles around is a bit dangerous because it could lead
to leaking file descriptors. Add a read() method that handles the simple
case of obtaining the full contents of a File instance.
This should ideally be a method on BaseFile. But this would require
extra work and isn't needed. So we've deferred it until bug 1170329.
Today, the *Finder classes are optimized for doing matching and
returning multiple results. However, sometimes only looking for a
single file is wanted.
This commit implements the "get" method on all the *Finder classes.
It returns a BaseFile or None.
FileFinder._find_file has been refactored into FileFinder.get
to avoid code duplication.
