The hglib Mercurial finder was nice. But it is somewhat slow, as it
involves a separate Mercurial process.
This commit introduces a native Mercurial finder that speaks directly
to a Mercurial repository instance. It is significantly faster.
The new code is isolated to its own file because it imports Mercurial
code, which is GPL.
This commit adds support for specifying a Mercurial revision with `mach
file-info`. Aside from being a potentially useful feature, it proves
that MercurialRevisionFinder works with BuildReader.
The moz.build reader uses absolute paths when referencing moz.build
files. *Finder classes expect paths to be relative to a base. When we
switched the reader to use *Finder instances for I/O, we simply provided
a default FileFinder based at the filesystem root. This was quick and
easy. Things worked.
Unfortunately, that solution isn't ideal because not all *Finder
instances can accept absolute paths like that. The
MercurialRevisionFinder is one of them.
Changing the moz.build reader to talk in terms of relative paths is a
lot of work. While this would be ideal, it is significantly easier to
defer the problem until later and hack up MercurialRevisionFinder to
accept absolute paths. This commit does exactly that.
Bug 1171069 has been filed to track converting BuildReader to relative
paths.
Now that moz.build files use finders for I/O, we can start reading
moz.build data from other sources.
Version control is essentially a filesystem. We implement a finder
that speaks to Mercurial to obtain file data. It is able to obtain
file data from a specific revision in the repository.
We use the hglib package (which uses the Mercurial command server) for
speaking with Mercurial. This adds overhead compared to consuming the
raw Mercurial APIs. However, it also avoids GPL side-effects of
importing Mercurial's Python modules.
Testing shows that performance is good but not great. A follow-up
commit will introduce a GPL licensed Mercurial finder. For now, get
the base functionality in place.
Sometimes moz.build data may not come from the local filesystem. To
support defining moz.build data in other backing stores, we switch
moz.build reading to use mozpack's *Finder classes for I/O.
The use of a FileFinder bound to / is sub-optimal. We should ideally
be creating a finder bound to topsrcdir. However, doing this would
require refactoring a lot of path handling in the reader, as that code
makes many assumptions that paths are absolute. This would be excellent
follow-up work.
While I was here, I cleaned up some linter warnings for unused imports.
On my machine, this commit results in a slight slowdown of moz.build
reading. Before, `mach build-backend` was consistently reporting 1.99s
for reading 2,572 moz.build files. After, it is consistently reporting
2.07s, an increase of 4%. This is likely due to a) function call
overhead b) the cost of instantiating a few thousand File instances
c) FileFinder performing an os.path.exists() before returning File
instances. I believe the regression is tolerable.
Passing raw file handles around is a bit dangerous because it could lead
to leaking file descriptors. Add a read() method that handles the simple
case of obtaining the full contents of a File instance.
This should ideally be a method on BaseFile. But this would require
extra work and isn't needed. So we've deferred it until bug 1170329.
Today, the *Finder classes are optimized for doing matching and
returning multiple results. However, sometimes only looking for a
single file is wanted.
This commit implements the "get" method on all the *Finder classes.
It returns a BaseFile or None.
FileFinder._find_file has been refactored into FileFinder.get
to avoid code duplication.
The current mode of execution of templates doesn't allow them to more advanced
control flow, like returning early, returning or yielding values, because that
mode of execution is equivalent to running the code at the top level of a .py
file.
Making the templates executed through a function call, although trickier,
allows those control flows, which will be useful for template as context
managers.
inspect.getsourcelines() and inspect.getfile() involve I/O out of our control.
Our use of those functions, however, doesn't require all their smarts. In fact,
we only use them on function objects, for which we can just do the work
ourselves without involving inspect functions that trigger I/O.
`mach help <command>` currently only displays a brief description of the
command along with its arguments. Sometimes more detailed help text is
needed.
With this commit, the docstrings of mach command handlers will appear in
the output of `mach help <command>` if they are defined.
I've implemented basic docstrings for the three flavors of mach commands
(normal command, main subcommand, subcommand) to demonstate things work.
My hope is others will start to fill in docstrings once this feature
lands so the output for `mach help` can serve as a better learning guide
for new contributors.
The old way of writing scripts for generated files would invoke the script thusly:
python script.py arg1...
Invoking the script this way means that the script's directory is
automatically added to sys.path, and importing modules from that
directory is easy. Let's make it equally easy in the new world for
GENERATED_FILES, too.
Add generic support for different forms of paths in moz.build:
'/topsrcdir/relative/paths'
'srcdir/relative/paths'
'!/topobjdir/relative/paths'
'!objdir/relative/paths'
This drops the use of UserString for performance reasons, which required
going around with a meta class.
This code has been commented since it landed, and enabling it requires
adding the proper conditionals around all non_unified_sources in gyp
configurations, which is a daunting task. OTOH, the commented code is
already outdated (it would need updates to work) and the related code
that is not commented gets in the way of upcoming changes, so remove
it.
This helps upcoming changes, and relieves backends from path resolution.
This has the side effect of chaning the order of some unified sources,
which consequently breaks nsTextFormatter because it declares snprintf
methods after nsStringAPI #defines it.
With TemplateContexts keeping the name of the associated template, and the
Gyp context being declared as a TemplateContext, it is now possible to know
the equivalent of IS_GYP_DIR just by looking at the template name.
Add .rs as a recognized file extension in SOURCES.
Propagate that through to the Makefile backend and add a dependency
generated and an explicit rule to call $(RUSTC) to compile them.
rustc builds static libraries, not obj files. At least, if one
asks it to output an obj file, I'm not clear how to get all the
compiler-specific runtime libraries the code will expect to link
to. Therefore we generate a static library for each rust source
file (which must be a complete crate for the time being) and link
that. Because of the extension it ends up on the LIBS line in the
the corresponding .desc file.
Note that the static library does still depend on some system
libraries, e.g. -ldl -lpthread -lm on linux. Gecko already
links to all of those, so we don't keep track of it here.
Should we need to add explicit linkage for other targets,
rustc does print a list to stderr which can be parsed.
Formerly, running |./mach test image/| would result in running a number of devtools tests
in addition to running tests under the image/ subdirectory. With this change, only tests
under the image/ directory would be run. Note that ./mach test animations or similar will
still run a variety of tests across the tree, because this input does not match a directory.
mach dispatch makes separate, independent calls to construct build
system state. Part of this resolution is determining the object
directory. For environments without an object directory defined, we must
execute config.guess to determine the object directory. This redundant
execution of config.guess can result in significant execution overhead.
Before this patch, `mach help` with no mozconfig took ~1.5s on my OS X
machine. After this patch, it goes down to ~0.750s. On Windows, the
difference is even more pronounced, with execution time dropping from
8.5s to 0.930s.
Now that moz.build can see EXTRA_*COMPONENTS and NO_JS_MANIFEST, we can
move some logic from rules.mk (executed every build) to moz.build's
emitter.py (executed only at build-backend time).
Interactive prompts make automation more difficult for developers looking to stand up machines using bootstrap en masse.
Two new options have been added to allow users with such needs to bypass all prompts: one for selecing an application
(desktop/mobile) and another for assuming yes to all questions posed by package managers (apt-get/yum).
This avoids duplicating the logic from SimplePackager to find base
directories, and fixes some cases where the l10n repack code wouldn't
find them properly.
Bug 910660 added a consistency check that rejects cases where a manifest
inside a directory detected as being the base of an addon is included from
outside the addon. Unfortunately, this triggered false positives when
a manifest is included from within the addon, but just happens to be at
the top-level of that addon.
This avoids duplicating the logic from SimplePackager to find base
directories, and fixes some cases where the l10n repack code wouldn't
find them properly.
Some file types, such as XPTFile, read their associated data during the
copy. In the case of XPTFiles, several original files are linked together
in one destination file. When a FileCopier is used in-place, those
original files are removed before XPTFile.copy is invoked, so XPTFile.copy
fails.
Generally speaking, there are many other reasons why FileCopier can fail
in-place, but the only active use of this mode is l10n repack code, which
actually doesn't do much of the dangerous uses. However, it can end up
linking XPTFiles for some reason, which fails for the reasons given above.
Back when mozpack.path was added, it was used as:
import mozpack.path
mozpack.path.func()
Nowadays, the common idiom is:
import mozpack.path as mozpath
mozpath.func()
because it's shorter.
$ git grep mozpath\\. | wc -l
423
$ git grep mozpack.path\\. | wc -l
123
This change was done with:
$ git grep -l mozpack.path\\. | xargs sed -i 's/mozpack\.path\./mozpath./g'
$ git grep -l 'import mozpack.path$' | xargs sed -i 's/import mozpack.path$/\0 as mozpath/'
$ (pat='import mozpack.path as mozpath'; git grep -l "$pat" | xargs sed -i "1,/$pat/b;/$pat/d")
