Both these files, are, after all, define files, like other CONFIGURE_DEFINE_FILES.
They only happen to have a special requirement for an expansion for all defines,
which doesn't need to happen through traditional preprocessing.
This change adds consistency in how configure-related headers are being handled.
This change allows specifying objdir-relative paths in EXPORTS to enable
exporting entries from GENERATED_FILES. Objdir paths in EXPORTS that are
not in GENERATED_FILES will raise an exception.
Example:
```
EXPORTS += ['!g.h', 'f.h']
GENERATED_FILES += ['g.h']
```
Given the implementation, this should also work for FINAL_TARGET_FILES,
FINAL_TARGET_PP_FILES, and TESTING_FILES, but those are not well-tested.
This patch also renames the install manifest for '_tests' to match the
directory name for convenience in some code I refactored.
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
The only use of BRANDING_FILES[...].source is in xulrunner/app/moz.build, for
the app.ico file.
This file has not been useful since the removal of the xpinstall-based
installer in bug 344236... 9 years ago.
Currently mach treats the first argument to eslint as the path and moves it to
the end of the arguments but this breaks usage like "mach eslint -f json browser".
It used to be necessary to change to the directory you wanted to lint but now
the .eslintignore is at the top level we just run from the top level. This means
the path argument doesn't need to be special anymore.
The current implementation of HierarchicalStringList allows the following:
FOO.bar = [
'foo',
'bar',
]
while
FOO.bar += [
'foo',
'bar',
]
would be invalid because of the StrictOrderingOnAppendList enforcement.
It also allows to overwrite the entire list with a subsequent
FOO.bar = [
'baz',
]
while we've explicitly forbidden such things for every other list.
While in the vicinity, fix HierarchicalStringList._get_export_variable to not
call the HierarchicalStringList constructor uselessly.
This will allow a new kind of special variable where it is possible to do
FOO += ['bar']
All the current special variables are either strings (for which __setitem__ would
be called with a different string object), or a read-only dict (which doesn't
allow modifications).
We have many unit tests in the tree for some small parts of the build system
pipeline, but we don't have anything that resembles an end to end test, and we
kind of rely on the resulting Firefox not being broken by our changes.
With the Faster make backend growing, I want to ensure it produces the same
thing as the recursive make backend, at least for the parts it supports.
This adds some kind of test that allows to check that.
The test I'm about to add doesn't have XPIDL files, and that currently avoids
the FasterMake backend to run properly. Also, in the future, when the FasterMake
grows the ability to build C++ files, it should be possible to build Spidermonkey
with the FasterMake backend, but it doesn't have XPIDL files either.
This makes it clearer that really it's the same thing as FINAL_TARGET,
with preprocessing.
We still keep DIST_FILES in backend.mk because it's shorter and doesn't
really matter.
This new ChromeManifestEntry object type is generic and can hold any kind of
chrome manifest entry, but we currently only emit them for binary components.
References to sub-directory manifests is left to the backend, for now, until
all manifest entries are emitted by the frontend.
Ideally, we should properly make and shell quote everything we print out
in makefiles, but that's a can of worms I don't want to open just yet. So
I'll limit myself to just passthru variables.
This further improves the changes from bug 1224460 to e.g. handle variable
references mixed with text, and to avoid adding empty strings to the
resulting flags variables when the expansion leads to an empty string.
Pymake's clinetoargv is very specific to pymake's use case, yet has been abused
as a replacement for shlex because shlex doesn't handle things properly for our
use cases.
Using pymake's clinetoargv, however, has shortcomings, and we're better off
importing its code in mozbuild, simplifying it a little, and using that
instead.
Plus, less dependencies on pymake will help kill it for good some day.
FlatFormatter, JarFormatter and OmniJarFormatter all, in some way, deal
with different pieces of the package being handled differently.
Instead of each of them dealing with their different pieces in some subtly
different way, instead, introduce a new base package formatter class that
will handle it for all of them.
Use this new PiecemealFormatter for the FlatFormatter.
Only directories containing chrome manifests are given as base to formatters,
but there can still be files given outside the bases, like, on mac builds,
all files in Content/MacOS, or Content/Info.plist, whereas chrome manifests
are under Content/Resources.
There is a lot of repetition across its various tests, and we're going to add
some more in a subsequent change, so it is desirable to make it a less
repetitive task.
This function was found to be a little slow while profiling due to repeated calls to
mozpath.dirname. This patch speeds up the function replacing dirname with string manipulation
(these paths are already normalized), by caching results on the basis of directory,
and converting from iteration to recursion to increase use of the cache.
This commit speeds up the "install tests" step run as a part of the build and running
tests by ~10% on a fast linux laptop.
Support for displaying docstrings in `mach help` was added relatively
recently. `mach build` was never documented. Let's document it.
There are a gazillion things we could put in the documentation. For now,
mainly focus on targets.
When a make target is generated with FileAvoidWrite, this can cause targets to
get rebuilt perpetually when a prerequisite is updated, because FileAvoidWrite
will leave the target's mtime older than the prerequisite's when the target's
contents are unchanged. To avoid this issue, GENERATED_FILES is modified to
unconditionally update its target's mtime.
Bug 118468 landed an option for FileAvoidWrite to always write to an output
file, whether or not the contents would be changed. This was to address a
problem caused by not updating mtimes when building GENERATED_FILES, but
undoes the purpose of FileAvoidWrite and isn't really necessary.
This is addressed in a subsequent commit by unconditionally updating
mtimes when processing GENERATED_FILES.
Calling CommonBackend.consume_object ensures that we process WebIDL and
IPDL files (and many other things) correctly. Calling
CommonBackend.consume_finished ensures that the CompileDB backend gets
to see the unified bindings and protocol files that we generate, and add
those files to the compilation database.
The only thing we need the obj for here is getting the objdir. Future
patches will just have an objdir when calling this function, and not a
proper mozbuild object. In light of these facts, let's change the
function to accept an objdir only, which will make those future patches
easier.
For GENERATED_FILES scripts that want to report dependencies, this
change makes it easy to use |preprocess|, rather than having to
construct and use |Preprocessor| manually.
In addition to their inputs declared in moz.build files, generated files
may also depend on other files, such as #includes in preprocessed files.
Let's provide a place for file_generate.py to write out those extra
dependencies.
Indicating a jar currently looks like the following in a jar manifest:
path/to/name.jar:
The `path/to` doesn't contain the implicit "chrome/" directory. This, in
turn, doesn't allow much flexibility to use the jar maker for what is not
necessarily under chrome/.
To use the jar maker to fill some chrome manifest for the default theme
extension, we currently use a hackish path to get to the right location,
and rely on the chrome.manifest file in the parent directory never to be
picked by the package manifest, which is a quite horrible way to do this,
but worked well enough for that specific use case.
With the need to handle system addons at the build system level, it
becomes necessary to come up with something less hackish.
What this change introduces is an additional syntax for the jar manifest,
in the following form:
[base/path] sub/path/to/name.jar:
Using this syntax, there is no implicit 'chrome' path. The `base/path` is
relative to the current DIST_SUBDIR, and the `sub/path` is relative to that
`base/path`. The distinction can be useful for build system backends.
The assumption that the "root" chrome.manifest is in the parent directory
of the implicit "chrome" directory dies, and the `base/path` is where the
root chrome.manifest is placed.
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
CLOSED TREE makes big refactorings like this a piece of cake.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
The configure option has explicitly thrown an error for more than a year now,
and it happens that the remaining way to still forcefully use it has been
broken for more than 8 months.
DONTBUILD NPOTB
This downloads to a temporary file named uniquely but consistently
based on the URL, and then extracts a build ID using mozversion to use
as a human readable and sortable prefix. This approach can be re-used
by |mach artifact| based Desktop builds.
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
It was added back in
5147d5c69f
for unclear reasons (and the lack of bug number doesn't help), and hasn't been
used, as far as I can see in the gecko-dev history, other than in bug 206029,
which is the only use currently in the tree.
Bug 206029 was working around the Flash player installer modifying Firefox's
prefs file and not dealing with it properly or something depending on the line
endings. 11 years later, all prefs files except channel-prefs.js are in
omni.ja, so obviously, bug 206029 doesn't actually apply anymore.
So, let's simplify it all and get rid of this.
Compressing C++ unit tests is a long pole when writing test archives.
Experimenting with various levels of compression revealed that
compression level 9 was providing minimal space savings for
significantly longer archiving times and greater CPU usage.
Results of our experimentation of `make -sj8 package-tests` on OS X
with various levels of compression are below. Note: these numbers were
accidentally obtained without JS tests being archived. This skews the
results a little but doesn't impact the analysis below.
ARCHIVE SIZE WALL CPU
(L=9)
cppunittest 76,806,629 30.6s
mochitest 61,276,928 9.4s
reftest 31,204,396 11.0s
ALL 228,146,761 31.2s 75.9s
(L=8)
cppunittest 76,851,593 24.1s
mochitest 61,279,322 8.9s
reftest 31,207,867 10.4s
ALL 228,228,096 24.9s 64.7s
(L=7)
cppunittest 77,102,292 14.3s
mochitest 61,305,147 8.2s
reftest 31,260,359 9.4s
ALL 228,717,803 15.0s 49.1s
(L=6)
cppunittest 77,321,408 11.5s
mochitest 61,336,539 8.2s
reftest 31,303,604 9.2s
ALL 229,123,307 12.2s 44.7s
(L=5)
cppunittest 78,226,404 8.2s
mochitest 61,483,804 7.6s
reftest 31,509,349 8.8s
ALL 230,725,600 9.6s 39.7s
(L=4)
cppunittest 79,733,669 6.3s
mochitest 61,825,519 7.6s
reftest 31,924,171 8.4s
ALL 233,669,991 9.0s 36.4s
(L=3)
cppunittest 82,380,731 5.8s
mochitest 62,554,431 7.1s
reftest 32,696,415 8.1s
ALL 239,180,168 8.9s 34.6s
Levels lower than 3 resulted in larger archives with no decreae in
wall time and marginal decrease in CPU time.
As we can see, lowering the compression level reduces archiving time by
>3x while only increasing total archive size by ~2.5 MB or ~1% for
compression level 5.
Total time hits a plateau around levels 4 and 5. After that, file size
increases faster for little decrease in wall time. I suspect that we're
hitting Python limits from having to process thousands of files: there's
only so fast Python can do I/O and make function calls.
I think choosing 4 or 5 for the new compression level are acceptable.
I went with 5 because the wall time savings from 5 to 4 are marginal and
the archive size does start to increase a bit faster at 4. That being
said, 4 does consume 10% less CPU. I could easily just 4 as well. 5 is
more conservative. We can always change to 4 after seeing results in the
wild.
The end result of this change is `make package-tests` is much faster:
Before: 228,146,761 bytes; 31.2s wall; 75.9s CPU
After: 230,725,600 bytes; 11.4s wall; 45.0s CPU
Delta: +2,578,839 bytes; -19.8s wall; -30.9s CPU
When you take the whole series into consideration:
Before: 44.2s wall; 84.6s CPU
After: 11.4s wall; 45.0s CPU
Lowering CPU is impressive considering we switched from the C `zip`
implementation to Python!
Keep in mind we were at ~78s wall before e87b74b3db43 introduced
concurrent archive generation!
And we still haven't eliminated the staging of JS tests, which are
several thousand files and a few dozen MB!
