This is two straightforward optimizations in FileCopier: avoiding a redundant iteration
over the directory structure to find destination files (which includes a
call to normpath) and avoiding redundant calls to determine directories to preserve
when remove_unaccounted is not specified (which include a call to dirname).
Running a no-op install of _tests with this patch results in a reduction of about
25,000 calls to normpath and remove about 220,000 calls to dirname, resulting in
an overall speedup of 10-20%.
Listing the tests in all-tests.json is prerequisite to adding support
for Marionette tests to `./mach test FILE`.
Marionette tests will be run from the source directory, as they do not
currently need packaging.
The current rule is only for "backend.RecursiveMakeBackend", but, with
the current default of generating both the RecursiveMake and FasterMake
backends, the command creates/refreshes both backends. This is, in fact,
how the FasterMake backend is refreshed in most cases.
Moreover, with an hybrid backends, the generated file is not
"backend.RecursiveMakeBackend" anymore, so we need a more generic way to
handle this.
Furthermore, it's not necessarily desirable for all backends to have a
dependency file to handle the dependencies to refresh the backend, so
generate a plain list instead. This has the side effect of making `mach
build-backend --diff` more readable for changes to that file.
Finally, make the backend.* files created like any other backend file,
such that its diff appears in the `mach build-backend --diff` output.
The FasterMake build system is meant to be invoked through `mach build
faster`, which does it already, or, in the near future, as part of an
hybrid build system, which will deal with it as well. People doing
`make -C objdir/faster` won't have the backend automatically refreshed,
but that's not a supported way to use it anyways.
Install manifests are not empty in normal conditions, apart a few
exceptions where they are only used for a "magic" `rm -rf`.
However, we're going to introduce changes that will empty some of
the install manifests and make their work happen from a different
backend, in which case we don't want them to correspond to a `rm -rf`.
When doing build system changes affecting backend-generated files, I
often use `mach build-backend --diff`. But most of the time I end up
wanting to look at the full diff again when doing further changes, which
leads me to stash my changes away, run `mach build-backend` to get the
initial state again, unstash and rerun `mach build-backend --diff`.
This has been a time drain for long enough :)
The code doing this was added before we had install manifests, back when
they were purge manifests, and before we tracked all files created by
the backend. Nowadays, if an install manifest is removed, it will be
removed in BuildBackend.consume.
In fact, purging the install manifests in the backend itself breaks the
deleted files count displayed after reticunating splines.
XPI_NAME affects no tier on its own, as it merely changes paths where
things end up that are defined by other variables.
FILES_PER_UNIFIED_FILE had an erroneous value.
Those are the worst offenders of affected_tiers. The rules to handle
them are all in directories that is not necessarily related to where
the variables are defined, each of which has a Makefile.in for those
rules, which forces export to go through them.
They are respectively using PP_TARGETS and INSTALL_TARGETS. Both affect
the misc tier since bug 1240671, per the *_TARGET value they set in the
backend.
This has the nice side effect of now skipping directories where test
harness files are handled by install manifests.
JAR_MANIFESTS affects the libs tiers through config/rules.mk rules.
While we could move the rules in the backend, they are too complex to
just do that now.
GENERATED_FILES impacts the export tier through the config/rules.mk
definitions, now moved to the backend itself, so that everything is
close to each other.
Those are non-passthru variables with the same property as
ANDROID_GENERATED_RESFILES, ANDROID_APK_NAME and ANDROID_APK_PACKAGE, in
that they don't affect tiers through the backend code itself, but from
the Makefile.in along the moz.build they are defined in.
The are passthru variables that don't actually affect any tier per the
backend itself. They do affect the `export` tier by way of the Makefile.in
along the moz.build defining them, and the existence of those
Makefile.in already guarantees those directories not to be skipped for
`export`.
This initiates a move off affected_tiers in VARIABLES definition to
explicit in-backend handling, which will hopfully make things clearer.
HAS_MISC_RULE is currently used to opt-in to the misc tier in a few
directories with a misc:: rule. It is, in fact, mostly used for custom
xpi creation, which will be separately addressed in bug 1240676, so it
will eventually go away entirely, but in the meantime, we send it as a
throwaway passthru.
Historically, all tiers were handled as opt-out (may_skip), until we
added the first opt-in tier (no_skip). It doesn't make much sense to
differentiate them anymore, so handle them all as opt-in.
When adding a backend, we currently have to add them in three different
places so that they appear in the right places.
Instead, keep the list in a single place.
Currently |./mach artifact| installs an old version of the taskcluster client,
which installs an old version of requests that's incompatible with commonly
installed python versions. This bumps to a version of taskcluster client that
accepts and requests version < 3, so we pick up the in-tree version instead of
installing 2.4.3.
generate_symbols_file only supports the global defines, and completely
ignores DEFINES from the same moz.build the SYMBOLS_FILE is defined.
This fixes this misfeature.
A recent change regressed this behavior -- while an artifact build runs, it
doesn't load certain "about:" pages due to missing libraries in subdirectories
of dist/bin.
AFAICT, we don't actually access web-platform test files from the
objdir for anything except test packaging. And we already have
a mechanism for creating test archives from files directly in the
source directory. So, let's stop copying them to the objdir and
package them directly from the source directory!
The _tests install manifest reports the following change:
before: 41,977 files installed
after: 24,537 files installed
delta: -17,440 files
We still copy some WPT files to the objdir. We might be able to
eliminate these as well. However, since there are only ~200 files,
I'm not too concerned.
I manually compared the resulting web-platform zip archives from before
and after. No files were removed from the archive. However, the new
archive does gain several hundred empty directories with .gitkeep
files. This feels weird, but it shouldn't break anything (I would
think). I'm inclined to leave them for now. I'll file a follow-up
bug to deal with them (preferably by removing them from version
control).
The faster make backend cannot support such things, so it's better to
avoid unsupported things to slip in because it happens that doesn't
break what automation runs.
Right now, each incremental build produces a new set of GeckoView
library files, but the previous files don't get cleaned up, and you end
up with a bunch of old libraries in your objdir. This patch cleans up
the old files before producing new ones.
