The hglib Mercurial finder was nice. But it is somewhat slow, as it
involves a separate Mercurial process.
This commit introduces a native Mercurial finder that speaks directly
to a Mercurial repository instance. It is significantly faster.
The new code is isolated to its own file because it imports Mercurial
code, which is GPL.
The moz.build reader uses absolute paths when referencing moz.build
files. *Finder classes expect paths to be relative to a base. When we
switched the reader to use *Finder instances for I/O, we simply provided
a default FileFinder based at the filesystem root. This was quick and
easy. Things worked.
Unfortunately, that solution isn't ideal because not all *Finder
instances can accept absolute paths like that. The
MercurialRevisionFinder is one of them.
Changing the moz.build reader to talk in terms of relative paths is a
lot of work. While this would be ideal, it is significantly easier to
defer the problem until later and hack up MercurialRevisionFinder to
accept absolute paths. This commit does exactly that.
Bug 1171069 has been filed to track converting BuildReader to relative
paths.
Now that moz.build files use finders for I/O, we can start reading
moz.build data from other sources.
Version control is essentially a filesystem. We implement a finder
that speaks to Mercurial to obtain file data. It is able to obtain
file data from a specific revision in the repository.
We use the hglib package (which uses the Mercurial command server) for
speaking with Mercurial. This adds overhead compared to consuming the
raw Mercurial APIs. However, it also avoids GPL side-effects of
importing Mercurial's Python modules.
Testing shows that performance is good but not great. A follow-up
commit will introduce a GPL licensed Mercurial finder. For now, get
the base functionality in place.
Passing raw file handles around is a bit dangerous because it could lead
to leaking file descriptors. Add a read() method that handles the simple
case of obtaining the full contents of a File instance.
This should ideally be a method on BaseFile. But this would require
extra work and isn't needed. So we've deferred it until bug 1170329.
Today, the *Finder classes are optimized for doing matching and
returning multiple results. However, sometimes only looking for a
single file is wanted.
This commit implements the "get" method on all the *Finder classes.
It returns a BaseFile or None.
FileFinder._find_file has been refactored into FileFinder.get
to avoid code duplication.
This avoids duplicating the logic from SimplePackager to find base
directories, and fixes some cases where the l10n repack code wouldn't
find them properly.
Bug 910660 added a consistency check that rejects cases where a manifest
inside a directory detected as being the base of an addon is included from
outside the addon. Unfortunately, this triggered false positives when
a manifest is included from within the addon, but just happens to be at
the top-level of that addon.
This avoids duplicating the logic from SimplePackager to find base
directories, and fixes some cases where the l10n repack code wouldn't
find them properly.
Some file types, such as XPTFile, read their associated data during the
copy. In the case of XPTFiles, several original files are linked together
in one destination file. When a FileCopier is used in-place, those
original files are removed before XPTFile.copy is invoked, so XPTFile.copy
fails.
Generally speaking, there are many other reasons why FileCopier can fail
in-place, but the only active use of this mode is l10n repack code, which
actually doesn't do much of the dangerous uses. However, it can end up
linking XPTFiles for some reason, which fails for the reasons given above.
Back when mozpack.path was added, it was used as:
import mozpack.path
mozpack.path.func()
Nowadays, the common idiom is:
import mozpack.path as mozpath
mozpath.func()
because it's shorter.
$ git grep mozpath\\. | wc -l
423
$ git grep mozpack.path\\. | wc -l
123
This change was done with:
$ git grep -l mozpack.path\\. | xargs sed -i 's/mozpack\.path\./mozpath./g'
$ git grep -l 'import mozpack.path$' | xargs sed -i 's/import mozpack.path$/\0 as mozpath/'
$ (pat='import mozpack.path as mozpath'; git grep -l "$pat" | xargs sed -i "1,/$pat/b;/$pat/d")
The regular expression cache for mozpack.path.match was keyed off the
original pattern. However, that variable was mutated as part of the
function and the mutated result was subsequently stored as the cache
key. This effectively resulted in a 0% cache hit rate.
On some tests being written for bug 1132111 which involve a full
filesystem traversal for moz.build files and subsequent execution of
those files, the following timings are indicative of the impact of this
patch.
Before:
real 16.082s
user 14.760s
sys 1.318s
After:
real 6.345s
user 5.085s
sys 1.257s
mopack.BaseFile.copy() performs a generic read/write file copy. Windows
has an explicit CopyFile() call that tests have shown to be
significantly faster. Let's use that instead via the magic of ctypes.
This is a straight copy from
a878bf0ba0
paired with a tiny change to use the new quote_chars option.
--HG--
extra : rebase_source : 75d604ffafc7062c663bca4242af35546d2c1e3a
The forward slash appears to be the standard path separator in zip/JAR
files. Accept back slashes when adding paths to a JAR.
--HG--
extra : rebase_source : bd94eab36b347006e65952d99b53dd397e2ca758
extra : amend_source : 2cefd887d8bb5d989fafb398a3464429ac376e2e
The install manifest processor starts with an empty InstallManifest and
uses |= to "concatenate" instances. It became pretty obvious when
developing some patches that add more preprocessed files to install
manifests that the source install manifest dependency was getting
lost during the |= operation. This patch fixes it.
The solution is not ideal performance wise. But slightly worse
performance (only after config.status, however) is better than
clobbers.
A test has been added to ensure this doesn't regress.
--HG--
extra : rebase_source : 848aebbbc935ce2bca2d3fcc85d1df534e734e0d
Smart xpt linking will keep around [scriptable] interfaces and anything
those interfaces depend on. Modify the tests that deal with xpt linking
so they use [scriptable] interfaces, ensuring that the tests continue to
work in the face of smarter linkers.
This already raised if the order was [foo, foo/bar]. But it didn't
prevent adding [foo/bar, foo].
The only sub-classes of FileRegistry are FileCopier and Jarrer.
FileCopier.copy threw in the previously unhandled case: the order of
creation is the same as the order of addition, so that foo is created
after foo/bar.
A zip file index can contain both foo and foo/bar. I don't think we
should rely on this property in our use of Jarrer, but if we already do,
I guess we need to move these guards into FileCopier. Let's hope that's
not the case!
(For the record: On my Mac OS X system, unzipping such a zip file
prompts the user for what to do, depending on the order of the entries
in the zip index.)
This patch adds pattern matching entries to install manifests. We store
metadata necessary to construct a pattern match at a later point in
time. When we convert the install manifest to a file registry, we
resolve the patterns using FileFinder.
The build config logic has been updated to store support-files values as
pattern entries. This should resolve the clobber needed issue and make
the local development experience more pleasant as well.
--HG--
extra : amend_source : 3fe659f7ad6930ef54316b5babac6b83bee240af
