For vertices in the plugin graph. This is so that the group and overlap
edges are evaluated in an order that does not depend on the current load
order. Tie-breaking still uses the current load order.
This is necessary because if the group and overlap edges that get added
depend on the current load order, sorting and applying changes the
current load order, so sorting again may give different results even
even though no plugin data or metadata has changed.
Rather than storing them in PluginSortingData, which is now immutable.
This also means the predecessor groups plugins map can use vertices
instead of plugin names, which is a
little simpler.
GCC 8 doesn't support C++17's parallel algorithms, which I'd like to
use, and GCC 10 is the newest major version available from Ubuntu
20.04's repositories.
This commit removes a Linux-only std::filesystem::path constructor test,
which fails due to a thrown exception when compiled with GCC 10. The
failure isn't important, because libloot avoids that particular
constructor: in a sense an exception is a bit better because it would
make any usage more obvious.
If a group edge exists due to the presence of some user metadata,
distinguish that from when the edge exists only due to masterlist
metadata.
The logic for detecting when a path between two groups involves user
metadata gives results that are stable but that do not consistently
paths that only consist of masterlist metadata. If there are multiple
paths between two groups and some involve user metadata, the same path
will be picked every time, but the path picked depends on the structure
of the groups graph.
In practice this shouldn't be much of a problem because the paths are
only exposed when there's a cycle, and in that case all paths between
the two groups need to be removed/broken anyway.
The use of FormIDs doesn't apply to Morrowind and it's really the
records that are significant - FormIDs are an implementation detail.
This also aligns with other use of records in names.
All master-flagged plugins must load before all non-master-flaggeg
plugins, and this means that most of the edges added in the graph
(about 2/3rds in large load orders) are just enforcing this.
Having lots of edges negatively impacts the performance of checking for
paths, and adding overlap edges is O(n^2), so instead of having one
graph containing all plugins, create one graph for masters and another
for plugins, and sort them independently, then append the non-masters
order to the masters order.
This speeds up my 1619 plugin sort from 44s to 34s, and larger load
orders should see more benefit.
This does introduce some behavioural changes though:
- any requirement or load after metadata that tries to put a master
after a non-master will now be ignored instead of causing a cyclic
interaction error. A master-flagged plugin that has a
non-master-flagged plugin will also no longer cause a cyclic
interaction error, but that scenario is much less likely.
- The resulting load order may differ slightly. When tie-breaking finds
a path that contradicts the old load order, it pins the positions of
plugins in the path. However, the lack of master flag edges causes
later edges to be added or skipped differently. This is all ultimately
down to the order of edge iteration mattering during path discovery
(since it stops at the first path discovered), so even though the two
approaches result in graphs that enforce the same relationships
between plugins at the point that tie-breaking starts, ties may be
broken differently due to differences in the edges enforcing those
relationships.
Instead of brute-forcing the existence of a Hamiltonian path, attempt
to create one by adding edges between each pair of adjacent vertices
in the old load order wherever possible without introducing a cycle,
and otherwise move plugins as necessary.
The bidirectional BFS has been refactored to use a visitor so that
the implementation can be shared between checking if a path exists and
finding a path without always incurring the cost of the latter.
This may result in a different sorted load order than the previous
method, but that shouldn't make a practical difference as tie-break edges
are only added between unrelated plugins.
For a load order containing 1619 plugins, this reduces the time taken
to sort them from around 400s to around 51s.
BA2 files contain 32-bit hashes, which are not enough to reliably avoid
hash collisions (the worst case amongst Fallout4.esm's BA2 files has a
~ 84% chance of collision).
This instead calculates 64-bit folder and file hashes from the file
paths stored at the end of the BA2 files, which means that worst case
becomes a ~ 0.00000004% chance of collision.
Despite the much lower odds, I still see the same number of warnings
about a hash pair already existing when trying to merge hashes from
different BA2 files, so the same file paths must be present across
multiple BA2 archives. I wasn't really expecting any different though,
as I'd expect the official archives not to contain any hash collisions:
the game developers have control over the contents so can choose names
to avoid collisions that would presumably reduce performance if not
cause issues. That's not the case when working with an arbitrary
collection of mods though, so I think using 64-bit hashes is worthwhile.
They're used by Fallout 4.
BA2 file and folder hashes are 32-bit, so collisions are much more
likely than with BSAs, which use 64-bit hashes. I put in some logging
to check the likelihood of collisions, and found that the largest
number of assets loaded was by Fallout4.esm (no surprise), which loaded
371182 files across 5746 folders, with the most files in one folder
being 124871. That puts the probability of a hash collision within that
folder at above 80%.
I did see different Fallout4 -*.ba2 files contain files with the same
combination of folder and file hashes, and similar for
DLCUltraHighResolution -*.ba2 files, so I'm going to try calculating
64-bit hashes from the file paths stored in the BA2 files.
This supports BSAs used from:
* Oblivion
* Fallout 3
* Fallout: New Vegas
* Skyrim
* Skyrim: Special Edition
If Skyrim VR uses the same BSA format as Skyrim SE, that's also supported.
Morrowind BSAs are intentionally not supported because they cannot be
loaded by plugins and so are of no interest to LOOT.
The BSA parsing code has been adapted from my abandoned libbsa library.
This doesn't bother reading folder and file names as hash collisions
seem pretty unlikely (2^64 possible folder hashes, and 2^64 possible
file hashes per folder).
The code checks and requires that BSAs use little-endian numbers, as
while big-endian BSAs are apparently possible I've never seen one,
and I don't want to try adding support without one to test against.
Oliver Hamlet