loot/libloot
0
0
Fork 0
mirror of https://github.com/loot/libloot.git synced 2026-07-27 14:16:01 -07:00
Code Issues Packages Projects Releases Wiki Activity

Split plugin graph in two

Browse Source 
All master-flagged plugins must load before all non-master-flaggeg
plugins, and this means that most of the edges added in the graph
(about 2/3rds in large load orders) are just enforcing this.

Having lots of edges negatively impacts the performance of checking for
paths, and adding overlap edges is O(n^2), so instead of having one
graph containing all plugins, create one graph for masters and another
for plugins, and sort them independently, then append the non-masters
order to the masters order.

This speeds up my 1619 plugin sort from 44s to 34s, and larger load
orders should see more benefit.

This does introduce some behavioural changes though:

- any requirement or load after metadata that tries to put a master
  after a non-master will now be ignored instead of causing a cyclic
  interaction error. A master-flagged plugin that has a
  non-master-flagged plugin will also no longer cause a cyclic
  interaction error, but that scenario is much less likely.
- The resulting load order may differ slightly. When tie-breaking finds
  a path that contradicts the old load order, it pins the positions of
  plugins in the path. However, the lack of master flag edges causes
  later edges to be added or skipped differently. This is all ultimately
  down to the order of edge iteration mattering during path discovery
  (since it stops at the first path discovered), so even though the two
  approaches result in graphs that enforce the same relationships
  between plugins at the point that tie-breaking starts, ties may be
  broken differently due to differences in the edges enforcing those
  relationships.
This commit is contained in:
Oliver Hamlet
2023-01-06 22:20:31 +00:00
parent ddf818ff51
commit 232202c17e
3 changed files with 94 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api/sorting.rst
+35 -11
View File
@@ -2,7 +2,7 @@
LOOT's Sorting Algorithm
************************
LOOT's sorting algorithm consists of four stages:
LOOT's sorting algorithm consists of the following stages:
.. contents::
:local:
@@ -21,8 +21,11 @@ it supports two threads).
When parsing plugins, all subrecords are skipped over for efficiency, apart from
the subrecords of the ``TES4`` header record.
Loading plugin data also involves loading any metadata that the plugin may have
in the masterlist and userlist.
Create plugin graph vertices
=================================
============================
Once the plugins have been loaded, they are sorted into their current load
order:
@@ -36,15 +39,25 @@ order:
to decide their order. If they are equal, a case-insensitive lexicographical
comparison of their file extensions is used.
After that, a directed graph is created and the plugins are added to it as
vertices in their sorted order.
After that, two graphs are created, and the plugins are added to them as
vertices in their sorted order. Plugins that have their master flag set go in
one graph, and plugins that do not have the flag set go in the other.
Any metadata a plugin has in the masterlist and userlist are then merged into
its vertex's data store. Plugin group dependencies are also resolved and added
as group-derived plugins.
Two graphs are used because master-flagged plugins must always load before non-master-flagged plugins, and it's much more efficient to sort them separately
and then combine their load orders than to enforce those relationships within a
single graph.
A consequence of using two separate graphs is that any plugin data or metadata
that involves a pair of plugins with and without their master flag set will be
silently ignored. For example: if plugin A is master-flagged and plugin B is
not, and plugin A has metadata saying it must load after plugin B, then that
metadata will be ignored because the two plugins are sorted independently, as if
the other plugin is not installed.
Create plugin graph edges
==============================
=========================
The steps described in this section are run on both graphs independently.
In this section, the terms *vertex* and *plugin* are used interchangeably, and
the iteration order 'for each plugin' is the order in which the vertices were
@@ -53,7 +66,9 @@ added to the graph.
For each plugin:
1. If the plugin is a master file, add edges going to all non-master files. If
the plugin is a non-master file, add edges coming from all master files.
the plugin is a non-master file, add edges coming from all master files. This
shouldn't result in any edges being added, since masters and non-masters are
sorted in separate graphs, but is done for completeness.
2. Add edges coming from all the plugin's masters. Missing masters have no edges
added.
3. Add edges coming from all the plugin's requirements. Missing requirements
@@ -154,8 +169,10 @@ add an edge going from the unpinned vertex to the vertex after the found vertex.
Then record the unpinned vertex's new position in the new load order list: the
vertex is now pinned.
Topologically sort the plugin graph
===================================
Topologically sort the plugin graphs
====================================
This is done for both graphs independently.
Note that edges for explicit interdependencies are the only edges allowed to
create cycles. However, the graph is again checked for cycles to guard against
@@ -163,3 +180,10 @@ potential logic bugs, and if a cycle is encountered an error is thrown.
Once the graph is confirmed to be cycle-free, a topological sort is performed on
the graph, outputting a list of plugins in their newly-sorted load order.
Combine the two load orders
===========================
Finally, the sorted load order for non-master-flagged plugins is appended to the
sorted load order for master-flagged plugins to give the complete sorted load
order.
src/api/sorting/plugin_graph.cpp
+3
View File
@@ -583,6 +583,9 @@ void PluginGraph::AddSpecificEdges() {
const auto& vertex = *vit;
const auto& plugin = GetPlugin(vertex);
// This loop should have no effect now that master-flagged and
// non-master-flagged plugins are sorted separately, but is kept
// as a safety net.
for (vertex_it vit2 = std::next(vit); vit2 != vitend; ++vit2) {
const auto& otherVertex = *vit2;
const auto& otherPlugin = GetPlugin(otherVertex);
src/api/sorting/plugin_sort.cpp
+56 -26
View File
@@ -192,31 +192,7 @@ std::vector<PluginSortingData> GetPluginsSortingData(
return pluginsSortingData;
}
std::vector<std::string> SortPlugins(
const Game& game,
const std::vector<std::string>& loadOrder) {
const auto pluginsSortingData = GetPluginsSortingData(game, loadOrder);
// If there aren't any plugins, exit early, because sorting assumes
// there is at least one plugin.
if (pluginsSortingData.empty()) {
return {};
}
const auto logger = getLogger();
if (logger) {
logger->debug("Current load order:");
for (const auto& plugin : loadOrder) {
logger->debug("\t{}", plugin);
}
}
PluginGraph graph;
for (const auto& plugin : pluginsSortingData) {
graph.AddVertex(plugin);
}
std::vector<std::string> SortPluginGraph(PluginGraph& graph, const Game& game) {
// Now add the interactions between plugins to the graph as edges.
graph.AddSpecificEdges();
graph.AddHardcodedPluginEdges(game);
@@ -243,6 +219,7 @@ std::vector<std::string> SortPlugins(
const auto path = graph.TopologicalSort();
const auto result = graph.IsHamiltonianPath(path);
const auto logger = getLogger();
if (result.has_value() && logger) {
logger->error("The path is not unique. No edge exists between {} and {}.",
graph.GetPlugin(result.value().first).GetName(),
@@ -250,7 +227,60 @@ std::vector<std::string> SortPlugins(
}
// Output a plugin list using the sorted vertices.
const auto newLoadOrder = graph.ToPluginNames(path);
return graph.ToPluginNames(path);
}
std::vector<std::string> SortPlugins(
const Game& game,
const std::vector<std::string>& loadOrder) {
auto pluginsSortingData = GetPluginsSortingData(game, loadOrder);
// If there aren't any plugins, exit early, because sorting assumes
// there is at least one plugin.
if (pluginsSortingData.empty()) {
return {};
}
const auto logger = getLogger();
if (logger) {
logger->debug("Current load order:");
for (const auto& plugin : loadOrder) {
logger->debug("\t{}", plugin);
}
}
const auto firstNonMasterIt = std::stable_partition(
pluginsSortingData.begin(),
pluginsSortingData.end(),
[](const PluginSortingData& plugin) { return plugin.IsMaster(); });
// Some parts of sorting are O(N^2) for N plugins, and master flags cause
// O(M*N) edges to be added for M masters and N non-masters, which can be
// two thirds of all edges added. The cost of each bidirectional search
// scales with the number of edges, so reducing edges makes searches
// faster.
// As such, sort plugins using two separate graphs for masters and
// non-masters. This means that any edges that go from a non-master to a
// master are effectively ignored, so won't cause cyclic interaction errors.
// Edges going the other way will also effectively be ignored, but that
// shouldn't have a noticeable impact.
PluginGraph mastersGraph;
PluginGraph nonMastersGraph;
for (auto it = pluginsSortingData.begin(); it != firstNonMasterIt; ++it) {
mastersGraph.AddVertex(*it);
}
for (auto it = firstNonMasterIt; it != pluginsSortingData.end(); ++it) {
nonMastersGraph.AddVertex(*it);
}
auto newLoadOrder = SortPluginGraph(mastersGraph, game);
const auto newNonMastersLoadOrder = SortPluginGraph(nonMastersGraph, game);
newLoadOrder.insert(newLoadOrder.end(),
newNonMastersLoadOrder.begin(),
newNonMastersLoadOrder.end());
if (logger) {
logger->debug("Calculated order:");