Path::is_dir() already resolves symlinks before checking if the path is a directory.
Passing down the resolved game path symlink to libloadorder means that when using MO2's VFS, libloadorder won't see the plugins that MO2 has "installed", because its VFS doesn't work with symlinks, so the files don't appear to be present in the symlink's target directory, they only appear to be present within the symlink directory path.
This avoids the issue of hash collisions, within any one archive, a plugin's archives, or between different plugins' archives. That means that sorting has accurate data on what the asset counts loaded by plugins are, and what the asset overlap between plugins, so it can make better decisions when adding overlap edges.
It does require that the archives include the folder and file names, which is apparently not strictly required, but it seems that having the names is practically required[1][2], and I verified that the parsing code can handle archives from all of the supported games (apart from Morrowind, which doesn't have its BSAs read by LOOT). That testing covered 321 archives containing 771246 folder records and 3404007 file records, including some mod BSAs/BA2s.
If an archive doesn't have the flags set for containing folder and file names, then libloot will log an error and effectively ignore its contents, so the new behaviour means that some archives that were (potentially inaccurately) taken into account before may now be ignored. Falling back to using the hashes for archives that don't have names would be an option, but that complicates comparison against archives that do have names, and I have no evidence that the fallback would be useful in practice.
This doubles the size of each map key and set entry (from 8 to 16 bytes), introduces another level of indirection when comparing keys or values, and also means that the name strings need to be stored, further increasing memory usage.
I tested the performance impact when sorting Skyrim SE and Starfield load orders, using LOOT v0.29.1 and comparing against using it with libloot v0.24.4:
- The Skyrim SE load order had 1630 plugins and 18 BSAs totalling 3.14 GB (not including Skyrim.esm's or its BSAs, which LOOT doesn't fully load).
- Loading plugins (which includes parsing BSAs) was ~ 2% (5 ms) faster
- Sorting was ~ 1% (21 ms) faster
- Memory usage after sorting was 1 MB (< 1%) higher
- These differences are probably within margin of error
- The Starfield load order had 31 plugins and 72 BA2s totalling 20.2 GB (not including Starfield.esm or its BA2s, which LOOT doesn't fully load).
- Loading plugins (which includes parsing BA2s) was ~ 96% (423 ms) faster
- Sorting speed was unchanged
- Memory usage after sorting was 11 MB (12%) higher
It looks like the accuracy improvement is also faster, and the memory usage cost is relatively modest.
[1]: https://en.uesp.net/wiki/Oblivion_Mod:BSA_File_Format
[2]: https://en.uesp.net/wiki/Skyrim_Mod:Archive_File_Format
Feedback from Robert on Discord[1] is that hash collisions can be very common for certain mods.
Since erroring causes that archive's assets to be ignored, it results in less accurate sorting behaviour than not erroring, and if there are many hash collisions then logging them just produces noise.
Instead, count the number of collisions and debug log a single statement if it's non-zero, so there's still some indication to aid debugging any unexpected sorting behaviour it causes.
[1]: https://discord.com/channels/473542112974077963/473542230095822848/1498060566344896673
The linked Discord message (and a few following messages) is:
> pretty sure for Skyblivion we have > Many duplicates all over our files
and i mean A LOT
> but 100s of thousands of files that i'd expect
> [...]
> quirks of the hashing formula the game uses
>
> -- Robert in #support at 2026-04-27, 21:36 PM
TL;DR: The warning added a lot more noise than expected, doesn't really
add value, and improving it isn't worth the effort.
The warning was logged when the same pair of folder hash and file hash
appeared more than once within the set of archive files being loaded
(as the function was used, the set would be the archives loaded by a
single plugin).
Hashes are used directly instead of asset file paths because the paths
are not necessarily present in BSA files, and while they are present in
BA2 files (at least, I'm not aware of any option to omit them or any
files that do omit them), hashes are calculated from the file paths they
contain for consistency with the approach needed for BSA files.
Without the file paths it's not possible to determine if the repeated
hashes represent the same asset file (possibly containing different
data) or different files that have colliding hashes.
I had assumed that it would be unusual for a single plugin to load more
than one archive file containing the same asset file path, so any
repeated hash pairs would likely indicate hash collisions, but feedback
from Pickysaurus on behalf of Vortex users indicates that's not true,
and that logging all the warnings significantly slows down fully loading
plugins.
There are a few ways that the logged warning could be improved: tracking
which archive file existing hashes were inserted from would make it
easier to identify the pair of archive files that might need
investigating; reading BA2 files could defer transforming their asset file
paths into hashes until after all the files for a plugin have been read
(or even past that, to account for hash collisions between different
plugins' assets); and reading BSA files could opportunistically store the
asset file paths if they are present, and fall back to comparing using
hashes if not.
However, even if the warning was logged for only true positive hash
collisions, this is all in service of a sorting heuristic that is only
used when adding overlap edges and a pair of plugins do not have
overlapping records but do both load assets, and a collision would mean
that a plugin might seem to load fewer assets than it does, and could
also appear to overlap with a plugin that doesn't actually load assets
with the same file paths. That in turn might result in the two plugins
loading in one order instead of the other, causing one's assets to
override the other's. If that's a problem, then it can be fixed using
load after metadata, and you can only really tell if it's a problem by
spotting something wrong in game, so the warning doesn't really add much
value, and although it indicates that there might be a problem, there's
enough conditions between the warning and there actually being an issue
that logging it as a warning is excessive anyway.
Some tests have been updated because UTF-8 is used as the native
path encoding with MinGW/Wine, unlike MSVC/Windows.
Some of the tests fail:
- 4 Rust tests fail because long paths are not enabled and so the
paths used when creating symlinks and junction paths are too
long. I've tested them with x86_64-pc-windows-gnu and
x86_64-pc-windows-gnullvm, and both see the same behaviour. The
tests pass when the MinGW-built executable is run on Windows, so
this is a Wine limitation.
- 12 C++ tests fail because directory symlink creation is not
implemented. They fail whether the MinGW-built executable is run
in Wine or on Windows, so this is a MinGW limitation.
- 1 C++ filesystem test fails because long paths are not enabled.
The failing tests are skipped at runtime when built with MinGW,
aside from the one test for long paths being enabled, which expects
them to be disabled when built with MinGW.
If long paths are enabled, e.g. by running
wine reg add HKLM\\System\\CurrentControlSet\\Control\\Filesystem /v LongPathsEnabled /t REG_DWORD /d 1 /f
then many more tests fail because the C++ tests create long paths
when that Registry value is set, but it doesn't seem to actually
enable long path support in Wine, so various filesystem operations
fail.
It includes new functionality that is required for the metadata syntax v0.29.
Add a couple of tests that rely on that new functionality so that the
dependency is backed by tests that fail if it's not met.
Some functionality is only supported because the regex is built with the
Unicode flag instead of the UnicodeSets flag, but that's done for
backwards compatibility as the latter disallows some syntax that may be
in use.
As they're relatively simple strings, the indirection introduced by using an alias may harm readability more than the alias helps to prevent typos, shorten the metadata doc and make repetition more obvious.
To match how they're organised in the masterlists. The order of the anchor types matters: contents and conditions go before messages and files because the latter can contain the former, so writing the former first ensures that all anchors are written at the same level of indentation.
Support is currently limited to aliasing:
- Condition and constraint strings
- File values
- Message values
- MessageContent arrays (including single-value arrays that are
serialised as strings)
An anchor is written if the same value appears more than once in the
metadata document being written, and if an anchor for that value has not
already been written. If a value has already been written with an
anchor, later appearances of that value will be written as aliases of
that anchor.
Anchors are named according to the type of data they're for, followed by
an incrementing number, e.g. file1, message1, contents1, condition1.
This behaviour is configurable within libloot, the configuration options
will be exposed externally once the functionality is more settled.
The C++ wrapper defaults it to true, so the behaviour hasn't changed for
existing callers that don't pass a value for the evaluateConditions
parameter, i.e. that call GetGeneralMessages().
Since the new parameter comes before evaluateConditions,
existing GetGeneralMessages(true) and GetGeneralMessages(false) calls
need to be replaced with GetGeneralMessages(true, true) and
GetGeneralMessages(true, false) to retain their existing behaviour. The
parameter order was chosen to match that of GetPluginMetadata(), at the
cost of this backwards incompatibility.
Oliver Hamlet