The script looks in message content, message subs and locations. If a
URL appears in the given plugins index file(s), it's assumed to be
valid.
The checks aren't perfect: some links look like they work, but the
actual page they go to doesn't have any relevant data on it.
I've hardcoded a bunch of hostnames that have gone defunct, usually
because the sites are gone, but some redirect to unrelated content.
There are also a few sites that respond to requests with 403s, but not
when I access them in my web browser.
Those sites (the defunct and the 403ing) are skipped, because there's no
point checking them, but the fact they're skipped is logged.
The data/urls.csv file was generated by running:
uv run -- scripts/check_urls.py -i mlox/mlox_base.txt.yaml -i mlox/mlox_base.danae.txt.yaml -i mlox/mlox_base_legacy.txt.yaml -i mlox/mlox_user.txt.yaml -p data/manual_mod_plugins.csv -p data/mmh_fliggerty_plugins.csv -p data/nexus_mods_morrowind_plugins.csv
Then making some manual edits, running again with "-u data/urls.tsv -w" added, and that loop repeated several times.
The input file is from <https://raw.githubusercontent.com/mlox/mlox/bd5b39e4704d4574e2a7be3d0a598a686a646214/data/mlox_base.txt>.
The mlox_base.txt.yaml file was generated by running:
uv run --with libloot@../libloot/target/wheels/libloot-0.28.4-cp314-cp314-win_amd64.whl --python python -- import_mlox.py -p mmh_fliggerty_plugins.csv -p nexus_mods_morrowind_plugins.csv -p manual_mod_plugins.csv
The script logs no errors, and the only warnings logged are related to
the substitution of known filenames in place of filename patterns.
This also introduces a hack to workaround the only piece of input data
that can't be represented by LOOT in any way. The version predicate:
[VER<1.5 Nevena's Twin Lamps & Slave Hunters *.esp]
is the only one that uses a filename pattern that does not include
"<VER>", which makes it unrepresentable in LOOT's condition syntax.
However, the pattern matches the following known filenames:
Nevena's Twin Lamps & Slave Hunters 1.0.esp
Nevena's Twin Lamps & Slave Hunters 1.2.esp
Nevena's Twin Lamps & Slave Hunters 1.5.esp
The 1.0 plugin is only available from the Morrowind Modding History
archived by the Wayback Machine, but the other two appear in
mlox_base.txt, and the 1.5 plugin is also available from Nexus
Mods. Since there are no other matching filenames that contain a version
string, it seems safe to replace the "*" in the pattern with "<VER>".
Plus a script to help generate the index TSV file. The filenames in that
index won't necessarily all match someone else's downloads, because some
were deduplicated by Windows, so their names depend on their download order.
This covers some filename patterns that were not previously matched.
The script takes the CSV export of the index of mods that were hosted on
Morrowind Modding History and Great House Fliggerty before they went
down, as downloaded from <https://modlist.altervista.org/mmh/>. That CSV
is included in this commit, and results in 61 GB of downloaded files.
This commit also includes a script that reads the index CSV and
downloads directory and outputs a TSV file of all the plugin files in
the downloaded files, looking inside archives. The output TSV file is
also included in this commit.
Both scripts include a few hacks to workaround issues affecting a small
minority of mods:
- The mod IDs 2770, 2771 and 2772 are three separate uploads but form
the three parts of a multipart 7z archive, so their downloaded files
are named to allow the archive to be read by 7-zip.
- There are a few downloads that are just bare plugin files: these are
written into zip files so that the plugin filenames can be preserved,
since the download file name needs to be the mod ID.
- Some download URLs use tinyurl: these have a hardcoded mapping to the
URLs that tinyurl points to, in case the tinyurl URLs stop working.
- Nexus Mods URLs are skipped as they're already covered by the
fetch_nexus_plugins_index.py script.
The download script also handles the many cases where mod IDs point to
downloads that are no longer available, by skipping a mod entirely if
its ID appears in a hardcoded whitelist of known invalid downloads.
This uses Nexus Mods' API to retrieve plugin filenames for the mod
uploads that are hosted there, and write them to a file as
tab-separated values.
The output TSV file is included in this commit. The source data is from
mid-October 2025.
Intern the values to force PyYAML to use anchors and aliases instead of
duplicating objects.
The effect on my test data output is:
No interning: 5559 KB, 94924 lines
+ Files: 3922 KB, 78568 lines
+ messages: 3808 KB, 77227 lines
+ file details: 2689 KB, 64467 lines
+ file conditions: 2268 KB, 64467 lines
+ file constraints: 2265 KB, 64467 lines
+ message contents: 2179 KB, 63451 lines
+ message conditions: 2179 KB, 63451 lines
That gives a 61% reduction in output size overall.
The script parses the mlox rules into an abstract representation. There
are two other parsers that I know of:
- mlox's is also written in Python, but mixes together parsing and
evaluation, so isn't suitable for this use case
- plox's seems like a better starting point, but isn't directly usable
in its current form
This script doesn't attempt to match either parser's behaviour exactly
(e.g. mlox treats whitespace as optional in more places), but instead
targets the syntax that is actually in use in the rules files present in
the https://github.com/mlox/mlox and
https://github.com/DanaePlays/mlox-rules repositories.
While the parser can handle everything in those files, not everything
supported by mlox is supported by LOOT, and the remaining differences
are intentional. See the comment at the top of the script for more
details on the differences and how they're handled.
Oliver Hamlet