Files
Oliver Hamlet 1bf45e5d46 Add a script to check URLs that appear in a given masterlist
The script looks in message content, message subs and locations. If a
URL appears in the given plugins index file(s), it's assumed to be
valid.

The checks aren't perfect: some links look like they work, but the
actual page they go to doesn't have any relevant data on it.

I've hardcoded a bunch of hostnames that have gone defunct, usually
because the sites are gone, but some redirect to unrelated content.

There are also a few sites that respond to requests with 403s, but not
when I access them in my web browser.

Those sites (the defunct and the 403ing) are skipped, because there's no
point checking them, but the fact they're skipped is logged.

The data/urls.csv file was generated by running:

uv run -- scripts/check_urls.py -i mlox/mlox_base.txt.yaml -i mlox/mlox_base.danae.txt.yaml -i mlox/mlox_base_legacy.txt.yaml -i mlox/mlox_user.txt.yaml  -p data/manual_mod_plugins.csv -p data/mmh_fliggerty_plugins.csv -p data/nexus_mods_morrowind_plugins.csv

Then making some manual edits, running again with "-u data/urls.tsv -w" added, and that loop repeated several times.
2026-01-16 09:21:58 +00:00
..