loot/libloot
0
0
Fork 0
mirror of https://github.com/loot/libloot.git synced 2026-07-27 14:16:01 -07:00
Code Issues Packages Projects Releases Wiki Activity

Notes updates.

Browse Source
This commit is contained in:
WrinklyNinja
2012-11-09 11:28:23 +00:00
parent 0143afac47
commit ccb8a1705b
2 changed files with 306 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files
File Format
+217
View File
@@ -0,0 +1,217 @@
LOOT Masterlist File Format
===========================
In YAML, the message object is:
---
!message:
condition: OPTIONAL !condition
type: REQUIRED !!string
content: REQUIRED !!string
...
Allowed values for the 'type' node are "SAY", "WARN" or "ERROR". The semantics
of the types are:
say Generic message type, carries no semantics.
warn A warning message, detailing a non-critical installation issue.
error An error message, detailing a critical installation issue.
The 'content' node string will treat all file:// , http:// and https:// links as
hyperlinks. A link can be given a label by enclosing it and the label in double
quotes, eg. "http://www.example.com example link".
The file object:
---
!file:
condition: OPTIONAL !condition
name: REQUIRED !!string
ver: OPTIONAL !!int or !!string
mod: OPTIONAL !!string
...
CRCs are given as hex integers prefixed by 0x. The 'mod' node allows the name
of the parent mod to be used instead of the filename in messages.
The condition object:
---
!condition:
exp: REQUIRED !!string
vars: REQUIRED [!file or !!string]
...
The [] denotes a list of the given type(s).
All 'file' objects used in a condition must have empty 'condition' nodes to
prevent recursion loops.
All conditions will have their results cached.
The 'exp' string must follow the following format:
IF|IFNOT FILE|VERSION|ACTIVE|LANG [AND|OR IF|IFNOT FILE|VERSION|ACTIVE|LANG]*
The four condition types are detailed below:
Type Args Details
FILE !file If the file object passed contains a CRC, this
condition checks for a CRC match, otherwise it
checks for existence.
VERSION !file, !!string The 'ver' node of the file object must contain
a version string. The string argument is '<', '=' or
'>', and is used to form an expression with the true
version on the left and the given version on the
right, which is then checked to see if it holds.
ACTIVE !file Checks if the given file is a plugin, and if that
plugin is active.
LANG !!string Checks to see if the given language string matches
the current language.
The 'vars' node lists the variables that are used in the expression in the order
that they are to be used.
The plugin object:
---
!plugin:
name: REQUIRED !!string
priority: OPTIONAL !undecided
reqs: OPTIONAL [!file]
incs: OPTIONAL [!file]
msgs: OPTIONAL [!message]
...
The 'name' node can be either an exact filename or a regular expression. The
difference will be determined by looking for the "\.esp" or "\.esm" substrings,
which are only valid for a regular expression.
The type of the 'priority' node has not been decided yet, as it's not yet clear
how much fidelity is required for it.
If any of the 'reqs' node objects are not present, an error message will be
displayed. If any of the 'incs' node objects are present, an error message will
be displayed.
The 'reqs' node objects must be listed in load order. The 'reqs' and 'incs'
nodes may contain a mix of plugin and non-plugin objects.
LOOT Userlist Format
====================
Same as masterlist format, but the plugin object is:
---
!plugin:
name: REQUIRED !!string
priority: OPTIONAL !undecided
enabled: OPTIONAL !bool
after: OPTIONAL !!string
reqs: OPTIONAL [!file]
incs: OPTIONAL [!file]
msgs: OPTIONAL [!message]
...
If the 'enabled' node is not specified, its value is assumed true. The 'after'
node is a tentative implementation of how users will specify custom positions,
and will take the filename of the plugin that the object plugin will load after.
Examples
========
Simple Example
--------------
---
- name: MyMod.esp
req:
- name: AnotherMod.esp
- name: AnotherModPatch.esp
msg:
- type: say
content: "This is my mod."
- type: say
content: "This is a second message."
...
Complex Example
---------------
---
# Note that classes are not specified, YAML will treat them as assoc. arrays,
# LOOT will assume they are file objects.
filevars:
- &GAME name: "TESV.exe"
- &MASTER {name: "Oblivion.esm", ver: "1.2.416"}
- &SE name: "obse_loader.exe"
- &TEST {name: "Test.esp", ver: 0x0537AB3C}
- &SE20 <<: [*SE, ver: "0.0.20.1"]
msgvars:
- &OBSOLETE {type: SAY, content: "Obsolete. Remove and upgrade to the latest version."}
globals:
- {type: SAY, content: "You're using LOOT!"}
- name: Oblivion.esm
msg:
- {condition: {exp: IF FILE, vars: *GAME}, type: WARN, content: "False alarm."}
- <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. }
...
LOOT Machine Database Format
============================
The formatting of this database has not yet been decided. It doesn't need to be
human-readable, but must contain the following information for each plugin:
* Filename (for roughly identifying the file)
* CRC (for precisely identifying the exact file)
* Masters in the order they appear in the plugin's master list.
* FormIDs of the records it edits (not added records).
However, the filename being a variable length byte string makes searching
through filenames more complicated than if a unique identifier of fixed length
generated from the filename were used. A hash is suitable, but it would have to
be of sufficient size to minimise the probability of hash collisions. There are
~25,000 mods for Oblivion on the Nexus, let's assume that each mod contains 4
plugins: many contain only one, but many others contain lots of plugins. Factor
in some room for growth, since Skyrim will likely have many more mods than
Oblivion does at its age, and having an upper bound of ~10,000,000 plugins,
including different versions of the same plugin does not seem wholey
unreasonable. It's unlikely that the number of plugins would reach over order
10^6, but this is an upper bound, after all.
A 32-bit hash has around 4 billion possible values, but due to the birthday
problem collisions become significant way below that, with there being a 1%
probability at ~10,000 values, assuming I've understood that right. A 64-bit
hash has a 1% probability at around 609 million values, so it's a much better
choice.
As for the choice of hash function, it doesn't need to be cryptographically
secure, and speed is favoured.
Since the use of the database is limited to searching for a specific file then
reading its data, it makes sense to optimise searching by storing all filename
hashes and CRCs in an index, with offsets to each file's data.
A 4 GB file can store of order 10^8 index entries, or ~10^7 plugin entries
including the index. It might be prudent to allow for some leeway by specifying
offsets using 64 bit integers, just in case my upper bound is too low, but we're
talking about a database file over 4 GB big. There's no way anyone is
downloading that. If the upper bound is surpassed, we'll just have multiple
files. That's something worth considering earlier too, because we want to avoid
people having to download things they'll never use.
Storing the index at the beginning of the file would require the entire file to
be rewritten whenever it is updated, so instead it will be stored at the end
of the file, so that a new entry can be appended then only the index rewritten.
So anyway, the plugin index:
Index entries index entry[Size]
Size uint32_t Last 4 bytes of the file.
A plugin index entry:
Filename hash uint64_t
File data CRC32 uint32_t
Data offset uint32_t From beginning of file.
A plugin data entry:
Size uint32_t Total size of data entry, not
including this value.
Masters length uint16_t Includes null characters.
Masters char * null-terminated filenames in order
of how they appeared in the plugin.
FormIDs uint32_t[len] len = (Size - Masters length) / 4
README
+89 -254
View File
@@ -34,7 +34,7 @@ The BOSS team's operational model is therefore unsustainable, and has
to change. The main issue is that it takes time for us to process all
these reports, and we only have so much free time: the team itself is the
weak point in the operating chain. It's a good idea, then, to automate
as much as possible, and so reduce the bottleneck.
as much as possible, and so reduce the need for the team to do things.
While that's being done, I might as well also make some improvements to
other areas of BOSS.
@@ -43,130 +43,68 @@ other areas of BOSS.
LOOT Design Overview
====================
BOSS sorts mods according to their positions on a "masterlist", which is
essentially a massive 'correct' load order. This masterlist is maintained
by the BOSS team, with input from the community via user submission of
plugins to be added and suggested improvements for existing positions.
LOOT's central design idea is that its functionality should be as independent of
external systems as possible. A system might be a human, a group of humans, or a
server somewhere. This rather vague idea can be broken down into a few design
goals:
To answer the question of how this may be automated, we first consider
how it is that a correct load order is obtained. A correct load order
satisfies the following conditions:
* LOOT should be able to figure out an optimum load order without any inputs
other than the plugins it will order.
* LOOT should be able to figure out a correct load order with a minimum of
supplementary data on author intent.
* LOOT should be able to obtain supplementary data from an arbitrary source,
or arbitrary sources, depending on the number of types of data.
* LOOT's effects should be customiseable by the user.
* LOOT should be scaleable, focussed and extensible.
Note that an optimum load order is not necessarily a correct load order. If
we define the impact of a plugin as the number of its edits that get applied to
a person's game, then an optimum load order is one with the highest average
impact from its constituent plugins whilst satisfiying all explicit dependencies.
However, some plugins may be made with the intent that they are to be overridden
by others, and so a correct load order is one that also takes into account
author intent. Given a set of plugins, their correct load order may have a lower
average impact than their optimum load order.
1. It satisfies all the explicit and implicit dependencies of all plugins
in the load order.
2. It maximises the impact of each plugin on the list. Every plugin has
a purpose, and given any two conflicting plugins there is a load order
for them that has the highest 'impact factor'. Eg. if a mod contains 5
armour records that conflict with another mod containing 10 armour
records and 15 weapon records, loading the first after the second will
result in a (100% + 80%)/2 = 90% impact, but loading the
second after the first will result in a (0% + 100%)/2 = 50% impact.
The design goal for a correct load order specifies the use of supplementary data
because the determination of author intent is not generally possible given only
a plugin. Such supplementary data might include any implicit masters, and
their intended order relative to any explicit masters. It might also include
whether the plugin was intended to be high or low priority.
The implicit dependencies mentioned in the first point are those that do
not appear in a plugin's master list, eg. target mods for landscape
patches. Such dependencies cannot be determined by examination of the
plugin's contents, but may be noted in the plugin's filename or in its
documentation.
Supplementary data can also include data not related to load ordering: one of
BOSS's great successes has been the highlighting of the issue of dirty plugins,
and the distribution of generally applicable information such as that is an
important feature to retain.
The second point has some additional subtleties associated with it:
it may be that a player may only want some changes from one mod, and some
from another, but the majority of changes from neither. It may also be
that while a plugin makes a lot of or a few changes, it is a low-priority
plugin by intent, eg. the Unofficial Patches are loaded early so that other
mods may override their fixes with other changes. There is probably
therefore an element of intent or user choice that cannot be determined from
examining the contents of the plugins.
The goal of arbitrary data sources is with LOOT's long-term success in mind. It
may be that the team responsible for it may suddenly drop off the face of the
Internet, as has happened with other utilities' teams in the past. In such a
circumstance, getting access to the data source LOOT uses may be difficult or
impossible, and so anyone wishing to pick the project up would have to set up
a new source, but if LOOT can't handle arbitrary data sources, then those
picking it up would also have to release a new version of LOOT and distribute
that. This may take a while if there is nobody available with the skills or
experience required. If LOOT can handle arbitrary data sources, then all that
would need to be done is for LOOT users to be made aware of the situation, and
for them to be given instructions on how to change the source LOOT looks at in
their copies.
These two points mean that a fully analytic load order solver is impossible
given an arbitrary set of plugins, so LOOT cannot be totally automated.
Some means of transferring data on implicit dependencies, intents and
user choices is required.
The goal of customiseability is because LOOT will never be able to automatically
get a load order quite how everyone wants it. There will always be an element of
choice involved in setting up a load order, so LOOT should accommodate for that.
Data on implicit dependencies and author intentions are suitable for
mass distribution to all LOOT users, as they are universally applicable,
while user choices are personal and so unsuitable for mass distribution.
This split can be neatly codified into a database of "master rules"
(masterlist) and a database of "user rules" (userlist). The masterlist
could also be used to distribute various messages as BOSS's masterlist does.
The final goal is basically to prevent the problems BOSS has happening again, by
implementing good systems design and coding practice. I'm always learning, and
now I realise some of the decisions I made working on BOSS weren't the best. I
probably won't make the best decisions this time around either, but they should
end up better.
Most plugins will likely not require additional data, and so not require
masterlist rules, so this will remove most of the strain on BOSS team members.
LOOT Masterlist & Userlist
==========================
As the purpose of LOOT's masterlist will be to provide additional data
for the sorting of plugins via algorithms, rather than to provide the
positions of plugins directly, the ordering of plugins loses semantic
meaning. As such, each plugin's entry becomes standalone from the rest
of the file, from a structure perspective. This leads to plugin grouping
also losing semantic meaning.
The userlist broadly maintains the purpose it has in BOSS, but expands to
mirror the masterlist, so that the two are equivalent, but with the
userlist overriding the masterlist. Userlist rules will be valid masterlist
rules and vice-versa.
To ease future development, the masterlist and userlist will be written
in YAML, which has a suitable mix of simplicity, power and flexibility.
The format is described later in this file.
Below is a run-down of the masterlist and userlist feature differences
between BOSS and LOOT:
Masterlist
----------
Plugins are unordered. No distinction is made in the syntax between regex
plugins and non-regex plugins. Detection of regex plugins will be done by
looking for a "\.esp" or "\.esm" in the filename.
Groups will no longer exist.
The requirement and incompatibility message types will be removed. Instead,
error messages will be generated if any requirements are not met or any
incompatibilities are present.
The Bash Tag suggestion message type will be removed. LOOT will be Bash
Tag agnostic. This is because BOSS's Bash Tag suggestions are very patchy,
and it would be more effective for Bash to implement some sort of scanning
mechanism to determine the correct Bash Tags for a plugin than for LOOT to
do the same.
Variables will no longer exist, as YAML's support for references and
aliases is far more flexible and powerful. 'Loose' file and message
definitions may be put at the beginning of the masterlist / userlist for
reference to later without requiring them to be attached to a plugin.
Web Link support will be retained with its current functionality.
Global message support will be retained with its current functionality. Global
messages will be required to be listed before all plugin entries.
Conditionals will be retained, though the hardcoded placeholders will be removed
and replaced with YAML references. The 'VAR' and 'REGEX' condition types will be
removed as the former will be unnecesssary and the latter included in the 'FILE'
type, since all file objects will potentially contain regex strings.
All conditionals will have their results cached.
Userlist
--------
There will be no distinction between those that add new rules, those that
override existing rules, and those that only supply messages.
Userlist rules will override masterlist rules on a per-component basis,
eg. if requirements are given, they will override the requirements given
in the masterlist, but if they are not given then the masterlist rule's
requirements will be used.
Rules will not be able to reference groups at all, as they will no longer
exist.
Rules will not be able to specify plugin positions, but will be able to
specify requirements, incompatibilities and priority instead.
I anticipate that by having LOOT be able to figure out an optimum load order
without any input from humans, ordering the vast majority of plugins correctly
can be fully automated, as most plugins make a small number of changes that
are highly compatible. More complex plugins will still require the maintainance
team to provide LOOT with supplementary data, but they are relatively few.
LOOT User Experience
@@ -174,7 +112,8 @@ LOOT User Experience
LOOT will require a greater degree of user interaction than BOSS does,
so an interface that facilitates this is required: as such, LOOT will
not have a CLI.
not have a CLI. A GUI that accepts command line parameters may be an option if
people want one.
A general workflow for a user running LOOT to sort would be:
@@ -235,10 +174,8 @@ accessing LOOT-specific functionality and data, eg. auto-sorting,
masterlist / userlist reading, masterlist download and user
edits submission.
To provide greater flexibility in the case of the LOOT team having to
relocate their hosting, the locations of the online masterlists will
not be hardcoded, but instead specified in LOOT's ini file. More generally,
hardcoding of potential variables will be avoided where possible.
The support for arbitrary data sources will be implemented by having the source
LOOT uses specified in a settings file, which will be editable via the GUI.
Because auto-sorting requires LOOT to know what is inside the plugins it sorts,
it must scan the plugins, and determine which of its records are new and which
@@ -273,139 +210,37 @@ written by the LOOT code. The list of metadata that is maintained by the LOOT
team will be referred to as the "masterlist", and the list of metadata that is
individual to each user will be referred to as the "userlist".
LOOT Masterlist & Userlist
==========================
LOOT Masterlist Format
======================
The purpose of the masterlist is to store the following data for plugins:
Simple Example
--------------
* Implicit and explicit requirements in load order.
* Incompatibilities, for catching unresolveable conflicts.
* Messages, for displaying information about the plugins.
* The intended priority of the plugin relative to others that make the same
changes.
* Possibly also Bash Tags to be applied.
It is also useful to store some data not associated with plugins, such as:
{{{
---
- name: MyMod.esp
req:
- name: AnotherMod.esp
- name: AnotherModPatch.esp
msg:
- type: say
content: "This is my mod."
- type: say
content: "This is a second message."
...
}}}
* Global messages.
* Message objects, for reference instead of rewriting the same message
multiple times.
* File objects, for reference instead of rewriting the same condition
argument multiple times.
All messages should also be optionally conditional on an arbitrary number of
conditions. More on that later.
The purpose of the userlist is to store modifications to masterlist plugin data,
and to also store user-specified positions of plugins. The userlist must also
support the disabling of plugin entries within it, so that users can turn off
their changes without deleting them.
Complex Example
---------------
As the masterlist and userlist are very similar, it makes sense for them to use
the same format. The userlist could then simply support two data fields that are
ignored by the masterlist parser. I think that YAML is a suitable format for
these files as it is simple, human-readable, yet quite powerful.
Note: The below example does not support compound conditionals:
I'm still trying to work out how to fit them in.
{{{
---
# Datatypes:
# !condition {key: REQUIRED, type: REQUIRED, arg: [REQUIRED, OPTIONAL]}
# !file { condition: OPTIONAL, name: REQUIRED, version: OPTIONAL, mod: OPTIONAL}
# !message { condition: OPTIONAL, type: REQUIRED, content: REQUIRED}
# !plugin
# name: REQUIRED
# priority: OPTIONAL
# req: [OPTIONAL !file, OPTIONAL !file]
# inc: [OPTIONAL !file, OPTIONAL !file]
# msg: [OPTIONAL !message, OPTIONAL !message]
#
# Variables can be implemented as references to file and message objects.
# Datatypes don't need to be explicitly declared, since the parser that will
# be used (yaml-cpp) will allow the querying of nodes that don't exist. The
# parser will simply assume that the correct data type is being used and fail
# if a required node is missing.
#
# The req, inc and msg lists in the plugin datatype should be omitted if they
# are empty, but can hold any number > 0 of nodes. The req list is ordered, in
# the load order of its contents, omitting any nodes that are for files that are
# not .esp or .esm files.
#
# For the condition datatype, 'key' is either "IF" or "IFNOT", 'type' is one
# of 'FILE', 'CRC', 'VERSION', 'ACTIVE' or 'LANG'. Depending on the value of
# 'type', 'arg' can be one of the following:
#
# Type Arg
# FILE a single file object
# CRC a single file object
# VERSION a single file object, followed by '<', '=' or '>' as the second arg.
# ACTIVE a single file object
# LANG a language name string
#
# CRCs and version strings can both be used in !file data structures' 'version'
# nodes, but strings should be wrapped in double quotes and CRCs unquoted. CRCs
# must be preceded by '0x' to tell the YAML parser that it's a hex integer and
# not a string.
#
# In general, strings should be double-quoted for consistency: they may be
# unquoted if they use no YAML syntax characters or single-quoted if that is
# convenient though. I haven't really decided yet. For instance, messages
# that include a web link but no apostrophes are better single-quoted, as that
# requires less escaping.
# These are all treated as file datatypes by LOOT, but they are not file
# datatypes in YAML.
filevars:
- &GAME name: "TESV.exe"
- &MASTER {name: "Oblivion.esm", version: "1.2.416"}
- &SE name: "obse_loader.exe"
- &TEST {name: "Test.esp", version: 0x0537AB3C}
- &SE20 <<: [*SE, version: "0.0.20.1"] # The result is {name: "obse_loader.exe", version: "0.0.20.1"}
msgvars:
- &OBSOLETE {type: SAY, content: "Obsolete. Remove and upgrade to the latest version."}
globals:
- {type: SAY, content: "You're using LOOT!"}
- name: Oblivion.esm
# Any unfulfilled 'req's will produce an error message, as will any 'inc's
# present. As an aside, it's OK for '#' to be the comment symbol, because
# while it can be used in filenames, plugins.txt also uses it for comments.
msg:
- {condition: {key: if, type: file, arg: *GAME}, type: WARN, content: "False alarm."}
- <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. }
...
}}}
LOOT Machine Database Format
============================
The formatting of this database has not yet been decided. It doesn't need to be
human-readable, but must contain the following information for each plugin:
* Filename
* CRC
* Masters in the order they appear in the plugin's master list.
* FormIDs of the records it edits (not added records).
Filename can be a null-terminated byte string, as can the master filenames. The
CRC is a 32 bit unsigned integer, as are the FormIDs. To aid searching of the
database, it might be worth including the filename byte length as an 8 bit
unsigned integer preceding the filename, and including the total byte length of
the master and FormID lists as a 32 bit unsigned integer following the CRC.
This would allow the search to skip the bulk of the database file.
So, one database entry might be as follows:
{{{
Filename length uint8_t Length includes null character.
Filename char[filename length + 1] Final char is NULL.
CRC uint32_t
Data size uint32_t Length of the Masters
+ FormIDs sections.
Masters length uint32_t Length of the Masters section.
Includes null characters.
Masters char * A list of null-terminated
filenames.
FormIDs uint32_t[FormIDs number]
}}}
It might be worth indexing the filename and CRCs, and storing offsets to the
data for each plugin.
Further details can be found in the LOOT File Format doc.