diff --git a/File Format b/File Format new file mode 100644 index 00000000..27bcd50e --- /dev/null +++ b/File Format @@ -0,0 +1,217 @@ +LOOT Masterlist File Format +=========================== + +In YAML, the message object is: +--- +!message: + condition: OPTIONAL !condition + type: REQUIRED !!string + content: REQUIRED !!string +... +Allowed values for the 'type' node are "SAY", "WARN" or "ERROR". The semantics +of the types are: + say Generic message type, carries no semantics. + warn A warning message, detailing a non-critical installation issue. + error An error message, detailing a critical installation issue. + +The 'content' node string will treat all file:// , http:// and https:// links as +hyperlinks. A link can be given a label by enclosing it and the label in double +quotes, eg. "http://www.example.com example link". + + +The file object: +--- +!file: + condition: OPTIONAL !condition + name: REQUIRED !!string + ver: OPTIONAL !!int or !!string + mod: OPTIONAL !!string +... +CRCs are given as hex integers prefixed by 0x. The 'mod' node allows the name +of the parent mod to be used instead of the filename in messages. + + +The condition object: +--- +!condition: + exp: REQUIRED !!string + vars: REQUIRED [!file or !!string] +... +The [] denotes a list of the given type(s). +All 'file' objects used in a condition must have empty 'condition' nodes to +prevent recursion loops. +All conditions will have their results cached. +The 'exp' string must follow the following format: + +IF|IFNOT FILE|VERSION|ACTIVE|LANG [AND|OR IF|IFNOT FILE|VERSION|ACTIVE|LANG]* + +The four condition types are detailed below: + Type Args Details + FILE !file If the file object passed contains a CRC, this + condition checks for a CRC match, otherwise it + checks for existence. + VERSION !file, !!string The 'ver' node of the file object must contain + a version string. The string argument is '<', '=' or + '>', and is used to form an expression with the true + version on the left and the given version on the + right, which is then checked to see if it holds. + ACTIVE !file Checks if the given file is a plugin, and if that + plugin is active. + LANG !!string Checks to see if the given language string matches + the current language. + +The 'vars' node lists the variables that are used in the expression in the order +that they are to be used. + +The plugin object: +--- +!plugin: + name: REQUIRED !!string + priority: OPTIONAL !undecided + reqs: OPTIONAL [!file] + incs: OPTIONAL [!file] + msgs: OPTIONAL [!message] +... +The 'name' node can be either an exact filename or a regular expression. The +difference will be determined by looking for the "\.esp" or "\.esm" substrings, +which are only valid for a regular expression. +The type of the 'priority' node has not been decided yet, as it's not yet clear +how much fidelity is required for it. +If any of the 'reqs' node objects are not present, an error message will be +displayed. If any of the 'incs' node objects are present, an error message will +be displayed. +The 'reqs' node objects must be listed in load order. The 'reqs' and 'incs' +nodes may contain a mix of plugin and non-plugin objects. + + +LOOT Userlist Format +==================== + +Same as masterlist format, but the plugin object is: +--- +!plugin: + name: REQUIRED !!string + priority: OPTIONAL !undecided + enabled: OPTIONAL !bool + after: OPTIONAL !!string + reqs: OPTIONAL [!file] + incs: OPTIONAL [!file] + msgs: OPTIONAL [!message] +... +If the 'enabled' node is not specified, its value is assumed true. The 'after' +node is a tentative implementation of how users will specify custom positions, +and will take the filename of the plugin that the object plugin will load after. + + +Examples +======== + +Simple Example +-------------- + +--- +- name: MyMod.esp + req: + - name: AnotherMod.esp + - name: AnotherModPatch.esp + msg: + - type: say + content: "This is my mod." + - type: say + content: "This is a second message." +... + +Complex Example +--------------- + +--- +# Note that classes are not specified, YAML will treat them as assoc. arrays, +# LOOT will assume they are file objects. +filevars: + - &GAME name: "TESV.exe" + - &MASTER {name: "Oblivion.esm", ver: "1.2.416"} + - &SE name: "obse_loader.exe" + - &TEST {name: "Test.esp", ver: 0x0537AB3C} + - &SE20 <<: [*SE, ver: "0.0.20.1"] + +msgvars: + - &OBSOLETE {type: SAY, content: "Obsolete. Remove and upgrade to the latest version."} + +globals: + - {type: SAY, content: "You're using LOOT!"} + +- name: Oblivion.esm + msg: + - {condition: {exp: IF FILE, vars: *GAME}, type: WARN, content: "False alarm."} + - <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. } +... + + +LOOT Machine Database Format +============================ + +The formatting of this database has not yet been decided. It doesn't need to be +human-readable, but must contain the following information for each plugin: + + * Filename (for roughly identifying the file) + * CRC (for precisely identifying the exact file) + * Masters in the order they appear in the plugin's master list. + * FormIDs of the records it edits (not added records). + +However, the filename being a variable length byte string makes searching +through filenames more complicated than if a unique identifier of fixed length +generated from the filename were used. A hash is suitable, but it would have to +be of sufficient size to minimise the probability of hash collisions. There are +~25,000 mods for Oblivion on the Nexus, let's assume that each mod contains 4 +plugins: many contain only one, but many others contain lots of plugins. Factor +in some room for growth, since Skyrim will likely have many more mods than +Oblivion does at its age, and having an upper bound of ~10,000,000 plugins, +including different versions of the same plugin does not seem wholey +unreasonable. It's unlikely that the number of plugins would reach over order +10^6, but this is an upper bound, after all. + +A 32-bit hash has around 4 billion possible values, but due to the birthday +problem collisions become significant way below that, with there being a 1% +probability at ~10,000 values, assuming I've understood that right. A 64-bit +hash has a 1% probability at around 609 million values, so it's a much better +choice. + +As for the choice of hash function, it doesn't need to be cryptographically +secure, and speed is favoured. + +Since the use of the database is limited to searching for a specific file then +reading its data, it makes sense to optimise searching by storing all filename +hashes and CRCs in an index, with offsets to each file's data. + +A 4 GB file can store of order 10^8 index entries, or ~10^7 plugin entries +including the index. It might be prudent to allow for some leeway by specifying +offsets using 64 bit integers, just in case my upper bound is too low, but we're +talking about a database file over 4 GB big. There's no way anyone is +downloading that. If the upper bound is surpassed, we'll just have multiple +files. That's something worth considering earlier too, because we want to avoid +people having to download things they'll never use. + +Storing the index at the beginning of the file would require the entire file to +be rewritten whenever it is updated, so instead it will be stored at the end +of the file, so that a new entry can be appended then only the index rewritten. + +So anyway, the plugin index: + + Index entries index entry[Size] + Size uint32_t Last 4 bytes of the file. + +A plugin index entry: + + Filename hash uint64_t + File data CRC32 uint32_t + Data offset uint32_t From beginning of file. + +A plugin data entry: + + Size uint32_t Total size of data entry, not + including this value. + Masters length uint16_t Includes null characters. + Masters char * null-terminated filenames in order + of how they appeared in the plugin. + FormIDs uint32_t[len] len = (Size - Masters length) / 4 + diff --git a/README b/README index 2320c54a..20e0e561 100644 --- a/README +++ b/README @@ -34,7 +34,7 @@ The BOSS team's operational model is therefore unsustainable, and has to change. The main issue is that it takes time for us to process all these reports, and we only have so much free time: the team itself is the weak point in the operating chain. It's a good idea, then, to automate -as much as possible, and so reduce the bottleneck. +as much as possible, and so reduce the need for the team to do things. While that's being done, I might as well also make some improvements to other areas of BOSS. @@ -43,130 +43,68 @@ other areas of BOSS. LOOT Design Overview ==================== -BOSS sorts mods according to their positions on a "masterlist", which is -essentially a massive 'correct' load order. This masterlist is maintained -by the BOSS team, with input from the community via user submission of -plugins to be added and suggested improvements for existing positions. +LOOT's central design idea is that its functionality should be as independent of +external systems as possible. A system might be a human, a group of humans, or a +server somewhere. This rather vague idea can be broken down into a few design +goals: -To answer the question of how this may be automated, we first consider -how it is that a correct load order is obtained. A correct load order -satisfies the following conditions: + * LOOT should be able to figure out an optimum load order without any inputs + other than the plugins it will order. + * LOOT should be able to figure out a correct load order with a minimum of + supplementary data on author intent. + * LOOT should be able to obtain supplementary data from an arbitrary source, + or arbitrary sources, depending on the number of types of data. + * LOOT's effects should be customiseable by the user. + * LOOT should be scaleable, focussed and extensible. + +Note that an optimum load order is not necessarily a correct load order. If +we define the impact of a plugin as the number of its edits that get applied to +a person's game, then an optimum load order is one with the highest average +impact from its constituent plugins whilst satisfiying all explicit dependencies. +However, some plugins may be made with the intent that they are to be overridden +by others, and so a correct load order is one that also takes into account +author intent. Given a set of plugins, their correct load order may have a lower +average impact than their optimum load order. -1. It satisfies all the explicit and implicit dependencies of all plugins - in the load order. -2. It maximises the impact of each plugin on the list. Every plugin has - a purpose, and given any two conflicting plugins there is a load order - for them that has the highest 'impact factor'. Eg. if a mod contains 5 - armour records that conflict with another mod containing 10 armour - records and 15 weapon records, loading the first after the second will - result in a (100% + 80%)/2 = 90% impact, but loading the - second after the first will result in a (0% + 100%)/2 = 50% impact. +The design goal for a correct load order specifies the use of supplementary data +because the determination of author intent is not generally possible given only +a plugin. Such supplementary data might include any implicit masters, and +their intended order relative to any explicit masters. It might also include +whether the plugin was intended to be high or low priority. -The implicit dependencies mentioned in the first point are those that do -not appear in a plugin's master list, eg. target mods for landscape -patches. Such dependencies cannot be determined by examination of the -plugin's contents, but may be noted in the plugin's filename or in its -documentation. +Supplementary data can also include data not related to load ordering: one of +BOSS's great successes has been the highlighting of the issue of dirty plugins, +and the distribution of generally applicable information such as that is an +important feature to retain. -The second point has some additional subtleties associated with it: -it may be that a player may only want some changes from one mod, and some -from another, but the majority of changes from neither. It may also be -that while a plugin makes a lot of or a few changes, it is a low-priority -plugin by intent, eg. the Unofficial Patches are loaded early so that other -mods may override their fixes with other changes. There is probably -therefore an element of intent or user choice that cannot be determined from -examining the contents of the plugins. +The goal of arbitrary data sources is with LOOT's long-term success in mind. It +may be that the team responsible for it may suddenly drop off the face of the +Internet, as has happened with other utilities' teams in the past. In such a +circumstance, getting access to the data source LOOT uses may be difficult or +impossible, and so anyone wishing to pick the project up would have to set up +a new source, but if LOOT can't handle arbitrary data sources, then those +picking it up would also have to release a new version of LOOT and distribute +that. This may take a while if there is nobody available with the skills or +experience required. If LOOT can handle arbitrary data sources, then all that +would need to be done is for LOOT users to be made aware of the situation, and +for them to be given instructions on how to change the source LOOT looks at in +their copies. -These two points mean that a fully analytic load order solver is impossible -given an arbitrary set of plugins, so LOOT cannot be totally automated. -Some means of transferring data on implicit dependencies, intents and -user choices is required. +The goal of customiseability is because LOOT will never be able to automatically +get a load order quite how everyone wants it. There will always be an element of +choice involved in setting up a load order, so LOOT should accommodate for that. -Data on implicit dependencies and author intentions are suitable for -mass distribution to all LOOT users, as they are universally applicable, -while user choices are personal and so unsuitable for mass distribution. -This split can be neatly codified into a database of "master rules" -(masterlist) and a database of "user rules" (userlist). The masterlist -could also be used to distribute various messages as BOSS's masterlist does. +The final goal is basically to prevent the problems BOSS has happening again, by +implementing good systems design and coding practice. I'm always learning, and +now I realise some of the decisions I made working on BOSS weren't the best. I +probably won't make the best decisions this time around either, but they should +end up better. -Most plugins will likely not require additional data, and so not require -masterlist rules, so this will remove most of the strain on BOSS team members. - - -LOOT Masterlist & Userlist -========================== - -As the purpose of LOOT's masterlist will be to provide additional data -for the sorting of plugins via algorithms, rather than to provide the -positions of plugins directly, the ordering of plugins loses semantic -meaning. As such, each plugin's entry becomes standalone from the rest -of the file, from a structure perspective. This leads to plugin grouping -also losing semantic meaning. - -The userlist broadly maintains the purpose it has in BOSS, but expands to -mirror the masterlist, so that the two are equivalent, but with the -userlist overriding the masterlist. Userlist rules will be valid masterlist -rules and vice-versa. - -To ease future development, the masterlist and userlist will be written -in YAML, which has a suitable mix of simplicity, power and flexibility. -The format is described later in this file. - -Below is a run-down of the masterlist and userlist feature differences -between BOSS and LOOT: - -Masterlist ----------- - -Plugins are unordered. No distinction is made in the syntax between regex -plugins and non-regex plugins. Detection of regex plugins will be done by -looking for a "\.esp" or "\.esm" in the filename. - -Groups will no longer exist. - -The requirement and incompatibility message types will be removed. Instead, -error messages will be generated if any requirements are not met or any -incompatibilities are present. - -The Bash Tag suggestion message type will be removed. LOOT will be Bash -Tag agnostic. This is because BOSS's Bash Tag suggestions are very patchy, -and it would be more effective for Bash to implement some sort of scanning -mechanism to determine the correct Bash Tags for a plugin than for LOOT to -do the same. - -Variables will no longer exist, as YAML's support for references and -aliases is far more flexible and powerful. 'Loose' file and message -definitions may be put at the beginning of the masterlist / userlist for -reference to later without requiring them to be attached to a plugin. - -Web Link support will be retained with its current functionality. - -Global message support will be retained with its current functionality. Global -messages will be required to be listed before all plugin entries. - -Conditionals will be retained, though the hardcoded placeholders will be removed -and replaced with YAML references. The 'VAR' and 'REGEX' condition types will be -removed as the former will be unnecesssary and the latter included in the 'FILE' -type, since all file objects will potentially contain regex strings. - -All conditionals will have their results cached. - -Userlist --------- - -There will be no distinction between those that add new rules, those that -override existing rules, and those that only supply messages. - -Userlist rules will override masterlist rules on a per-component basis, -eg. if requirements are given, they will override the requirements given -in the masterlist, but if they are not given then the masterlist rule's -requirements will be used. - -Rules will not be able to reference groups at all, as they will no longer -exist. - -Rules will not be able to specify plugin positions, but will be able to -specify requirements, incompatibilities and priority instead. +I anticipate that by having LOOT be able to figure out an optimum load order +without any input from humans, ordering the vast majority of plugins correctly +can be fully automated, as most plugins make a small number of changes that +are highly compatible. More complex plugins will still require the maintainance +team to provide LOOT with supplementary data, but they are relatively few. LOOT User Experience @@ -174,7 +112,8 @@ LOOT User Experience LOOT will require a greater degree of user interaction than BOSS does, so an interface that facilitates this is required: as such, LOOT will -not have a CLI. +not have a CLI. A GUI that accepts command line parameters may be an option if +people want one. A general workflow for a user running LOOT to sort would be: @@ -235,10 +174,8 @@ accessing LOOT-specific functionality and data, eg. auto-sorting, masterlist / userlist reading, masterlist download and user edits submission. -To provide greater flexibility in the case of the LOOT team having to -relocate their hosting, the locations of the online masterlists will -not be hardcoded, but instead specified in LOOT's ini file. More generally, -hardcoding of potential variables will be avoided where possible. +The support for arbitrary data sources will be implemented by having the source +LOOT uses specified in a settings file, which will be editable via the GUI. Because auto-sorting requires LOOT to know what is inside the plugins it sorts, it must scan the plugins, and determine which of its records are new and which @@ -273,139 +210,37 @@ written by the LOOT code. The list of metadata that is maintained by the LOOT team will be referred to as the "masterlist", and the list of metadata that is individual to each user will be referred to as the "userlist". +LOOT Masterlist & Userlist +========================== -LOOT Masterlist Format -====================== +The purpose of the masterlist is to store the following data for plugins: -Simple Example --------------- + * Implicit and explicit requirements in load order. + * Incompatibilities, for catching unresolveable conflicts. + * Messages, for displaying information about the plugins. + * The intended priority of the plugin relative to others that make the same + changes. + * Possibly also Bash Tags to be applied. + +It is also useful to store some data not associated with plugins, such as: -{{{ ---- -- name: MyMod.esp - req: - - name: AnotherMod.esp - - name: AnotherModPatch.esp - msg: - - type: say - content: "This is my mod." - - type: say - content: "This is a second message." -... -}}} + * Global messages. + * Message objects, for reference instead of rewriting the same message + multiple times. + * File objects, for reference instead of rewriting the same condition + argument multiple times. + +All messages should also be optionally conditional on an arbitrary number of +conditions. More on that later. + +The purpose of the userlist is to store modifications to masterlist plugin data, +and to also store user-specified positions of plugins. The userlist must also +support the disabling of plugin entries within it, so that users can turn off +their changes without deleting them. -Complex Example ---------------- +As the masterlist and userlist are very similar, it makes sense for them to use +the same format. The userlist could then simply support two data fields that are +ignored by the masterlist parser. I think that YAML is a suitable format for +these files as it is simple, human-readable, yet quite powerful. -Note: The below example does not support compound conditionals: -I'm still trying to work out how to fit them in. - -{{{ ---- - -# Datatypes: -# !condition {key: REQUIRED, type: REQUIRED, arg: [REQUIRED, OPTIONAL]} -# !file { condition: OPTIONAL, name: REQUIRED, version: OPTIONAL, mod: OPTIONAL} -# !message { condition: OPTIONAL, type: REQUIRED, content: REQUIRED} -# !plugin -# name: REQUIRED -# priority: OPTIONAL -# req: [OPTIONAL !file, OPTIONAL !file] -# inc: [OPTIONAL !file, OPTIONAL !file] -# msg: [OPTIONAL !message, OPTIONAL !message] -# -# Variables can be implemented as references to file and message objects. -# Datatypes don't need to be explicitly declared, since the parser that will -# be used (yaml-cpp) will allow the querying of nodes that don't exist. The -# parser will simply assume that the correct data type is being used and fail -# if a required node is missing. -# -# The req, inc and msg lists in the plugin datatype should be omitted if they -# are empty, but can hold any number > 0 of nodes. The req list is ordered, in -# the load order of its contents, omitting any nodes that are for files that are -# not .esp or .esm files. -# -# For the condition datatype, 'key' is either "IF" or "IFNOT", 'type' is one -# of 'FILE', 'CRC', 'VERSION', 'ACTIVE' or 'LANG'. Depending on the value of -# 'type', 'arg' can be one of the following: -# -# Type Arg -# FILE a single file object -# CRC a single file object -# VERSION a single file object, followed by '<', '=' or '>' as the second arg. -# ACTIVE a single file object -# LANG a language name string -# -# CRCs and version strings can both be used in !file data structures' 'version' -# nodes, but strings should be wrapped in double quotes and CRCs unquoted. CRCs -# must be preceded by '0x' to tell the YAML parser that it's a hex integer and -# not a string. -# -# In general, strings should be double-quoted for consistency: they may be -# unquoted if they use no YAML syntax characters or single-quoted if that is -# convenient though. I haven't really decided yet. For instance, messages -# that include a web link but no apostrophes are better single-quoted, as that -# requires less escaping. - - -# These are all treated as file datatypes by LOOT, but they are not file -# datatypes in YAML. -filevars: - - &GAME name: "TESV.exe" - - &MASTER {name: "Oblivion.esm", version: "1.2.416"} - - &SE name: "obse_loader.exe" - - &TEST {name: "Test.esp", version: 0x0537AB3C} - - &SE20 <<: [*SE, version: "0.0.20.1"] # The result is {name: "obse_loader.exe", version: "0.0.20.1"} - -msgvars: - - &OBSOLETE {type: SAY, content: "Obsolete. Remove and upgrade to the latest version."} - -globals: - - {type: SAY, content: "You're using LOOT!"} - -- name: Oblivion.esm - # Any unfulfilled 'req's will produce an error message, as will any 'inc's - # present. As an aside, it's OK for '#' to be the comment symbol, because - # while it can be used in filenames, plugins.txt also uses it for comments. - msg: - - {condition: {key: if, type: file, arg: *GAME}, type: WARN, content: "False alarm."} - - <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. } -... -}}} - - -LOOT Machine Database Format -============================ - -The formatting of this database has not yet been decided. It doesn't need to be -human-readable, but must contain the following information for each plugin: - - * Filename - * CRC - * Masters in the order they appear in the plugin's master list. - * FormIDs of the records it edits (not added records). - -Filename can be a null-terminated byte string, as can the master filenames. The -CRC is a 32 bit unsigned integer, as are the FormIDs. To aid searching of the -database, it might be worth including the filename byte length as an 8 bit -unsigned integer preceding the filename, and including the total byte length of -the master and FormID lists as a 32 bit unsigned integer following the CRC. -This would allow the search to skip the bulk of the database file. - -So, one database entry might be as follows: - -{{{ -Filename length uint8_t Length includes null character. -Filename char[filename length + 1] Final char is NULL. -CRC uint32_t -Data size uint32_t Length of the Masters - + FormIDs sections. -Masters length uint32_t Length of the Masters section. - Includes null characters. -Masters char * A list of null-terminated - filenames. -FormIDs uint32_t[FormIDs number] -}}} - -It might be worth indexing the filename and CRCs, and storing offsets to the -data for each plugin. +Further details can be found in the LOOT File Format doc.