From 94c83a7a6840b25814226e1a8e6da719890ffa63 Mon Sep 17 00:00:00 2001 From: WrinklyNinja Date: Tue, 6 Nov 2012 21:54:57 +0000 Subject: [PATCH] Initial commit of my design / planning notes. --- README | 341 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 README diff --git a/README b/README new file mode 100644 index 00000000..b04ff509 --- /dev/null +++ b/README @@ -0,0 +1,341 @@ +LOOT +==== + +Introduction +============ + +LOOT is BOSS v3. +See BOSS: http://code.google.com/p/better-oblivion-sorting-software/ + +LOOT has been given a different working name because it will be a very +different program to the one that people know and love as BOSS. + +LOOT is being designed from the ground up to address the shortcomings +that are inherent in how BOSS approaches load order optimisation. That's +another reason why it's being hosted on a different site, under a +different name, even though it is intended to be released as an update +to BOSS. While ideas and some code will be transferred, the distancing +will hopefully produce better results. + +So what's wrong with BOSS? Why do I feel the need to go back to the +drawing board? + +Well, Oblivion has accumulated around 25,000 mods on +Oblivion Nexus since its launch in 2006. Skyrim has accumulated around +19,000 mods on Skyrim Nexus since its launch in 2011. Growth models aside, +Skyrim has proved incredibly popular for modding. + +For BOSS to be as useful as possible, it needs to be able to sort as many +mods as possible. For Skyrim, we're getting user submissions of mods to +be sorted faster than we can add them, and our backlog is at 3,000+ +plugins and growing. That's after I've doubled the effective team size. + +The BOSS team's operational model is therefore unsustainable, and has +to change. The main issue is that it takes time for us to process all +these reports, and we only have so much free time: the team itself is the +weak point in the operating chain. It's a good idea, then, to automate +as much as possible, and so reduce the bottleneck. + + +LOOT Design Overview +==================== + +BOSS sorts mods according to their positions on a "masterlist", which is +essentially a massive 'correct' load order. This masterlist is maintained +by the BOSS team, with input from the community via user submission of +plugins to be added and suggested improvements for existing positions. + +To answer the question of how this may be automated, we first consider +how it is that a correct load order is obtained. A correct load order +satisfies the following conditions: + +1. It satisfies all the explicit and implicit dependencies of all plugins + in the load order. +2. It maximises the impact of each plugin on the list. Every plugin has + a purpose, and given any two conflicting plugins there is a load order + for them that has the highest 'impact factor'. Eg. if a mod contains 5 + armour records that conflict with another mod containing 10 armour + records and 15 weapon records, loading the first after the second will + result in a (100% + 80%)/2 = 90% impact, but loading the + second after the first will result in a (0% + 100%)/2 = 50% impact. + +The implicit dependencies mentioned in the first point are those that do +not appear in a plugin's master list, eg. target mods for landscape +patches. Such dependencies cannot be determined by examination of the +plugin's contents, but may be noted in the plugin's filename or in its +documentation. + +The second point has some additional subtleties associated with it: +it may be that a player may only want some changes from one mod, and some +from another, but the majority of changes from neither. It may also be +that while a plugin makes a lot of or a few changes, it is a low-priority +plugin by intent, eg. the Unofficial Patches are loaded early so that other +mods may override their fixes with other changes. There is probably +therefore an element of intent or user choice that cannot be determined from +examining the contents of the plugins. + +These two points mean that a fully analytic load order solver is impossible +given an arbitrary set of plugins, so LOOT cannot be totally automated. +Some means of transferring data on implicit dependencies, intents and +user choices is required. + +Data on implicit dependencies and author intentions are suitable for +mass distribution to all LOOT users, as they are universally applicable, +while user choices are personal and so unsuitable for mass distribution. +This split can be neatly codified into a database of "master rules" +(masterlist) and a database of "user rules" (userlist). The masterlist +could also be used to distribute various messages as BOSS's masterlist does. + +Most plugins will likely not require additional data, and so not require +masterlist rules, so this will remove most of the strain on BOSS team members. + + +LOOT Masterlist & Userlist +========================== + +As the purpose of LOOT's masterlist will be to provide additional data +for the sorting of plugins via algorithms, rather than to provide the +positions of plugins directly, the ordering of plugins loses semantic +meaning. As such, each plugin's entry becomes standalone from the rest +of the file, from a structure perspective. This leads to plugin grouping +also losing semantic meaning. + +The userlist broadly maintains the purpose it has in BOSS, but expands to +mirror the masterlist, so that the two are equivalent, but with the +userlist overriding the masterlist. Userlist rules will be valid masterlist +rules and vice-versa. + +To ease future development, the masterlist and userlist will be written +in YAML, which has a suitable mix of simplicity, power and flexibility. +The format is described later in this file. + +Below is a run-down of the masterlist and userlist feature differences +between BOSS and LOOT: + +Masterlist +---------- + +Plugins are unordered. No distinction is made in the syntax between regex +plugins and non-regex plugins. Detection of regex plugins will be done by +looking for a "\.esp" or "\.esm" in the filename. + +Groups will no longer exist. + +The requirement and incompatibility message types will be removed. Instead, +error messages will be generated if any requirements are not met or any +incompatibilities are present. + +The Bash Tag suggestion message type will be removed. LOOT will be Bash +Tag agnostic. This is because BOSS's Bash Tag suggestions are very patchy, +and it would be more effective for Bash to implement some sort of scanning +mechanism to determine the correct Bash Tags for a plugin than for LOOT to +do the same. + +Variables will no longer exist, as YAML's support for references and +aliases is far more flexible and powerful. 'Loose' file and message +definitions may be put at the beginning of the masterlist / userlist for +reference to later without requiring them to be attached to a plugin. + +Web Link support will be retained with its current functionality. + +Global message support will be retained with its current functionality. Global +messages will be required to be listed before all plugin entries. + +Conditionals will be retained, though the hardcoded placeholders will be removed +and replaced with YAML references. The 'VAR' and 'REGEX' condition types will be +removed as the former will be unnecesssary and the latter included in the 'FILE' +type, since all file objects will potentially contain regex strings. + +All conditionals will have their results cached. + +Userlist +-------- + +There will be no distinction between those that add new rules, those that +override existing rules, and those that only supply messages. + +Userlist rules will override masterlist rules on a per-component basis, +eg. if requirements are given, they will override the requirements given +in the masterlist, but if they are not given then the masterlist rule's +requirements will be used. + +Rules will not be able to reference groups at all, as they will no longer +exist. + +Rules will not be able to specify plugin positions, but will be able to +specify requirements, incompatibilities and priority instead. + + +LOOT User Experience +==================== + +LOOT will require a greater degree of user interaction than BOSS does, +so an interface that facilitates this is required: as such, LOOT will +not have a CLI. + +A general workflow for a user running LOOT to sort would be: + +1. Run LOOT: it auto-sorts based on plugin content plus and masterlist + and userlist rules. +2. LOOT finishes sorting, and displays the load order it sets. + Clicking on a plugin in the load order will highlight all those it + conflicts with. This provides a simple way for users to gauge conflicts + and so decide which plugin's changes they wish to apply to their games. + Dragging a plugin will display the positions to which it can be moved + (by dropping it there) whilst satisfying dependencies. + Plugins which have no conflicts are greyed out and cannot be selected, + to simplify this process. + There will be an information panel which will display the selected + plugin's requirements, incompatibilities, priority and messages. These + will be editable, upon exit of the load order window, the user will be + asked if they wish to submit any edits they make to these fields. + Edits to plugin positions will not be submitted. + If LOOT encounters any errors while sorting, they are displayed instead + of the load order. +3. The user exits the load order display, choosing either to apply or + discard changes. +4. LOOT asks the user if they want to submit their plugin data edits. If + so, it then asks the user to supply links for each of the plugins + with edited data. +4. LOOT then applies the resulting load order, saves any user choices + as userlist rules, and exits. + +Because auto-sorting will require significant computation (think CBash +Bashed Patch building), optimisations will be made where possible. For +example, if LOOT is run and the plugins have not changed since it was +last run, then it will use the last load order it calculated. It will +also do this if the only change was removal of plugin(s). Addition of +plugins will require a re-calculation, but it may be possible that only +a partial scan would be required. + +LOOT must also provide a means to manage user rules, to undo its +changes (only back one run though), and to edit its settings. + +The Userlist Syntax and Masterlist Syntax docs will be merged into a +"Rule Syntax" doc, as the syntax will be the same for both. The new doc +will be structured so that there is a progression in complexity from +simple rule syntax to complex rule syntax as the guide progresses. + + +Implementation +============== + +LOOT will be an ambitious project: to keep its code as simple as possible, +it will make use of external libraries to supply functionality where +possible, for example: + * yaml-cpp: for masterlist/userlist parsing. + * libespm: for plugin parsing. + * libloadorder: for reading and writing load order. + * cURL: for networking. + * wxWidgets: for the interface. + * Boost: for filesystem interaction, various helpful functions. + +As a reflection of this, LOOT's API will provide functions only for +accessing LOOT-specific functionality and data, eg. auto-sorting, +masterlist / userlist reading, masterlist download and user +edits submission. + +To provide greater flexibility in the case of the LOOT team having to +relocate their hosting, the locations of the online masterlists will +not be hardcoded, but instead specified in LOOT's ini file. More generally, +hardcoding of potential variables will be avoided where possible. + + +LOOT Database Format +==================== + +Simple Example +-------------- + +{{{ +--- +- name: MyMod.esp + req: + - name: AnotherMod.esp + - name: AnotherModPatch.esp + msg: + - type: say + content: "This is my mod." + - type: say + content: "This is a second message." +... +}}} + +Complex Example +--------------- + +Note: The below example does not support compound conditionals: +I'm still trying to work out how to fit them in. + +{{{ +--- + +# Datatypes: +# !condition {key: REQUIRED, type: REQUIRED, arg: [REQUIRED, OPTIONAL]} +# !file { condition: OPTIONAL, name: REQUIRED, version: OPTIONAL, mod: OPTIONAL} +# !message { condition: OPTIONAL, type: REQUIRED, content: REQUIRED} +# !plugin +# name: REQUIRED +# priority: OPTIONAL +# req: [OPTIONAL !file, OPTIONAL !file] +# inc: [OPTIONAL !file, OPTIONAL !file] +# msg: [OPTIONAL !message, OPTIONAL !message] +# +# Variables can be implemented as references to file and message objects. +# Datatypes don't need to be explicitly declared, since the parser that will +# be used (yaml-cpp) will allow the querying of nodes that don't exist. The +# parser will simply assume that the correct data type is being used and fail +# if a required node is missing. +# +# The req, inc and msg lists in the plugin datatype should be omitted if they +# are empty, but can hold any number > 0 of nodes. The req list is ordered, in +# the load order of its contents, omitting any nodes that are for files that are +# not .esp or .esm files. +# +# For the condition datatype, 'key' is either "IF" or "IFNOT", 'type' is one +# of 'FILE', 'CRC', 'VERSION', 'ACTIVE' or 'LANG'. Depending on the value of +# 'type', 'arg' can be one of the following: +# +# Type Arg +# FILE a single file object +# CRC a single file object +# VERSION a single file object, followed by '<', '=' or '>' as the second arg. +# ACTIVE a single file object +# LANG a language name string +# +# CRCs and version strings can both be used in !file data structures' 'version' +# nodes, but strings should be wrapped in double quotes and CRCs unquoted. CRCs +# must be preceded by '0x' to tell the YAML parser that it's a hex integer and +# not a string. +# +# In general, strings should be double-quoted for consistency: they may be +# unquoted if they use no YAML syntax characters or single-quoted if that is +# convenient though. I haven't really decided yet. For instance, messages +# that include a web link but no apostrophes are better single-quoted, as that +# requires less escaping. + + +# These are all treated as file datatypes by LOOT, but they are not file +# datatypes in YAML. +filevars: + - &GAME name: "TESV.exe" + - &MASTER {name: "Oblivion.esm", version: "1.2.416"} + - &SE name: "obse_loader.exe" + - &TEST {name: "Test.esp", version: 0x0537AB3C} + - &SE20 <<: [*SE, version: "0.0.20.1"] # The result is {name: "obse_loader.exe", version: "0.0.20.1"} + +msgvars: + - &OBSOLETE {type: SAY, content: "Obsolete. Remove and upgrade to the latest version."} + +globals: + - {type: SAY, content: "You're using LOOT!"} + +- name: Oblivion.esm + # Any unfulfilled 'req's will produce an error message, as will any 'inc's + # present. As an aside, it's OK for '#' to be the comment symbol, because + # while it can be used in filenames, plugins.txt also uses it for comments. + msg: + - {condition: {key: if, type: file, arg: *GAME}, type: WARN, content: "False alarm."} + - <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. } +... +}}}