diff --git a/File Format b/File Format index f8f06264..ee379b1c 100644 --- a/File Format +++ b/File Format @@ -1,5 +1,15 @@ -LOOT Masterlist File Format -=========================== +LOOT Metadata File Format +========================== + +The LOOT metadata file format is a YAML file which contains a specific set of +data in a specific structure. It is used by the masterlist, the userlist and the +lastrun file. + +General properties are: + 1. Keys and keywords (such as message types, conditional types) are lowercase. + 2. Whitespace is important: a two-space indentation should be used between + each level. + 3. All YAML properties. In YAML, the message object is: --- @@ -7,105 +17,86 @@ In YAML, the message object is: condition: OPTIONAL !condition type: REQUIRED !!string content: REQUIRED !!string + lang: OPTIONAL !!string ... -Allowed values for the 'type' node are "SAY", "WARN" or "ERROR". The semantics +Allowed values for the 'type' node are "say", "warn" or "error". The semantics of the types are: - say Generic message type, carries no semantics. - warn A warning message, detailing a non-critical installation issue. - error An error message, detailing a critical installation issue. - + say Generic message type, carries no semantics. + warn A warning message, detailing a non-critical installation issue. + error An error message, detailing a critical installation issue. + The 'content' node string will treat all file:// , http:// and https:// links as hyperlinks. A link can be given a label by enclosing it and the label in double quotes, eg. "http://www.example.com example link". +The 'lang' node takes an ISO 639-3 language string, of which there are a set defined for the languages LOOT supports (as yet, just 'eng'). The file object: --- !file: - condition: OPTIONAL !condition - name: REQUIRED !!string - ver: OPTIONAL !!int or !!string - mod: OPTIONAL !!string + condition: OPTIONAL !condition + name: REQUIRED !!string + ver: OPTIONAL !!int or !!string + display: OPTIONAL !!string + operator: OPTIONAL !!string ... -CRCs are given as hex integers prefixed by 0x. The 'mod' node allows the name -of the parent mod to be used instead of the filename in messages. +CRCs are given as hex integers prefixed by 0x. The 'display' node allows the name +of the parent mod or program to be used instead of the filename in messages. +'operator' is the comparison operator used for the version check (if performed). +'>=' is the default used if not specified, other possibilities include =, <, etc. - -The condition object: +The tag object: --- -!condition: - exp: REQUIRED !!string - vars: REQUIRED [!file or !!string] +!tag: + condition: OPTIONAL !condition + name: !!string ... -The [] denotes a list of the given type(s). -All 'file' objects used in a condition must have empty 'condition' nodes to -prevent recursion loops. -All conditions will have their results cached. -The 'exp' string must follow the following format: +This object can be used to represent a conditional tag. 'name' is the tag name, +eg. Graphics. -IF|IFNOT FILE|VERSION|ACTIVE|LANG [AND|OR IF|IFNOT FILE|VERSION|ACTIVE|LANG]* -The four condition types are detailed below: - Type Args Details - FILE !file If the file object passed contains a CRC, this - condition checks for a CRC match, otherwise it - checks for existence. - VERSION !file, !!string The 'ver' node of the file object must contain - a version string. The string argument is '<', '=' or - '>', and is used to form an expression with the true - version on the left and the given version on the - right, which is then checked to see if it holds. - ACTIVE !file Checks if the given file is a plugin, and if that - plugin is active. - LANG !!string Checks to see if the given language string matches - the current language. +The condition string: +--- +!condition: REQUIRED !!string +... +Syntax of the string is the same as is currently for BOSS, except the following: + 1. The VAR, REGEX and LANG condition types have been removed. + 2. The FILE condition type now evaluates regex path strings. + 3. The '&&' and '||' operators have been replaced with 'and' and 'or' respectively. + 4. Condition keywords, types and operators are all lowercase. + 5. Brackets may be used to logically group conditions in compound conditions. + 6. Evaluation of compound conditionals is done according to standard operator + precedence rules. -The 'vars' node lists the variables that are used in the expression in the order -that they are to be used. The plugin object: --- !plugin: - name: REQUIRED !!string - priority: OPTIONAL !undecided - reqs: OPTIONAL [!file] - incs: OPTIONAL [!file] - msgs: OPTIONAL [!message] + name: REQUIRED !!string + priority: OPTIONAL !undecided + enabled: OPTIONAL !bool + after: OPTIONAL [!file] + req: OPTIONAL [!file] + inc: OPTIONAL [!file] + msg: OPTIONAL [!message] + tag: OPTIONAL [!!string or !tag] ... -The 'name' node can be either an exact filename or a regular expression. The -difference will be determined by looking for the "\.esp" or "\.esm" substrings, +The 'name' node can be either an exact filename or a regular expression. The +difference will be determined by looking for the "\.esp" or "\.esm" substrings, which are only valid for a regular expression. The type of the 'priority' node has not been decided yet, as it's not yet clear how much fidelity is required for it. -If any of the 'reqs' node objects are not present, an error message will be +If any of the 'req' node objects are not present, an error message will be displayed. If any of the 'incs' node objects are present, an error message will be displayed. -The 'reqs' node objects must be listed in load order. The 'reqs' and 'incs' +The 'req' node objects must be listed in load order. The 'req' and 'inc' nodes may contain a mix of plugin and non-plugin objects. +If the 'enabled' node is not specified, its value is assumed true. It is +provided for use in the userlist. +The 'after' node is a list of plugins that the object plugin must load after, intended for use specifying implicit dependencies. It may also be used in the userlist to specify custom positions. +The 'tag' node holds an array of Bash Tags to be suggested. Tags that are prepended with a '-' (eg. '-Graphics') are suggested for removal, otherwise tags are suggested for addition. -LOOT Userlist Format -==================== - -Same as masterlist format, but the plugin object is: ---- -!plugin: - name: REQUIRED !!string - priority: OPTIONAL !undecided - enabled: OPTIONAL !bool - after: OPTIONAL !!string - reqs: OPTIONAL [!file] - incs: OPTIONAL [!file] - msgs: OPTIONAL [!message] -... -If the 'enabled' node is not specified, its value is assumed true. The 'after' -node is a tentative implementation of how users will specify custom positions, -and will take the filename of the plugin that the object plugin will load after. - - -Examples -======== - Simple Example -------------- @@ -121,6 +112,7 @@ Simple Example content: "This is a second message." ... + Complex Example --------------- @@ -142,107 +134,118 @@ globals: - name: Oblivion.esm msg: - - {condition: {exp: IF FILE, vars: *GAME}, type: WARN, content: "False alarm."} + - {condition: exp: "IF FILE("TESV.exe")", type: WARN, content: "False alarm."} - <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. } ... -LOOT Machine Database Format -============================ +LOOT Plugin Data Cache Format +============================= -The formatting of this database doesn't need to be human-readable, but must +The formatting of this cache doesn't need to be human-readable, but must contain the following information for each plugin: * Filename (for roughly identifying the file) - * CRC (for precisely identifying the exact file) + * CRC (for precisely identifying the exact file) * Masters in the order they appear in the plugin's master list. * FormIDs of the records it edits (not added records). - -Rather than have a single database file, it is more efficient past the short -term to use a different file for each plugin filename's data, even after taking -into account the additional networking cost of using many small files. See -the section after the divider below for the analysis. -Each database file will be named filename.txt, where filename is a plugin -filename including extension, eg. "Unofficial Oblivion Patch.esp.txt". In each -database file, the data will be divided into entries for each of the different -versions of that file, which are handled using their CRCs. These entries make -up the bulk of the file, and are followed by an index of CRCs and offsets to -their corresponding data at the end of the file. +Rather than have a single cache file, it is more efficient past the short +term to use a different file for each plugin filename's data, even after +taking into account the additional networking cost of using many small +files. See the section after the divider below for the analysis. + +The cache file extension will be ".dc", an acronym for "data cache". Each +cache file will share the filename of the plugin that it holds the data of, +including the plugin's file extension, eg. "Unofficial Oblivion Patch.esp.dc". + +In each cache file, the data will be divided into entries for each of the +different versions of that file, which are handled using their CRCs. These +entries make up the bulk of the file, and are followed by an index of CRCs +and offsets to their corresponding data at the end of the file. + + Name Type Details + --------------------------------------------------------------------------- + Version uint8_t Version number for the file's + format (in case it changes). + Name Length uint8_t The plugin filename's length. + Filename char[Name Length] Not null terminated. + Data Entries data entry[Entry Count] + Index index entry[Entry Count] + Entry Count uint32_t - Name Type Details - --------------------------------------------------------------------------- - Data Entries data entry[Entry Count] - Index index entry[Entry Count] - Entry Count uint32_t - Each data entry has the following structure: - Name Type Details - --------------------------------------------------------------------------- - Size uint32_t Total size of data entry, not - including this value. - Masters Length uint16_t Includes null characters. - Masters char[Masters Length] Null-terminated filenames in order - of how they appeared in the plugin. - FormIDs uint32_t[len] len = (Size - Masters Length) / 4 + Name Type Details + --------------------------------------------------------------------------- + Size uint32_t Total size of data entry, not + including this value. + Masters Length uint16_t Includes null characters. + Masters char[Masters Length] Null-terminated filenames in order + of how they appeared in the plugin. + FormIDs uint32_t[len] len = (Size - Masters Length) / 4 Each index entry has the following structure: - Name Type Details - --------------------------------------------------------------------------- - File CRC uint32_t - Data offset uint32_t From beginning of file. - + Name Type Details + --------------------------------------------------------------------------- + File CRC uint32_t + Data offset uint32_t From beginning of file. + All numbers are encoded low to high byte. All strings are encoded in UTF-8. The reason for this structure is to optimise the processes that will be -performed upon database files: searching, reading data and appending new data. +performed upon cache files: searching, reading data and appending new data. The index allows quick searching of the file as it is composed of a given -number of entries with defined size. It is located at the end of the file rather -than the more usual beginning so that appending new data does not need to -rewrite the whole file to also update the index: only the index will ever be -rewritten. +number of entries with defined size. It is located at the end of the file +rather than the more usual beginning so that appending new data does not +need to rewrite the whole file to also update the index: only the index and +filename data will ever be rewritten. -Each database file will be able to store up to ~10^7 entries, though the -probability of there being a CRC collision reaches 1% at ~10^4 entries, due to -the birthday problem. Still, it's unlikely that there will ever be several -thousand versions of a file with the same filename in use, so it's probably not -worth worrying about. +The plugin filename is contained within the cache file so that in case the +cache file's filename ever gets changed, it is still possible to work out +which plugins' data it holds. + +Each cache file will be able to store up to ~10^7 entries, though the +probability of there being a CRC collision reaches 1% at ~10^4 entries, due +to the birthday problem. Still, it's unlikely that there will ever be +several thousand versions of a file with the same filename in use, so it's +probably not worth worrying about. Single vs. Multi-File --------------------- -A single-file database would use the above structure, but with an additional -uint64_t in each index entry to store a filename hash. +A single-file cache would use the above structure, but with an additional +uint64_t in each index entry to store a filename hash, and without the filename +at the start of the file. -Let's say the database file holds info for 100,000 plugins (25,000 mods, each +Let's say the cache file holds info for 100,000 plugins (25,000 mods, each with two plugins, and having two versions released - probably still a -conservative estimate). Assuming each plugin changes 10 records, and has 2 +conservative estimate). Assuming each plugin changes 10 records, and has 2 masters with filenames 10 characters long, the size is ~11.4 MB. -Using a multi-file database and the same assumptions being made as above, +Using a multi-file cache and the same assumptions being made as above, the size of each file will be 390 bytes. However, because we're dealing with lots of small files, HTTP request/response -time has to be accounted for. Google says that the best average for the -upload/download speed ratio of most Internet connections is 1:4, and that -headers are roughly 300 bytes long without cookies. Each file's effective size -is therefore 1890 bytes. +time has to be accounted for. Google says that the best average for the +upload/download speed ratio of most Internet connections is 1:4, and that +headers are roughly 300 bytes long without cookies. Each file's effective size +is therefore 1890 bytes. For a load order of 250 plugins, the total effective size is ~461 kB. This is -~25x smaller than the equivalent single-file database that would need to be +~25x smaller than the equivalent single-file cache that would need to be transferred, but it doesn't take into account the HTTP connection time, which is significant for small files. I don't know how I can take that into account, but if I've understood things correctly, I don't need to, as a persistent connection could be used, which would require only one HTTP connection to be -made. HTTP pipelining could also be used to improve performance. A multi-file -database could also be cached, with new files only being downloaded if they had -changed or weren't already on the user's system, rather than the whole database -having to be re-downloaded even if the only change was the addition of a plugin -the user doesn't have. This would only save 95 kB over the assumed load order at +made. HTTP pipelining could also be used to improve performance. A multi-file +cache could also be cached, with new files only being downloaded if they had +changed or weren't already on the user's system, rather than the whole cache +having to be re-downloaded even if the only change was the addition of a plugin +the user doesn't have. This would only save 95 kB over the assumed load order at maximum though. -Given all that, it's probably better to use a multi-file database over a single -file database. +Given all that, it's probably better to use a multi-file cache over a single +file cache. diff --git a/README b/README index 4da4b6c7..13a294c9 100644 --- a/README +++ b/README @@ -52,17 +52,17 @@ goals: other than the plugins it will order. * LOOT should be able to figure out a correct load order with a minimum of supplementary data on author intent. - * LOOT should be able to obtain supplementary data from an arbitrary source, + * LOOT should be able to obtain supplementary data from an arbitrary source, or arbitrary sources, depending on the number of types of data. * LOOT's effects should be customiseable by the user. * LOOT should be scaleable, focussed and extensible. - -Note that an optimum load order is not necessarily a correct load order. If + +Note that an optimum load order is not necessarily a correct load order. If we define the impact of a plugin as the number of its edits that get applied to a person's game, then an optimum load order is one with the highest average impact from its constituent plugins whilst satisfiying all explicit dependencies. However, some plugins may be made with the intent that they are to be overridden -by others, and so a correct load order is one that also takes into account +by others, and so a correct load order is one that also takes into account author intent. Given a set of plugins, their correct load order may have a lower average impact than their optimum load order. @@ -117,21 +117,26 @@ people want one. A general workflow for a user running LOOT to sort would be: -1. Run LOOT: it auto-sorts based on plugin content plus and masterlist - and userlist rules. -2. LOOT finishes sorting, and displays the load order it sets. +1. Run LOOT: it auto-sorts based on plugin content plus supplementary data. +2. LOOT finishes sorting, and displays the load order it suggests. At this point + the load order has not yet been applied. + Clicking on a plugin in the load order will highlight all those it conflicts with. This provides a simple way for users to gauge conflicts and so decide which plugin's changes they wish to apply to their games. + Dragging a plugin will display the positions to which it can be moved (by dropping it there) whilst satisfying dependencies. + Plugins which have no conflicts are greyed out and cannot be selected, to simplify this process. + There will be an information panel which will display the selected plugin's requirements, incompatibilities, priority and messages. These - will be editable, upon exit of the load order window, the user will be - asked if they wish to submit any edits they make to these fields. - Edits to plugin positions will not be submitted. + will be editable, and upon exit of the load order window, the user will be + asked if they wish to submit any edits they make to these fields. Edits to + plugin positions will not be submitted. + If LOOT encounters any errors while sorting, they are displayed instead of the load order. 3. The user exits the load order display, choosing either to apply or @@ -139,21 +144,20 @@ A general workflow for a user running LOOT to sort would be: 4. LOOT asks the user if they want to submit their plugin data edits. If so, it then asks the user to supply links for each of the plugins with edited data. -4. LOOT then applies the resulting load order, saves any user choices - as userlist rules, and exits. +4. LOOT then applies the resulting load order, saves any user choices, and + exits. -LOOT must also provide a means to manage user rules, to undo its +LOOT must also provide a means to manage past user choices, to undo its changes (only back one run though), and to edit its settings. -The Userlist Syntax and Masterlist Syntax docs will be merged into a -"Rule Syntax" doc, as the syntax will be the same for both. The new doc -will be structured so that there is a progression in complexity from -simple rule syntax to complex rule syntax as the guide progresses. +LOOT should ask for permission to send plugin contents data on first run, and +to send plugin metadata whenever there are metadata changes to send, unless the +user selects a "do not ask me again" option. These settings should be +configurable at any time. -The drag 'n' drop plugin positioning will likely require userlist rules -to support positioning of plugins precisely relative to other plugins, but it -may be possible to achieve the positioning using only the priority setting. If -possible, that is preferred. +Documentation will be split between three readmes: one general readme, one API +readme and one rule syntax readme. The version history can be split into a +fourth readme when it grows sufficiently long. Implementation @@ -165,7 +169,8 @@ possible, for example: * yaml-cpp: for masterlist/userlist parsing. * libespm: for plugin parsing. * libloadorder: for reading and writing load order. - * cURL: for networking. + * libsvn: for accessing the Subversion repositories used. + * cURL: for misc. networking. * wxWidgets: for the interface. * Boost: for filesystem interaction, various helpful functions. @@ -196,19 +201,19 @@ used in the following manner: plugins, the database, the masterlist and the userlist. 6. The data from the scanned plugins is appended to the database file, which is then uploaded to its hosted location. - -In addition to having this database file as a sort of permanent and -self-updating cache, optimisations can also be made to skip load order + +In addition to having this database file as a sort of permanent and +self-updating cache, optimisations can also be made to skip load order calculation. For example, if the only changes from the last load order set by LOOT is that a plugin or plugins have been removed, then the last load order may be used. It might also be possible to only perform partial calculation when plugins are added, though this is not certain. Note: For clarity, the database that holds the results of plugin scanning will -be referred to as the "machine database" (or MDB) as it will only be read and -written by the LOOT code. The list of metadata that is maintained by the LOOT -team will be referred to as the "masterlist", and the list of metadata that is -individual to each user will be referred to as the "userlist". +be referred to as the "cache". The list of metadata that is maintained by the +LOOT team will be referred to as the "masterlist", and the list of metadata that +is individual to each user will be referred to as the "userlist". + LOOT Masterlist & Userlist ========================== @@ -221,7 +226,7 @@ The purpose of the masterlist is to store the following data for plugins: * The intended priority of the plugin relative to others that make the same changes. * Possibly also Bash Tags to be applied. - + It is also useful to store some data not associated with plugins, such as: * Global messages. @@ -229,10 +234,10 @@ It is also useful to store some data not associated with plugins, such as: multiple times. * File objects, for reference instead of rewriting the same condition argument multiple times. - -All messages should also be optionally conditional on an arbitrary number of -conditions. More on that later. - + +All messages and files should also be optionally conditional on an arbitrary +number of conditions. + The purpose of the userlist is to store modifications to masterlist plugin data, and to also store user-specified positions of plugins. The userlist must also support the disabling of plugin entries within it, so that users can turn off @@ -243,16 +248,332 @@ the same format. The userlist could then simply support two data fields that are ignored by the masterlist parser. I think that YAML is a suitable format for these files as it is simple, human-readable, yet quite powerful. -Further details can be found in the LOOT File Format doc. +The drag 'n' drop plugin positioning will likely require userlist rules +to support positioning of plugins precisely relative to other plugins, but it +may be possible to achieve the positioning using only the priority setting. If +possible, that is preferred. + +Further details can be found in the File Format doc. Networking ========== -LOOT needs to be able to download and upload specific files from a VCS -repository. (VCS to help deal with possible spam/vandalism, and to track -changes.) It also needs to be able to send/receive data from some sort of issue -tracker. +LOOT has four target networking functionalities: -As LOOT needs to do this for users without their own credentials for the -repository, it needs to do this as a robot. + * Remote / local cache sync. + * Masterlist updating. + * User feedback submission. + +None of these functionalities are critical to usage of LOOT. + +The masterlist and cache repositories will be Subversion repositories, as this +allows users to most easily download only the files they require, and only the +diffs of those files. + +As LOOT needs commit as well as checkout access to the cache repository, for +users without their own credentials, it must access the repository using a +robot account. Google Code does not allow the use of robot accounts, but +Sourceforge does. However, the actions performed by the robot account are the responsibility of the account creator. This brings up the issue of account +abuse. This will be covered in the 'Security' section below. + +Conflicts between different users' commits will probably be an issue. +Because the database files are binary, not text, and because they use an index +with data offsets, they can't be merged automatically. The database file +workflow is: + 1. User svn updates their working copy of a file. + 2. User performs scanning actions. + 3. User edits their working copy of the file. + 4. User commits their edit to the repository. + +If another user changes the same file between steps 1 and 4, the commit will +fail. If this happens, LOOT should svn revert followed by svn update, then +re-apply the user's edits to the updated file, checking to make sure that the +updated file doesn't already include the same data. A second commit should then +be attempted. If this commit also fails, it's best just to discard the edits. +While that means the user won't benefit from having a local cache, it becomes +too complicated to store the edits on disk and re-apply them for a potentially +uncapped number of failures. + +The database repository may store tens if not hundreds of thousands of files, +so they need to be organised well to ease browsing. Grouping the files into +the game they are for, then subgrouping them into folders based on the first +character of their filename, then subgrouping again based on the second +character would probably do. For example: + +repo/ + Oblivion/ + o/ + b/ + oblivion.esm.bpdc + +Files and the 'letter folders' will be in lowercase. + +"repo" does not have to be the repository root, but LOOT's settings should +use the URL pointing to that location. The repository will be required to use +the same game folder names as LOOT's install structure (see the section below). + +The masterlists would be stored using a folder structure that mirrors LOOT's +install structure, for example: + +repo/ + Oblivion/ + masterlist.yaml + +"repo" does not have to be the repository root, but LOOT's settings should +use the URL pointing to that location. The repo will be required to use the same +game folder names as LOOT's install structure. + +The masterlist update mechanism will feature a validator such that if the +masterlist downloaded cannot be parsed, it will not replace the current +masterlist (if exists). The log will display notification of validation failure +with instructions to inform the maintainers of this issue. This is a fairly +major improvement over BOSS currently, which becomes unusable when a masterlist +update is occassionally broken. + +The final bit of networking to be considered is the feedback mechanism. It's not +an initial development target, as initially I think it would be better for all +communication between users and the team to be two-way, which is easier +accomplished through our forum threads. It's probably best to leave design of +this until we see what sort of feedback users are giving. + +A program update checker is notably absent, as a fully automated updater is +not worth the trouble of implementing well, and notification of an available +update is easily performed through a global masterlist message and so does not +require an additional transport mechanism. + + +Security +======== + +The abuse of the database repository credentials to be distributed with LOOT is +a very real concern. Steps must be taken to prevent or minimise the associated +risk. Unfortunately, I have no experience with this sort of thing, so I could +be making some incorrect assumptions below. + +One possible way to make obtaining the SSH key for use outside of LOOT +non-trivial is to compile it into the executable. The key could be stored in a +header file, eg. "key.h", of which a template with an empty or dummy key is +committed to the source code repository, and then this header could be included +in the file(s) that require it. So long as the actual key is not distributed +beyond the team member who builds releases, this would then require people to +either reverse-engineer the executable or monitor its allocated memory to obtain +the key. + +However, this doesn't prevent the swapping out of files that LOOT uploads. It +may be possible for an attacker to replace the file that LOOT generates or edits +for upload with another containing rubbish data or malicious code. Attackers +could also "cut and paste" the section of the executable that deals with the +uploading into another executable that allows arbitrary uploads. I don't +actually know if the latter is possible, but it sounds like it could be. + +In that case, may we lose any upload safeguards, but the attacker could not +disable the download safeguards present in other users' copies of the LOOT +executable unless their systems were already compromised. We have to then assume +that the database is not necessarily safe. Given an unsafe database, we can do +the following: + + * Ensure that the robot only has repository commit/checkout permissions and + no admin abilities. Team members should be given repository access to + allow oversight, with admins having admin abilities. + * Check for the validity of database files. A database file should only ever + be read or generated by LOOT, so invalid database files should never + exist except on a compromised system. Any invalid files found could + cause a message to be displayed and a notification sent to the team using + the feedback mechanism. The invalid file would then be deleted. The parser + should fail immediately upon encountering invalid content. + * Don't execute database file content. The format doesn't have any + executable data, so this is pretty obvious in this case. + * Revoke access to the repository for the compromised key. If we only have + one valid robot key at any one time, this is pretty easy. The downside is + that it means that nobody can access the repository with the key, + effectively disabling the online cache. The procedure in case this was + done would be to then figure out how the compromise happened, then try to + stop it happening again, before releasing an update to LOOT with a new + key. + * Only download a database file if a plugin's data isn't found in the local + cache. This means that clients will only download database files for + plugins which are installed and which don't already have cached data. So + if A.esp and B.esp have database file updates available, but the specific + version of A.esp that the user has installed already has its data cached + locally, then only B.esp.lmdb will be updated. + * Users could disable online database access in their copies of LOOT. If + a user thinks that they have downloaded rubbish data, they can delete + their local cache and LOOT will regenerate it the next time it runs. + +The validity and non-execute points would not prevent an attacker from +distributing rubbish data, but it would only take one user noticing and +informing the team for this to be caught and dealt with. + +Since the database repository would be separate from the masterlist repository, +and the robot wouldn't have access to the latter, users can easily be informed +of any compromise via masterlist messages. + +An attacker may also upload files that may never be downloaded by users, but +which are still malicious or illegal in nature. As such, we can't rely on +users vetting the files they download as an indicator for a compromise. One +possibility is for the database to be monitored by having an email notification +system in place that would notify team members of any file uploads and their +content. That would allow us to detect database compromise without any user +involvement, though it does require team activity. + +All the above points are suggested on the assumption that we can't have a +dedicated machine to filter out malicious use. Such a machine would be likely +flawed anyway, so the above points are still worth considering even if we had +a dedicated machine. + + +Install Structure +================= + +LOOT will be a self-contained installation that can be dropped anywhere. It will +have an installer option that also installs some Start menu shortcuts and a +Registry entry, but these will not be required for LOOT to function. + +The directory structure will be as follows: + +LOOT/ + Morrowind/ + .svn/ + ... + Cache/ + .svn/ + ... + m/ + o/ + morrowind.esm.lmdb + ... + masterlist.yaml + userlist.yaml + lastrun.yaml + loadorder.old + Oblivion/ + .svn/ + ... + Cache/ + .svn/ + ... + o/ + b/ + oblivion.esm.lmdb + ... + masterlist.yaml + userlist.yaml + lastrun.yaml + loadorder.old + Skyrim/ + .svn/ + ... + Cache/ + .svn/ + ... + s/ + k/ + skyrim.esm.lmdb + ... + masterlist.yaml + userlist.yaml + lastrun.yaml + loadorder.old + Fallout3/ + .svn/ + ... + Cache/ + .svn/ + ... + f/ + a/ + fallout3.esm.lmdb + ... + masterlist.yaml + userlist.yaml + lastrun.yaml + loadorder.old + FalloutNewVegas/ + .svn/ + ... + Cache/ + .svn/ + ... + f/ + a/ + falloutNV.esm.lmdb + ... + masterlist.yaml + userlist.yaml + lastrun.yaml + loadorder.old + LOOT.exe + LOOT.ini + +lastrun.yaml stores the last load order set by LOOT, with the messages +displayed at that time, and uses the same format as the masterlist and userlist. + +loadorder.old stores the load order before LOOT was last run. + +The game caches are for storing database files so that they can be used when +there is no Internet connection. Each cache acts as separate working copy of +the database repository, holding the files it needs. They are separate to +help prevent name clashes between games. Each game folder also acts as a working +copy of the folder which stores the masterlist for that game on the repository. + + +Configuration +============= + +LOOT will store its settings in a LOOT.ini file in its install directory. This +file should contain the following settings: + +bEnableNetworking Boolean +bUseOnlineCache Boolean Cache = Database +bGetMetadataUpdates Boolean Metadata = Masterlist +bPromptMetadataSubmit Boolean +bAlwaysSubmitMetadata Boolean Only has effect if the above is false. +bDoStartupUpdateCheck Boolean Check for program updates. + +sOnlineCacheURL String URL to the base of the database repository. +sMetadataURL String +sVersionCheckerURL String URL to the file holding version info. +sFeedbackURL String URL to the issue tracker used for feedback. + +sGame String Same as in BOSS. +sLastGame String Same as in BOSS. +iDebugVerbosity Integer Debug output is logged if > 0. +bDoTrialRun Boolean Same as in BOSS. +sLanguage String Same as in BOSS. + + +Cross-Platform Support +====================== + +I develop on Linux, cross-compiling for Windows. LOOT will use the CMake build +system generator so that it can be compiled on multiple platforms, though some +functionality will require Windows-specific things, such as Registry queries. + + +Backwards Compatibility +======================= + +No backwards compatibility with previous versions of BOSS is planned: there are +simply too many changes. To ease the transition though, it may be beneficial for +there to be converters for the userlist and masterlist. + +Such converters would be limited in scope, however, due to the differences in +capabilities: only userlist rules which sort plugins after others and perform no +message changes have direct conversions. The masterlist may be adapted by having +rules created for those plugins with messages attached, but the content of those +messages would have to be adjusted manually, and all specific positioning +information would be lost too. + + +Misc +==== + +Here is the full list of BOSS members at the end of 2012. Any of these who lose +their membership status should still be credited in the BOSS readme: + +Random007, Arthmoor, WrinklyNinja, PacificMorrowind, aellis, Vacuity, Gabba, +ZiggyX200, RiddlingLynx, AliTheLord, Tokc.D.K., Valda, Space Oden69, Televator, +Leandro Conde, Psymon, Loucifer, Torrello, Malonn, Skyline, Sharlikran, Red Eye, +iyumichan, Peste, Calen Ellefson, SilentSpike, Arkangel, zyp, v111, Chevenga, +rowynyew diff --git a/src/legacy.py b/src/legacy.py new file mode 100755 index 00000000..abec1240 --- /dev/null +++ b/src/legacy.py @@ -0,0 +1,176 @@ +#! /usr/bin/env python3 +# BOSS +# +# A plugin load order optimiser for games that use the esp/esm plugin system. +# +# Copyright (C) 2012 WrinklyNinja +# +# This file is part of BOSS. +# +# BOSS is free software: you can redistribute +# it and/or modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation, either version 3 of +# the License, or (at your option) any later version. +# +# BOSS is distributed in the hope that it will +# be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with BOSS. If not, see +# . + +# This is a script that converts a MF 2.3 masterlist to a MF 3 masterlist. It's +# a bit hacky, but the YAML parser accepts it. Limitations are: +# - Doesn't lowercase condition statements. +# - Doesn't strip plugins that no longer belong in the masterlist, ie. without +# messages or positioning comments. +# - Requirements and incompatibilities do not use their new data structures, +# and are just converted to 'say' messages. +# - All strings except tags are enclosed in double quotes and escaped, even +# those that don't need to be. This means that any single backspaces used +# in a string will cause problems, but some strings are escaped where others +# aren't, so basically just test using the parser. +# - No extraneous comments/messages are removed. +# - VAR conditions are not replaced with their conditional statements. SET lines +# are stripped, so the old masterlist must be used to check what they are. +# - Each message or list of tags comes under a different 'msg' or 'tag' object. +# - Tag message text, aside from the tags themselves, is lost. +# - Conditional plugin positions are all made unconditional, so there will be +# duplicate entries. + +def escapeYAMLStr(s): + return '"' + s.replace('"', '\\"').strip(' ') + '"' + +def mContent(line): + content = line[line.find(':')+1:] + return escapeYAMLStr(content) + +inFile = open('masterlist.txt', 'r') +outFile = open('masterlist.yaml', 'w') + +outFile.write('---\n'); + +isInComment = False +lastLine = '' +lastLineType = 0 +for line in inFile: + + line = line.replace('\n', '') + + if (len(line) == 0): + continue + + # Skip group lines. + if ("BEGINGROUP:" in line or "ENDGROUP:" in line): + continue + + # Skip var lines. + if ('SET:' in line): + continue + + # Write comments. + if (line[0:2] == '//'): + outFile.write('#' + line[2:] + '\n') + continue + + if ('/*' in line): + isInComment = True + outFile.write('#' + line + '\n') + + if ('*/' in line): + isInComment = False + outFile.write('#' + line + '\n') + + if ('*/' in line or '/*' in line): + continue + + if (isInComment): + outFile.write('#' + line + '\n') + continue + + # Write plugin lines. + if (':' not in line): + outFile.write(' - name: ' + escapeYAMLStr(line) + '\n') + else: + pos = line.find(':') + if ('MOD:' in line[:pos+1]): + outFile.write(' - name: ' + escapeYAMLStr(line[pos+1:]) + '\n') + elif ('REGEX:' in line[:pos+1]): + outFile.write(' - name: ' + escapeYAMLStr(line[pos+1:]) + '\n') + + # Extract conditional from line. + if ('IF' in line and ':' in line): + key = line[:line.find(':')] + key = key.replace('GLOBAL', '') + condition = key[:key.rfind(' ')].strip(' ') + elif ('ELSE' in line and ':' in line): + condition = condition.replace('IF ', 'IFNOT2 ') + condition = condition.replace('IFNOT ', 'IF ') + condition = condition.replace('IFNOT2 ', 'IFNOT ') + else: + condition = '' + + # Write tag lines. + if ('TAG:' in line): + if ('{{BASH' in line): + tags = line[line.find('{{BASH:') + 7:line.find('}}') ] + tags = tags.split(',') + outFile.write(' tag:\n') + for tag in tags: + if (condition): + outFile.write(' - condition: ' + condition + '\n') + outFile.write(' name: ' + tag.strip(' ') + '\n') + else: + outFile.write(' - ' + tag.strip(' ') + '\n') + if ('[' in line): + tags = line[line.find('[')+1:line.find(']')] + tags = tags.split(',') + outFile.write(' tag:\n') + for tag in tags: + if (condition): + outFile.write(' - condition: ' + condition + '\n') + outFile.write(' name: -' + tag.strip(' ') + '\n') + else: + outFile.write(' - -' + tag.strip(' ') + '\n') + + # Write 'say' messages. + if ('SAY:' in line or 'INC:' in line or 'REQ:' in line): + if ('GLOBAL ' not in line): + outFile.write(' msg:\n') + indent = ' ' + else: + indent = ' ' + outFile.write(indent + '- type: say\n') + if (condition): + outFile.write(indent + ' condition: ' + condition + '\n') + outFile.write(indent + ' content: ' + mContent(line) + '\n') + + # Write 'warn' messages. + if ('DIRTY:' in line or 'WARN:' in line): + if ('GLOBAL ' not in line): + outFile.write(' msg:\n') + indent = ' ' + else: + indent = ' ' + outFile.write(indent + '- type: warn\n') + if (condition): + outFile.write(indent + ' condition: ' + condition + '\n') + outFile.write(indent + ' content: ' + mContent(line) + '\n') + + # Write 'error' messages. + if ('ERROR:' in line): + if ('GLOBAL ' not in line): + outFile.write(' msg:\n') + indent = ' ' + else: + indent = ' ' + outFile.write(indent + '- type: error\n') + if (condition): + outFile.write(indent + ' condition: ' + condition + '\n') + outFile.write(indent + ' content: ' + mContent(line) + '\n') + + +inFile.close(); +outFile.close();