loot/libloot
0
0
Fork 0
mirror of https://github.com/loot/libloot.git synced 2026-07-27 14:16:01 -07:00
Code Issues Packages Projects Releases Wiki Activity

Updated design, started writing masterlist converter.

Browse Source
This commit is contained in:
WrinklyNinja
2013-01-04 12:44:42 +00:00
parent 8c7a3da6b7
commit 380461f037
3 changed files with 674 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
File Format
+136 -133
View File
@@ -1,5 +1,15 @@
LOOT Masterlist File Format
===========================
LOOT Metadata File Format
==========================
The LOOT metadata file format is a YAML file which contains a specific set of
data in a specific structure. It is used by the masterlist, the userlist and the
lastrun file.
General properties are:
1. Keys and keywords (such as message types, conditional types) are lowercase.
2. Whitespace is important: a two-space indentation should be used between
each level.
3. All YAML properties.
In YAML, the message object is:
---
@@ -7,105 +17,86 @@ In YAML, the message object is:
condition: OPTIONAL !condition
type: REQUIRED !!string
content: REQUIRED !!string
lang: OPTIONAL !!string
...
Allowed values for the 'type' node are "SAY", "WARN" or "ERROR". The semantics
Allowed values for the 'type' node are "say", "warn" or "error". The semantics
of the types are:
say Generic message type, carries no semantics.
warn A warning message, detailing a non-critical installation issue.
error An error message, detailing a critical installation issue.
say Generic message type, carries no semantics.
warn A warning message, detailing a non-critical installation issue.
error An error message, detailing a critical installation issue.
The 'content' node string will treat all file:// , http:// and https:// links as
hyperlinks. A link can be given a label by enclosing it and the label in double
quotes, eg. "http://www.example.com example link".
The 'lang' node takes an ISO 639-3 language string, of which there are a set defined for the languages LOOT supports (as yet, just 'eng').
The file object:
---
!file:
condition: OPTIONAL !condition
name: REQUIRED !!string
ver: OPTIONAL !!int or !!string
mod: OPTIONAL !!string
condition: OPTIONAL !condition
name: REQUIRED !!string
ver: OPTIONAL !!int or !!string
display: OPTIONAL !!string
operator: OPTIONAL !!string
...
CRCs are given as hex integers prefixed by 0x. The 'mod' node allows the name
of the parent mod to be used instead of the filename in messages.
CRCs are given as hex integers prefixed by 0x. The 'display' node allows the name
of the parent mod or program to be used instead of the filename in messages.
'operator' is the comparison operator used for the version check (if performed).
'>=' is the default used if not specified, other possibilities include =, <, etc.
The condition object:
The tag object:
---
!condition:
exp: REQUIRED !!string
vars: REQUIRED [!file or !!string]
!tag:
condition: OPTIONAL !condition
name: !!string
...
The [] denotes a list of the given type(s).
All 'file' objects used in a condition must have empty 'condition' nodes to
prevent recursion loops.
All conditions will have their results cached.
The 'exp' string must follow the following format:
This object can be used to represent a conditional tag. 'name' is the tag name,
eg. Graphics.
IF|IFNOT FILE|VERSION|ACTIVE|LANG [AND|OR IF|IFNOT FILE|VERSION|ACTIVE|LANG]*
The four condition types are detailed below:
Type Args Details
FILE !file If the file object passed contains a CRC, this
condition checks for a CRC match, otherwise it
checks for existence.
VERSION !file, !!string The 'ver' node of the file object must contain
a version string. The string argument is '<', '=' or
'>', and is used to form an expression with the true
version on the left and the given version on the
right, which is then checked to see if it holds.
ACTIVE !file Checks if the given file is a plugin, and if that
plugin is active.
LANG !!string Checks to see if the given language string matches
the current language.
The condition string:
---
!condition: REQUIRED !!string
...
Syntax of the string is the same as is currently for BOSS, except the following:
1. The VAR, REGEX and LANG condition types have been removed.
2. The FILE condition type now evaluates regex path strings.
3. The '&&' and '||' operators have been replaced with 'and' and 'or' respectively.
4. Condition keywords, types and operators are all lowercase.
5. Brackets may be used to logically group conditions in compound conditions.
6. Evaluation of compound conditionals is done according to standard operator
precedence rules.
The 'vars' node lists the variables that are used in the expression in the order
that they are to be used.
The plugin object:
---
!plugin:
name: REQUIRED !!string
priority: OPTIONAL !undecided
reqs: OPTIONAL [!file]
incs: OPTIONAL [!file]
msgs: OPTIONAL [!message]
name: REQUIRED !!string
priority: OPTIONAL !undecided
enabled: OPTIONAL !bool
after: OPTIONAL [!file]
req: OPTIONAL [!file]
inc: OPTIONAL [!file]
msg: OPTIONAL [!message]
tag: OPTIONAL [!!string or !tag]
...
The 'name' node can be either an exact filename or a regular expression. The
difference will be determined by looking for the "\.esp" or "\.esm" substrings,
The 'name' node can be either an exact filename or a regular expression. The
difference will be determined by looking for the "\.esp" or "\.esm" substrings,
which are only valid for a regular expression.
The type of the 'priority' node has not been decided yet, as it's not yet clear
how much fidelity is required for it.
If any of the 'reqs' node objects are not present, an error message will be
If any of the 'req' node objects are not present, an error message will be
displayed. If any of the 'incs' node objects are present, an error message will
be displayed.
The 'reqs' node objects must be listed in load order. The 'reqs' and 'incs'
The 'req' node objects must be listed in load order. The 'req' and 'inc'
nodes may contain a mix of plugin and non-plugin objects.
If the 'enabled' node is not specified, its value is assumed true. It is
provided for use in the userlist.
The 'after' node is a list of plugins that the object plugin must load after, intended for use specifying implicit dependencies. It may also be used in the userlist to specify custom positions.
The 'tag' node holds an array of Bash Tags to be suggested. Tags that are prepended with a '-' (eg. '-Graphics') are suggested for removal, otherwise tags are suggested for addition.
LOOT Userlist Format
====================
Same as masterlist format, but the plugin object is:
---
!plugin:
name: REQUIRED !!string
priority: OPTIONAL !undecided
enabled: OPTIONAL !bool
after: OPTIONAL !!string
reqs: OPTIONAL [!file]
incs: OPTIONAL [!file]
msgs: OPTIONAL [!message]
...
If the 'enabled' node is not specified, its value is assumed true. The 'after'
node is a tentative implementation of how users will specify custom positions,
and will take the filename of the plugin that the object plugin will load after.
Examples
========
Simple Example
--------------
@@ -121,6 +112,7 @@ Simple Example
content: "This is a second message."
...
Complex Example
---------------
@@ -142,107 +134,118 @@ globals:
- name: Oblivion.esm
msg:
- {condition: {exp: IF FILE, vars: *GAME}, type: WARN, content: "False alarm."}
- {condition: exp: "IF FILE("TESV.exe")", type: WARN, content: "False alarm."}
- <<: {*OBSOLETE, content: Quotes aren't necessary, unless the message contains special chars. }
...
LOOT Machine Database Format
============================
LOOT Plugin Data Cache Format
=============================
The formatting of this database doesn't need to be human-readable, but must
The formatting of this cache doesn't need to be human-readable, but must
contain the following information for each plugin:
* Filename (for roughly identifying the file)
* CRC (for precisely identifying the exact file)
* CRC (for precisely identifying the exact file)
* Masters in the order they appear in the plugin's master list.
* FormIDs of the records it edits (not added records).
Rather than have a single database file, it is more efficient past the short
term to use a different file for each plugin filename's data, even after taking
into account the additional networking cost of using many small files. See
the section after the divider below for the analysis.
Each database file will be named filename.txt, where filename is a plugin
filename including extension, eg. "Unofficial Oblivion Patch.esp.txt". In each
database file, the data will be divided into entries for each of the different
versions of that file, which are handled using their CRCs. These entries make
up the bulk of the file, and are followed by an index of CRCs and offsets to
their corresponding data at the end of the file.
Rather than have a single cache file, it is more efficient past the short
term to use a different file for each plugin filename's data, even after
taking into account the additional networking cost of using many small
files. See the section after the divider below for the analysis.
The cache file extension will be ".dc", an acronym for "data cache". Each
cache file will share the filename of the plugin that it holds the data of,
including the plugin's file extension, eg. "Unofficial Oblivion Patch.esp.dc".
In each cache file, the data will be divided into entries for each of the
different versions of that file, which are handled using their CRCs. These
entries make up the bulk of the file, and are followed by an index of CRCs
and offsets to their corresponding data at the end of the file.
Name Type Details
---------------------------------------------------------------------------
Version uint8_t Version number for the file's
format (in case it changes).
Name Length uint8_t The plugin filename's length.
Filename char[Name Length] Not null terminated.
Data Entries data entry[Entry Count]
Index index entry[Entry Count]
Entry Count uint32_t
Name Type Details
---------------------------------------------------------------------------
Data Entries data entry[Entry Count]
Index index entry[Entry Count]
Entry Count uint32_t
Each data entry has the following structure:
Name Type Details
---------------------------------------------------------------------------
Size uint32_t Total size of data entry, not
including this value.
Masters Length uint16_t Includes null characters.
Masters char[Masters Length] Null-terminated filenames in order
of how they appeared in the plugin.
FormIDs uint32_t[len] len = (Size - Masters Length) / 4
Name Type Details
---------------------------------------------------------------------------
Size uint32_t Total size of data entry, not
including this value.
Masters Length uint16_t Includes null characters.
Masters char[Masters Length] Null-terminated filenames in order
of how they appeared in the plugin.
FormIDs uint32_t[len] len = (Size - Masters Length) / 4
Each index entry has the following structure:
Name Type Details
---------------------------------------------------------------------------
File CRC uint32_t
Data offset uint32_t From beginning of file.
Name Type Details
---------------------------------------------------------------------------
File CRC uint32_t
Data offset uint32_t From beginning of file.
All numbers are encoded low to high byte. All strings are encoded in UTF-8.
The reason for this structure is to optimise the processes that will be
performed upon database files: searching, reading data and appending new data.
performed upon cache files: searching, reading data and appending new data.
The index allows quick searching of the file as it is composed of a given
number of entries with defined size. It is located at the end of the file rather
than the more usual beginning so that appending new data does not need to
rewrite the whole file to also update the index: only the index will ever be
rewritten.
number of entries with defined size. It is located at the end of the file
rather than the more usual beginning so that appending new data does not
need to rewrite the whole file to also update the index: only the index and
filename data will ever be rewritten.
Each database file will be able to store up to ~10^7 entries, though the
probability of there being a CRC collision reaches 1% at ~10^4 entries, due to
the birthday problem. Still, it's unlikely that there will ever be several
thousand versions of a file with the same filename in use, so it's probably not
worth worrying about.
The plugin filename is contained within the cache file so that in case the
cache file's filename ever gets changed, it is still possible to work out
which plugins' data it holds.
Each cache file will be able to store up to ~10^7 entries, though the
probability of there being a CRC collision reaches 1% at ~10^4 entries, due
to the birthday problem. Still, it's unlikely that there will ever be
several thousand versions of a file with the same filename in use, so it's
probably not worth worrying about.
Single vs. Multi-File
---------------------
A single-file database would use the above structure, but with an additional
uint64_t in each index entry to store a filename hash.
A single-file cache would use the above structure, but with an additional
uint64_t in each index entry to store a filename hash, and without the filename
at the start of the file.
Let's say the database file holds info for 100,000 plugins (25,000 mods, each
Let's say the cache file holds info for 100,000 plugins (25,000 mods, each
with two plugins, and having two versions released - probably still a
conservative estimate). Assuming each plugin changes 10 records, and has 2
conservative estimate). Assuming each plugin changes 10 records, and has 2
masters with filenames 10 characters long, the size is ~11.4 MB.
Using a multi-file database and the same assumptions being made as above,
Using a multi-file cache and the same assumptions being made as above,
the size of each file will be 390 bytes.
However, because we're dealing with lots of small files, HTTP request/response
time has to be accounted for. Google says that the best average for the
upload/download speed ratio of most Internet connections is 1:4, and that
headers are roughly 300 bytes long without cookies. Each file's effective size
is therefore 1890 bytes.
time has to be accounted for. Google says that the best average for the
upload/download speed ratio of most Internet connections is 1:4, and that
headers are roughly 300 bytes long without cookies. Each file's effective size
is therefore 1890 bytes.
For a load order of 250 plugins, the total effective size is ~461 kB. This is
~25x smaller than the equivalent single-file database that would need to be
~25x smaller than the equivalent single-file cache that would need to be
transferred, but it doesn't take into account the HTTP connection time, which
is significant for small files. I don't know how I can take that into account,
but if I've understood things correctly, I don't need to, as a persistent
connection could be used, which would require only one HTTP connection to be
made. HTTP pipelining could also be used to improve performance. A multi-file
database could also be cached, with new files only being downloaded if they had
changed or weren't already on the user's system, rather than the whole database
having to be re-downloaded even if the only change was the addition of a plugin
the user doesn't have. This would only save 95 kB over the assumed load order at
made. HTTP pipelining could also be used to improve performance. A multi-file
cache could also be cached, with new files only being downloaded if they had
changed or weren't already on the user's system, rather than the whole cache
having to be re-downloaded even if the only change was the addition of a plugin
the user doesn't have. This would only save 95 kB over the assumed load order at
maximum though.
Given all that, it's probably better to use a multi-file database over a single
file database.
Given all that, it's probably better to use a multi-file cache over a single
file cache.
README
+362 -41
View File
@@ -52,17 +52,17 @@ goals:
other than the plugins it will order.
* LOOT should be able to figure out a correct load order with a minimum of
supplementary data on author intent.
* LOOT should be able to obtain supplementary data from an arbitrary source,
* LOOT should be able to obtain supplementary data from an arbitrary source,
or arbitrary sources, depending on the number of types of data.
* LOOT's effects should be customiseable by the user.
* LOOT should be scaleable, focussed and extensible.
Note that an optimum load order is not necessarily a correct load order. If
Note that an optimum load order is not necessarily a correct load order. If
we define the impact of a plugin as the number of its edits that get applied to
a person's game, then an optimum load order is one with the highest average
impact from its constituent plugins whilst satisfiying all explicit dependencies.
However, some plugins may be made with the intent that they are to be overridden
by others, and so a correct load order is one that also takes into account
by others, and so a correct load order is one that also takes into account
author intent. Given a set of plugins, their correct load order may have a lower
average impact than their optimum load order.
@@ -117,21 +117,26 @@ people want one.
A general workflow for a user running LOOT to sort would be:
1. Run LOOT: it auto-sorts based on plugin content plus and masterlist
and userlist rules.
2. LOOT finishes sorting, and displays the load order it sets.
1. Run LOOT: it auto-sorts based on plugin content plus supplementary data.
2. LOOT finishes sorting, and displays the load order it suggests. At this point
the load order has not yet been applied.
Clicking on a plugin in the load order will highlight all those it
conflicts with. This provides a simple way for users to gauge conflicts
and so decide which plugin's changes they wish to apply to their games.
Dragging a plugin will display the positions to which it can be moved
(by dropping it there) whilst satisfying dependencies.
Plugins which have no conflicts are greyed out and cannot be selected,
to simplify this process.
There will be an information panel which will display the selected
plugin's requirements, incompatibilities, priority and messages. These
will be editable, upon exit of the load order window, the user will be
asked if they wish to submit any edits they make to these fields.
Edits to plugin positions will not be submitted.
will be editable, and upon exit of the load order window, the user will be
asked if they wish to submit any edits they make to these fields. Edits to
plugin positions will not be submitted.
If LOOT encounters any errors while sorting, they are displayed instead
of the load order.
3. The user exits the load order display, choosing either to apply or
@@ -139,21 +144,20 @@ A general workflow for a user running LOOT to sort would be:
4. LOOT asks the user if they want to submit their plugin data edits. If
so, it then asks the user to supply links for each of the plugins
with edited data.
4. LOOT then applies the resulting load order, saves any user choices
as userlist rules, and exits.
4. LOOT then applies the resulting load order, saves any user choices, and
exits.
LOOT must also provide a means to manage user rules, to undo its
LOOT must also provide a means to manage past user choices, to undo its
changes (only back one run though), and to edit its settings.
The Userlist Syntax and Masterlist Syntax docs will be merged into a
"Rule Syntax" doc, as the syntax will be the same for both. The new doc
will be structured so that there is a progression in complexity from
simple rule syntax to complex rule syntax as the guide progresses.
LOOT should ask for permission to send plugin contents data on first run, and
to send plugin metadata whenever there are metadata changes to send, unless the
user selects a "do not ask me again" option. These settings should be
configurable at any time.
The drag 'n' drop plugin positioning will likely require userlist rules
to support positioning of plugins precisely relative to other plugins, but it
may be possible to achieve the positioning using only the priority setting. If
possible, that is preferred.
Documentation will be split between three readmes: one general readme, one API
readme and one rule syntax readme. The version history can be split into a
fourth readme when it grows sufficiently long.
Implementation
@@ -165,7 +169,8 @@ possible, for example:
* yaml-cpp: for masterlist/userlist parsing.
* libespm: for plugin parsing.
* libloadorder: for reading and writing load order.
* cURL: for networking.
* libsvn: for accessing the Subversion repositories used.
* cURL: for misc. networking.
* wxWidgets: for the interface.
* Boost: for filesystem interaction, various helpful functions.
@@ -196,19 +201,19 @@ used in the following manner:
plugins, the database, the masterlist and the userlist.
6. The data from the scanned plugins is appended to the database file, which
is then uploaded to its hosted location.
In addition to having this database file as a sort of permanent and
self-updating cache, optimisations can also be made to skip load order
In addition to having this database file as a sort of permanent and
self-updating cache, optimisations can also be made to skip load order
calculation. For example, if the only changes from the last load order set by
LOOT is that a plugin or plugins have been removed, then the last load order
may be used. It might also be possible to only perform partial calculation
when plugins are added, though this is not certain.
Note: For clarity, the database that holds the results of plugin scanning will
be referred to as the "machine database" (or MDB) as it will only be read and
written by the LOOT code. The list of metadata that is maintained by the LOOT
team will be referred to as the "masterlist", and the list of metadata that is
individual to each user will be referred to as the "userlist".
be referred to as the "cache". The list of metadata that is maintained by the
LOOT team will be referred to as the "masterlist", and the list of metadata that
is individual to each user will be referred to as the "userlist".
LOOT Masterlist & Userlist
==========================
@@ -221,7 +226,7 @@ The purpose of the masterlist is to store the following data for plugins:
* The intended priority of the plugin relative to others that make the same
changes.
* Possibly also Bash Tags to be applied.
It is also useful to store some data not associated with plugins, such as:
* Global messages.
@@ -229,10 +234,10 @@ It is also useful to store some data not associated with plugins, such as:
multiple times.
* File objects, for reference instead of rewriting the same condition
argument multiple times.
All messages should also be optionally conditional on an arbitrary number of
conditions. More on that later.
All messages and files should also be optionally conditional on an arbitrary
number of conditions.
The purpose of the userlist is to store modifications to masterlist plugin data,
and to also store user-specified positions of plugins. The userlist must also
support the disabling of plugin entries within it, so that users can turn off
@@ -243,16 +248,332 @@ the same format. The userlist could then simply support two data fields that are
ignored by the masterlist parser. I think that YAML is a suitable format for
these files as it is simple, human-readable, yet quite powerful.
Further details can be found in the LOOT File Format doc.
The drag 'n' drop plugin positioning will likely require userlist rules
to support positioning of plugins precisely relative to other plugins, but it
may be possible to achieve the positioning using only the priority setting. If
possible, that is preferred.
Further details can be found in the File Format doc.
Networking
==========
LOOT needs to be able to download and upload specific files from a VCS
repository. (VCS to help deal with possible spam/vandalism, and to track
changes.) It also needs to be able to send/receive data from some sort of issue
tracker.
LOOT has four target networking functionalities:
As LOOT needs to do this for users without their own credentials for the
repository, it needs to do this as a robot.
* Remote / local cache sync.
* Masterlist updating.
* User feedback submission.
None of these functionalities are critical to usage of LOOT.
The masterlist and cache repositories will be Subversion repositories, as this
allows users to most easily download only the files they require, and only the
diffs of those files.
As LOOT needs commit as well as checkout access to the cache repository, for
users without their own credentials, it must access the repository using a
robot account. Google Code does not allow the use of robot accounts, but
Sourceforge does. However, the actions performed by the robot account are the responsibility of the account creator. This brings up the issue of account
abuse. This will be covered in the 'Security' section below.
Conflicts between different users' commits will probably be an issue.
Because the database files are binary, not text, and because they use an index
with data offsets, they can't be merged automatically. The database file
workflow is:
1. User svn updates their working copy of a file.
2. User performs scanning actions.
3. User edits their working copy of the file.
4. User commits their edit to the repository.
If another user changes the same file between steps 1 and 4, the commit will
fail. If this happens, LOOT should svn revert followed by svn update, then
re-apply the user's edits to the updated file, checking to make sure that the
updated file doesn't already include the same data. A second commit should then
be attempted. If this commit also fails, it's best just to discard the edits.
While that means the user won't benefit from having a local cache, it becomes
too complicated to store the edits on disk and re-apply them for a potentially
uncapped number of failures.
The database repository may store tens if not hundreds of thousands of files,
so they need to be organised well to ease browsing. Grouping the files into
the game they are for, then subgrouping them into folders based on the first
character of their filename, then subgrouping again based on the second
character would probably do. For example:
repo/
Oblivion/
o/
b/
oblivion.esm.bpdc
Files and the 'letter folders' will be in lowercase.
"repo" does not have to be the repository root, but LOOT's settings should
use the URL pointing to that location. The repository will be required to use
the same game folder names as LOOT's install structure (see the section below).
The masterlists would be stored using a folder structure that mirrors LOOT's
install structure, for example:
repo/
Oblivion/
masterlist.yaml
"repo" does not have to be the repository root, but LOOT's settings should
use the URL pointing to that location. The repo will be required to use the same
game folder names as LOOT's install structure.
The masterlist update mechanism will feature a validator such that if the
masterlist downloaded cannot be parsed, it will not replace the current
masterlist (if exists). The log will display notification of validation failure
with instructions to inform the maintainers of this issue. This is a fairly
major improvement over BOSS currently, which becomes unusable when a masterlist
update is occassionally broken.
The final bit of networking to be considered is the feedback mechanism. It's not
an initial development target, as initially I think it would be better for all
communication between users and the team to be two-way, which is easier
accomplished through our forum threads. It's probably best to leave design of
this until we see what sort of feedback users are giving.
A program update checker is notably absent, as a fully automated updater is
not worth the trouble of implementing well, and notification of an available
update is easily performed through a global masterlist message and so does not
require an additional transport mechanism.
Security
========
The abuse of the database repository credentials to be distributed with LOOT is
a very real concern. Steps must be taken to prevent or minimise the associated
risk. Unfortunately, I have no experience with this sort of thing, so I could
be making some incorrect assumptions below.
One possible way to make obtaining the SSH key for use outside of LOOT
non-trivial is to compile it into the executable. The key could be stored in a
header file, eg. "key.h", of which a template with an empty or dummy key is
committed to the source code repository, and then this header could be included
in the file(s) that require it. So long as the actual key is not distributed
beyond the team member who builds releases, this would then require people to
either reverse-engineer the executable or monitor its allocated memory to obtain
the key.
However, this doesn't prevent the swapping out of files that LOOT uploads. It
may be possible for an attacker to replace the file that LOOT generates or edits
for upload with another containing rubbish data or malicious code. Attackers
could also "cut and paste" the section of the executable that deals with the
uploading into another executable that allows arbitrary uploads. I don't
actually know if the latter is possible, but it sounds like it could be.
In that case, may we lose any upload safeguards, but the attacker could not
disable the download safeguards present in other users' copies of the LOOT
executable unless their systems were already compromised. We have to then assume
that the database is not necessarily safe. Given an unsafe database, we can do
the following:
* Ensure that the robot only has repository commit/checkout permissions and
no admin abilities. Team members should be given repository access to
allow oversight, with admins having admin abilities.
* Check for the validity of database files. A database file should only ever
be read or generated by LOOT, so invalid database files should never
exist except on a compromised system. Any invalid files found could
cause a message to be displayed and a notification sent to the team using
the feedback mechanism. The invalid file would then be deleted. The parser
should fail immediately upon encountering invalid content.
* Don't execute database file content. The format doesn't have any
executable data, so this is pretty obvious in this case.
* Revoke access to the repository for the compromised key. If we only have
one valid robot key at any one time, this is pretty easy. The downside is
that it means that nobody can access the repository with the key,
effectively disabling the online cache. The procedure in case this was
done would be to then figure out how the compromise happened, then try to
stop it happening again, before releasing an update to LOOT with a new
key.
* Only download a database file if a plugin's data isn't found in the local
cache. This means that clients will only download database files for
plugins which are installed and which don't already have cached data. So
if A.esp and B.esp have database file updates available, but the specific
version of A.esp that the user has installed already has its data cached
locally, then only B.esp.lmdb will be updated.
* Users could disable online database access in their copies of LOOT. If
a user thinks that they have downloaded rubbish data, they can delete
their local cache and LOOT will regenerate it the next time it runs.
The validity and non-execute points would not prevent an attacker from
distributing rubbish data, but it would only take one user noticing and
informing the team for this to be caught and dealt with.
Since the database repository would be separate from the masterlist repository,
and the robot wouldn't have access to the latter, users can easily be informed
of any compromise via masterlist messages.
An attacker may also upload files that may never be downloaded by users, but
which are still malicious or illegal in nature. As such, we can't rely on
users vetting the files they download as an indicator for a compromise. One
possibility is for the database to be monitored by having an email notification
system in place that would notify team members of any file uploads and their
content. That would allow us to detect database compromise without any user
involvement, though it does require team activity.
All the above points are suggested on the assumption that we can't have a
dedicated machine to filter out malicious use. Such a machine would be likely
flawed anyway, so the above points are still worth considering even if we had
a dedicated machine.
Install Structure
=================
LOOT will be a self-contained installation that can be dropped anywhere. It will
have an installer option that also installs some Start menu shortcuts and a
Registry entry, but these will not be required for LOOT to function.
The directory structure will be as follows:
LOOT/
Morrowind/
.svn/
...
Cache/
.svn/
...
m/
o/
morrowind.esm.lmdb
...
masterlist.yaml
userlist.yaml
lastrun.yaml
loadorder.old
Oblivion/
.svn/
...
Cache/
.svn/
...
o/
b/
oblivion.esm.lmdb
...
masterlist.yaml
userlist.yaml
lastrun.yaml
loadorder.old
Skyrim/
.svn/
...
Cache/
.svn/
...
s/
k/
skyrim.esm.lmdb
...
masterlist.yaml
userlist.yaml
lastrun.yaml
loadorder.old
Fallout3/
.svn/
...
Cache/
.svn/
...
f/
a/
fallout3.esm.lmdb
...
masterlist.yaml
userlist.yaml
lastrun.yaml
loadorder.old
FalloutNewVegas/
.svn/
...
Cache/
.svn/
...
f/
a/
falloutNV.esm.lmdb
...
masterlist.yaml
userlist.yaml
lastrun.yaml
loadorder.old
LOOT.exe
LOOT.ini
lastrun.yaml stores the last load order set by LOOT, with the messages
displayed at that time, and uses the same format as the masterlist and userlist.
loadorder.old stores the load order before LOOT was last run.
The game caches are for storing database files so that they can be used when
there is no Internet connection. Each cache acts as separate working copy of
the database repository, holding the files it needs. They are separate to
help prevent name clashes between games. Each game folder also acts as a working
copy of the folder which stores the masterlist for that game on the repository.
Configuration
=============
LOOT will store its settings in a LOOT.ini file in its install directory. This
file should contain the following settings:
bEnableNetworking Boolean
bUseOnlineCache Boolean Cache = Database
bGetMetadataUpdates Boolean Metadata = Masterlist
bPromptMetadataSubmit Boolean
bAlwaysSubmitMetadata Boolean Only has effect if the above is false.
bDoStartupUpdateCheck Boolean Check for program updates.
sOnlineCacheURL String URL to the base of the database repository.
sMetadataURL String
sVersionCheckerURL String URL to the file holding version info.
sFeedbackURL String URL to the issue tracker used for feedback.
sGame String Same as in BOSS.
sLastGame String Same as in BOSS.
iDebugVerbosity Integer Debug output is logged if > 0.
bDoTrialRun Boolean Same as in BOSS.
sLanguage String Same as in BOSS.
Cross-Platform Support
======================
I develop on Linux, cross-compiling for Windows. LOOT will use the CMake build
system generator so that it can be compiled on multiple platforms, though some
functionality will require Windows-specific things, such as Registry queries.
Backwards Compatibility
=======================
No backwards compatibility with previous versions of BOSS is planned: there are
simply too many changes. To ease the transition though, it may be beneficial for
there to be converters for the userlist and masterlist.
Such converters would be limited in scope, however, due to the differences in
capabilities: only userlist rules which sort plugins after others and perform no
message changes have direct conversions. The masterlist may be adapted by having
rules created for those plugins with messages attached, but the content of those
messages would have to be adjusted manually, and all specific positioning
information would be lost too.
Misc
====
Here is the full list of BOSS members at the end of 2012. Any of these who lose
their membership status should still be credited in the BOSS readme:
Random007, Arthmoor, WrinklyNinja, PacificMorrowind, aellis, Vacuity, Gabba,
ZiggyX200, RiddlingLynx, AliTheLord, Tokc.D.K., Valda, Space Oden69, Televator,
Leandro Conde, Psymon, Loucifer, Torrello, Malonn, Skyline, Sharlikran, Red Eye,
iyumichan, Peste, Calen Ellefson, SilentSpike, Arkangel, zyp, v111, Chevenga,
rowynyew
src/legacy.py
Executable
+176
View File
@@ -0,0 +1,176 @@
#! /usr/bin/env python3
# BOSS
#
# A plugin load order optimiser for games that use the esp/esm plugin system.
#
# Copyright (C) 2012 WrinklyNinja
#
# This file is part of BOSS.
#
# BOSS is free software: you can redistribute
# it and/or modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# BOSS is distributed in the hope that it will
# be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with BOSS. If not, see
# <http://www.gnu.org/licenses/>.
# This is a script that converts a MF 2.3 masterlist to a MF 3 masterlist. It's
# a bit hacky, but the YAML parser accepts it. Limitations are:
# - Doesn't lowercase condition statements.
# - Doesn't strip plugins that no longer belong in the masterlist, ie. without
# messages or positioning comments.
# - Requirements and incompatibilities do not use their new data structures,
# and are just converted to 'say' messages.
# - All strings except tags are enclosed in double quotes and escaped, even
# those that don't need to be. This means that any single backspaces used
# in a string will cause problems, but some strings are escaped where others
# aren't, so basically just test using the parser.
# - No extraneous comments/messages are removed.
# - VAR conditions are not replaced with their conditional statements. SET lines
# are stripped, so the old masterlist must be used to check what they are.
# - Each message or list of tags comes under a different 'msg' or 'tag' object.
# - Tag message text, aside from the tags themselves, is lost.
# - Conditional plugin positions are all made unconditional, so there will be
# duplicate entries.
def escapeYAMLStr(s):
return '"' + s.replace('"', '\\"').strip(' ') + '"'
def mContent(line):
content = line[line.find(':')+1:]
return escapeYAMLStr(content)
inFile = open('masterlist.txt', 'r')
outFile = open('masterlist.yaml', 'w')
outFile.write('---\n');
isInComment = False
lastLine = ''
lastLineType = 0
for line in inFile:
line = line.replace('\n', '')
if (len(line) == 0):
continue
# Skip group lines.
if ("BEGINGROUP:" in line or "ENDGROUP:" in line):
continue
# Skip var lines.
if ('SET:' in line):
continue
# Write comments.
if (line[0:2] == '//'):
outFile.write('#' + line[2:] + '\n')
continue
if ('/*' in line):
isInComment = True
outFile.write('#' + line + '\n')
if ('*/' in line):
isInComment = False
outFile.write('#' + line + '\n')
if ('*/' in line or '/*' in line):
continue
if (isInComment):
outFile.write('#' + line + '\n')
continue
# Write plugin lines.
if (':' not in line):
outFile.write(' - name: ' + escapeYAMLStr(line) + '\n')
else:
pos = line.find(':')
if ('MOD:' in line[:pos+1]):
outFile.write(' - name: ' + escapeYAMLStr(line[pos+1:]) + '\n')
elif ('REGEX:' in line[:pos+1]):
outFile.write(' - name: ' + escapeYAMLStr(line[pos+1:]) + '\n')
# Extract conditional from line.
if ('IF' in line and ':' in line):
key = line[:line.find(':')]
key = key.replace('GLOBAL', '')
condition = key[:key.rfind(' ')].strip(' ')
elif ('ELSE' in line and ':' in line):
condition = condition.replace('IF ', 'IFNOT2 ')
condition = condition.replace('IFNOT ', 'IF ')
condition = condition.replace('IFNOT2 ', 'IFNOT ')
else:
condition = ''
# Write tag lines.
if ('TAG:' in line):
if ('{{BASH' in line):
tags = line[line.find('{{BASH:') + 7:line.find('}}') ]
tags = tags.split(',')
outFile.write(' tag:\n')
for tag in tags:
if (condition):
outFile.write(' - condition: ' + condition + '\n')
outFile.write(' name: ' + tag.strip(' ') + '\n')
else:
outFile.write(' - ' + tag.strip(' ') + '\n')
if ('[' in line):
tags = line[line.find('[')+1:line.find(']')]
tags = tags.split(',')
outFile.write(' tag:\n')
for tag in tags:
if (condition):
outFile.write(' - condition: ' + condition + '\n')
outFile.write(' name: -' + tag.strip(' ') + '\n')
else:
outFile.write(' - -' + tag.strip(' ') + '\n')
# Write 'say' messages.
if ('SAY:' in line or 'INC:' in line or 'REQ:' in line):
if ('GLOBAL ' not in line):
outFile.write(' msg:\n')
indent = ' '
else:
indent = ' '
outFile.write(indent + '- type: say\n')
if (condition):
outFile.write(indent + ' condition: ' + condition + '\n')
outFile.write(indent + ' content: ' + mContent(line) + '\n')
# Write 'warn' messages.
if ('DIRTY:' in line or 'WARN:' in line):
if ('GLOBAL ' not in line):
outFile.write(' msg:\n')
indent = ' '
else:
indent = ' '
outFile.write(indent + '- type: warn\n')
if (condition):
outFile.write(indent + ' condition: ' + condition + '\n')
outFile.write(indent + ' content: ' + mContent(line) + '\n')
# Write 'error' messages.
if ('ERROR:' in line):
if ('GLOBAL ' not in line):
outFile.write(' msg:\n')
indent = ' '
else:
indent = ' '
outFile.write(indent + '- type: error\n')
if (condition):
outFile.write(indent + ' condition: ' + condition + '\n')
outFile.write(indent + ' content: ' + mContent(line) + '\n')
inFile.close();
outFile.close();