diff --git a/README b/README index 86df5ced..ccd2f1ab 100644 --- a/README +++ b/README @@ -8,9 +8,7 @@ See BOSS: http://code.google.com/p/better-oblivion-sorting-software/ BOSSv3 is being developed on a separate repository from the rest of BOSS's code as it is essentially a new program, and is being designed to address the shortcomings that are inherent in how BOSS approaches load order optimisation. -So what's wrong with BOSS? Why do I feel the need to go back to the drawing board? - -Well, modding for Bethesda's games has proved to be incredibly popular, with over 25,000 mods each available for Oblivion and Skyrim. +Modding for Bethesda's games has proved to be incredibly popular, with over 25,000 mods each available for Oblivion and Skyrim. For BOSS to be as useful as possible, it needs to be able to sort as many mods as possible. For Skyrim, the backlog of mods that need adding to its masterlist is at 5,800+ and growing. Recruiting extra team members isn't a real solution, as I've already done that and frankly copy/pasting filenames into a massive text file is not a whole load of fun and any analysis takes a while to do properly. @@ -28,15 +26,15 @@ BOSSv3's masterlist will therefore be used to provide unordered data sets for pl If the 'impact' of a plugin is defined as the number of its edits that get applied to a person's game divided by the total number of edits it makes, then an 'optimum' load order is one that maximises the average impact of its constituent plugins whilst satisfying all explicit dependencies. A 'correct' load order is not necessarily an optimum load order though, as some plugins may be made with the intent that they are overridden by other plugins. -For flexibility, the 'masterlist' data file that gets updated by the BOSS Team should have its online location as a configurable option within BOSS, so that should its location change for whatever reason, users would be able to redirect their copy of BOSS to look in the new location. It might be a good idea to download it via a diff, or allow compression, somehow as the Skyrim masterlist is 1.7 MB. +For flexibility, the 'masterlist' data file that gets updated by the BOSS Team should have its online location as a configurable option within BOSS, so that should its location change for whatever reason, users would be able to redirect their copy of BOSS to look in the new location. It might be a good idea to download it via a diff, or allow compression somehow, as the Skyrim masterlist is 1.7 MB. BOSSv3 will also have an API for accessing BOSS data and functionality. The more general functionality found in v2's API has already been forked to libloadorder, which v3 shall make use of. Unlike v2, v3 will not have a built-in updater/update checker. It's just too much of a headache to code and manage for the payoff it gives, especially since notifications can be sent to users via the masterlist. -When run, BOSS shall output its results to a "log.yaml" file, which shall then be interpreted by the UI to display the BOSS Log. +When run, BOSS shall output its results to a report file, which shall then be interpreted by the UI to display the BOSS Log. -BOSSv3 won't have a command line interface. Command line options may be added to the GUI if requested. Users need to be made aware that what BOSS sets isn't the absolutely perfect load order, and is open to tweaking, which is being lost on some people currently, in part due to the 'one click' nature of BOSS. +BOSSv3 won't have a command line interface, to simplify things. It also allows BOSSv3 to have a greater focus on users making sure their load order is correct, which is something that is sadly lacking from most users at the moment. Most users don't look any further than assuming BOSS has set their order correctly, which is unfortunate. Roadmap/To Do @@ -45,40 +43,63 @@ Roadmap/To Do - Write new masterlist, userlist, settings file parsers. DONE - Implement per-game handling. DONE - Write API. DONE -- Develop sorting algorithm. - Tie together automatic 'simple' sorting with masterlist, userlist data parsing and usage. DONE -- Write masterlist updating code. -- Develop graphical user interface. -- Error handling. - Initialisation and finishing routines (read settings file, write log file, etc.). DONE -- Remove game detection override. +- Remove game detection override. DONE +- Include libespm initialisation into Game constructor. DONE +- Implement reading of plugin versions from their description field. DONE +- Develop UI - main window. DONE +- Develop UI - settings window. DONE +- Develop sorting algorithm. +- Write masterlist updating code. +- Develop UI - metadata editor window. DONE +- Develop UI - report viewer. +- Error handling. - Checks for cyclic dependencies and incompatibilities. - Setting load order. -- Include libespm initialisation into Game constructor. DONE - Optimisations to load ordering. - Implement logging. -- Implement reading of plugin versions from their description field. DONE +- Add a quick header-only plugin read for use when loading the metadata editor, so that existing Bash Tags can also be displayed. +- Add checks for "Deactivate" tag compliance? +- Make validity checks non-fatal. +- Add a massive "RUN THE GAME LAUNCHER IF YOUR GAME IS NOT DETECTED" message somewhere. -Sorting Algorithm -================= +BOSS Report +=========== -Given two plugins A and B, their load order only matters if they both change a third plugin's records or if one changes the other's records. In such a case, the plugins can be said to 'overlap'. +Once BOSS has applied a load order, it will display a report, similar to the BOSS Log, that covers the following: -The hard rules of load ordering are: - 1. Master files always load before non-master files. - 2. A plugin must load after all its masters. + - The number of messagesm, broken down into the total, the number of warnings and the number of errors. + - The masterlist version used and whether the masterlist was updated or not. + - Any parser or validity checker errors encountered. + - If there were any changes since BOSS was last run. + - The new load order. + - The messages attached to plugins, version numbers read, CRCs calculated and whether or not each plugin is active. -Beyond that, the following can be used to maximise the impact of two plugins: - 1. Compute the overlap between A and B. Divide the overlap size by the number of records in each plugin, and the plugin with the larger ratio should load later, as more of its overall changes would be lost if it loaded first than if the other plugin does. +The current BOSS Log has a number of useful filters. Depending on the UI used for the report, some or all of these filters may be useful for implementation: -There are issues related to author intent: for example, the Unofficial Patches contain a large number of mostly edited records, as does a Bashed Patch, but the former is intended to load early and be overridden by other plugins while the latter is designed to load late. It's not clear how such situations can be resolved. + - Hide versions + - Hide whether a plugin is active or not. + - Hide CRCs + - Hide "say" messages. + - Hide Bash Tag suggestions. + - Hide all messages. + - Hide "do not clean" messages. + - Hide inactive plugins. + - Hide messageless plugins. -One possibility is to have a 'priority' metadata field, which has a default value, a value signifying the plugin should load late, and a value signifying the plugin should load early. If left at the default value, the plugin would be sorted by the algorithm normally, but if set to the early value, the plugin would be moved as far up the load order as the hard rules allow, and if set to the late value, would be moved as far down the load order as the hard rules allow. If multiple plugins are present that load early or late, then within that set of plugins the algorithm could be applied normally. -It's not a great solution though, being far too imprecise for my liking, but then the loading intention is also vague. +Optimisation +============ -Optimisations may also be made, such as not performing any new sorting if the only differences since previous sorting are the removal of plugins, or if the plugins added only add new records they can just be placed at the end. +Some ideas that have yet to be tested or implemented: + + - If no changes have been made to the installed plugins since previous sorting, then the same order could be applied. This would require CRC checks to make sure plugins haven't been edited though, so it's not clear without profiling whether it would improve or worsen performance. + - If plugins have only been removed since last sort, the same order could be used, except that some conditions might be evaluated differently in their absence, so again this isn't clear-cut. + - If plugins have been added but those plugins contain only new records, then they can be positioned alphabetically at the end without any trouble. + - Masterlist updating using the Subversion API to get only diffs of the file. This would require the masterlists to be always hosted in an SVN repository, which reduces flexibility somewhat, and I've always found SVN to be really slow in practice. If the API isn't too complicated, it might be worth a shot though. + - Masterlist updating could also just use a straightforward HTTP GET request, and if this method is chosen then the Accept-Encoding header should be sent with appropriate values so that the server can use compression when serving the file. Unfortunately, Google Code doesn't seem to use it, though other sites do. Data Files Format diff --git a/docs/BOSS Metadata File Syntax.html b/docs/BOSS Metadata File Syntax.html index 0c216770..6085be06 100644 --- a/docs/BOSS Metadata File Syntax.html +++ b/docs/BOSS Metadata File Syntax.html @@ -109,7 +109,7 @@ This documentation is a work in progress, covering a syntax that is also still a
  • Strings should be enclosed in single quotes if they contain any of the following characters: -?:,[]{}#&*!|>"%@`, unless they also contain any single quotes, in which case the string should be enclosed in double quotes instead. Any backslashes or double quotes within a double-quoted string should be escaped with a backslash, ie. " → \" and \ → \\.
  • Strings are case-sensitive, apart from file paths, regular expressions and checksums. -

    These docs give all example syntax in YAML's least compact representation: you can also use the compact representation, but as it is less readable, this is not recommended. +

    These docs give all example syntax in YAML's least compact representation: you can also use a more compact representation, but it's better for the masterlist to be readable than compact.

    Some properties of file paths as used by BOSS:

    +

    Please test any changes you make before uploading them. One way of doing this is to run BOSS, another is to copy/paste what you've changed into an online parser such as this one, though it won't catch condition syntax errors or non-syntax errors such as having the same file both required and incompatible.

    Metadata File Structure

    The root of a metadata file is a key-value map. BOSS will recognise the following keys, none of which are required: diff --git a/docs/BOSS Readme.html b/docs/BOSS Readme.html index 86d55f69..51dec250 100644 --- a/docs/BOSS Readme.html +++ b/docs/BOSS Readme.html @@ -110,6 +110,8 @@ This documentation is a work in progress, covering an application that is also s

  • Dirty Edits, Mod Cleaning & CRCs
  • File Permissions +
  • API Readme (separate document) +
  • Metadata File Syntax (separate document)

    Introduction

    @@ -153,6 +155,55 @@ This documentation is a work in progress, covering an application that is also s

    Sorting Plugins

    Customising BOSS's Results

    +
    + +
    BOSS's metadata editor.
    +
    +

    BOSS's sorting algorithm uses the contents of plugins to work out where they should load relative to one another. However, this is sometimes not enough for it to produce a correct load order. In such cases, additional metadata can be supplied so that the plugins get sorted correctly. +

    This metadata is supplied to BOSS by two sources: one is the masterlist, which is maintained by the BOSS Team, and the other is the userlist, which you can edit as you desire. +

    The metadata editor allows you to manage the metadata stored in their userlist without having to use a text editor. When opened, it lists all the plugins you have installed, and any that are missing but have existing user-added metadata, in alphabetical order. Clicking on a plugin then opens its metadata for editing, and displays any existing metadata in the fields to the right of the plugin list. These fields are explained in the table below. +

    The Add …, Edit … and Remove … buttons are used to edit the contents of the list field currently visible. The Edit … and Remove … buttons are greyed out if no row is selected or if the selected row is not user-added metadata, but comes from the masterlist or the plugin itself. Metadata that comes from the masterlist or the plugin itself cannot be edited or removed, only added to. This is to prevent users from accidently overriding important metadata. +

    The Save Changes button will save any user-added metadata to your userlist, including any changes made, then exit the metadata editor. The Cancel button will exit the editor without saving any changes. + + + +
    FieldDescription +
    Enable User ChangesIf this is checked, BOSS will use the metadata you supplied when sorting the current plugin. Otherwise, your metadata will be ignored. +
    PriorityThis is a very rough control for determining how far up or down the load order a plugin should go. Plugins with equal priority will be compared as normal by the sorting algorithm, but plugins with a larger priority value will load later in the load order than plugins with a smaller priority value, unless the former depends on the latter or the latter is explicity told to load after the former. Plugins have a default priority of 0. +
    RequirementsThis is a list of files that are required by the current plugin for it to function correctly. BOSS will refuse to sort your load order if any of the listed files are missing, and will instead display an error message detailing the problem. +

    Any file, not just plugins, can be listed here, and each file has three sub-fields: +

      +
    • The filename is the path, relative to the game's Data folder, of the file to be checked for. This sub-field is required. +
    • The display name is optional, and if specified will be used instead of the filename in any error messages that are displayed if a problem is encountered relating to the file. +
    • The condition is the optional condition string that is used to determine if the file should be checked for. If left blank, the file will always be checked for. Condition strings are a relatively advanced part of BOSS's functionality, and their syntax is covered in the Metadata Metadata Syntax documentation. +
    +

    Note that listing a plugin's masters as requirements is pointless, as BOSS already checks them. +

    IncompatibilitiesThis is a list of files that are incompatible with the plugin. BOSS will refuse to sort your load order if any of the listed files are found, and will instead display an error message detailing the problem. +

    Any file, not just plugins, can be listed here, and each file has the same three sub-fields as for the requirements field. +

    Load AfterThis is a list of plugins which, if present, the current plugin must load after, but which are not required. This field can be used for resolving specific compatibility issues. +
    MessagesA list of messages that are to be displayed for the plugin in BOSS's report. These have no bearing on a plugin's position in the load order, but the messages BOSS provides for plugins are one of its most important features. +

    Each message has four sub-fields: +

      +
    • The type is fairly self-explanatory, and is used to provide messages of varying severity with appropriate emphasis in BOSS's report. +
    • The content is the actual message text. +
    • The condition is, like for the corresponding file sub-field, used to determine if the message should be displayed. If left blank, the message is displayed. +
    • The language is the language BOSS runs in that the message will be displayed for. For example, if the message language is set to English and BOSS is running in German, then the message will not be displayed. If left unspecified, the message will be shown no matter what language BOSS is running in. +
    +

    If a message's condition determines that the message should be displayed, but the language is specified and doesn't match the language BOSS is running in, then the message will not be displayed. However, if the condition determines the message should not be displayed and the language matches the language BOSS is running in, the message will still not be displayed. +

    Bash TagsA list of Bash Tags. These are used by the Wrye *ash utilities when generating a Bashed Patch. BOSS's metadata includes Bash Tag addition and removal suggestions, and any Bash Tags that came with the plugin are also displayed. +

    As BOSS can suggest Bash Tags be added or removed, it is possible for the same Tag to appear twice, being suggested for addition and removal. In such cases, removal overrides addition. +

    Each Bash Tag has three sub-fields: +

      +
    • The state determines whether the Tag is to be suggested for addition or removal. +
    • The name is the actual Bash Tag name. +
    • The condition decides if the Tag is to be suggested or not. It functions as for files and messages. +
    +

    A few Bash Tags are used by BOSS in its load order error checking: +

      +
    • If an active mod has the Deactivate tag applied (not merely suggested), an error message will be displayed for it. +
    • If a plugin's masters are missing, an error message will be displayed for it. Filter patches are special mods designed for use with a Bashed Patch that do not require all their masters to be present, and so any plugin with the Filter tag applied and missing masters will not cause any errors to be displayed. +
    +

    The Results Report