Change gtest branch from master to main

Add Nehrim repo

README.md

@@ -18,34 +18,37 @@ Open there a console and copy in: ``git clone https://github.com/Modorganizer2/m
 Now you need to install all required software to build Mod Organizer.
 
 #### Manual Requirements
-* Qt 5.13.2 http://download.qt.io/official_releases/online_installers/qt-unified-windows-x86-online.exe
+* Qt 5.15.2: [Download Manager](http://download.qt.io/official_releases/online_installers/qt-unified-windows-x86-online.exe)
   #### Qt Packages required:
-  * msvc2017-64
+  * msvc2019-64
   * qtwebengine
+  * Optional: Sources and Windows debug files
   
   Be sure to expand the options to show ALL the available choices and then select those packages. Failure to do so will result in a missing qmake.exe error or other build errors.
-* Visual Studio Community 2019 Link: https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community&rel=16
+* Visual Studio Community 2019: [Download Link](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community&rel=16)
   #### Visual Studio Packages required:
   * .NET desktop development
   * Desktop development for C++
   
   Additionally you need to activate under Individual components:
   * "Windows Universal C Runtime" (Under SDKs, libraries, and frameworks)
-  * "Windows 10 SDK (10.0.17763.0) for Desktop C++ [x86 and x64]" (Under SDKs, libraries, and frameworks)
+  * "Windows 10 SDK (10.0.19041.0) for Desktop C++ \[x86 and x64\]" (Under SDKs, libraries, and frameworks)
   * C++ ATL for v141/v142 build tools (x86 & x64) (Under SDKs, libraries, and frameworks)
   * Ensure v141 build tools are selected for some dependencies that may not support v142 yet
-    * Note: If you have problems building Python due to a "missing SDK version", edit the registry key [HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Microsoft SDKs\Windows\v10.0] to point to the the 10.0.17763.0 SDK.
-* .NET Framework 3.5 Link: https://www.microsoft.com/en-us/download/details.aspx?id=21
+    * *v141 build tools may no longer be needed*
+    * Note: If you have problems building Python due to a "missing SDK version", edit the registry key \[HKEY_LOCAL_MACHINE\\SOFTWARE\\WOW6432Node\\Microsoft\\Microsoft SDKs\\Windows\\v10.0\] to point to the the 10.0.19041.0 SDK.
+* .NET Framework 3.5: [Download Link](https://www.microsoft.com/en-us/download/details.aspx?id=21)
+  * *This is a dependency of the NCC install tool, which may be deprecated soon*
 
 #### Additional Requirements
 ##### Manual Install
-* 7zip (Latest 64Bit) Link: http://www.7-zip.org/a/7z1900-x64.exe
-* Inno 5 or 6(Only required to build Installer) Setup Link: http://www.jrsoftware.org/download.php/is.exe
-* CMake (latest 64bit)  Link: https://github.com/Kitware/CMake/releases/download/v3.14.5/cmake-3.14.5-win64-x64.msi
-* Git Link: https://github.com/git-for-windows/git/releases/download/v2.22.0.windows.1/Git-2.22.0-64-bit.exe
-* Python 3.7.4 (64Bit) Link: https://www.python.org/ftp/python/3.7.4/python-3.7.4-amd64.exe
+* 7zip (Latest 64Bit): [Download Link](https://www.7-zip.org/a/7z1900-x64.exe)
+* Inno 5 or 6 (Only required to build Installer): [Download Link](https://www.jrsoftware.org/download.php/is.exe)
+* CMake (latest 64bit):  [Download Link](https://github.com/Kitware/CMake/releases/download/v3.18.3/cmake-3.18.3-win64-x64.msi)
+* Git: [Download Link](https://github.com/git-for-windows/git/releases/download/v2.28.0.windows.1/Git-2.28.0-64-bit.exe)
+* Python 3 (64Bit): [Download Link](https://www.python.org/ftp/python/3.8.6/python-3.8.6-amd64.exe)
   * *If you've previously used a Python 2 version of Umbrella, you may need to run `git clean -xdf` to remove cached data which interferes with Python 3.*
-* Strawberry Perl Link: http://strawberryperl.com/download/5.30.0.1/strawberry-perl-5.30.0.1-64bit.msi
+* Strawberry Perl: [Download Link](http://strawberryperl.com/download/5.32.0.1/strawberry-perl-5.32.0.1-64bit.msi)
 ##### Chocolatey
 Optionally, you can use [Chocolatey](https://chocolatey.org/install) to install these dependencies. The packages are:
 * 7zip
@@ -63,6 +66,8 @@ You can specify a build location using ``-d <directory path>``
 If you wish to rebuild only one target once everything is complete, you simply delete the relevant txt file or directory in the progress folder, e.g C:\modorganizer-umbrella\progress\modorganizer
 Then rerun the build script and it will rebuild the relevant project.
 
+To build an individual project once the umbrella script has completed successfully, you can also use the relative VS solution, making sure to set the configuration to RelWithDebugInfo.
+
 ## Purpose
 This repository contains a meta build-system that is able to download and build MO subprojects and dependencies as necessary.
 It can be used to build the whole project to produce a build that should be equivalent to the release or to build subprojects (i.e. plugins) with the minimum of dependencies.
@@ -118,4 +123,4 @@ optional arguments:
 I'd suggest to use a destination folder that isn't too deep, some dependencies don't handle long paths well.
 If the make target is left empty, everything is built. A incomplete lists of targets can be found [here](targets.md).
 
-Here are the dependency graphs that currently unimake takes care of: https://imgur.com/TT7TFGg.png
+Here are the dependency graphs that currently unimake takes care of: https://www.dropbox.com/s/rqi1wrboevrxelu/graph.pdf?dl=1

config.py

@@ -46,6 +46,9 @@ config = {
     'Release_Build': False,  #Used to override certain versions in umbrella when doing an officail release
 
                             #eg. Using the usvfs_version below instead of the Main_Branch config
+
+    'override_build_version': '', #used to override the modorganizer.exe output version, useful for Appveyor nightlies and dev builds.
+
     'vc_CustomInstallPath': '',  # If you installed VC to a custom location put the full path here
                                  # eg.  'E:\Microsoft Visual Studio 14.0'
     'qt_CustomInstallPath': r'',  # If you installed QT to a custom location put the full path here
@@ -71,58 +74,63 @@ config = {
 
     'Main_Author': 'ModOrganizer2',  # the current maintainer
     'Dev_Branch': "master",
-    'Release_Branch': "2_2_2",
+    'Release_Branch': "2_3_0",
+    'Feature_Branch': None, # Specify a branch or None to disable, used in place of the build branch when found
     'Distrib_Author': 'TanninOne',  # the current distribution (and the original Author)
     'Work_Author': '',  # yourself
 
     # manualy set all versions
     '7zip_version': '19.00',
-    'boost_version': '1.71.0',
+    'boost_version': '1.75.0',
     'boost_version_tag': '',
-    'fmt_version': '6.0.0',
+    'fmt_version': '7.1.3',
     'googletest_version': '1.8.0', # unused. We use the latest source
     'grep_version': '3.3',
-    'icu_version': '65',
-    'icu_version_minor': '1',
-    'loot_version': '0.15.0',
-    'loot_commit': 'g629fdab',
-    'loot_branch': 'master',
-    'lz4_version': '1.9.2',
+    'icu_version': '68',
+    'icu_version_minor': '2',
+    'loot_version': '0.16.1',
+    'loot_commit': 'gd8c9b98',
+    'loot_branch': '0.16.1',
+    'lz4_version': '1.9.3',
     'lz4_version_minor': '', # leave empty if no patch version (1.2.3.x)
-    'nasm_version': '2.14.02',
-    'nuget_version': '4.9.3',
-    'nmm_version': '0.70.9',
-    'openssl_version': '1.1.1d',
-    'pyqt_version': '5.13.1',
+    'nasm_version': '2.15.05',
+    'nuget_version': '5.8.1',
+    'nmm_version': '0.71.2',
+    'openssl_version': '1.1.1i',
+    'pyqt_version': '5.15.2',
+    'pyqt_pypi_hash': '28/6c/640e3f5c734c296a7193079a86842a789edb7988dca39eab44579088a1d1',
     'pyqt_dev_version': '', # leave empty for a standard release
-    'python_version': '3.7',
-    'python_version_minor': '.5',
+    'pyqt_builder_version': '1.7.0',
+    'pyqt_sip_version': '12.8.1',
+    'python_version': '3.8',
+    'python_version_minor': '.7',
     'bzip2_version': '1.0.6', # For python
-    'sip_version': '4.19.19',
+    'sip_version': '5.5.0',
     'sip_dev_version': '', # leave empty for a standard release
-    'spdlog_version': 'v1.4.2',
-    'qt_version': '5.13',
+    'spdlog_version': 'v1.8.2',
+    'qt_version': '5.15',
     'qt_version_minor': '2',
-    'qt_version_appveyor': '5.13',
+    'qt_version_appveyor': '5.15',
     'qt_version_minor_appveyor': '2',
-    'qt_vc_version': '15.0',
+    'qt_vs_version': '16.0',
     'vc_platformtoolset': 'v142',
-    'vc_TargetPlatformVersion': '10.0.17763.0',
-    'vc_version': '16.0',
+    'vc_TargetPlatformVersion': '10.0.19041.0',
+    'vs_version': '16.0',
     'vc_version_for_boost': '14.2',
     'WixToolset_version': '311',
     'zlib_version': '1.2.11',
     'explorer++_version': '1.3.5',
-    'libbsarch_version': '0.0.7',
+    'libbsarch_version': '0.0.8',
 
     #stylesheets
-    'paper-light-and-dark_version': '5.3',
+    'paper-light-and-dark_version': '6.0',
     'paper-automata_version': '2.2',
     'paper-mono_version': '2.1',
-    '1809-dark-mode_version': '1.3',
+    '1809-dark-mode_version': '2.0',
+    'ModOrganizer_Style_Morrowind_version': '1.0',
 
     #the usvfs version below will only be used if
-    'usvfs_version': 'v0.4.6',
+    'usvfs_version': 'v0.4.8',
 
     'optimize': True,  # activate link-time code generation and other optimization.  This massively increases build time but
                        # produces smaller binaries and marginally faster code

eggs/__init__.py

@@ -25,7 +25,9 @@ def download(url, filename):
     if os.path.exists(filename):
         return False
 
-    data = urllib.request.urlopen(url)
+    user_agent = 'Mod Organizer 2/Umbrella'
+    req = urllib.request.Request(url, headers={'User-Agent' : user_agent}) 
+    data = urllib.request.urlopen(req)
     with open(filename, 'wb') as outfile:
         while True:
             block = data.read(4096)

makefile.uni.py

@@ -20,11 +20,12 @@ import shutil
 import zipfile
 
 from config import config
+from pathlib import Path
 from string import Formatter
 from glob import glob
 from unibuild import Project
-from unibuild.modules import build, cmake, git, github, urldownload, msbuild, appveyor
-from unibuild.projects import boost, fmt, googletest, libloot, lz4, nasm, ncc, openssl, sevenzip, sip, usvfs, python, pyqt5, qt5, spdlog, zlib, nuget, libbsarch, boost_di, stylesheets
+from unibuild.modules import build, cmake, git, github, urldownload, msbuild, appveyor, Patch
+from unibuild.projects import boost, fmt, googletest, libloot, lz4, nasm, ncc, usvfs_bin, openssl, sevenzip, sip, usvfs, python, pyqt5, qt5, spdlog, zlib, nuget, libbsarch, boost_di, stylesheets
 from unibuild.utility import FormatDict
 from unibuild.utility.config_utility import cmake_parameters, qt_inst_path
 
@@ -42,21 +43,32 @@ def gen_userfile_content(project):
         return res
 
 
+# Set the feature branch on appveyor if the build branch is not the default branch
+if config['Appveyor_Build']:
+    branch = os.getenv('APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH')
+    branch = branch if branch is not None else os.getenv('APPVEYOR_REPO_BRANCH')
+    if branch is not None and branch != 'master':
+        config['Feature_Branch'] = branch
+
 for author, git_path, path, branch, dependencies, Build in [
-    (config['Main_Author'], "modorganizer-game_features", "game_features", config['Build_Branch'], [], False),
-    (config['Main_Author'], "modorganizer-archive", "archive", config['Build_Branch'], ["7zip", "Qt5", "boost"], True),
-    (config['Main_Author'], "modorganizer-uibase", "uibase", config['Build_Branch'], ["Qt5", "boost", "fmt", "spdlog"], True),
-    (config['Main_Author'], "modorganizer-lootcli", "lootcli", config['Build_Branch'], ["libloot", "boost"], True),
-    (config['Main_Author'], "modorganizer-esptk", "esptk", config['Build_Branch'], ["boost"], True),
-    (config['Main_Author'], "modorganizer-bsatk", "bsatk", config['Build_Branch'], ["zlib", "boost", "lz4"], True),
+    (config['Main_Author'], "cmake_common", "cmake_common", config['Build_Branch'], ["boost", "zlib", "fmt"], False),
+    (config['Main_Author'], "modorganizer-uibase", "uibase", config['Build_Branch'], ["Qt5", "boost", "fmt", "spdlog", "lz4", "zlib", "GTest"], True),
+    (config['Main_Author'], "modorganizer-game_features", "game_features", config['Build_Branch'], ["modorganizer-uibase"], False),
+    (config['Main_Author'], "modorganizer-archive", "archive", config['Build_Branch'], ["7zip", "Qt5", "boost", "modorganizer-uibase"], True),
+    (config['Main_Author'], "modorganizer-lootcli", "lootcli", config['Build_Branch'], ["libloot", "boost", "modorganizer-uibase"], True),
+    (config['Main_Author'], "modorganizer-esptk", "esptk", config['Build_Branch'], ["boost", "modorganizer-uibase"], True),
+    (config['Main_Author'], "modorganizer-bsatk", "bsatk", config['Build_Branch'], ["zlib", "boost", "lz4", "modorganizer-uibase"], True),
     (config['Main_Author'], "modorganizer-nxmhandler", "nxmhandler", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-helper", "helper", config['Build_Branch'], ["Qt5","boost"], True),
+    (config['Main_Author'], "modorganizer-helper", "helper", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
     (config['Main_Author'], "modorganizer-game_gamebryo", "game_gamebryo", config['Build_Branch'], ["Qt5", "modorganizer-uibase",
                                                                                                    "modorganizer-game_features",
                                                                                                    "lz4"], True),
     (config['Main_Author'], "modorganizer-game_oblivion", "game_oblivion", config['Build_Branch'], ["Qt5", "modorganizer-uibase",
                                                                                                    "modorganizer-game_gamebryo",
                                                                                                    "modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-game_nehrim", "game_nehrim", config['Build_Branch'], ["Qt5", "modorganizer-uibase",
+                                                                                                "modorganizer-game_gamebryo",
+                                                                                                "modorganizer-game_features"], True),
     (config['Main_Author'], "modorganizer-game_fallout3", "game_fallout3", config['Build_Branch'], ["Qt5", "modorganizer-uibase",
                                                                                                    "modorganizer-game_gamebryo",
                                                                                                    "modorganizer-game_features"], True),
@@ -89,25 +101,29 @@ for author, git_path, path, branch, dependencies, Build in [
                                                                                          "modorganizer-game_features"], True),
     (config['Main_Author'], "modorganizer-tool_inieditor", "tool_inieditor", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
     (config['Main_Author'], "modorganizer-tool_inibakery", "tool_inibakery", config['Build_Branch'], ["modorganizer-uibase", "modorganizer-game_features"], True),
-    (config['Main_Author'], "modorganizer-tool_configurator", "tool_configurator", config['Build_Branch'], ["PyQt5"], True),
-    (config['Main_Author'], "modorganizer-fnistool", "fnistool", config['Build_Branch'],  ["PyQt5"], True),
     (config['Main_Author'], "modorganizer-preview_base", "preview_base", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-preview_dds", "preview_dds", config['Build_Branch'],  ["PyQt5"], True),
     (config['Main_Author'], "modorganizer-diagnose_basic", "diagnose_basic", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-script_extender_plugin_checker", "script_extender_plugin_checker", config['Build_Branch'],  ["PyQt5"], True),
-    (config['Main_Author'], "modorganizer-form43_checker", "form43_checker", config['Build_Branch'],  ["PyQt5"], True),
     (config['Main_Author'], "modorganizer-check_fnis", "check_fnis", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-installer_bain", "installer_bain", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-installer_manual", "installer_manual", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
+    (config['Main_Author'], "modorganizer-installer_bain", "installer_bain", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-installer_manual", "installer_manual", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "modorganizer-game_features"], True),
     (config['Main_Author'], "modorganizer-installer_bundle", "installer_bundle", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-installer_quick", "installer_quick", config['Build_Branch'], ["Qt5", "modorganizer-uibase"], True),
-    (config['Main_Author'], "modorganizer-installer_fomod", "installer_fomod", config['Build_Branch'], ["Qt5", "modorganizer-uibase","modorganizer-game_features"], True),
-    (config['Main_Author'], "modorganizer-installer_ncc", "installer_ncc", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "ncc","modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-installer_quick", "installer_quick", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-installer_fomod", "installer_fomod", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-installer_fomod_csharp", "installer_fomod_csharp", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-installer_wizard", "installer_wizard", config['Build_Branch'], ["Python", "PyQt5"], True),
+    (config['Main_Author'], "modorganizer-installer_omod", "installer_omod", config["Build_Branch"], ["Qt5", "modorganizer-uibase", "modorganizer-game_features"], True),
+    (config['Main_Author'], "modorganizer-installer_ncc", "installer_ncc", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "ncc", "modorganizer-game_features"], True),
     (config['Main_Author'], "modorganizer-bsa_extractor", "bsa_extractor", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "modorganizer-bsatk"], True),
-    (config['Main_Author'], "modorganizer-plugin_python", "plugin_python", config['Build_Branch'], ["Qt5", "boost", "Python",
-                                                                                                   "modorganizer-uibase", "sip","modorganizer-game_features"], True),
-    (config['Main_Author'], "githubpp", "githubpp", config['Build_Branch'], ["Qt5"], True),
+    (config['Main_Author'], "modorganizer-plugin_python", "plugin_python", config['Build_Branch'], ["Qt5", "boost", "modorganizer-uibase", "modorganizer-game_features",
+                                                                                                    "Python", "PyQt5", "sip"], True),
+    (config['Main_Author'], "modorganizer-tool_configurator", "tool_configurator", config['Build_Branch'], ["modorganizer-plugin_python"], True),
+    (config['Main_Author'], "modorganizer-fnistool", "fnistool", config['Build_Branch'],  ["modorganizer-plugin_python"], True),
+    (config['Main_Author'], "modorganizer-script_extender_plugin_checker", "script_extender_plugin_checker", config['Build_Branch'],  ["modorganizer-plugin_python"], True),
+    (config['Main_Author'], "modorganizer-form43_checker", "form43_checker", config['Build_Branch'],  ["modorganizer-plugin_python"], True),
+    (config['Main_Author'], "modorganizer-preview_dds", "preview_dds", config['Build_Branch'],  ["modorganizer-plugin_python"], True),
+    (config['Main_Author'], "githubpp", "githubpp", config['Build_Branch'], ["Qt5", "boost"], True),
     (config['Main_Author'], "modorganizer-bsapacker", "bsapacker", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "libbsarch", "boost_di"], True),
+    (config['Main_Author'], "modorganizer-preview_bsa", "preview_bsa", config['Build_Branch'], ["Qt5", "modorganizer-uibase", "libbsarch" ], True),
     (config['Main_Author'], "modorganizer", "modorganizer", config['Build_Branch'], ["Qt5", "boost", "usvfs_32",
                                                                                     "modorganizer-uibase",
                                                                                     "modorganizer-archive",
@@ -118,7 +134,7 @@ for author, git_path, path, branch, dependencies, Build in [
                                                                                     "modorganizer-lootcli",
                                                                                     "usvfs", "githubpp",
                                                                                     "ncc", "openssl",
-                                                                                    "paper-light-and-dark", "paper-automata", "paper-mono", "1809-dark-mode"], True),
+                                                                                    "paper-light-and-dark", "paper-automata", "paper-mono", "1809-dark-mode", "ModOrganizer_Style_Morrowind"], True),
 ]:
     cmake_param = cmake_parameters() + ["-DCMAKE_INSTALL_PREFIX:PATH={}".format(config["paths"]["install"])]
     # build_step = cmake.CMake().arguments(cmake_param).install()
@@ -129,43 +145,38 @@ for author, git_path, path, branch, dependencies, Build in [
     project = Project(git_path)
 
     if Build:
-        if config['Appveyor_Build']:
-            appveyor_cmake_step = cmake.CMakeJOM().arguments(cmake_param).install()
+        #prepare build version override step
+        patch_version_step = Patch.RegexReplace("src/version.rc",
+                                                "#define VER_FILEVERSION_STR.*\n",
+                                                "#define VER_FILEVERSION_STR \"{}\"\n"
+                                                .format(config['override_build_version']))
 
-            for dep in dependencies:
-                appveyor_cmake_step.depend(dep)
-            if os.getenv("APPVEYOR_PROJECT_NAME","") == git_path:
-                project.depend(
-                    appveyor_cmake_step.depend(
-                        appveyor.SetProjectFolder(os.getenv("APPVEYOR_BUILD_FOLDER", ""))
-                    )
-                )
-            else:
-                project.depend(
-                    appveyor_cmake_step.depend(
-                        github.Source(author, git_path, branch, super_repository=tl_repo).set_destination(path)
-                    )
-                )
+        #begin build
+        vs_cmake_step = cmake.CMakeVS().arguments(cmake_param).install()
+
+        if git_path != "cmake_common":
+            vs_cmake_step.depend("cmake_common")
+
+        for dep in dependencies:
+            vs_cmake_step.depend(dep)
+
+        #vs_target = "Clean;Build" if config['rebuild'] else "Build"
+        vs_msbuild_step = msbuild.MSBuild(os.path.join("vsbuild", "INSTALL.vcxproj"), None, None,
+                                          "{}".format("x64" if config['architecture'] == 'x86_64' else "x86"),
+                                          config['build_type'])
+
+        source_retrieval_step = github.Source(author, git_path, branch, feature_branch=config['Feature_Branch'], super_repository=tl_repo).set_destination(path)
+
+        if git_path == "modorganizer" and config['override_build_version']:
+            #add patching step that depends on source retrieval
+            patch_version_step.depend(source_retrieval_step)
+            vs_cmake_step.depend(patch_version_step)
         else:
-            vs_cmake_step = cmake.CMakeVS().arguments(cmake_param).install()
+            vs_cmake_step.depend(source_retrieval_step)
 
-            for dep in dependencies:
-                vs_cmake_step.depend(dep)
-
-            #vs_target = "Clean;Build" if config['rebuild'] else "Build"
-            vs_msbuild_step = msbuild.MSBuild(os.path.join("vsbuild", "INSTALL.vcxproj"), None, None,
-                                              "{}".format("x64" if config['architecture'] == 'x86_64' else "x86"),
-                                              config['build_type'])
-
-            project.depend(
-                vs_msbuild_step.depend(
-                    vs_cmake_step.depend(
-                        github.Source(author, git_path, branch, super_repository=tl_repo).set_destination(path)
-                    )
-                )
-            )
+        project.depend(vs_msbuild_step.depend(vs_cmake_step))
     else:
-        project.depend(github.Source(author, git_path, branch, super_repository=tl_repo)
+        project.depend(github.Source(author, git_path, branch, feature_branch=config['Feature_Branch'], super_repository=tl_repo)
                        .set_destination(path))
 
 
@@ -222,16 +233,16 @@ def copy_licenses(context):
         sip_path = os.path.join(build_path, "sip-{}.dev{}".format(config['sip_version'], config['sip_dev_version']))
     else:
         sip_path = os.path.join(build_path, "sip-{}".format(config['sip_version']))
+    shutil.copy(os.path.join(sip_path, "LICENSE"), os.path.join(license_path, "sip.txt"))
+    shutil.copy(os.path.join(sip_path, "LICENSE-GPL2"), os.path.join(license_path, "GPL-v2.0.txt"))
+    # license path is different in the prebuilt archive
     if config['Appveyor_Build']:
-        shutil.copy(os.path.join(build_path, "python-{}{}".format(config['python_version'], config['python_version_minor']),"Licenses","Sip", "LICENSE"), os.path.join(license_path, "sip.txt"))
-        shutil.copy(os.path.join(build_path, "python-{}{}".format(config['python_version'], config['python_version_minor']),"Licenses","Sip", "LICENSE-GPL2"), os.path.join(license_path, "GPL-v2.0.txt"))
         shutil.copy(os.path.join(build_path, "python-{}{}".format(config['python_version'], config['python_version_minor']),"Licenses","Python","LICENSE"), os.path.join(license_path, "python.txt"))
     else:
-        shutil.copy(os.path.join(sip_path, "LICENSE"), os.path.join(license_path, "sip.txt"))
-        shutil.copy(os.path.join(sip_path, "LICENSE-GPL2"), os.path.join(license_path, "GPL-v2.0.txt"))
         shutil.copy(os.path.join(build_path, "python-{}{}".format(config['python_version'], config['python_version_minor']),"LICENSE"), os.path.join(license_path, "python.txt"))
-    shutil.copy(os.path.join(build_path, "openssl-{}".format(config['openssl_version']), "LICENSE"), os.path.join(license_path, "openssl.txt"))
     shutil.copy(os.path.join(build_path, "modorganizer_super", "lootcli", "vsbuild", "src", "external", "src", "cpptoml", "LICENSE"), os.path.join(license_path, "cpptoml.txt"))
+    shutil.copy(os.path.join(build_path, "openssl-{}".format(config['openssl_version']), "LICENSE"), os.path.join(license_path, "openssl.txt"))
+
     shutil.copy(os.path.join(build_path, "boost_{}".format(boost_tag_version.replace(".", "_")), "LICENSE_1_0.txt"), os.path.join(license_path, "boost.txt"))
     shutil.copy(os.path.join(build_path, "7zip-{}".format(config['7zip_version']), "DOC", "License.txt"), os.path.join(license_path, "7zip.txt"))
     shutil.copy(os.path.join(build_path, "7zip-{}".format(config['7zip_version']), "DOC", "copying.txt"), os.path.join(license_path, "GNU-LGPL-v2.1.txt"))
@@ -247,28 +258,29 @@ Project("licenses") \
         .depend(urldownload.URLDownload("https://www.gnu.org/licenses/gpl-3.0.txt", 0))
         .depend(urldownload.URLDownload("https://raw.githubusercontent.com/Microsoft/DirectXTex/master/LICENSE", 0).set_download_filename("DXTex.txt"))
         .depend(urldownload.URLDownload("https://creativecommons.org/licenses/by-sa/3.0/legalcode.txt", 0).set_download_filename("BY-SA-v3.0.txt"))
+        .depend("sip")
         .depend("modorganizer"))
 
 
 def copy_explorerpp(context):
-    target_path = os.path.join(config["paths"]["install"], "bin", "explorer++")
-    build_path = os.path.join(config["paths"]["build"], "explorer++")
-    shutil.copytree(build_path, target_path)
+    target_path = Path(config["paths"]["install"]) / "bin" / "explorer++"
+    build_path = Path(config["paths"]["build"]) / "explorer++"
+    shutil.copytree(build_path, target_path, dirs_exist_ok = True)
     return True
 
 
 Project("explorerpp") \
     .depend(build.Execute(copy_explorerpp)
-        .depend(urldownload.URLDownload("https://explorerplusplus.com/software/explorer++_{}_x64.zip".format(config["explorer++_version"]), 0)
-            .set_destination("explorer++")))
+            .depend(urldownload.URLDownload("https://explorerplusplus.com/software/explorer++_{}_x64.zip".format(config["explorer++_version"]), 0)
+                    .set_destination("explorer++")))
 
 
 if config['Installer']:
-    build_installer =  build.Run(r'"{}" {}'.format(config["paths"]["InnoSetup"],"dist/MO2-Installer.iss"),
-              name="Build MO2 Installer")
+    build_installer = build.Run(r'"{}" {}'.format(config["paths"]["InnoSetup"],"dist/MO2-Installer.iss"),
+                                name="Build MO2 Installer")
 
     installer = Project("Installer") \
         .depend(build_installer
-            .depend(github.Source(config['Main_Author'], "modorganizer-Installer", config['Build_Branch'], super_repository=tl_repo)
-                .set_destination("installer")) \
-                    .depend("modorganizer").depend("usvfs").depend("usvfs_32").depend("translationsBuild").depend("modorganizer-fnistool"))
+                .depend(github.Source(config['Main_Author'], "modorganizer-Installer", config['Build_Branch'], feature_branch=config['Feature_Branch'], super_repository=tl_repo)
+                        .set_destination("installer"))
+                .depend("modorganizer").depend("usvfs").depend("usvfs_32").depend("translationsBuild").depend("modorganizer-fnistool"))

patches/PyQt5_init.py

@@ -1,49 +0,0 @@
-# Copyright (c) 2019 Riverbank Computing Limited <info@riverbankcomputing.com>
-#
-# This file is part of PyQt5.
-#
-# This file may be used under the terms of the GNU General Public License
-# version 3.0 as published by the Free Software Foundation and appearing in
-# the file LICENSE included in the packaging of this file.  Please review the
-# following information to ensure the GNU General Public License version 3.0
-# requirements will be met: http://www.gnu.org/copyleft/gpl.html.
-#
-# If you do not wish to use this file under the terms of the GPL version 3.0
-# then you may purchase a commercial license.  For more information contact
-# info@riverbankcomputing.com.
-#
-# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
-# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
-
-# flake8: noqa
-
-from ctypes import CDLL
-
-
-def find_qt():
-    import os
-
-    try:
-        qt5 = CDLL("Qt5Core")
-    except OSError as e:
-        path = os.environ['PATH']
-
-        dll_dir = os.path.dirname(__file__) + '\\Qt\\bin'
-        if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
-            path = dll_dir + ';' + path
-            os.environ['PATH'] = path
-        else:
-            for dll_dir in path.split(';'):
-                if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
-                    break
-            else:
-                raise ImportError("unable to find Qt5Core.dll on PATH")
-
-        try:
-            os.add_dll_directory(dll_dir)
-        except AttributeError:
-            pass
-
-
-find_qt()
-del find_qt

patches/pyqt5_configure.patch

@@ -0,0 +1,36 @@
+--- builder.py	(date 1575938575861)
++++ builder.py	(date 1575938575861)
+@@ -441,17 +441,29 @@
+                     'QMAKE_LFLAGS += -Wl,--version-script={}.exp'.format(
+                             buildable.target))
+ 
+-        pro_lines.append(
++        if project.py_platform == 'win32':
++            for include_dir in project.py_include_dir.split(';'):
++                pro_lines.append(
++                        'INCLUDEPATH += {}'.format(
++                                self.qmake_quote(include_dir)))
++        else:
++            pro_lines.append(
+                 'INCLUDEPATH += {}'.format(
+-                        self.qmake_quote(project.py_include_dir)))
++                    self.qmake_quote(project.py_include_dir)))
+ 
++        if project.py_pylib_shlib != "":
++            pro_lines.append(
++                    'DEFINES += {}'.format(
++                        self.qmake_quote("PYTHON_LIB=\"{}\"".format(
++                                project.py_pylib_shlib))))
++
+         # Python.h on Windows seems to embed the need for pythonXY.lib, so tell
+         # it where it is.
+         # TODO: is this still necessary for Python v3.8?
+         if not buildable.static:
+             pro_lines.extend(['win32 {',
+-                    '    LIBS += -L{}'.format(
+-                            self.qmake_quote(project.py_pylib_dir)),
++                    '    LIBS += -L{}{}'.format(
++                            self.qmake_quote(project.py_pylib_dir), '' if project.py_pylib_lib == '' else ' -l' + project.py_pylib_lib),
+                     '}'])
+ 
+         # Add any installables from the buildable

patches/pyqt5_configure_init.patch

@@ -1,55 +0,0 @@
-Index: configure.py
-IDEA additional info:
-Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
-<+>UTF-8
-===================================================================
---- configure.py	(date 1562041436573)
-+++ configure.py	(date 1562041436573)
-@@ -1552,26 +1552,32 @@
-     if target_config.py_platform == 'win32':
-         inf.write("""
- 
-+from ctypes import CDLL
-+
-+
- def find_qt():
-     import os
- 
--    path = os.environ['PATH']
-+    try:
-+        qt5 = CDLL("Qt5Core")
-+    except OSError as e:
-+        path = os.environ['PATH']
- 
--    dll_dir = os.path.dirname(__file__) + '\\\\Qt\\\\bin'
--    if os.path.isfile(dll_dir + '\\\\Qt5Core.dll'):
--        path = dll_dir + ';' + path
--        os.environ['PATH'] = path
--    else:
--        for dll_dir in path.split(';'):
--            if os.path.isfile(dll_dir + '\\\\Qt5Core.dll'):
--                break
--        else:
--            raise ImportError("unable to find Qt5Core.dll on PATH")
-+        dll_dir = os.path.dirname(__file__) + '\\Qt\\bin'
-+        if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
-+            path = dll_dir + ';' + path
-+            os.environ['PATH'] = path
-+        else:
-+            for dll_dir in path.split(';'):
-+                if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
-+                    break
-+            else:
-+                raise ImportError("unable to find Qt5Core.dll on PATH")
- 
--    try:
--        os.add_dll_directory(dll_dir)
--    except AttributeError:
--        pass
-+        try:
-+            os.add_dll_directory(dll_dir)
-+        except AttributeError:
-+            pass
- 
- 
- find_qt()

patches/python_openssl_path.patch

@@ -0,0 +1,10 @@
+--- PCbuild/openssl.props	(revision 5c02a39a0b31a330e06b4d6f44835afb205dc7cc)
++++ PCbuild/openssl.props	(date 1575994670797)
+@@ -13,6 +13,7 @@
+     <_DLLSuffix>-1_1</_DLLSuffix>
+     <_DLLSuffix Condition="$(Platform) == 'ARM'">$(_DLLSuffix)-arm</_DLLSuffix>
+     <_DLLSuffix Condition="$(Platform) == 'ARM64'">$(_DLLSuffix)-arm64</_DLLSuffix>
++    <_DLLSuffix Condition="$(Platform) == 'x64'">$(_DLLSuffix)-x64</_DLLSuffix>
+   </PropertyGroup>
+   <ItemGroup>
+     <_SSLDLL Include="$(opensslOutDir)\libcrypto$(_DLLSuffix).dll" />

targets.md

@@ -50,6 +50,8 @@ modorganizer-game_gamebryo
 
 modorganizer-game_oblivion
 
+modorganizer-game_nehrim
+
 modorganizer-game_fallout3
 
 modorganizer-game_fallout4

unibuild/modules/Patch.py

@@ -17,6 +17,7 @@
 # along with Mod Organizer.  If not, see <http://www.gnu.org/licenses/>.
 import os.path
 import shutil
+import re
 
 from unibuild.task import Task
 from unibuild.utility.lazy import Lazy
@@ -44,6 +45,27 @@ class Replace(Task):
             f.write(data)
         return True
 
+class RegexReplace(Task):
+    def __init__(self, filename, search, substitute):
+        super(RegexReplace, self).__init__()
+        self.__file = filename
+        self.__search = search
+        self.__substitute = substitute
+
+    @property
+    def name(self):
+        return "Replace in {}".format(self.__file)
+
+    def process(self, progress):
+        full_path = os.path.join(self._context["build_path"], self.__file)
+        with open(full_path, "r") as f:
+            data = f.read()
+
+        data = re.sub(self.__search, self.__substitute, data)
+
+        with open(full_path, "w") as f:
+            f.write(data)
+        return True
 
 class Copy(Task):
     def __init__(self, source, destination):

unibuild/modules/build.py

@@ -223,11 +223,12 @@ class Execute(Builder):
 
 class Run(Builder):
     def __init__(self, command, fail_behaviour=Task.FailBehaviour.FAIL, environment=None, working_directory=None,
-                 name=None):
+                 name=None, retries=0):
         super(Run, self).__init__()
         self.__command = Lazy(command)
         self.__name = name
         self.__fail_behaviour = fail_behaviour
+        self.__retries = retries
         self.__environment = Lazy(environment)
         self.__working_directory = Lazy(working_directory)
 
@@ -242,6 +243,8 @@ class Run(Builder):
             logging.error("source path not known for {},"
                           " are you missing a matching retrieval script?".format(self.name))
 
+        tries = 0
+        return_code = -1
         soutpath = os.path.join(self._context["build_path"], "stdout.log")
         serrpath = os.path.join(self._context["build_path"], "stderr.log")
         with open(soutpath, "w") as sout:
@@ -254,13 +257,18 @@ class Run(Builder):
                           else self._context["build_path"])
 
                 sout.write("running {} in {}".format(self.__command(), cwd))
-                proc = Popen(self.__command(),
-                             env=environment,
-                             cwd=cwd,
-                             shell=True,
-                             stdout=sout, stderr=serr)
-                proc.communicate()
-                if proc.returncode != 0:
+
+                while tries <= self.__retries and return_code != 0:
+                    tries += 1
+                    proc = Popen(self.__command(),
+                                 env=environment,
+                                 cwd=cwd,
+                                 shell=True,
+                                 stdout=sout, stderr=serr)
+                    proc.communicate()
+                    return_code = proc.returncode
+
+                if return_code != 0:
                     logging.error("failed to run %s (returncode %s), see %s and %s",
                                   self.__command(), proc.returncode, soutpath, serrpath)
                     return False

unibuild/modules/cmake.py

@@ -175,7 +175,7 @@ class CMakeEdit(Builder):
     def __generator_name(self):
         if self.__type == CMakeEdit.Type.VC:
             return "Visual Studio {} {}" \
-                .format(config['vc_version'].split('.')[0], self.__vc_year(config['vc_version']))
+                .format(config['vs_version'].split('.')[0], self.__vc_year(config['vs_version']))
         elif self.__type == CMakeEdit.Type.CodeBlocks:
             return "CodeBlocks - NMake Makefiles"
 
@@ -254,7 +254,7 @@ class CMakeVS(Builder):
         soutpath = os.path.join(self._context["build_path"], "vs_stdout.log")
         serrpath = os.path.join(self._context["build_path"], "vs_stderr.log")
 
-        vs_generator = "Visual Studio {0} {1}".format(config['vc_version'].split(".", 1)[0],vc_year(config['vc_version']))
+        vs_generator = "Visual Studio {0} {1}".format(config['vs_version'].split(".", 1)[0],vc_year(config['vs_version']))
         # Updated for latest versions of CMake > 3.1 - VS 2019 does not allow use of the deprecated method
         vs_generator_arch = "x64"
         if config["architecture"] == "x86":

unibuild/modules/git.py

@@ -80,15 +80,24 @@ class SuperRepository(Task):
 
 
 class Clone(Repository):
-    def __init__(self, url, branch, super_repository=None, update=True, commit=None, shallowclone=False):
+    def __init__(self, url, branch, feature_branch=None, super_repository=None, update=True, commit=None, shallowclone=False):
         if config['shallowclone']:
             self.shallowclone = True
-        super(Clone, self).__init__(url, branch)
+        super(Clone, self).__init__(url, branch, feature_branch)
         self.__super_repository = super_repository
         self.__base_name = os.path.basename(self._url)
         self.__update = update
         self.__commit = commit
         self.__shallowclone = shallowclone
+
+        if self._feature_branch is not None:
+            proc = Popen([config['paths']['git'], "ls-remote", "--heads", "--exit-code", self._url, self._feature_branch],
+                     env=config["__environment"])
+            if proc is not None:
+                proc.communicate()
+                if proc.returncode != 2:
+                    self._branch = self._feature_branch
+
         if self.__super_repository is not None:
             self._output_file_path = os.path.join(self.__super_repository.path, self.__determine_name())
             self.depend(super_repository)

unibuild/modules/github.py

@@ -15,11 +15,31 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Mod Organizer.  If not, see <http://www.gnu.org/licenses/>.
+import sys
+import os
+import urllib.request
+import json
+import subprocess
+import logging
 from config import config
 from unibuild.modules.git import Clone
 from unibuild.modules.urldownload import URLDownload
 
 
+def Popen(cmd, **kwargs):
+    pc = ''
+    if 'cwd' in kwargs:
+        try:
+            pc += os.path.relpath(kwargs['cwd'], os.path.abspath('..'))
+        except ValueError:
+            pc += kwargs['cwd']
+    pc += '>'
+    for arg in cmd:
+        pc += ' ' + arg
+    print(pc)
+    return subprocess.Popen(cmd,**kwargs)
+
+
 class Release(URLDownload):
     def __init__(self, author, project, version, filename, extension="zip", tree_depth=0):
         super(Release, self) \
@@ -43,10 +63,45 @@ class Tag(URLDownload):
 
 
 class Source(Clone):
-    def __init__(self, author, project, branch="master", super_repository=None, update=True, commit=None, shallowclone=False):
+    def __init__(self, author, project, branch="master", feature_branch=None, super_repository=None, update=True, commit=None, pr_label=None, shallowclone=False):
+        self.__author = author
+        self.__project = project
         if config['shallowclone']:
             self.shallowclone = True
+        self.__pr_label = pr_label
 
         super(Source, self).__init__("https://github.com/{author}/{project}.git".format(author=author, project=project),
-                                     branch, super_repository, update, commit, shallowclone)
+                                     branch, feature_branch, super_repository, update, commit, shallowclone)
+
         self.set_destination(project)
+
+    def process(self, progress):
+        result = super(Source, self).process(progress)
+        if result is False:
+            return False
+
+        if self.__pr_label is not None:
+            url = "https://api.github.com/repos/{}/{}/pulls".format(self.__author, self.__project)
+            response = urllib.request.urlopen(url)
+            data = json.loads(response.read())
+            filtered_data = [x for x in data if x["head"]["label"] == self.__pr_label]
+            sys.stdout.write(filtered_data.__str__())
+            if len(filtered_data) > 0:
+                pr_number = filtered_data[0]["number"]
+                proc = Popen([config['paths']['git'], "fetch", "-q", "origin", "+refs/pull/{}/merge:".format(pr_number)],
+                             cwd=self._context["build_path"],
+                             env=config["__environment"])
+                proc.communicate()
+                if proc.returncode != 0:
+                    logging.error("failed to fetch pr %s (returncode %s)", pr_number, proc.returncode)
+                    return False
+
+                proc = Popen([config['paths']['git'], "checkout", "-qf", "FETCH_HEAD"],
+                             cwd=self._context["build_path"],
+                             env=config["__environment"])
+                proc.communicate()
+                if proc.returncode != 0:
+                    logging.error("failed to checkout pr branch (returncode %s)", proc.returncode)
+                    return False
+
+        return True

unibuild/modules/msbuild.py

@@ -88,6 +88,8 @@ class MSBuild(Builder):
                           "/verbosity:" + verbosity,
                           "/consoleloggerparameters:Summary",
                           "/fileLogger",
+                          "/property:RunCodeAnalysis=false",
+                          "/restore",
                           "/fileloggerparameters:Summary;Verbosity=" + lverbosity]
 
 

unibuild/modules/pipdownload.py

@@ -0,0 +1,145 @@
+# Copyright (C) 2015 Sebastian Herbord.  All rights reserved.
+# Copyright (C) 2016 - 2019 Mod Organizer contributors.
+#
+# This file is part of Mod Organizer.
+#
+# Mod Organizer is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Mod Organizer is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Mod Organizer.  If not, see <http://www.gnu.org/licenses/>.
+import logging
+import os
+import shutil
+import subprocess
+import tarfile
+
+from config import config
+from unibuild.retrieval import Retrieval
+from unibuild.utility import ProgressFile
+from unibuild.utility.context_objects import on_failure
+
+logging.getLogger("requests").setLevel(logging.WARNING)
+
+class PIPDownload(Retrieval):
+
+    def __init__(self, package, version, tree_depth=0, name=None, clean=True):
+        super(PIPDownload, self).__init__()
+        self.__package = package
+        self.__version = version
+        self.__tree_depth = tree_depth
+        self.__clean = clean
+        self.__file_path = os.path.join(config['paths']['download'], "{}-{}.tar.gz".format(package, version))
+        self.__name = "{}-{}".format(package, version) if name is None else name
+
+    @property
+    def name(self):
+        if self.__name is None:
+            return "download {0}".format(self.__file_name)
+        return "download {0}".format(self.__name)
+
+    @name.setter
+    def name(self, value):
+        self.__name = value
+
+    @property
+    def package(self):
+        return self.__package
+
+    @package.setter
+    def package(self, value):
+        self.__package = value
+
+    @property
+    def version(self):
+        return self.__version
+
+    @version.setter
+    def version(self, value):
+        self.__version = value
+
+    def prepare(self):
+        if 'build_path' not in self._context:
+            output_file_path = os.path.join(config['paths']['build'], "{}-{}".format(self.__package, self.__version))
+            self._context['build_path'] = output_file_path
+
+    def process(self, progress):
+        logging.info("processing download")
+        output_file_path = self._context['build_path']
+        #archive_file_path = os.path.join(config['paths']['download'], self.__file_name)
+
+        if os.path.isfile(output_file_path):
+            logging.info("File already extracted: %s", self.__file_path)
+        else:
+            if os.path.isfile(self.__file_path):
+                logging.info("File already downloaded: %s", self.__file_path)
+            else:
+                logging.info("File not yet downloaded: %s", self.__file_path)
+                self.download(output_file_path, progress)
+            progress.finish()
+
+        if not self.extract(self.__file_path, output_file_path, progress):
+            return False
+        progress.finish()
+
+        builddir = os.listdir(self._context["build_path"])
+        if len(builddir) == 1:
+            self._context["build_path"] = os.path.join(self._context["build_path"], builddir[0])
+        return True
+
+    def download(self, output_file_path, progress):
+        logging.info("Downloading {}-{} to {}".format(self.__package, self.__version, output_file_path))
+        #progress.job = "Downloading"
+        subprocess.check_call(["python", "-m", "pip", "download", "--no-binary=:all:", "--no-deps", "-d", "{}".format(config['paths']['download']), "{}=={}".format(self.__package, self.__version)])
+
+    def extract(self, archive_file_path, output_file_path, progress):
+        def progress_func(pos, size):
+            progress.value = int(pos * 100 / size)
+
+        try:
+            os.makedirs(output_file_path)
+        except Exception:
+            # it does matter if the directory already exists otherwise downloads with tree_depth will fail if the are
+            # extracted a second time on a dirty build environment
+            sub_dirs = os.listdir(output_file_path)
+            if not len(sub_dirs) == 0 and self.__clean:
+                logging.info("Cleaning {}".format(output_file_path))
+                for ls in sub_dirs:
+                    try:
+                        shutil.rmtree(os.path.join(output_file_path, ls))
+                    except Exception:
+                        os.remove(os.path.join(output_file_path, ls))
+
+        logging.info("Extracting {}".format(self.__file_path))
+        output_file_path = "\\\\?\\" + os.path.abspath(output_file_path)
+
+        def extractProgress():
+            progress.value = 0
+            progress.job = "Extracting"
+
+        with on_failure(lambda: shutil.rmtree(output_file_path)):
+            extractProgress()
+            archive_file = ProgressFile(archive_file_path, progress_func)
+            with tarfile.open(fileobj=archive_file, mode='r:gz') as arch:
+                arch.extractall(output_file_path)
+            archive_file.close()
+
+            for i in range(self.__tree_depth):
+                sub_dirs = os.listdir(output_file_path)
+                if len(sub_dirs) != 1:
+                    raise ValueError("unexpected archive structure,"
+                                     " expected exactly one directory in {}".format(output_file_path))
+                source_dir = os.path.join(output_file_path, sub_dirs[0])
+
+                for src in os.listdir(source_dir):
+                    shutil.move(os.path.join(source_dir, src), output_file_path)
+
+                shutil.rmtree(source_dir)
+        return True

unibuild/modules/repository.py

@@ -22,10 +22,11 @@ from unibuild.retrieval import Retrieval
 
 
 class Repository(Retrieval):
-    def __init__(self, url, branch):
+    def __init__(self, url, branch, feature_branch=None):
         super(Repository, self).__init__()
         self._url = url
         self._branch = branch
+        self._feature_branch = feature_branch
         self._dir_name = os.path.basename(self._url)
         self._output_file_path = os.path.join(config["paths"]["build"], self._dir_name)
 

unibuild/modules/urldownload.py

@@ -19,6 +19,7 @@ import logging
 import os
 import io
 import shutil
+import stat
 import subprocess
 import sys
 import tarfile
@@ -164,7 +165,10 @@ class URLDownload(Retrieval):
                 logging.info("Cleaning {}".format(output_file_path))
                 for ls in sub_dirs:
                     try:
-                        shutil.rmtree(os.path.join(output_file_path, ls))
+                        def ensureWritable(function, path, exc_info):
+                            os.chmod(path, stat.S_IWRITE)
+                            function(path)
+                        shutil.rmtree(os.path.join(output_file_path, ls), onerror = ensureWritable)
                     except Exception:
                         os.remove(os.path.join(output_file_path, ls))
 
@@ -180,9 +184,15 @@ class URLDownload(Retrieval):
             if extension == ".gz" or extension == ".tgz":
                 extractProgress()
                 archive_file = ProgressFile(archive_file_path, progress_func)
-                with tarfile.open(fileobj=archive_file, mode='r:gz') as arch:
-                    arch.extractall(output_file_path)
-                archive_file.close()
+                try:
+                    with tarfile.open(fileobj=archive_file, mode='r:gz') as arch:
+                        arch.extractall(output_file_path)
+                    archive_file.close()
+                except tarfile.ReadError:
+                    logging.info("Extracting {} failed due to a Read Error".format(self.__file_path))
+                    archive_file.close()
+                    os.remove(archive_file_path)
+                    return False
             elif extension == ".bz2":
                 extractProgress()
                 archive_file = ProgressFile(archive_file_path, progress_func)
@@ -202,7 +212,7 @@ class URLDownload(Retrieval):
             elif extension in [".exe", ".msi"]:
                 # installers need to be handled by the caller
                 return True
-            elif extension in [".md", ".txt", ""]:
+            elif extension in [".md", ".txt", ".pdb", ".lib", ".dll", ""]:
                 # we don't need todo anything
                 return True
             else:

unibuild/projects/boost.py

@@ -34,7 +34,8 @@ boost_components = ["date_time",
     "filesystem",
     "thread",
     "log",
-    "locale"]
+    "locale",
+    "program_options"]
 boost_components_shared = ["python"]
 
 user_config_jam = "user-config-{}.jam".format("64" if config['architecture'] == "x86_64" else "32")

unibuild/projects/googletest.py

@@ -22,5 +22,5 @@ from unibuild.modules import cmake, github, build
 
 Project("GTest") \
   .depend(cmake.CMake().arguments(["-DCMAKE_BUILD_TYPE={0}".format(config["build_type"])])
-    .depend(github.Source("google", "googletest", "master")))
+    .depend(github.Source("google", "googletest", "main")))
       #.depend(github.Source("google", "googletest", "release-{}".format(googletest_version))))