added DownloadInfo for headers and user agent

added IPluginRepository to container
downloads now have an IRepositoryDownload to manage the actual downloads curl downloader uses IDownload and IDownloader download stats, allow downloading to buffer
2026-07-27 13:58:24 -07:00 · 2021-03-02 19:14:14 -05:00 · 2021-03-02 17:15:52 -05:00 · 2021-02-17 14:16:55 -05:00 · 2021-02-17 11:18:28 -05:00 · 2021-02-16 04:53:49 -05:00
91 changed files with 6386 additions and 3468 deletions
@@ -2,7 +2,7 @@
 name: Game support Request
 about: Request support for a new game
 title: Add support for game [GAME NAME]
-labels: 'Feature Request, additional games support, area: mo2 game plugins'
+labels: 'additional games support, Feature Request, area: mo2 game plugins'
 assignees: ''

 ---
@@ -18,7 +18,7 @@ assignees: ''
 - **Nexus ID [optional]:** ID of the game on Nexus (you can usually find this in Nexus URL).
 - **Executable:** Name of the main executable for the game (relative to the game folder).
 - **Launcher [optional]:** Name of the game launcher (relative to the game folder).
- **Data path:** Path to the data folder of the game (relative to the game folder). Please note that the Virtual Files System often does not work for top level dlls or exe files.
+- **Data path:** Path to the data folder of the game (relative to the game folder).
 - **Documents path:** Path to folder containing INI files, etc., for the game (usually under "My Games", or the game folder itself).
 - **Saves directory [optional]:** Path to the folder containing save games (this default to the path above).
 - **Save extension [optional]:** Extension of the saves
@@ -8,7 +8,7 @@ assignees: ''
 ---

 ## The problem:
-Briefly describe the issue you are experiencing. Tell us what you were trying to do and what happened instead. Remember, this is _not_ a place to ask questions. For that, go to our [Discord server](https://discord.gg/6GKR9jZ).
+Briefly describe the issue you are experiencing. Tell us what you were trying to do and what happened instead. Remember, this is _not_ a place to ask questions. For that, go to our [discord channel](https://discord.gg/5tCqt6V)


 ## To Reproduce:
@@ -12,7 +12,7 @@ The project took up speed again after a few more coders showed up in late 2017,
 ## Help Wanted!
 Mod Organizer 2 is an open project in the hands of the community, there are problems that need to be solved and things that could be added. MO2 really needs developers and if you have the programming skills and some free time you can really improve the experience of the modding community.

-To have more information, please join the open MO2 Development discord server: [Mod Organizer 2](https://discord.gg/6GKR9jZ)
+To have more information, please join the open MO2 Development discord server: [Mod Organizer 2](https://discord.gg/vD2ZbfX)
 If you want to help translate MO2 to your language you should join the discord server too and head to the #translation channel.
 To setup a development environment on your machine, there is the [mob project](https://github.com/modorganizer2/mob) that handles that.
 If you want to submit your code changes, please use a good formatting style like the default one in Visual Studio.
@@ -20,7 +20,7 @@ If you want to submit your code changes, please use a good formatting style like
 Through the work of a few people of the community MO2 has come quite far, now it needs some more of those people to go further.

 ## Reporting Issues:
-Issues should be reported to the GitHub page or on the open discord server: [Mod Organizer 2](https://discord.gg/6GKR9jZ). Here is also where dev builds are tested, bugs are reported and investigated, suggestions are discussed and a lot more.
+Issues should be reported to the GitHub page or on the open discord server: [Mod Organizer 2](https://discord.gg/vD2ZbfX). Here is also where dev builds are tested, bugs are reported and investigated, suggestions are discussed and a lot more.

 Credits to Tannin, LePresidente, Silarn, erasmux, AL12, LostDragonist, AnyOldName3, isa, Holt59 and many others for the development.

@@ -37,7 +37,6 @@ add_filter(NAME src/core GROUPS
 	apiuseraccount
 	processrunner
 	qdirfiletree
-	virtualfiletree
 	uilocker
 )

@@ -66,6 +65,7 @@ add_filter(NAME src/downloads GROUPS
 	downloadlist
 	downloadlistview
 	downloadmanager
+	downloadmanager2
 )

 add_filter(NAME src/env GROUPS
@@ -127,7 +127,7 @@
         <item>
          <widget class="QLabel" name="label_3">
           <property name="text">
-            <string notr="true">&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;Copyright © 2011-2016 Sebastian Herbord&lt;br/&gt;Copyright © 2016-2021 Mod Organizer 2 Contributors&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
+            <string notr="true">&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;Copyright © 2011-2016 Sebastian Herbord&lt;br/&gt;Copyright © 2016-2019 Mod Organizer 2 Contributors&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
           </property>
          </widget>
         </item>
@@ -261,11 +261,6 @@
                <string notr="true">przester</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Qudix</string>
-               </property>
-              </item>
             </widget>
            </item>
           </layout>
@@ -297,31 +292,11 @@
                <string notr="true">miraclefreak (Czech)</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Meiton (Czech)</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string>Cyb3r (Dutch)</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">madmedic (Dutch)</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Ilja (Finnish)</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Mikmet (Finnish)</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string notr="true">Alyndiar (French)</string>
@@ -347,11 +322,6 @@
                <string>Yoplala (French)</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">CircleOfDao (German)</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string>Faron (German)</string>
@@ -367,11 +337,6 @@
                <string notr="true">Xahtax (German)</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Ypselon (German)</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string>yohru (Japanese)</string>
@@ -435,11 +400,6 @@
                <string notr="true">Hakan &quot;Nyks45&quot; Albayrak (Turkish)</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Kato (Turkish)</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string>...and all other contributors!</string>
@@ -466,7 +426,7 @@
              </property>
              <item>
               <property name="text">
-                <string notr="true">Tannin (Original Creator)</string>
+                <string>Tannin (Original Creator)</string>
               </property>
              </item>
              <item>
@@ -494,21 +454,11 @@
                <string notr="true">Brixified</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">ddbb07</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string notr="true">deathneko11</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">dekart811</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string notr="true">DoubleYou</string>
@@ -519,26 +469,6 @@
                <string notr="true">Drew Warwick</string>
               </property>
              </item>
-              <item>
-               <property name="text">
-                <string notr="true">erri120</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">ianpatt</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Falsellyu</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">foresto</string>
-               </property>
-              </item>
              <item>
               <property name="text">
                <string notr="true">GamerPoet</string>
@@ -561,17 +491,7 @@
              </item>
              <item>
               <property name="text">
-                <string notr="true">JayLCypher</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">jimfcarroll</string>
-               </property>
-              </item>
-              <item>
-               <property name="text">
-                <string notr="true">Luca|EzioTheDeadPoet</string>
+                <string>Luca|EzioTheDeadPoet</string>
               </property>
              </item>
              <item>
@@ -586,7 +506,7 @@
              </item>
              <item>
               <property name="text">
-                <string notr="true">Patchier</string>
+                <string>Patchier</string>
               </property>
              </item>
              <item>
@@ -594,6 +514,11 @@
                <string notr="true">PurpleFez</string>
               </property>
              </item>
+              <item>
+               <property name="text">
+                <string notr="true">Qudix</string>
+               </property>
+              </item>
              <item>
               <property name="text">
                <string notr="true">reedts</string>
@@ -158,14 +158,14 @@ protected:
        currentName = std::get<0>(p)[0];
      }

-      // If the name is different, we need to create a directory from what we have
+      // If the name is different, we need to create a directory from what we have 
      // accumulated:
      if (currentName != std::get<0>(p)[0]) {

        // We may or may not have an index here, it depends on the type of archive (some archives list
        // intermediate non-empty folders, some don't):
        entries.push_back(std::make_shared<ArchiveFileTreeImpl>(parent, currentName, currentIndex, std::move(currentFiles)));
-
+        
        currentFiles.clear(); // Back to a valid state.

        // Reset the index:
@@ -200,7 +200,7 @@ protected:
    if (currentName != "") {
      entries.push_back(std::make_shared<ArchiveFileTreeImpl>(parent, currentName, currentIndex, std::move(currentFiles)));
    }
-
+    
    // Let the parent class sort the entries:
    return false;
  }
@@ -214,7 +214,7 @@ private:
  mutable std::vector<File> m_Files;
 };

-std::shared_ptr<ArchiveFileTree> ArchiveFileTree::makeTree(Archive const& archive)
+std::shared_ptr<ArchiveFileTree> ArchiveFileTree::makeTree(Archive const& archive) 
 {
  auto const& data = archive.getFileList();

@@ -222,16 +222,9 @@ std::shared_ptr<ArchiveFileTree> ArchiveFileTree::makeTree(Archive const& archiv
  files.reserve(data.size());

  for (size_t i = 0; i < data.size(); ++i) {
-    // Ignore "." and ".." as they're useless and muck things up
-    if (data[i]->getArchiveFilePath().compare(L".") == 0 ||
-      data[i]->getArchiveFilePath().compare(L"..") == 0)
-    {
-      continue;
-    }
-
    files.push_back(std::make_tuple(
-      QString::fromStdWString(data[i]->getArchiveFilePath()).replace("\\", "/").split("/", Qt::SkipEmptyParts),
-      data[i]->isDirectory(),
+      QString::fromStdWString(data[i]->getArchiveFilePath()).replace("\\", "/").split("/", Qt::SkipEmptyParts), 
+      data[i]->isDirectory(), 
      (int) i));
  }

@@ -232,7 +232,7 @@ void CategoryFactory::loadDefaultCategories()
  addCategory(44, "Equipment", { 1, 44 }, 43);
  addCategory(45, "Home/Settlement", { 1, 45 }, 43);
  addCategory(10, "Body, Face, & Hair", { 2, 17, 26 }, 0);
-  addCategory(56, "Tattoos", { 1, 57 }, 10);
+  addCategory(39, "Tattoos", { 1, 57 }, 10);
  addCategory(40, "Character Presets", { 1, 58 }, 0);
  addCategory(11, "Items", { 2, 27, 85 }, 0);
  addCategory(32, "Mercantile", { 2, 23, 69 }, 0);
@@ -341,25 +341,28 @@ void CreateInstanceDialog::finish()
      s.game().setEdition(ci.gameVariant);
    }

-    if (ci.paths.base != ci.dataPath) {
-      s.paths().setBase(ci.paths.base);
+    if (ci.type == Global) {
+      if (ci.paths.base != ci.dataPath) {
+        s.paths().setBase(ci.paths.base);
+      }
+
+      if (ci.paths.downloads != cid::makeDefaultPath(AppConfig::downloadPath())) {
+        s.paths().setDownloads(ci.paths.downloads);
+      }
+
+      if (ci.paths.mods != cid::makeDefaultPath(AppConfig::modsPath())) {
+        s.paths().setMods(ci.paths.mods);
+      }
+
+      if (ci.paths.profiles != cid::makeDefaultPath(AppConfig::profilesPath())) {
+        s.paths().setProfiles(ci.paths.profiles);
+      }
+
+      if (ci.paths.overwrite != cid::makeDefaultPath(AppConfig::overwritePath())) {
+        s.paths().setOverwrite(ci.paths.overwrite);
+      }
    }

-    if (ci.paths.downloads != cid::makeDefaultPath(AppConfig::downloadPath())) {
-      s.paths().setDownloads(ci.paths.downloads);
-    }
-
-    if (ci.paths.mods != cid::makeDefaultPath(AppConfig::modsPath())) {
-      s.paths().setMods(ci.paths.mods);
-    }
-
-    if (ci.paths.profiles != cid::makeDefaultPath(AppConfig::profilesPath())) {
-      s.paths().setProfiles(ci.paths.profiles);
-    }
-
-    if (ci.paths.overwrite != cid::makeDefaultPath(AppConfig::overwritePath())) {
-      s.paths().setOverwrite(ci.paths.overwrite);
-    }

    logCreation(tr("Writing %1...").arg(ci.iniPath));

@@ -67,7 +67,7 @@
     <property name="currentIndex">
      <number>0</number>
     </property>
-     <widget class="QWidget" name="page_1">
+     <widget class="QWidget" name="page_10">
      <layout class="QVBoxLayout" name="verticalLayout_21">
       <item>
        <widget class="QLabel" name="label_15">
@@ -115,7 +115,7 @@
       </item>
      </layout>
     </widget>
-     <widget class="QWidget" name="page_2">
+     <widget class="QWidget" name="page">
      <layout class="QVBoxLayout" name="verticalLayout_3">
       <item>
        <widget class="QWidget" name="widget_7" native="true">
@@ -226,7 +226,7 @@
       </item>
      </layout>
     </widget>
-     <widget class="QWidget" name="page_3">
+     <widget class="QWidget" name="page_2">
      <layout class="QVBoxLayout" name="verticalLayout_7">
       <item>
        <widget class="QWidget" name="widget_5" native="true">
@@ -367,7 +367,7 @@
       </item>
      </layout>
     </widget>
-     <widget class="QWidget" name="page_4">
+     <widget class="QWidget" name="page_9">
      <layout class="QVBoxLayout" name="verticalLayout_19">
       <item>
        <widget class="QWidget" name="widget_16" native="true">
@@ -594,7 +594,7 @@
            <property name="currentIndex">
             <number>0</number>
            </property>
-            <widget class="QWidget" name="page_6_simple">
+            <widget class="QWidget" name="page_7">
             <layout class="QVBoxLayout" name="verticalLayout_12">
              <property name="leftMargin">
               <number>0</number>
@@ -676,7 +676,7 @@
              </item>
             </layout>
            </widget>
-            <widget class="QWidget" name="page_6_advanced">
+            <widget class="QWidget" name="page_8">
             <layout class="QVBoxLayout" name="verticalLayout_13">
              <property name="leftMargin">
               <number>0</number>
@@ -906,7 +906,7 @@
       </item>
      </layout>
     </widget>
-     <widget class="QWidget" name="page_7">
+     <widget class="QWidget" name="page_3">
      <layout class="QVBoxLayout" name="verticalLayout_4">
       <item>
        <widget class="QWidget" name="widget_4" native="true">
@@ -1048,7 +1048,7 @@
       </item>
      </layout>
     </widget>
-     <widget class="QWidget" name="page_8">
+     <widget class="QWidget" name="page_11">
      <layout class="QVBoxLayout" name="verticalLayout_22">
       <item>
        <widget class="QWidget" name="widget_18" native="true">
@@ -1232,38 +1232,6 @@
   <header location="global">linklabel.h</header>
  </customwidget>
 </customwidgets>
- <tabstops>
-  <tabstop>next</tabstop>
-  <tabstop>cancel</tabstop>
-  <tabstop>hideIntro</tabstop>
-  <tabstop>createGlobal</tabstop>
-  <tabstop>createPortable</tabstop>
-  <tabstop>scrollArea_3</tabstop>
-  <tabstop>showAllGames</tabstop>
-  <tabstop>gamesFilter</tabstop>
-  <tabstop>scrollArea_2</tabstop>
-  <tabstop>instanceName</tabstop>
-  <tabstop>location</tabstop>
-  <tabstop>browseLocation</tabstop>
-  <tabstop>advancedPathOptions</tabstop>
-  <tabstop>base</tabstop>
-  <tabstop>browseBase</tabstop>
-  <tabstop>downloads</tabstop>
-  <tabstop>browseDownloads</tabstop>
-  <tabstop>mods</tabstop>
-  <tabstop>browseMods</tabstop>
-  <tabstop>profiles</tabstop>
-  <tabstop>browseProfiles</tabstop>
-  <tabstop>overwrite</tabstop>
-  <tabstop>browseOverwrite</tabstop>
-  <tabstop>nexusConnect</tabstop>
-  <tabstop>nexusManual</tabstop>
-  <tabstop>nexusLog</tabstop>
-  <tabstop>review</tabstop>
-  <tabstop>creationLog</tabstop>
-  <tabstop>launch</tabstop>
-  <tabstop>back</tabstop>
- </tabstops>
 <resources>
  <include location="resources.qrc"/>
 </resources>
@@ -8,7 +8,6 @@
 #include <iplugingame.h>
 #include <report.h>
 #include <utility.h>
-#include "filesystemutilities.h"

 namespace cid
 {
@@ -293,7 +292,7 @@ void GamePage::select(IPluginGame* game, const QString& dir)
  Game* checked = findGame(game);

  if (checked) {
-    if (!checked->installed || (detectMicrosoftStore(checked->dir) && !confirmMicrosoftStore(checked->dir, checked->game))) {
+    if (!checked->installed) {
      if (dir.isEmpty()) {
        // the selected game has no installation directory and none was given,
        // ask the user
@@ -305,9 +304,6 @@ void GamePage::select(IPluginGame* game, const QString& dir)
        if (path.isEmpty()) {
          // cancelled
          checked = nullptr;
-        } else if (detectMicrosoftStore(path) && !confirmMicrosoftStore(path, game)) {
-          // cancelled
-          checked = nullptr;
        } else {
          // check whether a plugin supports the given directory; this can
          // return the same plugin, a different one, or null
@@ -357,13 +353,6 @@ void GamePage::selectCustom()
    return;
  }

-  // Microsoft store games are not supported
-  if (detectMicrosoftStore(path) && !confirmMicrosoftStore(path, nullptr)) {
-    // reselect the previous button
-    selectButton(m_selection);
-    return;
-  }
-
  // try to find a plugin that likes this directory
  for (auto& g : m_games) {
    if (g->game->looksValid(path)) {
@@ -474,7 +463,7 @@ void GamePage::updateButton(Game* g)
    return;
  }

-  g->button->setText(g->game->gameName().replace("&", "&&"));
+  g->button->setText(g->game->gameName());
  g->button->setIcon(g->game->gameIcon());

  if (g->installed) {
@@ -604,11 +593,6 @@ GamePage::Game* GamePage::checkInstallation(const QString& path, Game* g)
    return g;
  }

-  if (detectMicrosoftStore(path) && confirmMicrosoftStore(path, g->game)) {
-    // okay
-    return g;
-  }
-
  // the selected game can't use that folder, find another one
  IPluginGame* otherGame = nullptr;

@@ -655,35 +639,6 @@ GamePage::Game* GamePage::checkInstallation(const QString& path, Game* g)
  return g;
 }

-bool GamePage::detectMicrosoftStore(const QString& path)
-{
-  return path.contains("/ModifiableWindowsApps/") ||
-         path.contains("/WindowsApps/");
-}
-
-bool GamePage::confirmMicrosoftStore(const QString& path, IPluginGame* game)
-{
-  const auto r = TaskDialog(&m_dlg)
-    .title(QObject::tr("Microsoft Store game"))
-    .main(QObject::tr("Microsoft Store game"))
-    .content(QObject::tr(
-      "The folder %1 seems to be a Microsoft Store game install.  Games"
-      " installed through the Microsoft Store are not supported by Mod Organizer"
-      " and will not work properly.")
-      .arg(path))
-    .button({
-      game ? QObject::tr("Use this folder for %1").arg(game->gameName())
-           : QObject::tr("Use this folder"),
-      QObject::tr("I know what I'm doing"),
-      QMessageBox::Ignore})
-    .button({
-      QObject::tr("Cancel"),
-      QMessageBox::Cancel})
-    .exec();
-
-  return (r == QMessageBox::Ignore);
-}
-
 bool GamePage::confirmUnknown(const QString& path, IPluginGame* game)
 {
  const auto r = TaskDialog(&m_dlg)
@@ -901,7 +856,7 @@ QString NamePage::selectedInstanceName() const
  }

  const auto text = ui->instanceName->text().trimmed();
-  return MOBase::sanitizeFileName(text);
+  return InstanceManager::singleton().sanitizeInstanceName(text);
 }

 void NamePage::onChanged()
@@ -928,7 +883,7 @@ bool NamePage::checkName(QString parentDir, QString name)
  if (name.isEmpty()) {
    empty = true;
  } else {
-    if (MOBase::validFileName(name)) {
+    if (InstanceManager::singleton().validInstanceName(name)) {
      exists = QDir(parentDir).exists(name);
    } else {
      invalid = true;
@@ -1153,7 +1108,7 @@ bool PathsPage::checkPath(
  } else {
    const QDir d(path);

-    if (MOBase::validFileName(d.dirName())) {
+    if (m.validInstanceName(d.dirName())) {
      if (m_dlg.rawCreationInfo().type == CreateInstanceDialog::Portable) {
        // the default data path for a portable instance is the application
        // directory, so it's not an error if it exists
@@ -354,15 +354,6 @@ private:
  MOBase::IPluginGame* confirmOtherGame(
    const QString& path,
    MOBase::IPluginGame* selectedGame, MOBase::IPluginGame* guessedGame);
-
-  // detects if the given path likely contains a Microsoft Store game
-  //
-  bool detectMicrosoftStore(const QString& path);
-
-  // tells the user that the path probably contains a Microsoft Store game that
-  // is not supported, returns true if the user decides to accept anyway.
-  //
-  bool confirmMicrosoftStore(const QString& path, MOBase::IPluginGame* game);
 };


@@ -0,0 +1,212 @@
+#ifndef MODORGANIZER_CURLDOWNLOADER_INCLUDED
+#define MODORGANIZER_CURLDOWNLOADER_INCLUDED
+
+#include <ipluginrepository.h>
+
+namespace dm::curl
+{
+
+namespace fs = std::filesystem;
+using hr_clock = std::chrono::high_resolution_clock;
+
+struct defer_t {};
+extern const defer_t defer;
+
+class GlobalHandle
+{
+public:
+  GlobalHandle();
+  ~GlobalHandle();
+
+  GlobalHandle(const GlobalHandle&) = delete;
+  GlobalHandle& operator=(const GlobalHandle) = delete;
+};
+
+
+class EasyHandle
+{
+public:
+  EasyHandle();
+  EasyHandle(defer_t);
+  ~EasyHandle();
+
+  EasyHandle(const EasyHandle&) = delete;
+  EasyHandle& operator=(const EasyHandle&) = delete;
+
+  bool create();
+  CURL* get() const;
+
+private:
+  CURL* m_handle;
+};
+
+
+class MultiHandle
+{
+public:
+  MultiHandle();
+  MultiHandle(defer_t);
+  ~MultiHandle();
+
+  MultiHandle(const MultiHandle&) = delete;
+  MultiHandle& operator=(const MultiHandle&) = delete;
+
+  bool create();
+  CURLM* get() const;
+
+private:
+  CURLM* m_handle;
+};
+
+
+class FileHandle
+{
+public:
+  FileHandle();
+  ~FileHandle();
+
+  FileHandle(const FileHandle&) = delete;
+  FileHandle& operator=(const FileHandle&) = delete;
+
+  bool opened() const;
+  std::size_t open(fs::path p, bool append);
+  void close();
+
+  bool write(std::string_view sv);
+
+private:
+  HANDLE m_handle;
+  fs::path m_path;
+
+  bool doOpen(bool append);
+};
+
+
+struct SListDeleter
+{
+  void operator()(curl_slist* p)
+  {
+    if (p) {
+      curl_slist_free_all(p);
+    }
+  }
+};
+
+using SList = std::unique_ptr<curl_slist, SListDeleter>;
+
+
+class Download : public MOBase::IDownload
+{
+public:
+  Download(std::string url, Info info);
+
+  CURL* setup(curl_off_t maxSpeed);
+
+  CURL* handle() const;
+  States state() const override;
+  Stats stats() const override;
+  std::string stealBuffer();
+  QByteArray buffer() const override;
+  int httpCode() const override;
+  std::string error() const;
+  std::string debugName() const;
+
+  void start();
+  void stop() override;
+  bool finish(CURLcode code);
+
+  bool xfer(
+    curl_off_t dltotal, curl_off_t dlnow,
+    curl_off_t ultotal, curl_off_t ulnow);
+
+  bool header(std::string_view sv);
+  bool write(std::string_view data);
+  void debug(curl_infotype t, std::string_view data);
+
+private:
+  std::string m_url;
+  Info m_info;
+  EasyHandle m_handle;
+  FileHandle m_out;
+  SList m_headers;
+  std::string m_buffer;
+  States m_state;
+  std::string m_error;
+
+  hr_clock::time_point m_lastCheck;
+  std::size_t m_bytes;
+  std::atomic<double> m_bytesPerSecond;
+  std::atomic<double> m_progress;
+
+  std::size_t resumeFrom();
+  bool rename();
+  fs::path outputFile() const;
+
+  static int s_xfer(
+    void* p,
+    curl_off_t dltotal, curl_off_t dlnow,
+    curl_off_t ultotal, curl_off_t ulnow);
+
+  static size_t s_header(char* data, size_t size, size_t n, void* p);
+  static size_t s_write(char* data, size_t size, size_t n, void* p);
+  static int s_debug(CURL* h, curl_infotype t, char* data, size_t n, void *p);
+};
+
+
+class Downloader : public MOBase::IDownloader
+{
+public:
+  static const std::size_t NoLimit =
+    std::numeric_limits<std::size_t>::max();
+
+  Downloader();
+  ~Downloader();
+
+  void cancel();
+  void stop();
+  void join();
+
+  void maxSpeed(std::size_t bytesPerSecond);
+
+  bool finished() const;
+
+  std::shared_ptr<MOBase::IDownload> add(
+    const QUrl& url, const MOBase::IDownload::Info& info={}) override;
+
+private:
+  using DownloadList = std::list<std::shared_ptr<Download>>;
+
+  std::shared_ptr<GlobalHandle> m_global;
+  MultiHandle m_handle;
+
+  std::vector<std::shared_ptr<Download>> m_temp;
+  std::mutex m_tempMutex;
+
+  std::thread m_thread;
+  std::atomic<bool> m_cancel, m_stop, m_finished;
+  std::condition_variable m_cv;
+  DownloadList m_list;
+  std::map<CURL*, DownloadList::iterator> m_map;
+
+  std::atomic<std::size_t> m_maxSpeed;
+
+
+  void run();
+  void checkTemp();
+  void perform();
+  void poll();
+  bool start(std::shared_ptr<Download> d);
+  void setLimits();
+  void checkCancel();
+
+  void checkQueue();
+  bool cleanupActive();
+  DownloadList::iterator removeFromActive(DownloadList::iterator itor);
+  void stopOverMax(std::size_t max);
+  bool addFromQueue(std::size_t max);
+  curl_off_t maxSpeedPer() const;
+};
+
+} // namespace
+
+#endif // MODORGANIZER_CURLDOWNLOADER_INCLUDED
@@ -265,11 +265,6 @@ void DirectoryRefresher::stealModFilesIntoStructure(
  for (const QString &filename : stealFiles) {
    if (filename.isEmpty()) {
      log::warn("Trying to find file with no name");
-      log::warn(" . modName: {}", modName);
-      log::warn(" . directory: {}", directory);
-      log::warn(" . priority: {}", priority);
-      for (int i = 0; i < stealFiles.length(); ++i)
-        log::warn(" . stealFiles[{}]: {}", i, stealFiles[i]);
      continue;
    }
    QFileInfo fileInfo(filename);
@@ -226,7 +226,7 @@ bool DownloadList::lessThanPredicate(const QModelIndex &left, const QModelIndex
  if ((leftIndex < m_manager.numTotalDownloads())
      && (rightIndex < m_manager.numTotalDownloads())) {
    if (left.column() == DownloadList::COL_NAME) {
-      return left.data(Qt::DisplayRole).toString().compare(right.data(Qt::DisplayRole).toString(), Qt::CaseInsensitive) < 0;
+      return m_manager.getFileName(left.row()).compare(m_manager.getFileName(right.row()), Qt::CaseInsensitive) < 0;
    } else if (left.column() == DownloadList::COL_MODNAME) {
      QString leftName, rightName;

@@ -273,9 +273,9 @@ bool DownloadList::lessThanPredicate(const QModelIndex &left, const QModelIndex
      DownloadManager::DownloadState leftState = m_manager.getState(left.row());
      DownloadManager::DownloadState rightState = m_manager.getState(right.row());
      if (leftState == rightState)
-        return m_manager.getFileTime(left.row()) > m_manager.getFileTime(right.row());
+        return m_manager.getFileTime(left.row()) < m_manager.getFileTime(right.row());
      else
-        return leftState < rightState;
+        return leftState > rightState;
    } else if (left.column() == DownloadList::COL_SIZE) {
      return m_manager.getFileSize(left.row()) < m_manager.getFileSize(right.row());
    } else if (left.column() == DownloadList::COL_FILETIME) {
@@ -33,7 +33,6 @@ along with Mod Organizer.  If not, see <http://www.gnu.org/licenses/>.
 #include "shared/util.h"
 #include <utility.h>
 #include <report.h>
-#include "filesystemutilities.h"

 #include <QTimer>
 #include <QFileInfo>
@@ -1223,7 +1222,6 @@ QString DownloadManager::getFileTypeString(int fileType)
    case 4: return tr("Old");
    case 5: return tr("Miscellaneous");
    case 6: return tr("Deleted");
-    case 7: return tr("Archived");
    default: return tr("Unknown");
  }
 }
@@ -1458,7 +1456,7 @@ void DownloadManager::markUninstalled(QString fileName)

 QString DownloadManager::getDownloadFileName(const QString &baseName, bool rename) const
 {
-  QString fullPath = m_OutputDirectory + "/" + MOBase::sanitizeFileName(baseName);
+  QString fullPath = m_OutputDirectory + "/" + baseName;
  if (QFile::exists(fullPath) && rename) {
    int i = 1;
    while (QFile::exists(QString("%1/%2_%3").arg(m_OutputDirectory).arg(i).arg(baseName))) {
@@ -1478,7 +1476,7 @@ QString DownloadManager::getFileNameFromNetworkReply(QNetworkReply *reply)

    std::cmatch result;
    if (std::regex_search(reply->rawHeader("Content-Disposition").constData(), result, exp)) {
-      return MOBase::sanitizeFileName(QString::fromUtf8(result.str(1).c_str()));
+      return QString::fromUtf8(result.str(1).c_str());
    }
  }

@@ -1724,8 +1722,7 @@ void DownloadManager::nxmFilesAvailable(QString, int, QVariant userData, QVarian
        {return lhs.toMap()["uploaded_timestamp"].toInt() > rhs.toMap()["uploaded_timestamp"].toInt();});
      for (QVariant file : files) {
        QVariantMap fileInfo = file.toMap();
-        if (fileInfo["category_id"].toInt() != NexusInterface::FileStatus::REMOVED &&
-          fileInfo["category_id"].toInt() != NexusInterface::FileStatus::ARCHIVED)
+        if (fileInfo["category_id"].toInt() != 6)
          selection.addChoice(fileInfo["file_name"].toString(), "", file);
      }
      if (selection.exec() == QDialog::Accepted) {
@@ -1960,8 +1957,7 @@ void DownloadManager::nxmFileInfoFromMd5Available(QString gameName, QVariant use
      auto results = resultlist[i].toMap();
      auto fileDetails = results["file_details"].toMap();

-      if (fileDetails["category_id"].toInt() != NexusInterface::FileStatus::REMOVED &&
-          fileDetails["category_id"].toInt() != NexusInterface::FileStatus::ARCHIVED) {
+      if (fileDetails["category_id"].toInt() != 6) { //6 = deleted
        if (chosenIdx < 0) {
          chosenIdx = i; //intentional to not break in order to check other results
        } else {
@@ -0,0 +1,114 @@
+#ifndef MODORGANIZER_DOWNLOADEDMANAGER2_INCLUDED
+#define MODORGANIZER_DOWNLOADEDMANAGER2_INCLUDED
+
+#include <ipluginrepository.h>
+
+class PluginContainer;
+namespace dm::curl { class Downloader; class Download; }
+
+
+namespace dm
+{
+
+namespace fs = std::filesystem;
+
+class DownloadManager2;
+
+class Download
+{
+public:
+  enum States
+  {
+    None = 0,
+
+    Queueing,
+    Queued,
+
+    Finished,
+    Errored,
+
+    Running,
+
+    Pausing,
+    Paused,
+
+    Cancelling,
+    Cancelled
+  };
+
+  Download(DownloadManager2& dm, MOBase::IPluginRepository& repo, QString what);
+
+  const QString& what() const;
+  const QString& error() const;
+
+  States state() const;
+
+  bool start();
+  void cancel();
+  void pause();
+  void queue();
+  void tick();
+
+  QString debugName() const;
+
+private:
+  DownloadManager2& m_dm;
+  MOBase::IPluginRepository& m_repo;
+  std::unique_ptr<MOBase::IRepositoryDownload> m_download;
+  QString m_what;
+  States m_state;
+  QString m_error;
+
+  void setState(States s);
+  void next();
+};
+
+
+class DownloadManager2
+{
+public:
+  static const std::size_t NoLimit =
+    std::numeric_limits<std::size_t>::max();
+
+  DownloadManager2(PluginContainer& pc);
+  ~DownloadManager2();
+
+  void add(QString what);
+
+  void maxActive(std::size_t n);
+  void maxSpeed(std::size_t bytesPerSecond);
+
+  bool hasActive() const;
+
+  curl::Downloader& downloader();
+
+private:
+  using DownloadList = std::list<std::shared_ptr<Download>>;
+
+  PluginContainer& m_pc;
+
+  std::thread m_thread;
+  std::atomic<bool> m_stop;
+
+  std::unique_ptr<curl::Downloader> m_downloader;
+
+  std::vector<std::shared_ptr<Download>> m_temp;
+  std::mutex m_tempMutex;
+  std::condition_variable m_cv;
+
+  DownloadList m_queued, m_active, m_inactive;
+  std::atomic<std::size_t> m_maxActive, m_maxSpeed;
+  std::atomic<bool> m_hasActive;
+
+
+  void run();
+  void checkTemp();
+  void checkQueue();
+  void cleanupActive();
+  void stopOverMax(std::size_t max);
+  void addFromQueue(std::size_t max);
+};
+
+} // namespace
+
+#endif // MODORGANIZER_DOWNLOADEDMANAGER2_INCLUDED
@@ -232,8 +232,8 @@ void forEachEntryImpl(

  if (status < 0) {
    log::error(
-      "failed to open directory '{}': {}",
-      toString(poa), formatNtMessage(status));
+      "NtOpenFile() failed for '{}', {}",
+      toString(poa), formatSystemMessage(status));

    return;
  }
@@ -264,9 +264,8 @@ void forEachEntryImpl(
      break;
    } else if (status < 0) {
      log::error(
-        "failed to read directory '{}': {}",
-        toString(poa), formatNtMessage(status));
-
+        "NtQueryDirectoryFile() failed for '{}', {}",
+        toString(poa), formatSystemMessage(status));
      break;
    }

@@ -322,23 +321,6 @@ void forEachEntryImpl(
  }
 }

-std::wstring makeNtPath(const std::wstring& path)
-{
-  constexpr const wchar_t* nt_prefix = L"\\??\\";
-  constexpr const wchar_t* nt_unc_prefix = L"\\??\\UNC\\";
-  constexpr const wchar_t* share_prefix = L"\\\\";
-
-  if (path.starts_with(nt_prefix)) {
-    // already an nt path
-    return path;
-  } else if (path.starts_with(share_prefix)) {
-    // network shared need \??\UNC\ as a prefix
-    return nt_unc_prefix + path.substr(2);
-  } else {
-    // prepend the \??\ prefix
-    return nt_prefix + path;
-  }
-}

 void DirectoryWalker::forEachEntry(
  const std::wstring& path, void* cx,
@@ -353,7 +335,7 @@ void DirectoryWalker::forEachEntry(
    NtClose = (NtClose_type)::GetProcAddress(m.get(), "NtClose");
  }

-  const std::wstring ntpath = makeNtPath(path);
+  const std::wstring ntpath = std::wstring(L"\\??\\") + path;

  UNICODE_STRING ObjectName = {};
  ObjectName.Buffer = const_cast<wchar_t*>(ntpath.c_str());
@@ -130,7 +130,7 @@ Module::FileInfo Module::getFileInfo() const
      return {};
    }

-    log::debug(
+    log::error(
      "GetFileVersionInfoSizeW() failed on '{}', {}",
      m_path, formatSystemMessage(e));

@@ -268,7 +268,7 @@ QDateTime Module::getTimestamp(const VS_FIXEDFILEINFO& fi) const
    if (h.get() == INVALID_HANDLE_VALUE) {
      const auto e = GetLastError();

-      log::debug(
+      log::error(
        "can't open file '{}' for timestamp, {}",
        m_path, formatSystemMessage(e));

--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
isanae	08e0d85d85	added DownloadInfo for headers and user agent	2021-03-02 19:14:14 -05:00
isanae	2094d7d15f	added IPluginRepository to container downloads now have an IRepositoryDownload to manage the actual downloads curl downloader uses IDownload and IDownloader download stats, allow downloading to buffer	2021-03-02 17:15:52 -05:00
isanae	25a69bccd3	split queue back into manager	2021-02-17 14:16:55 -05:00
isanae	a39a80ec4c	log wrapper, verbose log log curl version	2021-02-17 11:18:28 -05:00
isanae	42d432e5aa	moved to curldownloader	2021-02-16 04:53:49 -05:00
isanae	acb55d431c	fixed output file staying opened when moving to queue refactored removing from queue	2021-02-16 04:43:26 -05:00
isanae	95a2cc93b9	queue management	2021-02-16 04:23:20 -05:00
isanae	35a0474ac1	initial new download manager	2021-02-16 02:33:20 -05:00