#rb CarlMagnus.Nordin, Per.Larsson
#rnx
#preflight 63496fcf1f6054a99fe8bd0c
- When parsing the paklist, any file with a -rehydrate flag will be considered for rehydration
- Add the virtualization module to the PakFileUtilities module
- Add the source control module to UnrealPak (the virtualization module should be taking care of this)
- While parsing the files to be included in the pak file, we will record if any of the files require rehydration, if so this will be noted in FPakCommandLineParameters::bRequiresRehydration and used to initialize the virtualization system.
-- We only need to initialize the system once, even if we detect that multiple pak files have files that need rehydration.
-- If no pak file needs rehydration we do not initialize the system and the virtualization module is never loaded.
- FOutputPakFileEntry::CompressedFileBuffer was made private with accessors to make the refactor easier.
- Now when we start to compress a file, we always load it entirely into memory (along with any potential padding needed), if we then detect that the file doesn't require padding then we just use the in memory buffer as the output and if we do need compression we compress the buffer.
-- This means that there is only one place we load the file from disk, meaing only one place we need to insert the rehydration code to. This load is where the rehydration occurs.
-- In the previous code we would load the file into memory and then retain this copy during the compression pass, so no additional memory should be used.
- At the moment when we get the buffer back fromt he rehydration pass it will be in the form of a FSharedBuffer and so for now we need to memcpy to FCompressedFileBuffer::UncompressedBuffer, which is a waste of cpu cycles.
-- In a future change we should change UncompressedBuffer from TArray to FSharedBuffer to avoid this.
[CL 22595708 by paul chipchase in ue5-main branch]
#rb CarlMagnus.Nordin
#rnx
#preflight 6346ab768a0a7b2adc72cce4
### Problem
- The bug was originally introduced in CL 21791765
- Each file to be compressed calls FPakWriterContext::BeginCompress which will create a FMemoryCompressor and assign it to the files FOutputPakFileEntry. The problem is that the work done by the FMemoryCompressor can potentially complete and signal EndCompressionBarrier before the FMemoryCompresseor has been assigned. This can be due to the work itself being very small, or that closing a file handle when FCompressedFileBuffer::BeginCompressFileToWorkingBuffer returns (which creates the FMemoryCompressor) stalls against system resources. This potentially allows FPakWriterContext::EndCompres to execute concurrently and incorrectly believe that the file is not being compressed as FOutputPakFileEntry::MemoryCompressor is still nullptr.
- Not only does this mean that some files end up being paked uncompressed when they should be compressed, the results are not deterministic which means we can end up with different binary output when paking the exact same data with the exact same settings.
### Solution
- The constructor of FMemoryCompressor no longer adds its work to the task graph system. Instead the tasks are only added after the compressor has been assigned to FOutputPakFileEntry.
- Replaced auto in ranged for loops with the type, as per recent coding standards discussions.
[CL 22505090 by paul chipchase in ue5-main branch]
thread-safe delegates are not zero-initializable and so can't be used as global vars because they are vulnerable to static initialization order fiasco.
#jira UE-163668
#rb steve.robb
#preflight 632462db3752284a3179ec02
[CL 22094531 by Andriy Tylychko in ue5-main branch]
* Added -cryptofile documentation line when starting unrealpak without parameters
#rb Erik.Knapik
#preflight skip
#ROBOMERGE-AUTHOR: henrik.karlsson
#ROBOMERGE-SOURCE: CL 21183712 via CL 21195199 via CL 21195379 via CL 21195491
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21196924 by henrik karlsson in ue5-main branch]
* Contains some selected back outs from other cls that included config context dependent changes, though it's limited to only backing out broken parts as much as possible to minimize surface area.
[Backout] - CL20029182
[FYI] josh.adams
Original CL Desc
-----------------------------------------------------------------
- Adding FConfigContext which is used to repalce LoadExternalIniFile, LoadLocalIniFile, etc, as well as have localized data for all configs read on a thread (like the other platform configs loaded in the editor)
- The Load*IniFile functions will create a Context, but eventually those APIs will go away and the Context will be the only way to load ini files
- Simplified some of the ini loading code, like removing the HierarchyCache (it wasn't helping editor load times, and added much complexity, and was not thread-safe, and it shouldn't actually be helpful because all the calls to Load*IniFile should eventually be replaced with either GConfig or FCOnfigCacheIni::ForPlatform(), which won't need to re-read in files
- Ini reading time actually went down due to the simplification, including Cache removal
- Left in old code for now behing a #define (USE_CONTEXT) in case something goes wrong
- Added in VERIFY_CONTEXT mode which I used to run original and Context modes and compare (including preflighting builds) (I also added a Compare function that we may want to keep around to use for future debugging)
- Added a separate set of config layers for plugins which speeds up plugin parsing, but also will fix the issue with BaseEngine.ini vs DefaultEngine.ini in Engine vs Project plugins (this shows how Contexts can bring useful information down into the guts - however a later change will enable it)
- Once this is all seen to be working, I will clean up the non-Context functions, and some globals vs static members, etc
#rb paul.chipchase
#preflight 62716c1c5e6ce673f452005a
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 20029165 via CL 20029180 via CL 20905839 via CL 20905880
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)
[CL 20908094 by mic rooney in ue5-main branch]
- Deprecated some global functions now in ConfigUtilities
- Deleted a couple of old deprecated stuff from 3.24
#rb chris.waters
#preflight 628415e1ba3597a030b3b900
[CL 20259749 by Josh Adams in ue5-main branch]
TInlineAlloctor changed to be an alias of TSizedInlineAllocator, and TInlineAlloctor64 added.
Fix for Pakfile creation when the uncompressed buffer size exceeds int32 capacity.
#rb devin.doucette
[FYI] scott.lindeneau
#jira UE-141912
#preflight 6203bc9b7244040418573f53
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 18921401 in //UE5/Release-5.0/... via CL 18926938 via CL 18928756
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v916-18915374)
[CL 18929198 by steve robb in ue5-main branch]
- Fixes for the ini console command for other platforms
- Allow for reading Config files from another project (for instance from a program like UnrelPak)
#rb matt.peters
#preflight 61fae9189a71b11fd38faa0e
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 18834077 via CL 18835502 via CL 18835961 via CL 18836096 via CL 18844966 via CL 18845578
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18845601 by josh adams in ue5-main branch]
because they are needed in startup phase (ini,res,uplugin,etc.)
it's now fine to use Oodle there
this commit does not enable the new option so behavior stays the same for now
#preflight 61ba896058796f05e14b6931
#rb none
#ROBOMERGE-AUTHOR: charles.bloom
#ROBOMERGE-SOURCE: CL 18480192 in //UE5/Release-5.0/... via CL 18481555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18481827 by charles bloom in ue5-release-engine-test branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
