Output is the same as when executing the -list command separately.
Running with -csv=<dir> will create one <pakchunkfilename>.pak.csv file per generated pak and one <pakchunkfilename>.utoc.csv file per generated container.
Running with -csv=<file> will create one single <file>.pak.csv for all paks and one <file>.utoc.csv for all containers.
Details:
The container output (the big set of data) is generated based on in-memory writer TOC data once the IoStoreWriters have been finalized.
The pak output (smaller set of data) is generated using the existing list command and loads the newly generated files from disk.
Cleanup the remains of three obsolete csv output features for containers (-csvoutput, -writefinalorder and OUTPUT_CHUNKID_DIRECTORY).
Optimize container output generation by extending FIoStoreTocChunkInfo with OffsetOnDisk and NumCompressedBlocks filled directly by EnumerateChunks.
Add a ChunkIdToFileNameMap to FCookedPackageStore.
Use the FCookedPackageStore to output the actual package ids and package names (this is currently broken when running -list as a separate command).
#rb carlmagnus.nordin
#rnx
#preflight 63e5eacb98775169f8dea431
[CL 24118941 by pj kack in ue5-main branch]
Add support for diffing pak directories for pak diff.
Add support for different cryptokeys for pak diff and legacy iostore diff.
#jira UE-175144
#rb carlmagnus.nordin
#rnx
#preflight 63d9259e7a39a18021d4f997
[CL 23945218 by pj kack in ue5-main branch]
Let CollectFilesToAdd fallback to the old logic of using the source file name when the FPakInputPair.Dest specifies a directory, i.e. is ending with a "/".
Let ExtractFilesFromPak start using destination file names just as ProcessCommandLine reading from response files is doing so that -Extract with -responsefile will generate the same format as UAT with file names.
Remove dead code in ProcessPakFileSpecificCommandLine.
#jira UE-171678
#rb carlmagnus.nordin
#lockdown mark.lintott
#preflight 6390679d1776b8c21cf42bac
[CL 23427029 by pj kack in ue5-main branch]
#rb CarlMagnus.Nordin, Per.Larsson
#rnx
#preflight 63496fcf1f6054a99fe8bd0c
- When parsing the paklist, any file with a -rehydrate flag will be considered for rehydration
- Add the virtualization module to the PakFileUtilities module
- Add the source control module to UnrealPak (the virtualization module should be taking care of this)
- While parsing the files to be included in the pak file, we will record if any of the files require rehydration, if so this will be noted in FPakCommandLineParameters::bRequiresRehydration and used to initialize the virtualization system.
-- We only need to initialize the system once, even if we detect that multiple pak files have files that need rehydration.
-- If no pak file needs rehydration we do not initialize the system and the virtualization module is never loaded.
- FOutputPakFileEntry::CompressedFileBuffer was made private with accessors to make the refactor easier.
- Now when we start to compress a file, we always load it entirely into memory (along with any potential padding needed), if we then detect that the file doesn't require padding then we just use the in memory buffer as the output and if we do need compression we compress the buffer.
-- This means that there is only one place we load the file from disk, meaing only one place we need to insert the rehydration code to. This load is where the rehydration occurs.
-- In the previous code we would load the file into memory and then retain this copy during the compression pass, so no additional memory should be used.
- At the moment when we get the buffer back fromt he rehydration pass it will be in the form of a FSharedBuffer and so for now we need to memcpy to FCompressedFileBuffer::UncompressedBuffer, which is a waste of cpu cycles.
-- In a future change we should change UncompressedBuffer from TArray to FSharedBuffer to avoid this.
[CL 22595708 by paul chipchase in ue5-main branch]
#rb CarlMagnus.Nordin
#rnx
#preflight 6346ab768a0a7b2adc72cce4
### Problem
- The bug was originally introduced in CL 21791765
- Each file to be compressed calls FPakWriterContext::BeginCompress which will create a FMemoryCompressor and assign it to the files FOutputPakFileEntry. The problem is that the work done by the FMemoryCompressor can potentially complete and signal EndCompressionBarrier before the FMemoryCompresseor has been assigned. This can be due to the work itself being very small, or that closing a file handle when FCompressedFileBuffer::BeginCompressFileToWorkingBuffer returns (which creates the FMemoryCompressor) stalls against system resources. This potentially allows FPakWriterContext::EndCompres to execute concurrently and incorrectly believe that the file is not being compressed as FOutputPakFileEntry::MemoryCompressor is still nullptr.
- Not only does this mean that some files end up being paked uncompressed when they should be compressed, the results are not deterministic which means we can end up with different binary output when paking the exact same data with the exact same settings.
### Solution
- The constructor of FMemoryCompressor no longer adds its work to the task graph system. Instead the tasks are only added after the compressor has been assigned to FOutputPakFileEntry.
- Replaced auto in ranged for loops with the type, as per recent coding standards discussions.
[CL 22505090 by paul chipchase in ue5-main branch]
thread-safe delegates are not zero-initializable and so can't be used as global vars because they are vulnerable to static initialization order fiasco.
#jira UE-163668
#rb steve.robb
#preflight 632462db3752284a3179ec02
[CL 22094531 by Andriy Tylychko in ue5-main branch]
* Added -cryptofile documentation line when starting unrealpak without parameters
#rb Erik.Knapik
#preflight skip
#ROBOMERGE-AUTHOR: henrik.karlsson
#ROBOMERGE-SOURCE: CL 21183712 via CL 21195199 via CL 21195379 via CL 21195491
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21196924 by henrik karlsson in ue5-main branch]
* Contains some selected back outs from other cls that included config context dependent changes, though it's limited to only backing out broken parts as much as possible to minimize surface area.
[Backout] - CL20029182
[FYI] josh.adams
Original CL Desc
-----------------------------------------------------------------
- Adding FConfigContext which is used to repalce LoadExternalIniFile, LoadLocalIniFile, etc, as well as have localized data for all configs read on a thread (like the other platform configs loaded in the editor)
- The Load*IniFile functions will create a Context, but eventually those APIs will go away and the Context will be the only way to load ini files
- Simplified some of the ini loading code, like removing the HierarchyCache (it wasn't helping editor load times, and added much complexity, and was not thread-safe, and it shouldn't actually be helpful because all the calls to Load*IniFile should eventually be replaced with either GConfig or FCOnfigCacheIni::ForPlatform(), which won't need to re-read in files
- Ini reading time actually went down due to the simplification, including Cache removal
- Left in old code for now behing a #define (USE_CONTEXT) in case something goes wrong
- Added in VERIFY_CONTEXT mode which I used to run original and Context modes and compare (including preflighting builds) (I also added a Compare function that we may want to keep around to use for future debugging)
- Added a separate set of config layers for plugins which speeds up plugin parsing, but also will fix the issue with BaseEngine.ini vs DefaultEngine.ini in Engine vs Project plugins (this shows how Contexts can bring useful information down into the guts - however a later change will enable it)
- Once this is all seen to be working, I will clean up the non-Context functions, and some globals vs static members, etc
#rb paul.chipchase
#preflight 62716c1c5e6ce673f452005a
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 20029165 via CL 20029180 via CL 20905839 via CL 20905880
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)
[CL 20908094 by mic rooney in ue5-main branch]
- Deprecated some global functions now in ConfigUtilities
- Deleted a couple of old deprecated stuff from 3.24
#rb chris.waters
#preflight 628415e1ba3597a030b3b900
[CL 20259749 by Josh Adams in ue5-main branch]
