#rb steve.robb
#ROBOMERGE-SOURCE: CL 12450992 via CL 12450999 via CL 12457993
#ROBOMERGE-BOT: (v672-12450963)
[CL 12458149 by johan torp in Release-Engine-Staging branch]
[FYI] johan.torp, steve.robb, eric.knapik
#rnx
#ROBOMERGE-SOURCE: CL 12434782 via CL 12434783 via CL 12437070
#ROBOMERGE-BOT: (v671-12333473)
[CL 12443982 by david hamm in Release-Engine-Staging branch]
#rb steve.robb, eric.knapik
#ROBOMERGE-SOURCE: CL 12371098 via CL 12372591 via CL 12388959
#ROBOMERGE-BOT: (v671-12333473)
[CL 12390565 by johan torp in Release-Engine-Staging branch]
Legacy pak file versions encode pak entries in FPakFile::LoadLegacyIndex. That function repeatedly serializes bytes from the pakfile on disk into a reused FPakEntry. Serialize expects the FPakEntry to be in an empty constructed state, and it was not empty due to serializing previous PakEntries into it.
Added a reset function and call it in this one place where FPakEntry is reused.
#rb Graeme.Thornton
#rnx
[CL 12000363 by Matt Peters in Main branch]
#rb none
#testing iOS Client
#ROBOMERGE-SOURCE: CL 11843182 via CL 11843195 via CL 11843203
#ROBOMERGE-BOT: (v656-11643781)
[CL 11879820 by thomas ross in Main branch]
This was originally fixed in CL 11458603 but it did not fix the issue entirely and in any case I was not happy with the manual reset event approach.
The problem with the manual reset event was that there is no actual safe point at which we can reset the event without risking some requesting thread missing it and waiting forever, it is just a lot rarer than the original thread lock.
- Each FChunkRequest now can be given an event that is triggered, so the correct thread is always woken up if waiting.
- Since the signals are 1-1 we no longer risk having threads never being woken.
- We only wake the thread that has a request to process so don't risk flooding the cpu.
- Removed FChunkCacheWorker::SetupDecryptionKey as it was not implemented/used.
- Removed FChunkCacheWorker::ChunkRequestAvailable and its associated methods ::WaitForNextChunk/::FlushRemainingChunkCompletionEvents.
In testing this version runs slightly faster on loading tests (0.2seconds on average), although just removing the events entirely and running a busy loop with a yield ran slightly faster on machines with less cores (4 cores with hyperthreading or less) albiet with a much higher variation and so potentially could end up a lot worse.
#rb Graeme.Thornton
#jira UE-89165
#ushell-cherrypick of 11724338 by paul.chipchase
#ROBOMERGE-SOURCE: CL 11724753 in //UE4/Release-4.25/... via CL 11724760
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11724764 by paul chipchase in Main branch]
Fix incorrect use of static constexpr without inline.
#rb trivial
#rnx
#ROBOMERGE-SOURCE: CL 11465526 via CL 11465539
#ROBOMERGE-BOT: (v654-11333218)
[CL 11465545 by matt peters in Main branch]
FChunkCacheWorker::ChunkRequestAvailable could have many threads waiting on it and would only wake up a single thread each time a FChunkBuffer is processed. There would be no guarantee that the correct thread would be woken up or that each thread would be woken a single time, meaning that sooner or later we would end up with the FChunkCacheWorker having no more work to process but a number of threads still idle, waiting on the event to wake them, which would never happen.
Switching to a manually reset event will cause all waiting threads to be woken each time the event is triggered and will make sure that we don't deadlock.
This issue showed up now due to changes else where in the code which resulted in more threads waiting on FChunkCacheWorker than we previously had.
#jira UE-88734
#rb graeme.thornton
#ROBOMERGE-SOURCE: CL 11458603 in //UE4/Release-4.25/... via CL 11458605
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v654-11333218)
[CL 11462181 by paul chipchase in Main branch]
FChunkCacheWorker::ChunkRequestAvailable could have many threads waiting on it and would only wake up a single thread each time a FChunkBuffer is processed. There would be no guarantee that the correct thread would be woken up or that each thread would be woken a single time, meaning that sooner or later we would end up with the FChunkCacheWorker having no more work to process but a number of threads still idle, waiting on the event to wake them, which would never happen.
Switching to a manually reset event will cause all waiting threads to be woken each time the event is triggered and will make sure that we don't deadlock.
This issue showed up now due to changes else where in the code which resulted in more threads waiting on FChunkCacheWorker than we previously had.
#jira UE-88734
#rb graeme.thornton
#ROBOMERGE-SOURCE: CL 11458603 in //UE4/Release-4.25/...
#ROBOMERGE-BOT: RELEASE (Release-4.25 -> Release-4.25Plus) (v654-11333218)
[CL 11458605 by paul chipchase in 4.25-Plus branch]
#ROBOMERGE-SOURCE: CL 11458135 via CL 11458136 via CL 11458137 via CL 11458138
#ROBOMERGE-BOT: (v654-11333218)
[CL 11458139 by matt peters in Main branch]
OptimizedPakIndex: Bump the PakIndex version from 9 to 10, to avoid conflict with PakFile_Version_FrozenIndex = 9 from mainline. This change will remain in Fortnite only, and unlike UE::Main will refuse to load version 9 paks. This change can be clobbered with UE::Main's version after a few days when we have deleted all of our version 9 paks.
#ROBOMERGE-SOURCE: CL 11458117 via CL 11458118 via CL 11458119 via CL 11458120
#ROBOMERGE-BOT: (v654-11333218)
[CL 11458121 by matt peters in Main branch]
[REVIEW]
#rb Graeme.Thornton
Optimize Pak file indices by computing the PathHashIndex in unrealpak rather than using strings at runtime.
This also removes the need to shrink the string indices for performance, which was triggering a bug due to unsynchronized multithreaded access.
#ROBOMERGE-SOURCE: CL 11442058 in //Fortnite/Release-12.00/... via CL 11442063 via CL 11442073 via CL 11442081
[CL 11443295 by Matt Peters in Main branch]
#rb steve.robb
#ROBOMERGE-SOURCE: CL 11303014 via CL 11303031 via CL 11303037 via CL 11303042
#ROBOMERGE-BOT: (v0-11244347)
[CL 11303201 by graeme thornton in Main branch]
Name the number in pakchunk file pakchunk index and the number of a platform chunk chunk id to avoid confusion. Also keep this naming convention consistent across all platform code.
#ROBOMERGE-SOURCE: CL 11131543 via CL 11132441 via CL 11132504
#ROBOMERGE-BOT: (v640-11091645)
[CL 11132564 by hongyi yu in Main branch]
Partially back out changelist 11106714, only keeping intended merged code
[FYI] Dan.Phillips
#ROBOMERGE-SOURCE: CL 11107908 via CL 11107911
#ROBOMERGE-BOT: (v640-11091645)
[CL 11107914 by bob tellez in Main branch]
Allow a read request to tell the pak cache not to keep the memory after the request is completed.
Make virtual textures give up their pak cache memory immediately after loading.
[REVIEW] ben.woodhouse, steve.robb, david.harvey
#ROBOMERGE-OWNER: dan.phillips
#ROBOMERGE-AUTHOR: dan.phillips
#ROBOMERGE-SOURCE: CL 11106543 via CL 11106714 via CL 11106736 via CL 11106740
#ROBOMERGE-BOT: (v640-11091645)
[CL 11106744 by dan phillips in Main branch]
#rb Justin.Marcus
#ROBOMERGE-SOURCE: CL 11064634 via CL 11064641 via CL 11064653
#ROBOMERGE-BOT: (v637-11041722)
[CL 11064661 by hongyi yu in Main branch]
