Commit Graph

398 Commits

Author SHA1 Message Date
tomasz obrebski
5cce972555 Zen pak file streaming:
- -ZenPak used to indicate streaming using pak files instead of Zen loose file streaming
 - added pak file hardware decompression on supported platforms

#jira UE-221577
#rb David.Harvey, daniele.pieroni

[CL 36761256 by tomasz obrebski in 5.5 branch]
2024-10-01 20:04:53 -04:00
tomasz obrebski
54a30bd892 IFileHandle seekless ReadAt
#jira UE-213809
#rb brian.white, daniele.pieroni, David.Harvey, Josie.Yang
#rnx

[CL 36757440 by tomasz obrebski in 5.5 branch]
2024-10-01 19:29:48 -04:00
Ryan Hummer
4af2fd066d Updating Dev-Release-5.5 from Main at CL #36144969
#okforversepublic

[CL 36146571 by Ryan Hummer in Dev-5.5 branch]
2024-09-10 10:26:02 -04:00
per larsson
95a0b062e0 Notify I/O service thread when cancelling read requests
#rb mickael.gilabert, paul.chipchase
#rnx

[CL 35903347 by per larsson in ue5-main branch]
2024-08-29 15:02:23 -04:00
mickael gilabert
e0dc61b0b9 Enable processing cancelled file read requests before starting I/O (s.IoDispatcherCullCancelledReadRequests=1)
[REVIEW] [at]per.larsson
#rnx
#rb Per.Larsson

[CL 35744281 by mickael gilabert in ue5-main branch]
2024-08-22 12:17:41 -04:00
martin ridgers
1ed154e414 Reunite all FPakPlatformFile methods
#rnx
#rb pj.kack

[CL 35734382 by martin ridgers in ue5-main branch]
2024-08-22 04:40:41 -04:00
martin ridgers
103e889f71 Split a large file with components multiplexed together into smaller parts.
#rnx
#rb pj.kack

[CL 35733610 by martin ridgers in ue5-main branch]
2024-08-22 03:33:21 -04:00
daniele vettorel
a4e0b45042 Correctly determine IO scatter task priority from CVar.
#rb tom.holmes, tyler.staples

[CL 35722676 by daniele vettorel in ue5-main branch]
2024-08-21 17:32:54 -04:00
per larsson
31d38f1835 Process cancelled file read requests before performing file I/O to reduce the risk of stalling the I/O dispatcher thread and GC hang.
- disabled by default, enable with CVAR s.IoDispatcherCullCancelledReadRequests = True

#rb pj.kack
#rnx
[FYI] mickael.gilabert

[CL 35708446 by per larsson in ue5-main branch]
2024-08-21 12:58:23 -04:00
danny couture
17e411509c [IoDispatcher]
- Fix potential deadlock on low core count when all workers are waiting on IO and the IoDispatcher is waiting on decompression tasks

#rb Francis.Hurteau
[RN] Fix potential deadlock on low core count

[CL 35174606 by danny couture in ue5-main branch]
2024-07-30 07:59:31 -04:00
pj kack
c7e041ef02 FPackageId: Deprecate and replace ValueForDebugging with LexToString.
#rb per.larsson
#rnx

[CL 34702722 by pj kack in ue5-main branch]
2024-06-27 04:15:42 -04:00
pj kack
7099f09264 IoDispatcher: Use the new Tasks API to launch scatter block decompression tasks.
Reduce risk of deadlock if other background worker tasks are waiting for io by allowing decompression tasks to run on both foreground and background workers.
Respect the existing priority cvar "TaskGraph.TaskPriorities.IoDispatcherAsyncTasks".

#rb per.larsson, danny.couture
#tests pc and console replays

[CL 34673303 by pj kack in ue5-main branch]
2024-06-26 09:11:36 -04:00
per larsson
129585ee43 Added option -CookSoftPackageReferences to save soft package references when cooking
#rb Matt.Peters, Paul.Chipchase
#rnx

[CL 34503130 by per larsson in ue5-main branch]
2024-06-19 12:16:51 -04:00
ben woodhouse
2ea61ce0c6 Switch to CSV_PROFILER_STATS instead of CSV_PROFILER for various profiling subsystems. This allows them to be compiled out when CSV_PROFILER_MINIMAL is defined.
As part of this change we also promote dynamic resolution and IO/PackageQueueDepth stats to Minimal since they're important for high level performance reporting.
Also fix up a few places that were redundantly using #if CSV_PROFILER around CSV macros.

#rb mickael.gilabert

[CL 34386798 by ben woodhouse in ue5-main branch]
2024-06-14 18:16:47 -04:00
pj kack
3ad849a688 UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Always run UnrealPak itself with -nopak to avoid picking up and mounting any existing pak files in the project directory.

Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.

Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.

#rb per.larsson
#tests identical binary output

[CL 34266589 by pj kack in ue5-main branch]
2024-06-11 03:37:13 -04:00
brandon schaefer
d1becd40a2 Remove the now deleted option to EnableAutoRTFM, this is default and now youve to explicitly disable it for a module
#rb neil.henning

[CL 34015057 by brandon schaefer in ue5-main branch]
2024-05-30 18:00:50 -04:00
anton dunchev
7837872066 [Trace] Fix CIS Add TraceLog module to linker
[CL 33361272 by anton dunchev in ue5-main branch]
2024-04-30 20:27:05 -04:00
anton dunchev
69fd53db6a [LLM] Scope filesystem callbacks to shader code into Shader LLM, added more asset tags
#rb robert.millar
#tests PIE

[CL 33360979 by anton dunchev in ue5-main branch]
2024-04-30 20:09:23 -04:00
pj kack
1754180bc2 ZenLoader: Extend container mount logs to get some more information on errors of type "Failed reading chunk for package: (Not Found)" when there is no ExportBundleData chunk for a package known by the loader.
Add LexToString(const FIoContainerId& ContainerId).

#rb Per.Larsson
#rnx

[CL 33193351 by pj kack in ue5-main branch]
2024-04-24 02:52:07 -04:00
dan thompson
2b3a8d681c FCompression interface update to 64 bit.
-> Done because UE disables 32 bit down cast - silently could overflow.
-> General use should automatically work.
-> Deprecate some old 32 bit functions not generally used.
-> Deprecate/rename some ancient flags for future removal.
-> Update CompressorData to uint ptr for future use
-> Remove some stuff deprecated in 4.21 (2018)
#rb fabian.giesen
#jira UE-173911

[CL 31688300 by dan thompson in ue5-main branch]
2024-02-21 12:49:23 -05:00
steve robb
f8d47335a4 Replaced RemoveAt(N, 1, EAllowShrinking::*) with RemoveAt(N, EAllowShrinking::*).
[CL 31626444 by steve robb in ue5-main branch]
2024-02-19 16:51:58 -05:00
matt peters
65511c6a9b Fix crash when running game with legacy pakfiles (-skipiostore). FPakReadRequestBase and all subclasses of IAsyncReadRequest need a default implementation for the new EnsureCompletion function rather than a fatal error if called.
#rnx
#rb danny.couture

[CL 31567969 by matt peters in ue5-main branch]
2024-02-16 11:40:30 -05:00
matt peters
149d6dc477 PakFiles: DeleteRecords: Allow files that are duplicated between chunks to still be found if one chunk deleted them in a patch but the other chunk did not.
#rnx
#rb David.Harvey

[CL 31563886 by matt peters in ue5-main branch]
2024-02-16 09:47:47 -05:00
per larsson
dccb6df43e I/O - changed dispatcher backend interface from resolving a single to a batch of I/O requests.
#rb Paul.Chipchase, Martin.Ridgers
#rnx

[CL 31469283 by per larsson in ue5-main branch]
2024-02-14 08:43:21 -05:00
matt peters
398a33a33c PakFiles: Suppress spurious log message "Mount point '%s' is not mounted to a valid Root Path yet" when the mount point is a parent path of existing root paths.
#rnx
#rb Sebastian.Thomeczek

[CL 31396542 by matt peters in ue5-main branch]
2024-02-12 15:50:54 -05:00