354 Commits

Author SHA1 Message Date
UnrealBot
e332249d7b Branch snapshot for CL 26969186
[CL 26969186 in 5.3 branch]
2023-08-09 20:15:04 +00:00
henrik karlsson
562dc45ab9 [Engine]
* Moved dllexport from type to methods/staticvar in all Engine runtime code. This improves compile times, memory and performance in dll builds

[CL 26082288 by henrik karlsson in 5.3 branch]
2023-06-17 18:14:12 -04:00
eric knapik
bc3b5dd786 #jira: none
Adding some Pak initialization and deinitialization logging to track down a horde only ensure.

#rb: trivial

[CL 25795014 by eric knapik in ue5-main branch]
2023-06-05 12:36:27 -04:00
Wojciech Krywult
d1baf24722 AsyncIO: Fixed incorrect handling of the STAT_AsyncFileMemory stat.
I've reviewed all code using async requests and we had multiple places incorrectly handling the stat: some placed failed to decrease the counter, while others would decrease it when they shouldn't. The main cause was the fact that when we create an async request and pass nullptr as the target memory, the request allocates the memory itself and increases the STAT_AsyncFileMemory. However, when we call GetReadResults() on such requests, the responsibility is on the caller to balance the calls and decrease the stat, which is not very obvious and leads to the mentioned bugs. On top of that, when we call GetReadResults() we may not have the knowledge as to whether the returned memory was allocated by the request or not (which may affect whether we should touch the stat at all).

Resolved by adding IAsyncReadRequest::ReleaseMemoryOwnershipImpl member, which may be used by request implementations to do the proper clean up (balance the stat). This way requests take the full responsibility for their changes and users of the requests don't need to know their implementation details.

#preflight none
#rb Patrick.Laflamme
#jira UE-185064

[CL 25664733 by Wojciech Krywult in ue5-main branch]
2023-05-29 12:11:37 -04:00
dan thompson
5ecef89101 Validating index loading for upacks.
#jira UE-173038
#rb charles.bloom
#preflight 646d0bc96c2a2532b1747645

[CL 25593251 by dan thompson in ue5-main branch]
2023-05-23 18:33:52 -04:00
dan thompson
aba11d90b1 Sanitize inputs for UncompressCopyFile
#jira UE-173032
#jira UE-173029
#jira UE-173028
#rb fabian.giesen
#preflight 646bccd2576becd7b57ad6e1

[CL 25572218 by dan thompson in ue5-main branch]
2023-05-22 19:17:22 -04:00
per larsson
f72cc84a5a I/O - move slab allocator to internal for reuse between backends
#rb PJ.Kack
#preflight 645dc3106b4bdec267c9cfd2

[CL 25444366 by per larsson in ue5-main branch]
2023-05-12 03:50:13 -04:00
pj kack
12064cb1d3 IoDispatcher: Improve shutdown logic.
First reset the GIoDispatcher pointer.
Then stop the IoDispatcher thread and wait for all pending requests to finish.
Then shutdown the backends.
Then destroy the IoDispatcher itself.

#rb per.larsson
#rnx
#preflight 644bbbe91c2846595c308b2a

[CL 25298458 by pj kack in ue5-main branch]
2023-05-02 03:28:46 -04:00
matt peters
b6bd711eaa PakFiles: If FChunkCacheWorker's construction fails and the FChunkCacheWorker is invalid, fail the entire Pak by returning nullptr from CreatePakReader rather than allowing creation of an FSignedArchiveReader that will crash in FChunkCacheWorker::CheckSignature.
#rb Robert.Millar
#rnx
#preflight 6446b821dc71c13446eed5d1

[CL 25175481 by matt peters in ue5-main branch]
2023-04-24 23:21:31 -04:00
bryan sefcik
c7696eaa09 Updated several modules to set UnsafeTypeCastWarningLevel to WarningLevel.Error and not WarningLevel.Warning. The use of WarningLevel.Warning makes separate shared PCHs.
#preflight 643718baf12d5de7058cd78c

[CL 25015908 by bryan sefcik in ue5-main branch]
2023-04-12 17:09:36 -04:00
carlmagnus nordin
ecfc01c258 ZenLoader: Include preload dependencies in package headers instead of relying on predetermined export bundle dependencies. This means that each package can be treated in isolation when staging. Also removes the need to keep track of export and bundle counts in the package store which saves some memory.
#preflight 642e8838441e8b2f5506dbb4
#rb pk.kack,per.larsson,francis.hurteau

[CL 24944233 by carlmagnus nordin in ue5-main branch]
2023-04-06 10:50:30 -04:00
carlmagnus nordin
462cc2a438 IoStore: Include imported package names in the header data. This removes the need to keep track of uncooked packages in the package store for editor builds.
#rnx
#rb pj.kack
#preflight 64256a01b01a25d6a044758d

[CL 24869323 by carlmagnus nordin in ue5-main branch]
2023-03-31 07:20:51 -04:00
Nicholas Frechette
471ad92304 Prevent memory mapping compressed or encrypted memory with IoStore
#jira UE-146471
#rb carlmagnus.nordin
#preflight 641da9bdc44ce895fc9ac839

[CL 24778890 by Nicholas Frechette in ue5-main branch]
2023-03-24 10:06:39 -04:00
Andriy Tylychko
33551a6eaf deprecated FCoreDelegates::OnFileOpenedForReadFromPakFile because it's thread-safe but is used concurrently. Added a thread-safe version FCoreDelegates::GetOnFileOpenedForReadFromPakFile() and switched all use cases to the new one
#rb francis.hurteau
#preflight 641d9f8d7a393e211af9829b
#preflight 641d9f8d7a393e211af9829b

[CL 24778456 by Andriy Tylychko in ue5-main branch]
2023-03-24 09:22:49 -04:00
Andriy Tylychko
7b5ca318e0 deprecated FCoreDelegates::OnPakFileMounted2 because it's thread-safe but is used concurrently. Added a thread-safe version FCoreDelegates::GetOnPakFileMounted2() and switched all use cases to the new one
#rb francis.hurteau
#preflight 641d99d214423a492d30c75a

[CL 24778298 by Andriy Tylychko in ue5-main branch]
2023-03-24 08:57:53 -04:00
carlmagnus nordin
3b38a4e17d IoDispatcherFileBackend: Ensure that all decompression tasks have finished when shutting down
#preflight 641c50cabb1320be41f9eb44
#rb pj.kack
#rnx

[CL 24762343 by carlmagnus nordin in ue5-main branch]
2023-03-23 11:22:59 -04:00
devin doucette
4e02304b05 Moved API macro from types to members to fix compiling for Windows with Clang
An optimization to UE_LOG exposed an issue in Clang when targeting Windows. Log statements in inline functions of imported types fail to compile.

#preflight 63f7f350ef1b24bf94367191
#rb Zousar.Shaker
#rnx

[CL 24397809 by devin doucette in ue5-main branch]
2023-02-24 03:58:21 -05:00
james doverspike
90c0ddf9ba [Backout] - CL24328738
[FYI] james.doverspike
Original CL Desc
-----------------------------------------------------------------
Cosmetic bulk asset streaming

Adds support for marking skeletal meshes and textures as optional files that stream from the cloud, typically used for cosmetics. Meshes and textures will now upload all of their non-inlined LODs as optional bulk files to the cloud, removing them from the pak files. The client will download into the VirtualFileCache in PersistentDownloadDir/VFC, which evicts least recently used LODs. These assets will have a longer delay to update to the higher quality LOD when their LODs are not in cache because they are now downloaded on-demand.

[CL 24373261 by james doverspike in ue5-main branch]
2023-02-22 19:19:46 -05:00
james doverspike
6a8a19a4b7 Cosmetic bulk asset streaming
Adds support for marking skeletal meshes and textures as optional files that stream from the cloud, typically used for cosmetics. Meshes and textures will now upload all of their non-inlined LODs as optional bulk files to the cloud, removing them from the pak files. The client will download into the VirtualFileCache in PersistentDownloadDir/VFC, which evicts least recently used LODs. These assets will have a longer delay to update to the higher quality LOD when their LODs are not in cache because they are now downloaded on-demand.

[CL 24331735 by james doverspike in ue5-main branch]
2023-02-20 20:22:07 -05:00
carlmagnus nordin
ccaaec5e2e IoDispatcher: Keep priority value on resolved requests to avoid having to go out and read from the dispatcher requests
#preflight 63eb5a7ee92f139c51943811
#rb pj.kack

[CL 24253463 by carlmagnus nordin in ue5-main branch]
2023-02-16 04:19:30 -05:00
bob tellez
79d7201510 [Backout] - CL24230614
[FYI] james.doverspike
Original CL Desc
-----------------------------------------------------------------
Cosmetic bulk asset streaming

Adds support for marking skeletal meshes and textures as optional files that stream from the cloud, typically used for cosmetics. Meshes and textures will now upload all of their non-inlined LODs as optional bulk files to the cloud, removing them from the pak files. The client will download into the VirtualFileCache in PersistentDownloadDir/VFC, which evicts least recently used LODs. These assets will have a longer delay to update to the higher quality LOD when their LODs are not in cache because they are now downloaded on-demand.

[CL 24253381 by bob tellez in ue5-main branch]
2023-02-16 04:15:55 -05:00
james doverspike
4e78faf0de Cosmetic bulk asset streaming
Adds support for marking skeletal meshes and textures as optional files that stream from the cloud, typically used for cosmetics. Meshes and textures will now upload all of their non-inlined LODs as optional bulk files to the cloud, removing them from the pak files. The client will download into the VirtualFileCache in PersistentDownloadDir/VFC, which evicts least recently used LODs. These assets will have a longer delay to update to the higher quality LOD when their LODs are not in cache because they are now downloaded on-demand.

[CL 24251508 by james doverspike in ue5-main branch]
2023-02-16 01:47:06 -05:00
anton dunchev
990e8f0d44 MemQuery use PackageStore to resolve dependencies
#tests replay
#rb Robert.Millar
[FYI] andrew.ladenberger, simon.orr

[CL 24202843 by anton dunchev in ue5-main branch]
2023-02-13 22:45:46 -05:00
Matt Peters
7dfa3846ff PakFilePrecacher: Fix deadlock by avoiding calling the body of ClearOldBlockTasks while holding CachedFilesScopeLock.
ClearOldBlockTasks can BusyWait, which can call FAsyncIOCPUWorkTask::DoTask, which can try to take a FPakAsyncReadFileHandle's CriticalSection. FPakAsyncReadFileHandle's CriticalSection must be entered first when holding both CriticalSection and CachedFilesScopeLock to prevent deadlocks.
#rb Per.Larsson
#rnx
#preflight 63e144cdc2257e56f4ee08a3

[CL 24035541 by Matt Peters in ue5-main branch]
2023-02-06 14:17:52 -05:00
jared cotton
33f6bf1377 FORT-550853
Update to make IsSymlink() return an enum which allows callers to identify whether or not it is implemented for the calling PlatformFile. Also, added appropriate implementation for FPakPlatformFile which was the cause of ensure's on previous submission which was backed-out.

(see review https://p4-swarm.epicgames.net/reviews/23757906 for usage in SolarisEditorModule.cpp)

#rnx
#rb yiliang.siew
#rb brandon.schaefer
#rb calvin.zheng
#rb zack.neyland
#preflight 63d186e0ef20a5272dba95ae

[CL 23863390 by jared cotton in ue5-main branch]
2023-01-25 20:42:04 -05:00