Commit Graph

729 Commits

Author SHA1 Message Date
Devin Doucette
9b5b232cca Texture: Integrated the derived data build jobs with the asset compiling manager in a two-phase approach
The texture build task queues with the texture compiling manager to block on concurrency limits, or when texture builds are paused. The build scheduler will queue with the asset compiling manager to reserve memory when the build needs to execute locally. With this approach, resuming one texture will complete one texture, even though multiple separate build jobs may be executed.

#preflight 6381820933774509009e6e7d
#rb Danny.Couture, Zousar.Shaker

[CL 23271848 by Devin Doucette in ue5-main branch]
2022-11-25 22:18:09 -05:00
Devin Doucette
110afa128b DDC: Added new build scheduler modular features to queue by type and reserve memory
#preflight 638128f3cc307d6fa5b4ee80
#rb Zousar.Shaker

[CL 23271781 by Devin Doucette in ue5-main branch]
2022-11-25 21:38:59 -05:00
Devin Doucette
e2483014be DDC: Added LaunchTaskInThreadPool as a replacement for the two existing thread pool task requests
#preflight 6380f7c4170bc34a93b9104b
#rb Zousar.Shaker

[CL 23267593 by Devin Doucette in ue5-main branch]
2022-11-25 12:35:29 -05:00
Devin Doucette
d53ba8513d DDC: Allow the build system to be created without a cache
#preflight 637fa7e518176c67c20951a6
#rb Zousar.Shaker

[CL 23262694 by Devin Doucette in ue5-main branch]
2022-11-24 17:23:49 -05:00
Devin Doucette
d508667187 DDC: Added a missing read lock in the cache hierarchy
#preflight 63751a331d25fe8b93371452
#rb Matt.Peters
#rnx

[CL 23161581 by Devin Doucette in ue5-main branch]
2022-11-16 15:43:58 -05:00
Matt Peters
dd0f970188 HttpCacheStore: When a record is missing data and we return EStatus::Error and an empty record, fix the FCacheRecord passed into OnComplete to not have metadata or any values. The FCacheStoreHierarchy client of FHttpCacheStore assumes that metadata or values are only present on a successful record, and if it is present, incorrectly Puts the record as a valid record into the other cachestores in the hierarchy. This then breaks clients of DDC, because the record is invalid since it is missing some values.
This change fixes the warning messages when using EditorDomain: LogEditorDomain: Warning: Package %s received an invalid CacheRecord...
#rb Zousar.Shaker, Devin.Doucette
#rnx
#preflight 63753d95953c19d4351d8f11

[CL 23160500 by Matt Peters in ue5-main branch]
2022-11-16 15:12:51 -05:00
Devin Doucette
dcf192e97a DDC: Remove any trailing slash from the Jupiter URI
#preflight 636d11ed0f51d9ed4afbf185
#rb Joakim.Lindqvist
#rnx

[CL 23079894 by Devin Doucette in ue5-main branch]
2022-11-10 10:57:37 -05:00
Devin Doucette
165d24ede5 DDC: Made the cache optional for the build system
#preflight 636c045ef56cab38c78f8b47
#rb Tim.Smith
#rnx

[CL 23062223 by Devin Doucette in ue5-main branch]
2022-11-09 15:53:21 -05:00
Devin Doucette
381a587dcd DDC: Invoke the build completion callback after storing to the cache
This avoids a race condition where a build completes and then another process is unable to retrieve the data from the cache.

#preflight 636b2a547c2b505190e7d036
#rb Zousar.Shaker
#rnx

[CL 23059312 by Devin Doucette in ue5-main branch]
2022-11-09 13:57:59 -05:00
Zousar Shaker
ac3fc5db61 Print more detailed timing metrics for HTTP operations in the HttpCacheStore.
#rb devin.doucette
#preflight 636acc25dc30a4ce96bbe43c

[CL 23040720 by Zousar Shaker in ue5-main branch]
2022-11-08 16:52:32 -05:00
henrik karlsson
4feb09e0b3 Compile time optimizations Managed to reduce unnamed game's biggest file compiler frontend cost from 113 seconds to 60 seconds. This also impact other things such as pch size and memory footprint when compiling which is great.
Takeaways. Try to use "friend" on functions/operators that are frequently overloaded EXCEPT if they are in a templated type that is frequently instantiated. So do not put friends in TMap, TSet, TObjectPtr etc, this will slow down compile times. There is a break-even somewhere and hard to tell where it is but taking a class that is templatized on character type probably don't matter either way and then it is nicer to use a friend since that simplies error messages when compiler can't resolve functions/operators.

If it is possible to use member functions instead of friend that is the best option in terms of compile time performance. With c++20 you only have to write operator==(Foo, Bar) and the compiler will automatically provide operator==(Bar, Foo), operator!=(Foo, Bar) and operator!=(Bar, Foo).

Changes in this changelist involes
* Making operator<< friends in non-template types and not friends in template types
* Making operator==/!= members where possible and if not possible moved out if type is a frequently instantiated templated type.

#preflight 636970f5376a9cd6a80da54a
#rb steve.robb

[CL 23038965 by henrik karlsson in ue5-main branch]
2022-11-08 15:59:46 -05:00
Devin Doucette
6c33c94b36 DDC: Added experimental interface IBuildSchedulerThreadPoolProvider to integrate the build scheduler with the asset compiling managers
#jira UE-141137
#jira UE-141139
#preflight 63696f5e843e6ac79482dad1
#rb Zousar.Shaker

[CL 23027624 by Devin Doucette in ue5-main branch]
2022-11-08 09:45:50 -05:00
Devin Doucette
bfd3e68dc5 DDC: Fixed invalid put stats in the Jupiter client
#preflight 63691e65f56cab38c7c75f83
#rb Zousar.Shaker
#rnx

[CL 23010981 by Devin Doucette in ue5-main branch]
2022-11-07 10:33:06 -05:00
dan engelbrecht
d81aef1e6c Allow support for receiving local file references for cache items
Companion PR in zen: https://github.com/EpicGames/zen/pull/184
#rb stefan.boberg
#preflight 6368ce57d0174259cc1e7b7a

[CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
Zousar Shaker
8b31a778bf Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only.
#rb devin.doucette
#preflight 63643f6d882365b8596ce430

[CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
devin doucette
41c5373ebd DDC: Fixed parameter names related to OIDC to be compliant with the coding standard and simplified analytics formatting
#preflight 636403b3de2c4dbb5d3a5b11
#rb Zousar.Shaker
#rnx

[CL 22968397 by devin doucette in ue5-main branch]
2022-11-03 18:00:06 -04:00
Devin Doucette
7962247e4e DDC: Added a DeleteOnly mode to FileSystemCacheStore
This allows a cache to be kept in the hierarchy solely to ensure that it is cleaned up as its contents expire. Maintenance has been improved to delete more empty folders and to count every scanned folder.

#preflight 6363370154471d10be395619
#rb Zousar.Shaker

[CL 22953493 by Devin Doucette in ue5-main branch]
2022-11-03 11:01:04 -04:00
mark lintott
5ae432fc55 Addeed specific Unreal Cloud DDC analytics for core.cooking and core.loading events ( Domain, Effective Domain, Cache Hit/MIss for Cache Puts & Gets, FailedOIDC login attempts )
#rb joakim.lindqvist
#jira none
#preflight 6363c123d308b70e66f4cf33

[CL 22951911 by mark lintott in ue5-main branch]
2022-11-03 09:39:24 -04:00
Devin Doucette
7945169d6c DDC: Fixed the async cache replay to be tracked as async work by the cache
#preflight 6362d7f69160784fc32ea869
#rb Zousar.Shaker
#rnx

[CL 22926967 by Devin Doucette in ue5-main branch]
2022-11-02 17:13:24 -04:00
Devin Doucette
0db4050542 Zen: Changed the Analytics dependency from public to private
#preflight 6362b15954471d10be06613a
#rb none
#rnx

[CL 22924597 by Devin Doucette in ue5-main branch]
2022-11-02 15:43:23 -04:00
Devin Doucette
f1d54c3a16 DDC: Sorted and removed duplicate includes in DerivedDataCache.cpp
#preflight 63627f0254471d10bef32ed7
#rb none
#rnx

[CL 22918352 by Devin Doucette in ue5-main branch]
2022-11-02 11:58:13 -04:00
mark lintott
490070be3c Decoupled Zen, DerivedDataCache and Virtualization dependency from Studio Analytics. Systems can now register callbacks to append or modify Event payloads to analytics events by name via the FAnalytics interface.
#rb Nick.Darnell
#fyi Paul.Chipchase,Zousar.Shaker
#preflight 636248d054471d10bedc3749

[CL 22909100 by mark lintott in ue5-main branch]
2022-11-02 06:56:10 -04:00
henrik karlsson
4a36cfe8ff Moved operator== to be hidden friend instead of put directly in global namespace
Moved GetTypeHash function to be hidden friend instead of put directly in global namespace.

Note that the function/operator needs to be fully inlined in the type or placed in the cpp. If the function is added as friend but then implemented outside the type then hidden friend optimization won't work.

This should improve compile time somewhat according to msvc devs.

#rb Steve.Robb
#preflight 6360b7052b5338aceb26471b

[CL 22889837 by henrik karlsson in ue5-main branch]
2022-11-01 15:50:27 -04:00
graeme thornton
e7986ac620 [Backout] - CL22808749
[FYI] Steve.Robb
Original CL Desc
-----------------------------------------------------------------
TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

[FYI] steve.robb
#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22850782 by graeme thornton in ue5-main branch]
2022-10-29 02:59:59 -04:00
devin doucette
271b9555dc Added Unreal Cloud DDC to the base cache graphs
It is disabled by default outside of Epic, and can be enabled by configuring the Default key in the StorageServers section of the Engine config.

#rb Zousar.Shaker

[CL 22836839 by devin doucette in ue5-main branch]
2022-10-28 16:07:04 -04:00