Commit Graph

212 Commits

Author SHA1 Message Date
Devin Doucette
9b5b232cca Texture: Integrated the derived data build jobs with the asset compiling manager in a two-phase approach
The texture build task queues with the texture compiling manager to block on concurrency limits, or when texture builds are paused. The build scheduler will queue with the asset compiling manager to reserve memory when the build needs to execute locally. With this approach, resuming one texture will complete one texture, even though multiple separate build jobs may be executed.

#preflight 6381820933774509009e6e7d
#rb Danny.Couture, Zousar.Shaker

[CL 23271848 by Devin Doucette in ue5-main branch]
2022-11-25 22:18:09 -05:00
Devin Doucette
110afa128b DDC: Added new build scheduler modular features to queue by type and reserve memory
#preflight 638128f3cc307d6fa5b4ee80
#rb Zousar.Shaker

[CL 23271781 by Devin Doucette in ue5-main branch]
2022-11-25 21:38:59 -05:00
Devin Doucette
e2483014be DDC: Added LaunchTaskInThreadPool as a replacement for the two existing thread pool task requests
#preflight 6380f7c4170bc34a93b9104b
#rb Zousar.Shaker

[CL 23267593 by Devin Doucette in ue5-main branch]
2022-11-25 12:35:29 -05:00
Devin Doucette
165d24ede5 DDC: Made the cache optional for the build system
#preflight 636c045ef56cab38c78f8b47
#rb Tim.Smith
#rnx

[CL 23062223 by Devin Doucette in ue5-main branch]
2022-11-09 15:53:21 -05:00
henrik karlsson
4feb09e0b3 Compile time optimizations Managed to reduce unnamed game's biggest file compiler frontend cost from 113 seconds to 60 seconds. This also impact other things such as pch size and memory footprint when compiling which is great.
Takeaways. Try to use "friend" on functions/operators that are frequently overloaded EXCEPT if they are in a templated type that is frequently instantiated. So do not put friends in TMap, TSet, TObjectPtr etc, this will slow down compile times. There is a break-even somewhere and hard to tell where it is but taking a class that is templatized on character type probably don't matter either way and then it is nicer to use a friend since that simplies error messages when compiler can't resolve functions/operators.

If it is possible to use member functions instead of friend that is the best option in terms of compile time performance. With c++20 you only have to write operator==(Foo, Bar) and the compiler will automatically provide operator==(Bar, Foo), operator!=(Foo, Bar) and operator!=(Bar, Foo).

Changes in this changelist involes
* Making operator<< friends in non-template types and not friends in template types
* Making operator==/!= members where possible and if not possible moved out if type is a frequently instantiated templated type.

#preflight 636970f5376a9cd6a80da54a
#rb steve.robb

[CL 23038965 by henrik karlsson in ue5-main branch]
2022-11-08 15:59:46 -05:00
Devin Doucette
6c33c94b36 DDC: Added experimental interface IBuildSchedulerThreadPoolProvider to integrate the build scheduler with the asset compiling managers
#jira UE-141137
#jira UE-141139
#preflight 63696f5e843e6ac79482dad1
#rb Zousar.Shaker

[CL 23027624 by Devin Doucette in ue5-main branch]
2022-11-08 09:45:50 -05:00
henrik karlsson
4a36cfe8ff Moved operator== to be hidden friend instead of put directly in global namespace
Moved GetTypeHash function to be hidden friend instead of put directly in global namespace.

Note that the function/operator needs to be fully inlined in the type or placed in the cpp. If the function is added as friend but then implemented outside the type then hidden friend optimization won't work.

This should improve compile time somewhat according to msvc devs.

#rb Steve.Robb
#preflight 6360b7052b5338aceb26471b

[CL 22889837 by henrik karlsson in ue5-main branch]
2022-11-01 15:50:27 -04:00
graeme thornton
e7986ac620 [Backout] - CL22808749
[FYI] Steve.Robb
Original CL Desc
-----------------------------------------------------------------
TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

[FYI] steve.robb
#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22850782 by graeme thornton in ue5-main branch]
2022-10-29 02:59:59 -04:00
steve robb
e980405a8f TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22810695 by steve robb in ue5-main branch]
2022-10-27 11:20:32 -04:00
Devin Doucette
74ce7089b4 DDC: Fixed SharedString reference counting flagged by TSan
#preflight 633f1a48437603cb0edd72a4
#rb Zousar.Shaker
#rnx

[CL 22384667 by Devin Doucette in ue5-main branch]
2022-10-06 17:28:03 -04:00
Devin Doucette
7f98c7fe6c DDC: Added functions to convert between derived data priority and queued work priority
#preflight 632b42d4b4515b7e226ffae8
#rb Zousar.Shaker
#rnx

[CL 22118431 by Devin Doucette in ue5-main branch]
2022-09-21 14:25:32 -04:00
devin doucette
abd22903b7 Fixed forwarding of -DDC=<Graph> to avoid initializing DDC and only forward non-default graph names
#preflight 631f926c1e6887c4a6d4c30d
#rb Zousar.Shaker

[CL 21984731 by devin doucette in ue5-main branch]
2022-09-13 12:12:58 -04:00
devin doucette
35be596466 DDC: Reverted unnecessary changes made by IWYU
There are several cases where includes were added that are not directly used (only by reference or pointer), and where forward declarations were reordered in a way that worsens readability.

#rb none
#rnx

[CL 21499953 by devin doucette in ue5-main branch]
2022-08-22 21:23:18 -04:00
bryan sefcik
de1956f47b Ran IWYU on Public headers under Engine/Source/Developer/...
Headers are updated to contain any missing #includes needed to compile and #includes are sorted.  Nothing is removed.

#ushell-cherrypick of 21064294 by bryan.sefcik
#jira
#preflight 62d5c2111062f2e63015e598

#ROBOMERGE-OWNER: bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21155249 via CL 21158121 via CL 21161259
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21182053 by bryan sefcik in ue5-main branch]
2022-07-20 12:03:45 -04:00
zousar shaker
75d0e7277b Add status icon to indicate DDC cache store status with tooltip text for details. Currently used to show when a cache store has been deactivated for performance.
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 21152573 via CL 21155094 via CL 21156160
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21181761 by zousar shaker in ue5-main branch]
2022-07-20 11:55:16 -04:00
devin doucette
6366a19bda DDC: Added functions to serialize cache types to and from compact binary, and strings where appropriate
#rb Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741868 via CL 20744192 via CL 20745275
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751385 by devin doucette in ue5-main branch]
2022-06-20 23:40:01 -04:00
devin doucette
3933b8f407 DDC: Added TryLoadFromCompactBinary for FCacheKey
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20705430 via CL 20705467 via CL 20705498
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)

[CL 20707645 by devin doucette in ue5-main branch]
2022-06-17 14:45:17 -04:00
steve robb
0edc9ec830 Some missing TEXT macros added.
#rb trivial
#jira UE-132142
#preflight none

#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20671620 via CL 20671646 via CL 20671659
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20675152 by steve robb in ue5-main branch]
2022-06-15 15:09:53 -04:00
devin doucette
3e2538a313 DDC: Removed the legacy cache store implementations and limit cache bucket names to 63 characters
The cache bucket name was already limited to 63 characters in practice by Horde Storage. This makes it consistent across every cache store.

#preflight 629e6f0a1c7e8f9a180e192a
#rb Zousar.Shaker

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20536454 via CL 20536805 via CL 20536993
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20538633 by devin doucette in ue5-main branch]
2022-06-07 11:17:10 -04:00
Devin Doucette
633521c203 DDC: Exposed IsCanceled and LaunchTask through FRequestOwner
#preflight 62958df65370042eb050c1c3
#rb Zousar.Shaker
#rnx

[CL 20436079 by Devin Doucette in ue5-main branch]
2022-05-31 09:01:19 -04:00
mark lintott
da46ade47c Added Cache Speed Stats to Analytics
Added Zen Server Stats to Analytics
#rb zousar.shaker, devin.doucette
#jira none
#preflight 6295d84bd57da28cc8ee9eab

[CL 20434901 by mark lintott in ue5-main branch]
2022-05-31 05:11:05 -04:00
Dan Thompson
7f5df95274 no functionality change, misc comments
#rb none
#preflight 62869724f15d967ff4d1e712

[CL 20286015 by Dan Thompson in ue5-main branch]
2022-05-19 15:44:07 -04:00
Devin Doucette
92a888c1f6 Replaced uses of _ASV with ANSITEXTVIEW
#preflight 6266acf50634d0904cdfe91c
#rb Steve.Robb
#rnx

[CL 19903251 by Devin Doucette in ue5-main branch]
2022-04-25 11:31:36 -04:00
Devin Doucette
ceabbe51e8 DDC: Added more documentation to the cache policy
#preflight 626183407d162829ae811783
#rb Zousar.Shaker
#rnx

[CL 19851224 by Devin Doucette in ue5-main branch]
2022-04-21 12:40:49 -04:00
Devin Doucette
8c27714a84 DDC: Added ConvertLegacyCacheKey to construct new cache keys from legacy cache keys
The function is shared with the legacy cache store to allow callers of the legacy API to migrate to the value API using this conversion function without invalidating existing cache keys.

#preflight 624c9ab68d1db441a913137e
#rb Zousar.Shaker

[CL 19630681 by Devin Doucette in ue5-main branch]
2022-04-05 16:27:17 -04:00