Commit Graph

224 Commits

Author SHA1 Message Date
devin doucette
e675fcb5b5 DDC: Exposed metadata more extensively in the build API
#rb Zousar.Shaker

[CL 30691096 by devin doucette in ue5-main branch]
2024-01-18 14:06:50 -05:00
dan engelbrecht
ba66b561de For local Zen DDC connections show the path where data is store in the Cache Statistics tab for DDC instead of local ip address
For Zen DDC connection show the storage size
#jira UE-199929
#rb Devin.Doucette

[CL 30501101 by dan engelbrecht in ue5-main branch]
2024-01-09 06:08:29 -05:00
mark lintott
b99d530ce0 Moved EpicStudioAnalytics plugin to StudioTelemetry which is now public facing.
The intention is to provide a set of "in the box" telemetry hooks to our most common developer workflows for use internal Epic use and licensees
Moved startup of StudioTelemetry plugin much earlier to PostConfigInit stage so that sessions can start much earlier in the workflow
Added WIP Client support to telemetry plugin
Added data driven provider support for FAnlayiticProvidersET interfaces via BaseEngine.ini for common games and sample projects. Settings and URLs for EPic are only avaliable in Resttricted/NotForLicensee config folders for Lyra and Shooter game.
Added support for Horde telemetry and fixed up various problems with duplicate attributes being sent.
Added Core.VirtualAssets telemetry event
Added Core.Zen telemetry event
Added Core.IAS telemetry event
Added check for IOStoreOnDemand IsEnabled to avoid sending empty IAS events
Added FAnalyticsMulticastProvider to forward telemetry events to multiple providers contained within
Removed deprecated Fire_LoadingEvent from StudioAnalytics
[FYI] paul.chipchase
#rb Wes.Hunt

[CL 29908039 by mark lintott in ue5-main branch]
2023-11-23 07:06:10 -05:00
danny couture
5c1c6cd4b4 Fix TSAN warning about potential race when releasing shared string
#rnx
#rb Devin.Doucette

[CL 26735342 by danny couture in ue5-main branch]
2023-08-01 07:49:14 -04:00
devin doucette
3df0bf7f14 DDC: Gather bucket stats from the new ICacheStats
#jira UE-133395
#rb Matt.Peters
#rnx

[CL 26555188 by devin doucette in ue5-main branch]
2023-07-24 16:52:05 -04:00
mark lintott
e63c334048 EpicStudioAnalytics plugin rewrite
High Level Vision:
https://docs.google.com/document/d/13R4nJ2UNMJZlf6JCjYssFMSY-5CrURazE4tqxW77yBw/edit#heading=h.bpycydduomgi

Details of changes:
Removed "most" references to FStudioAnalytics from Engine code.
Deprecated use of FStudioAnalytics::FireEvent_Loading(), will deprecate FStudioAnalytics in due course.
Added FAnalyticsFlowTracker to the engine code base based on the flow system used in UEFN.
Added explicit DDC Resource and DDC Summary events to track DDC usage by resource type.
Moved common Editor analytics callbacks into EpicStudioAnalytics plugin
Added a variety of delegates to the engine code to so that analytics capture can be defined in the plugin code or within the game code instead of the engine code ( eg. load map, editor initialization, cooking, slow task )
Added JSON serializer for analytics events to provide facility to capture analytics in restriced environments ( eg. UEFN CookWorker ). Though this fetaure is not enabled yet.
This will temporarily break Loading and Cooking analytics for Lyra, ShooterGame, CitySample but not Fortnite or UEFN. The remainder will need to be "fixed up" in UE5/Main once the changes have propogated (stakeholders have been notified)

[FYI] zousar.shaker, devin.doucette, eric.knapik, francis.hurteau
#rb wes.hunt
#preflight 643e45150a5a4b944e603be8

[CL 25083739 by mark lintott in ue5-main branch]
2023-04-18 07:16:36 -04:00
Devin Doucette
32c049c3f5 Removed some includes of Function.h in Core
#preflight 6435cf83127980724b0f16b9
#rnx

[CL 25006462 by Devin Doucette in ue5-main branch]
2023-04-12 10:05:50 -04:00
devin doucette
592c55ccc8 DDC: Removed deprecated code
#preflight 6410d4770e1f02786b69ac5a
#rb Zousar.Shaker
#rnx

[CL 24654027 by devin doucette in ue5-main branch]
2023-03-15 10:15:30 -04:00
charles bloom
5e488c3f89 LaunchTaskInThreadPool API change needs to be properly deprecated
add pass-through for old args

#rb devin.doucette
#preflight none

[CL 24263337 by charles bloom in ue5-main branch]
2023-02-16 13:36:05 -05:00
charles bloom
048e8bc0fd AssetCompilingManager track down and log about resources that don't set RequiredMemory
mark places that need fixing with [at]todo RequiredMemory
add DebugName to Task and Work to track down sources of the work items

#preflight https://horde.devtools.epicgames.com/job/63eabe61b96a6b2718336f8f
#rb danny.couture

[CL 24201094 by charles bloom in ue5-main branch]
2023-02-13 22:01:48 -05:00
Devin Doucette
82621298bf DDC: Added -ExportBuildOutput to include outputs in exported builds
#preflight 63d804445c69f453c111cfd0
#rb Zousar.Shaker

[CL 23917590 by Devin Doucette in ue5-main branch]
2023-01-30 17:44:07 -05:00
mark lintott
8bad2c0365 Removed the callback system from core Analytics system. Fixed up analytics gathering in DerivedDataCache, Virtualization and HttpCacheStore
#rb wes.hunt
#preflight 63be95bd68068a8bd6a4438e

[CL 23651011 by mark lintott in ue5-main branch]
2023-01-11 14:28:01 -05:00
Devin Doucette
9b5b232cca Texture: Integrated the derived data build jobs with the asset compiling manager in a two-phase approach
The texture build task queues with the texture compiling manager to block on concurrency limits, or when texture builds are paused. The build scheduler will queue with the asset compiling manager to reserve memory when the build needs to execute locally. With this approach, resuming one texture will complete one texture, even though multiple separate build jobs may be executed.

#preflight 6381820933774509009e6e7d
#rb Danny.Couture, Zousar.Shaker

[CL 23271848 by Devin Doucette in ue5-main branch]
2022-11-25 22:18:09 -05:00
Devin Doucette
110afa128b DDC: Added new build scheduler modular features to queue by type and reserve memory
#preflight 638128f3cc307d6fa5b4ee80
#rb Zousar.Shaker

[CL 23271781 by Devin Doucette in ue5-main branch]
2022-11-25 21:38:59 -05:00
Devin Doucette
e2483014be DDC: Added LaunchTaskInThreadPool as a replacement for the two existing thread pool task requests
#preflight 6380f7c4170bc34a93b9104b
#rb Zousar.Shaker

[CL 23267593 by Devin Doucette in ue5-main branch]
2022-11-25 12:35:29 -05:00
Devin Doucette
165d24ede5 DDC: Made the cache optional for the build system
#preflight 636c045ef56cab38c78f8b47
#rb Tim.Smith
#rnx

[CL 23062223 by Devin Doucette in ue5-main branch]
2022-11-09 15:53:21 -05:00
henrik karlsson
4feb09e0b3 Compile time optimizations Managed to reduce unnamed game's biggest file compiler frontend cost from 113 seconds to 60 seconds. This also impact other things such as pch size and memory footprint when compiling which is great.
Takeaways. Try to use "friend" on functions/operators that are frequently overloaded EXCEPT if they are in a templated type that is frequently instantiated. So do not put friends in TMap, TSet, TObjectPtr etc, this will slow down compile times. There is a break-even somewhere and hard to tell where it is but taking a class that is templatized on character type probably don't matter either way and then it is nicer to use a friend since that simplies error messages when compiler can't resolve functions/operators.

If it is possible to use member functions instead of friend that is the best option in terms of compile time performance. With c++20 you only have to write operator==(Foo, Bar) and the compiler will automatically provide operator==(Bar, Foo), operator!=(Foo, Bar) and operator!=(Bar, Foo).

Changes in this changelist involes
* Making operator<< friends in non-template types and not friends in template types
* Making operator==/!= members where possible and if not possible moved out if type is a frequently instantiated templated type.

#preflight 636970f5376a9cd6a80da54a
#rb steve.robb

[CL 23038965 by henrik karlsson in ue5-main branch]
2022-11-08 15:59:46 -05:00
Devin Doucette
6c33c94b36 DDC: Added experimental interface IBuildSchedulerThreadPoolProvider to integrate the build scheduler with the asset compiling managers
#jira UE-141137
#jira UE-141139
#preflight 63696f5e843e6ac79482dad1
#rb Zousar.Shaker

[CL 23027624 by Devin Doucette in ue5-main branch]
2022-11-08 09:45:50 -05:00
henrik karlsson
4a36cfe8ff Moved operator== to be hidden friend instead of put directly in global namespace
Moved GetTypeHash function to be hidden friend instead of put directly in global namespace.

Note that the function/operator needs to be fully inlined in the type or placed in the cpp. If the function is added as friend but then implemented outside the type then hidden friend optimization won't work.

This should improve compile time somewhat according to msvc devs.

#rb Steve.Robb
#preflight 6360b7052b5338aceb26471b

[CL 22889837 by henrik karlsson in ue5-main branch]
2022-11-01 15:50:27 -04:00
graeme thornton
e7986ac620 [Backout] - CL22808749
[FYI] Steve.Robb
Original CL Desc
-----------------------------------------------------------------
TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

[FYI] steve.robb
#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22850782 by graeme thornton in ue5-main branch]
2022-10-29 02:59:59 -04:00
steve robb
e980405a8f TMap and TSet can now be declared as members with forward-declared key and value parameters.
KeyFuncs::KeyInitType and KeyFuncs::ElementInitType typedefs are no longer used and user-defined KeyFuncs do not need to provide them.  Deprecated placeholders for these typedefs exist though they may not be defined exactly as they were before.  A new KeyType typedef needs to be provided by custom KeyFuncs which don't already inherit from BaseKeyFuncs or TDefaultMapKeyFuncs.
KeyConstPointerType, KeyInitType, ValueInitType and ElementInitType typedefs have been deprecated across TMap, TSet and TSortedMap.  Regular C++ parameter-passing semantics should be used instead (const T& Ref or T Value, depending on T).
Added missing FSetElementId::operator!=().

#rb james.hopkin
[FYI] henrik.karlsson
#preflight 635a56c15d49a96f7b31938f

[CL 22810695 by steve robb in ue5-main branch]
2022-10-27 11:20:32 -04:00
Devin Doucette
74ce7089b4 DDC: Fixed SharedString reference counting flagged by TSan
#preflight 633f1a48437603cb0edd72a4
#rb Zousar.Shaker
#rnx

[CL 22384667 by Devin Doucette in ue5-main branch]
2022-10-06 17:28:03 -04:00
Devin Doucette
7f98c7fe6c DDC: Added functions to convert between derived data priority and queued work priority
#preflight 632b42d4b4515b7e226ffae8
#rb Zousar.Shaker
#rnx

[CL 22118431 by Devin Doucette in ue5-main branch]
2022-09-21 14:25:32 -04:00
devin doucette
abd22903b7 Fixed forwarding of -DDC=<Graph> to avoid initializing DDC and only forward non-default graph names
#preflight 631f926c1e6887c4a6d4c30d
#rb Zousar.Shaker

[CL 21984731 by devin doucette in ue5-main branch]
2022-09-13 12:12:58 -04:00
devin doucette
35be596466 DDC: Reverted unnecessary changes made by IWYU
There are several cases where includes were added that are not directly used (only by reference or pointer), and where forward declarations were reordered in a way that worsens readability.

#rb none
#rnx

[CL 21499953 by devin doucette in ue5-main branch]
2022-08-22 21:23:18 -04:00