Commit Graph

36 Commits

Author SHA1 Message Date
mark lintott
b99d530ce0 Moved EpicStudioAnalytics plugin to StudioTelemetry which is now public facing.
The intention is to provide a set of "in the box" telemetry hooks to our most common developer workflows for use internal Epic use and licensees
Moved startup of StudioTelemetry plugin much earlier to PostConfigInit stage so that sessions can start much earlier in the workflow
Added WIP Client support to telemetry plugin
Added data driven provider support for FAnlayiticProvidersET interfaces via BaseEngine.ini for common games and sample projects. Settings and URLs for EPic are only avaliable in Resttricted/NotForLicensee config folders for Lyra and Shooter game.
Added support for Horde telemetry and fixed up various problems with duplicate attributes being sent.
Added Core.VirtualAssets telemetry event
Added Core.Zen telemetry event
Added Core.IAS telemetry event
Added check for IOStoreOnDemand IsEnabled to avoid sending empty IAS events
Added FAnalyticsMulticastProvider to forward telemetry events to multiple providers contained within
Removed deprecated Fire_LoadingEvent from StudioAnalytics
[FYI] paul.chipchase
#rb Wes.Hunt

[CL 29908039 by mark lintott in ue5-main branch]
2023-11-23 07:06:10 -05:00
devin doucette
592c55ccc8 DDC: Removed deprecated code
#preflight 6410d4770e1f02786b69ac5a
#rb Zousar.Shaker
#rnx

[CL 24654027 by devin doucette in ue5-main branch]
2023-03-15 10:15:30 -04:00
mark lintott
8bad2c0365 Removed the callback system from core Analytics system. Fixed up analytics gathering in DerivedDataCache, Virtualization and HttpCacheStore
#rb wes.hunt
#preflight 63be95bd68068a8bd6a4438e

[CL 23651011 by mark lintott in ue5-main branch]
2023-01-11 14:28:01 -05:00
steve robb
0edc9ec830 Some missing TEXT macros added.
#rb trivial
#jira UE-132142
#preflight none

#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20671620 via CL 20671646 via CL 20671659
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20675152 by steve robb in ue5-main branch]
2022-06-15 15:09:53 -04:00
Johan Torp
175fbdc61b Optimized BuildCacheKey using FAsciiSet for ~4x speedup
Estimated to save 45s in a 1h50m cook of a larger internal project

#rb devin.doucette
#preflight 6238648a10251d53d55d3d12

[CL 19450563 by Johan Torp in ue5-main branch]
2022-03-21 07:56:39 -04:00
devin doucette
70d0aea9a6 DDC: Added support for 64-bit sizes in the legacy cache
Includes an optimization to avoid an extra copy of data loaded from the cache.

#jira UE-103039
#preflight 6205f379b84973a2bb76d942
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18954059 in //UE5/Release-5.0/... via CL 18954686 via CL 18955089
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)

[CL 18955195 by devin doucette in ue5-main branch]
2022-02-11 12:56:47 -05:00
charles bloom
d6c8d22b81 Fix DDC should not sanitize $ characters because it makes $ characters
SanitizeCacheKey and ValidateCacheKey should check the same condition

#rb devin.doucette, danny.couture

#ROBOMERGE-AUTHOR: charles.bloom
#ROBOMERGE-SOURCE: CL 18455963 in //UE5/Release-5.0/... via CL 18455975
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v898-18417669)

[CL 18455987 by charles bloom in ue5-release-engine-test branch]
2021-12-14 12:35:14 -05:00
devin doucette
53f316587b DDC: New filesystem cache store directory layout and file format
- Implemented only for the structured cache initially.
- Records are stored as compact binary packages with payloads stored within the record if small enough.
- Records are stored in Records/{Bucket}/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash in the cache key.
- Content that exceeds the size allowed for storage within the record is stored in Content/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash of the content.
- Both records and content are written to disk with their hash as a suffix, and are always checked on load.
- Maximum size of compressed data stored in the package can be configured by MaxRecordSizeKB and MaxValueSizeKB.
- Maintenance of the cache, to delete unused files, has been rewritten and can now scan ~175k files/second on modern hardware, though is limited to 10k files/second in the config to avoid adding too much overhead.

#rb Zousar.Shaker
#preflight 618b75c14a0c609a29204032
#preflight 618bfb5ec717ba4e929b7ac0

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18133085 in //UE5/Release-5.0/... via CL 18133356
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 18133430 by devin doucette in ue5-release-engine-test branch]
2021-11-10 13:08:00 -05:00
devin doucette
d50e330b94 DDC: Split the new API from the old API
The new API uses exported functions and cannot be included with the old API without compile errors in existing code that has an include-only dependency on DDC.

#rb Zousar.Shaker
#rnx
#preflight 610c01e3aeb05700011dc5ab

#ROBOMERGE-SOURCE: CL 17071263 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071271 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:12:08 -04:00
devin doucette
77b340efea DDC: Moved new stats functions into FDerivedDataCacheInterface to remove module dependency
#rb Aurel.Cordonnier
#rnx

#ROBOMERGE-SOURCE: CL 16997387 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16997407 by devin doucette in ue5-release-engine-test branch]
2021-07-29 13:07:00 -04:00
aurel cordonnier
02c0f425e8 Copy up from Release-Engine-Staging @ 16738359
This represents UE4/Main @ 16738161 and Dev-PerfTest @ 16737719

[CL 16738582 by aurel cordonnier in ue5-release-engine-test branch]
2021-06-22 00:27:54 -04:00
Devin Doucette
e9c43a79ae DDC: Implemented ICache on FDerivedDataCacheInterface
Moved IDerivedDataCacheModule to its own header and renamed CreateOrGetDDC to CreateOrGetCache.

#rb Zousar.Shaker
#rnx
#preflight 6089d5a036afba00015c0348

[CL 16153121 by Devin Doucette in ue5-main branch]
2021-04-28 23:25:28 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Devin Doucette
7d5d9f3798 DDC: Clean up a few details in the new Cache API and remove the last exported type from DDC
#rb Zousar.Shaker
#rnx

[CL 15670275 by Devin Doucette in ue5-main branch]
2021-03-10 17:33:38 -04:00
devin doucette
0f5201b8a1 DDC: Added CreateOrGetDDC to the module and changed creation of DDC to occur there instead of on module startup
#rb Arciel.Rekman, Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 15530385 via CL 15530408 via CL 15530432 via CL 15530444 via CL 15531592
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Main) (v771-15082668)
#ROBOMERGE[STARSHIP]: Release-5.0-EarlyAccess

[CL 15531598 by devin doucette in ue5-main branch]
2021-02-25 14:50:04 -04:00
Devin Doucette
6394ccc1cd DDC: Added TryToPrefetch to check multiple keys for existence and prefetch the data into a fast cache backend
#rb Zousar.Shaker

[CL 15422719 by Devin Doucette in ue5-main branch]
2021-02-16 17:24:48 -04:00
Devin Doucette
7bd5b197e8 DDC: Added the new cache interface UE::DerivedData::ICache
ICache will eventually replace the existing cache access functions on FDerivedDataCacheInterface. The notable differences relative to the existing interface are:

- It is asynchronous with completion callbacks by default, and allows waiting when necessary.
- It allows batch requests containing multiple cache records by default.
- It uses a structured cache key made up of a bucket name and a hash.
- It supports prioritization and cancellation of requests.
- It supports structured cache records with metadata.
- It supports attachments to allow multi-part cache records.
- It uses a typed request object instead of an opaque integer handle.
- It provides cache policy flags to control how to query and store cache records, and which parts of a cache record to fetch.

This initial implementation of ICache is totally synchronous and uses the existing cache backends for storage.

#rb Zousar.Shaker

[CL 15152247 by Devin Doucette in ue5-main branch]
2021-01-21 01:57:01 -04:00
Devin Doucette
e6401b0897 DDC: Added variants of CachedDataProbablyExists that check multiple keys
#rb Matt.Peters
#fyi Joakim.Lindqvist, Stefan.Boberg

[CL 14998596 by Devin Doucette in ue5-main branch]
2021-01-06 09:27:11 -04:00
Marcus Wassmer
3b81cf8201 Merging using //UE5/Main_to_//UE5/Release-Engine-Staging @14384769
autoresolved files
#rb none

[CL 14384911 by Marcus Wassmer in ue5-main branch]
2020-09-24 00:43:27 -04:00
Devin Doucette
e8ba41d54f Changed FStringView parameters to virtual functions to pass by value instead of by reference
This is the preferred way to use string views. These interfaces were added in 4.25 so this is the time to fix them without breaking user code.

#jira none
#rb trivial
#rnx

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 12236111 in //UE4/Release-4.25/... via CL 12236115
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v664-12234175)

[CL 12238549 by Devin Doucette in Main branch]
2020-03-17 22:32:03 -04:00
nick darnell
7fb5ae96d7 Changing the DDC interface to use TArrayView instead of TArray for placing data into the DDC.
Updating the ArrayView with new comparison operators to have greater replacement for what TArray provides.

#rb Marc.Audy, Steve.Robb


#ROBOMERGE-OWNER: nick.darnell
#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-SOURCE: CL 11843381 via CL 11843463 via CL 11843492
#ROBOMERGE-BOT: (v656-11643781)

[CL 11879995 by nick darnell in Main branch]
2020-03-03 14:33:45 -05:00
Devin Doucette
070f44a07a Add debug context to DDC requests that take the key as a parameter
The existing GetSynchronous, GetAsynchronous, and Put functions are deprecated in favor of the DebugContext overloads. It is unlikely that a licensee has derived from FDerivedDataCacheInterface, but anyone who does will have a compile error that they override a non-virtual function and that they don't implement a pure virtual function, which will be straightforward to fix.

#rb Zousar.Shaker

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 11210386 via CL 11210387
#ROBOMERGE-BOT: (v643-11205221)

[CL 11210609 by Devin Doucette in Main branch]
2020-02-03 17:17:56 -05:00
ryan durand
471d972e62 Updating copyright for Engine Developer.
#rnx
#rb none


#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)

[CL 10870584 by ryan durand in Main branch]
2019-12-26 15:32:37 -05:00
Stefan Boberg
d2f9a61b06 Copy-up from Dev-Core
#rb none

[CL 10419044 by Stefan Boberg in Main branch]
2019-11-25 12:03:09 -05:00
Ben Marsh
7598af0532 Update copyright notices to 2019.
#rb none
#lockdown Nick.Penwarden

[CL 4662404 by Ben Marsh in Main branch]
2018-12-14 13:41:00 -05:00