Commit Graph

742 Commits

Author SHA1 Message Date
zousar shaker
d78e86696f Exclude Zen from HttpCacheStore test on non-windows platforms as it is not ready for use on non-windows platforms.
#rb trivial
[FYI] devin.doucette
[FYI] matt.peters

[CL 23658577 by zousar shaker in ue5-main branch]
2023-01-11 20:57:41 -05:00
zousar shaker
bfbf54d4ca ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold).
#rb devin.doucette

[CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
mark lintott
8bad2c0365 Removed the callback system from core Analytics system. Fixed up analytics gathering in DerivedDataCache, Virtualization and HttpCacheStore
#rb wes.hunt
#preflight 63be95bd68068a8bd6a4438e

[CL 23651011 by mark lintott in ue5-main branch]
2023-01-11 14:28:01 -05:00
Devin Doucette
d081f5a477 DDC: Fixed the file system cache EditorOverrideSetting not handling paths with \ in them
#jira none
#preflight 63b78d3ad862fdd347ea642b
#rb Zousar.Shaker

[CL 23595626 by Devin Doucette in ue5-main branch]
2023-01-05 22:10:29 -05:00
zousar shaker
d1afea0f1f Allow ReadOnly mode for a DDC FileSystemCacheStore to have maintenance performed if Clean=true or DeleteUnused=true. When ReadOnly, they are now both false by default, but usable if explicitly set to true. When not ReadOnly, DeleteUnused remains true by default.
#rb devin.doucette

[CL 23592522 by zousar shaker in ue5-main branch]
2023-01-05 16:11:30 -05:00
Matt Peters
6f0b8a2906 HttpCacheStore: Fix backwards if statement for whether the compactbinary is valid in GetBodyAsString
#preflight 63b70052221aab21e01200c0

[CL 23589159 by Matt Peters in ue5-main branch]
2023-01-05 12:01:37 -05:00
matt peters
71802fb86c Analytics: Backed out 22909100 and 22951911
#rnx

[CL 23583550 by matt peters in ue5-main branch]
2023-01-04 18:51:41 -05:00
Zousar Shaker
624d3b7dbf Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default.
#rb devin.doucette
#preflight 63b493d5202bee5e27a645ed

[CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
Zousar Shaker
e27b568f2e Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine.
#rb devin.doucette
#preflight 639cc26343330e63e5db1750

[CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Zousar Shaker
e7fb0b1751 Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store.
#rb devin.doucette
#preflight 6398ea4e680483bcb1057b0a

[CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
Zousar Shaker
b9548ddad4 Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status.
Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes.  No functional change, purely a move, search+replace, and sort of includes.

#rb matt.peters
#preflight 63939eebc415e8dc78d35d63

[CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
Zousar Shaker
54d9057532 Fixes for UE Cloud DDC client:
- Print CbObject responses to the log/console as JSON, not raw binary
- Reduce the thresholds for connect timeout and slow transfer rate time window
- Instead of one request pool of size 128*2 requests, split GET and PUT operations into their own limited request pools and have the PUT request pool (size 20) be set smaller than the GET one (size 128)
- Stop using unbounded overflow for chained sequences of HTTP requests to avoid slow/failed requests due to many upload operations happening concurrently during high upload quantity scenarios (eg: shader version bump, editor domain version bump)

Testing:
- Cook of CitySample with shader version bumped to ensure there are no failed requests due to slow transfer
- Launch of CitySample editor with no local cache to observe throughput when all DDC data has to be fetched from upstream

#rb devin.doucette
#preflight 638c2d86cb3b754183f2e574

[CL 23385459 by Zousar Shaker in ue5-main branch]
2022-12-04 02:26:39 -05:00
Andriy Tylychko
871aabab2e renamed TDepletableMpscQueue to TDepletableMpmcQueue as it supports multiple concurrent consumers after the last change.
Deprecated the old name.
Updated existing use cases to use the new name or to switch to `TConsumeAllMpmcQueue` if the order of consumption doesn't matter.
Added `IsEmpty()` method to `TConsumeAllMpmcQueue`
#rb devin.doucette, arne.schober
#preflight 6389f53f9242c013261fca6b

[CL 23371804 by Andriy Tylychko in ue5-main branch]
2022-12-02 08:47:16 -05:00
Devin Doucette
9b5b232cca Texture: Integrated the derived data build jobs with the asset compiling manager in a two-phase approach
The texture build task queues with the texture compiling manager to block on concurrency limits, or when texture builds are paused. The build scheduler will queue with the asset compiling manager to reserve memory when the build needs to execute locally. With this approach, resuming one texture will complete one texture, even though multiple separate build jobs may be executed.

#preflight 6381820933774509009e6e7d
#rb Danny.Couture, Zousar.Shaker

[CL 23271848 by Devin Doucette in ue5-main branch]
2022-11-25 22:18:09 -05:00
Devin Doucette
110afa128b DDC: Added new build scheduler modular features to queue by type and reserve memory
#preflight 638128f3cc307d6fa5b4ee80
#rb Zousar.Shaker

[CL 23271781 by Devin Doucette in ue5-main branch]
2022-11-25 21:38:59 -05:00
Devin Doucette
e2483014be DDC: Added LaunchTaskInThreadPool as a replacement for the two existing thread pool task requests
#preflight 6380f7c4170bc34a93b9104b
#rb Zousar.Shaker

[CL 23267593 by Devin Doucette in ue5-main branch]
2022-11-25 12:35:29 -05:00
Devin Doucette
d53ba8513d DDC: Allow the build system to be created without a cache
#preflight 637fa7e518176c67c20951a6
#rb Zousar.Shaker

[CL 23262694 by Devin Doucette in ue5-main branch]
2022-11-24 17:23:49 -05:00
Devin Doucette
d508667187 DDC: Added a missing read lock in the cache hierarchy
#preflight 63751a331d25fe8b93371452
#rb Matt.Peters
#rnx

[CL 23161581 by Devin Doucette in ue5-main branch]
2022-11-16 15:43:58 -05:00
Matt Peters
dd0f970188 HttpCacheStore: When a record is missing data and we return EStatus::Error and an empty record, fix the FCacheRecord passed into OnComplete to not have metadata or any values. The FCacheStoreHierarchy client of FHttpCacheStore assumes that metadata or values are only present on a successful record, and if it is present, incorrectly Puts the record as a valid record into the other cachestores in the hierarchy. This then breaks clients of DDC, because the record is invalid since it is missing some values.
This change fixes the warning messages when using EditorDomain: LogEditorDomain: Warning: Package %s received an invalid CacheRecord...
#rb Zousar.Shaker, Devin.Doucette
#rnx
#preflight 63753d95953c19d4351d8f11

[CL 23160500 by Matt Peters in ue5-main branch]
2022-11-16 15:12:51 -05:00
Devin Doucette
dcf192e97a DDC: Remove any trailing slash from the Jupiter URI
#preflight 636d11ed0f51d9ed4afbf185
#rb Joakim.Lindqvist
#rnx

[CL 23079894 by Devin Doucette in ue5-main branch]
2022-11-10 10:57:37 -05:00
Devin Doucette
165d24ede5 DDC: Made the cache optional for the build system
#preflight 636c045ef56cab38c78f8b47
#rb Tim.Smith
#rnx

[CL 23062223 by Devin Doucette in ue5-main branch]
2022-11-09 15:53:21 -05:00
Devin Doucette
381a587dcd DDC: Invoke the build completion callback after storing to the cache
This avoids a race condition where a build completes and then another process is unable to retrieve the data from the cache.

#preflight 636b2a547c2b505190e7d036
#rb Zousar.Shaker
#rnx

[CL 23059312 by Devin Doucette in ue5-main branch]
2022-11-09 13:57:59 -05:00
Zousar Shaker
ac3fc5db61 Print more detailed timing metrics for HTTP operations in the HttpCacheStore.
#rb devin.doucette
#preflight 636acc25dc30a4ce96bbe43c

[CL 23040720 by Zousar Shaker in ue5-main branch]
2022-11-08 16:52:32 -05:00
henrik karlsson
4feb09e0b3 Compile time optimizations Managed to reduce unnamed game's biggest file compiler frontend cost from 113 seconds to 60 seconds. This also impact other things such as pch size and memory footprint when compiling which is great.
Takeaways. Try to use "friend" on functions/operators that are frequently overloaded EXCEPT if they are in a templated type that is frequently instantiated. So do not put friends in TMap, TSet, TObjectPtr etc, this will slow down compile times. There is a break-even somewhere and hard to tell where it is but taking a class that is templatized on character type probably don't matter either way and then it is nicer to use a friend since that simplies error messages when compiler can't resolve functions/operators.

If it is possible to use member functions instead of friend that is the best option in terms of compile time performance. With c++20 you only have to write operator==(Foo, Bar) and the compiler will automatically provide operator==(Bar, Foo), operator!=(Foo, Bar) and operator!=(Bar, Foo).

Changes in this changelist involes
* Making operator<< friends in non-template types and not friends in template types
* Making operator==/!= members where possible and if not possible moved out if type is a frequently instantiated templated type.

#preflight 636970f5376a9cd6a80da54a
#rb steve.robb

[CL 23038965 by henrik karlsson in ue5-main branch]
2022-11-08 15:59:46 -05:00
Devin Doucette
6c33c94b36 DDC: Added experimental interface IBuildSchedulerThreadPoolProvider to integrate the build scheduler with the asset compiling managers
#jira UE-141137
#jira UE-141139
#preflight 63696f5e843e6ac79482dad1
#rb Zousar.Shaker

[CL 23027624 by Devin Doucette in ue5-main branch]
2022-11-08 09:45:50 -05:00