Commit Graph

352 Commits

Author SHA1 Message Date
Stefan Boberg
55c818a30b Remove prototype "ZenHttp" DDC client code - this was never meant to propagate to main
#rb zousar.shaker

[CL 17015794 by Stefan Boberg in ue5-main branch]
2021-08-02 10:24:04 -04:00
Devin Doucette
a1fb614f8a DDC: Moved new stats functions into FDerivedDataCacheInterface to remove module dependency
#rb Aurel.Cordonnier
#rnx

[CL 16997387 by Devin Doucette in ue5-main branch]
2021-07-29 13:05:41 -04:00
dmytro vovk
e0e8407cc0 Fixed static events initialization crash
#jira none
#rb Andriy.Tylychko

[CL 16995074 by dmytro vovk in ue5-main branch]
2021-07-29 09:55:50 -04:00
danny couture
59d6d0b089 Fix double Jupiter download caused by improper usage of async put underneath each backend in hierarchy
- Remove one copy of the data in the AsyncPut task when used in conjonction with an Inflight cache
  - Make sure each async put in the hierarchical backend can fullfill request from it's Inflight cache
  - Make sure to hit the asyncput backends for exists, trytoprefetch and get queries to avoid repetitive miss while the put is still in flight
  - Remove the code that tried to delete cache miss as it caused many deletes and redownload of the same asset, that duty is fulfilled by the corruption wrapper anyway
  - Fix race condition when mounting and unmounting Pak files or other modifications applied to the hierarchical backend

#rnx
#rb Devin.Doucette, Francis.Hurteau
#preflight 61009a8d1e63730001d339a2

[CL 16977632 by danny couture in ue5-main branch]
2021-07-27 21:19:59 -04:00
Francis Hurteau
79415aabf8 Refix crash in http request batch when the batch contains both a head and get request for the same cache keys
Added a test that could reliably reproduce the problem to validate it is properly fixed

#rb Danny.Couture, Zousar.Shaker
#preflight 610010b247c93a0001335319


#ROBOMERGE-OWNER: Francis.Hurteau
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 16968978
#ROBOMERGE-BOT: (v838-16927207)
#ROBOMERGE-CONFLICT from-shelf

[CL 16972104 by Francis Hurteau in ue5-main branch]
2021-07-27 15:24:32 -04:00
danny couture
f283881ae0 Fix thread starvation for request in http backend by using a FIFO for thread waiting for a connection to be freed
Replace the sleep by a wait on an event triggered by the thread freeing the connection
Improve the test to output stats seen below and use the proper amount of threads on low core systems

Automation RunTests System.DerivedDataCache.HttpDerivedDataBackend.ConcurrentCachedDataProbablyExistsBatch

Before
   RPS: 238, AvgLatency: 90.09 ms, MaxLatency: 7.77 s
   RPS: 231, AvgLatency: 87.66 ms, MaxLatency: 7.77 s
   RPS: 245, AvgLatency: 139.07 ms, MaxLatency: 10.08 s

After
   RPS: 634, AvgLatency: 102.97 ms, MaxLatency: 0.26 s
   RPS: 653, AvgLatency: 102.51 ms, MaxLatency: 0.26 s
   RPS: 630, AvgLatency: 102.29 ms, MaxLatency: 0.26 s

#rnx
#rb Zousar.Shaker, Francis.Hurteau
#preflight 60ffe8317f21c90001f6116a

[CL 16968010 by danny couture in ue5-main branch]
2021-07-27 09:25:29 -04:00
Devin Doucette
89c7e45694 DDC: Compress payloads using Oodle Mermaid VeryFast
#rb Zousar.Shaker
#rnx

[CL 16956293 by Devin Doucette in ue5-main branch]
2021-07-26 10:28:55 -04:00
Simon Tourangeau
f723c252dc Fix DDCUtils build error in engine
#fyi zousar.shaker
#rb none

[CL 16951592 by Simon Tourangeau in ue5-main branch]
2021-07-25 14:09:32 -04:00
francis hurteau
ed059fdb34 Undo changelist 16938823 until investigation of cook failures are done
[FYI] Zousar.Shaker
#rb trivial
#preflight skip
discussed with Zoursar.Shaker, undo to fix cook failure


#ROBOMERGE-SOURCE: CL 16941401
#ROBOMERGE-BOT: (v838-16927207)

[CL 16941404 by francis hurteau in ue5-main branch]
2021-07-23 16:16:08 -04:00
geoff evans
89385463ec Add trace counter increments for more cache interaction prototypes in FileSystemDDC.
This fixes bytes read not matching the reported get hit count, and likely the same for put hits.

#jira none
#rb francis.hurteau
#preflight 60fad90ee885f200013f6939
#robomerge Private-Frosty

[CL 16939581 by geoff evans in ue5-main branch]
2021-07-23 14:20:45 -04:00
francis hurteau
c7436d7678 Fix crash in http request batch when the batch contains both a head and get request for the same cache keys
Added a test that could reliably reproduce the problem to validate it is properly fixed
#preflight 60faeafc1f926d000113425b


#ROBOMERGE-SOURCE: CL 16938823
#ROBOMERGE-BOT: (v838-16927207)

[CL 16938832 by francis hurteau in ue5-main branch]
2021-07-23 13:14:33 -04:00
danny couture
c2a09f533e Add a DDC throttling layer that can be used to approximate Jupiter performance locally by adding latency and limiting bandwidth
This is useful for benchmarking as it reduce noise caused by variable Internet performance

#rnx
#rb Devin.Doucette
#preflight 60fa02ea1f926d0001f9f9b6

[CL 16934015 by danny couture in ue5-main branch]
2021-07-22 22:55:55 -04:00
mark lintott
d790fa8c84 Updated DDC Widget
Added detailed resource and cache backend information.
Added new indicators for Busy, Upload, Download, LocalCache and RemoteCache
Minor modifications to StudioAnalytics to re-use the stats gathering code
#rb francis.hurteau
[FYI] jeff.farris


#ROBOMERGE-SOURCE: CL 16929122
#ROBOMERGE-BOT: (v838-16927207)

[CL 16929180 by mark lintott in ue5-main branch]
2021-07-22 15:09:44 -04:00
sebastien lussier
e16a51bf54 DDC backends - removed speed class tests in implementations of TryToPrefetch()
#rb danny.couture, devin.doucette


#ROBOMERGE-SOURCE: CL 16900182
#ROBOMERGE-BOT: (v836-16769935)

[CL 16917763 by sebastien lussier in ue5-main branch]
2021-07-21 19:08:29 -04:00
Devin Doucette
bea14d56ce DDC: Changed TRequest to cast to IRequest before invoking its functions
This allows the request type to make its functions on IRequest protected or private, or allows use with a request type that hides its IRequest behind an implicit conversion.

Also disabled operator-> and operator* for the base IRequest because it is preferred to use the functions on TRequest to the IRequest interface.

#rb Matt.Peters
#rnx

[CL 16911217 by Devin Doucette in ue5-main branch]
2021-07-21 12:42:51 -04:00
danny couture
989c3cb117 Fix behavior of CachedDataProbablyExists so it can reach the slow cache to avoid always recompiling stuff that exists in Jupiter but not locally
#rnx
#rb Devin.Doucette, Sebastien.Lussier

[CL 16893978 by danny couture in ue5-main branch]
2021-07-20 05:56:37 -04:00
Zousar Shaker
1571c9f36e Fix linux compile error and bug with my last submits related to function version iteration for build functions.
#rb none

[CL 16889472 by Zousar Shaker in ue5-main branch]
2021-07-19 17:40:45 -04:00
Devin Doucette
ca1582337c DDC: Switched to XXH3 for FPayloadId::FromName
#rb Zousar.Shaker
#rnx

[CL 16889028 by Devin Doucette in ue5-main branch]
2021-07-19 17:05:40 -04:00
Zousar Shaker
3f35355cc1 Add methods for iterating through build functions in the function registry.
#rb devin.doucette

[CL 16888770 by Zousar Shaker in ue5-main branch]
2021-07-19 16:47:17 -04:00
Devin Doucette
0748494fc6 DDC: Exposed the cache key and build status in the build complete callback params
#rb Zousar.Shaker
#rnx
#preflight 60f5c7a78da1560001c68086

[CL 16888763 by Devin Doucette in ue5-main branch]
2021-07-19 16:46:58 -04:00
Zousar Shaker
bd3886b31f Fix bug where queued batch request failure in HttpDerivedDataBackend would cause incorrect DDC output (erroneous cache misses).
Adding basic unit test for HttpDerivedDataBackend.

#rb devin.doucette

[CL 16885295 by Zousar Shaker in ue5-main branch]
2021-07-19 13:09:40 -04:00
Devin Doucette
672da1da55 DDC: Fixed a crash when creating the build context fails
#rb Zousar.Shaker
#rnx

[CL 16883576 by Devin Doucette in ue5-main branch]
2021-07-19 10:51:21 -04:00
Devin Doucette
431dab26ea DDC: Added EPutStatus::Skipped to avoid slow retry when simulating cache misses
#rb Zousar.Shaker
#preflight 60f12443a34bf50001654df1

[CL 16883247 by Devin Doucette in ue5-main branch]
2021-07-19 10:11:49 -04:00
Devin Doucette
090bdb178e DDC: Fixed a missing include in DerivedDataBuildVersion.h
#rb trivial
#rnx

[CL 16883233 by Devin Doucette in ue5-main branch]
2021-07-19 10:10:26 -04:00
Devin Doucette
c9ef86c7d2 DDC: Schedule non-blocking builds using tasks
#rb Zousar.Shaker
#rnx

[CL 16870688 by Devin Doucette in ue5-main branch]
2021-07-16 01:23:29 -04:00