Commit Graph

369 Commits

Author SHA1 Message Date
Devin Doucette
5bdd0e67b3 DDC: Add FBuildConfigContext::SetRequiredMemory() and forward the estimate to the build scheduler
#rb Zousar.Shaker
#rnx

[CL 17088443 by Devin Doucette in ue5-main branch]
2021-08-06 17:32:53 -04:00
Devin Doucette
2d827a0a7a DDC: Added flags to control request barrier behavior
#rb Matt.Peters
#rnx

[CL 17088062 by Devin Doucette in ue5-main branch]
2021-08-06 17:11:24 -04:00
Devin Doucette
30760f545b DDC: Split DerivedDataRequest.h and rename DerivedDataBuildPolicy.h
#rb none
#rnx

[CL 17086960 by Devin Doucette in ue5-main branch]
2021-08-06 15:49:14 -04:00
Devin Doucette
48d8d5f7eb DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

[CL 17084521 by Devin Doucette in ue5-main branch]
2021-08-06 12:51:28 -04:00
Devin Doucette
c4e3574345 DDC: Removed IBuild::Load[Type] in favor of exported functions
#rb Zousar.Shaker
#rnx

[CL 17083958 by Devin Doucette in ue5-main branch]
2021-08-06 11:57:55 -04:00
Devin Doucette
7f0835f571 DDC: Fixed non-unity compile errors
#rb trivial
#rnx

[CL 17079642 by Devin Doucette in ue5-main branch]
2021-08-05 22:29:19 -04:00
Devin Doucette
0167604e91 DDC: Rename FRequestGroup to FRequestOwner and export its constructor
#rb Zousar.Shaker
#rnx

[CL 17074555 by Devin Doucette in ue5-main branch]
2021-08-05 15:45:41 -04:00
Devin Doucette
2d9736cdea DDC: Added conversions between FPayloadId and FCbObjectId
#rb Zousar.Shaker
#rnx

[CL 17071877 by Devin Doucette in ue5-main branch]
2021-08-05 13:36:44 -04:00
Devin Doucette
87792e1b5e DDC: Split the new API from the old API
The new API uses exported functions and cannot be included with the old API without compile errors in existing code that has an include-only dependency on DDC.

#rb Zousar.Shaker
#rnx
#preflight 610c01e3aeb05700011dc5ab

[CL 17071263 by Devin Doucette in ue5-main branch]
2021-08-05 13:11:50 -04:00
Stefan Boberg
a4e1247a67 Another NonUnity build fix (ZenServerHttp.h)
#rb fbcli

[CL 17068679 by Stefan Boberg in ue5-main branch]
2021-08-05 11:23:25 -04:00
Stefan Boberg
7192f2be8a Zen HTTP interface refactored into separate header/source files
#rb none

[CL 17066037 by Stefan Boberg in ue5-main branch]
2021-08-05 05:30:53 -04:00
Stefan Boberg
d20e67e203 More defensive #if placement, to address EngineTest build issue
#rb none

[CL 17064898 by Stefan Boberg in ue5-main branch]
2021-08-05 02:26:16 -04:00
Devin Doucette
3513768988 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

[CL 17060470 by Devin Doucette in ue5-main branch]
2021-08-04 17:52:40 -04:00
aurel cordonnier
dc856801cd Merge from Release-Engine-Test @ 17059716 to UE5/Main
This represents UE4/Main @ 17030256 and Dev-PerfTest @ 17029914

[CL 17060422 by aurel cordonnier in ue5-main branch]
2021-08-04 17:46:20 -04:00
Stefan Boberg
970aad0412 Refactored ZenDDC communication. HTTP interface code is now simplified and separated out from the ZenDDC client code as a step towards a shared communication layer for the oplog/cache/etc as well as instance lifetime management. There should be no functional differences.
#rb devin.doucette
#jira none

[CL 17056895 by Stefan Boberg in ue5-main branch]
2021-08-04 14:55:30 -04:00
Stefan Boberg
4656bf2930 General HTTP zen client implementation simplification
* Rename FRequest -> FZenHttpRequest to make it easier to move the implementation elsewhere
* Simplified implementation of PUT/HEAD/DELETE operations by turning template functions into plain member functions to make it easier to hide the implementation details when the class is moved into a header

#rb none
#jira none

[CL 17048397 by Stefan Boberg in ue5-main branch]
2021-08-04 08:08:03 -04:00
Steve Robb
203cacf399 FNames can now be constructed with UTF8CHAR strings.
UTF-8 support added to FString::AppendChars() and made able to correctly handle variably-encoded source strings (both wider and narrower than TCHAR).
FUTF8ToTCHAR now accepts either UTF8CHAR* or ANSICHAR* for better compatibility.
FUtf8StringView now binds to UTF8CHAR* or any UTF8CHAR range, and can be constructed via a _U8SV string literal suffix.
CompactBinary now supports UTF-8.
Miscellaneous other UTF8CHAR fixes.

#rb devin.doucette

[CL 17017066 by Steve Robb in ue5-main branch]
2021-08-02 12:15:16 -04:00
Stefan Boberg
55c818a30b Remove prototype "ZenHttp" DDC client code - this was never meant to propagate to main
#rb zousar.shaker

[CL 17015794 by Stefan Boberg in ue5-main branch]
2021-08-02 10:24:04 -04:00
Devin Doucette
a1fb614f8a DDC: Moved new stats functions into FDerivedDataCacheInterface to remove module dependency
#rb Aurel.Cordonnier
#rnx

[CL 16997387 by Devin Doucette in ue5-main branch]
2021-07-29 13:05:41 -04:00
dmytro vovk
e0e8407cc0 Fixed static events initialization crash
#jira none
#rb Andriy.Tylychko

[CL 16995074 by dmytro vovk in ue5-main branch]
2021-07-29 09:55:50 -04:00
danny couture
59d6d0b089 Fix double Jupiter download caused by improper usage of async put underneath each backend in hierarchy
- Remove one copy of the data in the AsyncPut task when used in conjonction with an Inflight cache
  - Make sure each async put in the hierarchical backend can fullfill request from it's Inflight cache
  - Make sure to hit the asyncput backends for exists, trytoprefetch and get queries to avoid repetitive miss while the put is still in flight
  - Remove the code that tried to delete cache miss as it caused many deletes and redownload of the same asset, that duty is fulfilled by the corruption wrapper anyway
  - Fix race condition when mounting and unmounting Pak files or other modifications applied to the hierarchical backend

#rnx
#rb Devin.Doucette, Francis.Hurteau
#preflight 61009a8d1e63730001d339a2

[CL 16977632 by danny couture in ue5-main branch]
2021-07-27 21:19:59 -04:00
Francis Hurteau
79415aabf8 Refix crash in http request batch when the batch contains both a head and get request for the same cache keys
Added a test that could reliably reproduce the problem to validate it is properly fixed

#rb Danny.Couture, Zousar.Shaker
#preflight 610010b247c93a0001335319


#ROBOMERGE-OWNER: Francis.Hurteau
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 16968978
#ROBOMERGE-BOT: (v838-16927207)
#ROBOMERGE-CONFLICT from-shelf

[CL 16972104 by Francis Hurteau in ue5-main branch]
2021-07-27 15:24:32 -04:00
danny couture
f283881ae0 Fix thread starvation for request in http backend by using a FIFO for thread waiting for a connection to be freed
Replace the sleep by a wait on an event triggered by the thread freeing the connection
Improve the test to output stats seen below and use the proper amount of threads on low core systems

Automation RunTests System.DerivedDataCache.HttpDerivedDataBackend.ConcurrentCachedDataProbablyExistsBatch

Before
   RPS: 238, AvgLatency: 90.09 ms, MaxLatency: 7.77 s
   RPS: 231, AvgLatency: 87.66 ms, MaxLatency: 7.77 s
   RPS: 245, AvgLatency: 139.07 ms, MaxLatency: 10.08 s

After
   RPS: 634, AvgLatency: 102.97 ms, MaxLatency: 0.26 s
   RPS: 653, AvgLatency: 102.51 ms, MaxLatency: 0.26 s
   RPS: 630, AvgLatency: 102.29 ms, MaxLatency: 0.26 s

#rnx
#rb Zousar.Shaker, Francis.Hurteau
#preflight 60ffe8317f21c90001f6116a

[CL 16968010 by danny couture in ue5-main branch]
2021-07-27 09:25:29 -04:00
Devin Doucette
89c7e45694 DDC: Compress payloads using Oodle Mermaid VeryFast
#rb Zousar.Shaker
#rnx

[CL 16956293 by Devin Doucette in ue5-main branch]
2021-07-26 10:28:55 -04:00
Simon Tourangeau
f723c252dc Fix DDCUtils build error in engine
#fyi zousar.shaker
#rb none

[CL 16951592 by Simon Tourangeau in ue5-main branch]
2021-07-25 14:09:32 -04:00