Commit Graph

371 Commits

Author SHA1 Message Date
devin doucette
d5a632673b DDC: Fixed a build job assert caused by canceling at the wrong time
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17108375 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17108399 by devin doucette in ue5-release-engine-test branch]
2021-08-09 17:56:38 -04:00
charles bloom
b9f505a152 Log DerivedDataCache when registry overrides dirs
make editor dialog update when string is empty

#rb devin.doucette

#ROBOMERGE-SOURCE: CL 17107690 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17107751 by charles bloom in ue5-release-engine-test branch]
2021-08-09 17:06:13 -04:00
devin doucette
d55593526e DDC: Add FBuildConfigContext::SetRequiredMemory() and forward the estimate to the build scheduler
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17088443 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17089009 by devin doucette in ue5-release-engine-test branch]
2021-08-06 17:53:47 -04:00
devin doucette
af0f2d277f DDC: Added flags to control request barrier behavior
#rb Matt.Peters
#rnx

#ROBOMERGE-SOURCE: CL 17088062 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17088067 by devin doucette in ue5-release-engine-test branch]
2021-08-06 17:11:37 -04:00
devin doucette
c67fcf1702 DDC: Split DerivedDataRequest.h and rename DerivedDataBuildPolicy.h
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17086960 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17086977 by devin doucette in ue5-release-engine-test branch]
2021-08-06 15:50:00 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
7ae4b64cd6 DDC: Removed IBuild::Load[Type] in favor of exported functions
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17083958 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17083970 by devin doucette in ue5-release-engine-test branch]
2021-08-06 11:58:24 -04:00
devin doucette
e3b266c064 DDC: Fixed non-unity compile errors
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 17079642 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17079646 by devin doucette in ue5-release-engine-test branch]
2021-08-05 22:30:50 -04:00
devin doucette
f9a760f21f DDC: Rename FRequestGroup to FRequestOwner and export its constructor
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17074555 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17074567 by devin doucette in ue5-release-engine-test branch]
2021-08-05 15:46:22 -04:00
devin doucette
733d9c1a97 DDC: Added conversions between FPayloadId and FCbObjectId
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17071877 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071890 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:37:13 -04:00
devin doucette
d50e330b94 DDC: Split the new API from the old API
The new API uses exported functions and cannot be included with the old API without compile errors in existing code that has an include-only dependency on DDC.

#rb Zousar.Shaker
#rnx
#preflight 610c01e3aeb05700011dc5ab

#ROBOMERGE-SOURCE: CL 17071263 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071271 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:12:08 -04:00
stefan boberg
efa91a8fac Another NonUnity build fix (ZenServerHttp.h)
#rb fbcli

#ROBOMERGE-SOURCE: CL 17068679 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17068688 by stefan boberg in ue5-release-engine-test branch]
2021-08-05 11:23:42 -04:00
stefan boberg
fdffa5c30d Zen HTTP interface refactored into separate header/source files
#rb none

#ROBOMERGE-SOURCE: CL 17066037 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v852-17065384)

[CL 17066041 by stefan boberg in ue5-release-engine-test branch]
2021-08-05 05:31:06 -04:00
stefan boberg
5f8cef4af5 More defensive #if placement, to address EngineTest build issue
#rb none

#ROBOMERGE-SOURCE: CL 17064898 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17064901 by stefan boberg in ue5-release-engine-test branch]
2021-08-05 02:27:45 -04:00
devin doucette
bfda02d384 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17060649 by devin doucette in ue5-release-engine-test branch]
2021-08-04 18:08:50 -04:00
stefan boberg
01f3343c82 Refactored ZenDDC communication. HTTP interface code is now simplified and separated out from the ZenDDC client code as a step towards a shared communication layer for the oplog/cache/etc as well as instance lifetime management. There should be no functional differences.
#rb devin.doucette
#jira none

#ROBOMERGE-SOURCE: CL 17056895 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17056908 by stefan boberg in ue5-release-engine-test branch]
2021-08-04 14:56:02 -04:00
stefan boberg
de2a508d79 General HTTP zen client implementation simplification
* Rename FRequest -> FZenHttpRequest to make it easier to move the implementation elsewhere
* Simplified implementation of PUT/HEAD/DELETE operations by turning template functions into plain member functions to make it easier to hide the implementation details when the class is moved into a header

#rb none
#jira none

#ROBOMERGE-SOURCE: CL 17048397 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17049901 by stefan boberg in ue5-release-engine-test branch]
2021-08-04 09:33:43 -04:00
aurel cordonnier
dc4bf61540 Merge from Release-Engine-Staging @ 17030559 to Release-Engine-Test
This represents UE4/Main @ 17030256 and Dev-PerfTest @ 17030553

[CL 17031509 by aurel cordonnier in ue5-release-engine-test branch]
2021-08-03 11:56:47 -04:00
steve robb
fdd30297da FNames can now be constructed with UTF8CHAR strings.
UTF-8 support added to FString::AppendChars() and made able to correctly handle variably-encoded source strings (both wider and narrower than TCHAR).
FUTF8ToTCHAR now accepts either UTF8CHAR* or ANSICHAR* for better compatibility.
FUtf8StringView now binds to UTF8CHAR* or any UTF8CHAR range, and can be constructed via a _U8SV string literal suffix.
CompactBinary now supports UTF-8.
Miscellaneous other UTF8CHAR fixes.

#rb devin.doucette

#ROBOMERGE-SOURCE: CL 17017066 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17017071 by steve robb in ue5-release-engine-test branch]
2021-08-02 12:15:42 -04:00
stefan boberg
2255568c4e Remove prototype "ZenHttp" DDC client code - this was never meant to propagate to main
#rb zousar.shaker

#ROBOMERGE-SOURCE: CL 17015794 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17015806 by stefan boberg in ue5-release-engine-test branch]
2021-08-02 10:24:54 -04:00
devin doucette
77b340efea DDC: Moved new stats functions into FDerivedDataCacheInterface to remove module dependency
#rb Aurel.Cordonnier
#rnx

#ROBOMERGE-SOURCE: CL 16997387 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16997407 by devin doucette in ue5-release-engine-test branch]
2021-07-29 13:07:00 -04:00
dmytro vovk
3137bf8531 Fixed static events initialization crash
#jira none
#rb Andriy.Tylychko

#ROBOMERGE-SOURCE: CL 16995074 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16995075 by dmytro vovk in ue5-release-engine-test branch]
2021-07-29 09:56:01 -04:00
danny couture
3cc87f486a Fix double Jupiter download caused by improper usage of async put underneath each backend in hierarchy
- Remove one copy of the data in the AsyncPut task when used in conjonction with an Inflight cache
  - Make sure each async put in the hierarchical backend can fullfill request from it's Inflight cache
  - Make sure to hit the asyncput backends for exists, trytoprefetch and get queries to avoid repetitive miss while the put is still in flight
  - Remove the code that tried to delete cache miss as it caused many deletes and redownload of the same asset, that duty is fulfilled by the corruption wrapper anyway
  - Fix race condition when mounting and unmounting Pak files or other modifications applied to the hierarchical backend

#rnx
#rb Devin.Doucette, Francis.Hurteau
#preflight 61009a8d1e63730001d339a2

#ROBOMERGE-SOURCE: CL 16977632 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16977643 by danny couture in ue5-release-engine-test branch]
2021-07-27 21:20:40 -04:00
francis hurteau
86b3fd8de8 Refix crash in http request batch when the batch contains both a head and get request for the same cache keys
Added a test that could reliably reproduce the problem to validate it is properly fixed

#rb Danny.Couture, Zousar.Shaker
#preflight 610010b247c93a0001335319

#ROBOMERGE-OWNER: francis.hurteau
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 16968978 via CL 16972104
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16972120 by francis hurteau in ue5-release-engine-test branch]
2021-07-27 15:25:39 -04:00
danny couture
950c53fc2b Fix thread starvation for request in http backend by using a FIFO for thread waiting for a connection to be freed
Replace the sleep by a wait on an event triggered by the thread freeing the connection
Improve the test to output stats seen below and use the proper amount of threads on low core systems

Automation RunTests System.DerivedDataCache.HttpDerivedDataBackend.ConcurrentCachedDataProbablyExistsBatch

Before
   RPS: 238, AvgLatency: 90.09 ms, MaxLatency: 7.77 s
   RPS: 231, AvgLatency: 87.66 ms, MaxLatency: 7.77 s
   RPS: 245, AvgLatency: 139.07 ms, MaxLatency: 10.08 s

After
   RPS: 634, AvgLatency: 102.97 ms, MaxLatency: 0.26 s
   RPS: 653, AvgLatency: 102.51 ms, MaxLatency: 0.26 s
   RPS: 630, AvgLatency: 102.29 ms, MaxLatency: 0.26 s

#rnx
#rb Zousar.Shaker, Francis.Hurteau
#preflight 60ffe8317f21c90001f6116a

#ROBOMERGE-SOURCE: CL 16968010 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16968015 by danny couture in ue5-release-engine-test branch]
2021-07-27 09:26:42 -04:00