Commit Graph

76 Commits

Author SHA1 Message Date
devin doucette
a066ec0c49 DDC: Implemented existing cache debug options for the new cache stores
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17308400 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17308416 by devin doucette in ue5-release-engine-test branch]
2021-08-25 16:08:33 -04:00
charles bloom
b9f505a152 Log DerivedDataCache when registry overrides dirs
make editor dialog update when string is empty

#rb devin.doucette

#ROBOMERGE-SOURCE: CL 17107690 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17107751 by charles bloom in ue5-release-engine-test branch]
2021-08-09 17:06:13 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
f9a760f21f DDC: Rename FRequestGroup to FRequestOwner and export its constructor
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17074555 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17074567 by devin doucette in ue5-release-engine-test branch]
2021-08-05 15:46:22 -04:00
devin doucette
bfda02d384 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17060649 by devin doucette in ue5-release-engine-test branch]
2021-08-04 18:08:50 -04:00
stefan boberg
2255568c4e Remove prototype "ZenHttp" DDC client code - this was never meant to propagate to main
#rb zousar.shaker

#ROBOMERGE-SOURCE: CL 17015794 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17015806 by stefan boberg in ue5-release-engine-test branch]
2021-08-02 10:24:54 -04:00
danny couture
a3ed621ba9 Add a DDC throttling layer that can be used to approximate Jupiter performance locally by adding latency and limiting bandwidth
This is useful for benchmarking as it reduce noise caused by variable Internet performance

#rnx
#rb Devin.Doucette
#preflight 60fa02ea1f926d0001f9f9b6

#ROBOMERGE-SOURCE: CL 16934015 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16934016 by danny couture in ue5-release-engine-test branch]
2021-07-22 22:56:48 -04:00
zousar shaker
c27625b056 Integrating //UE5/Dev-Cooker [at] 16678003 to //UE5/Main (Zousar.Shaker-YEG-0943-Quaternary)
Non-DevIteration_ShooterGame


#ROBOMERGE-SOURCE: CL 16678907 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16678928 by zousar shaker in ue5-release-engine-test branch]
2021-06-15 16:38:03 -04:00
Devin Doucette
600ee5377a DDC: Added serialization of FCacheRecord to/from FCbPackage
#rb Devin.Doucette
#rnx

#ushell-cherrypick of 16476127 by Matt.Peters

[CL 16483925 by Devin Doucette in ue5-main branch]
2021-05-27 10:16:47 -04:00
Zousar Shaker
86d1fe0f4f Allow for the use of -DDC=None commandline switch to permit DDC to be used without having a backend graph from the standalone build workers.
#rb devin.doucette
#preflight 60ad051f3df6e30001896e21

[CL 16449857 by Zousar Shaker in ue5-main branch]
2021-05-25 11:40:26 -04:00
Devin Doucette
e42bbbedab DDC: Implement ICacheStore on FDerivedDataBackendInterface
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.

The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.

The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.

The HTTP, S3, and pak file backends are not implemented for the new interface.

The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.

The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.

#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517

[CL 16148296 by Devin Doucette in ue5-main branch]
2021-04-28 16:22:18 -04:00
Zousar Shaker
8b5a796e2a Add support for ReadOnly={true|false} and EnvHostOverride={string} as configurable parameters on an HTTP DDC backend.
#rb devin.doucette

[CL 16076564 by Zousar Shaker in ue5-main branch]
2021-04-21 13:19:11 -04:00
Zousar Shaker
3149d95ca3 Since we are now waiting for quiescence from DDC backends during shutdown even in interactive sessions, reduce the sleep time before checking for task completion from 1s to 0.1s.
#rb trivial

[CL 15934332 by Zousar Shaker in ue5-main branch]
2021-04-06 16:13:44 -04:00
Zousar Shaker
dc68fab37b Ensure HTTP DDC backend doesn't delay process shutdown due to a queue of pending PUTs for normal users.
Still allows build machine runs to be thorough about completing PUTs.
Doesn't interrupt any in-flight PUTs, just prevents processing of any remaining queued PUTs.

#jira UE-112179
#rb devin.doucette

[CL 15920539 by Zousar Shaker in ue5-main branch]
2021-04-05 14:36:58 -04:00
nick darnell
832cc08a58 DDC - Don't look up the store for DDC paths that don't have environment variable overrides.
#jira UE-110702

#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 15715366 via CL 15715370 via CL 15715374 via CL 15715407 via CL 15743064
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Main) (v781-15675533)

[CL 15747819 by nick darnell in ue5-main branch]
2021-03-19 11:39:18 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Joakim Lindqvist
5957a9c659 Added option to override Jupiter speed class
#rb devin.doucette

[CL 15736124 by Joakim Lindqvist in ue5-main branch]
2021-03-18 05:51:42 -04:00
Joakim Lindqvist
c159fac11a Adding command line override for Jupiter Host in ShooterGame, DDCHttpHost
Changed the Dev toggle for CookCompare to use this.
This can also be used to locally toggle Jupiter to run against a local deployment.

[CL 15527157 by Joakim Lindqvist in ue5-main branch]
2021-02-25 06:12:30 -04:00
Marcus Wassmer
3b81cf8201 Merging using //UE5/Main_to_//UE5/Release-Engine-Staging @14384769
autoresolved files
#rb none

[CL 14384911 by Marcus Wassmer in ue5-main branch]
2020-09-24 00:43:27 -04:00
danny couture
0655582bd0 Fix stutters in the game-thread while streaming texture mips from DDC.
This will fix a good bunch of ForceRebuildPlatformData in UTexture2D::UpdateStreamingStatus because of the Boot DDC telling it has some mips cached effectively preventing other cache layers from receiving them.
When the Boot DDC is disabled after the editor boot, all the mips will go into oblivion, and nobody will have them anymore, causing stutters in the game-thread while streaming because a second full rebuild is now required.

Any DDC that plans on disappearing should never be alone in the cache hierarchy and should give other levels the opportunity to properly cache data before its disapperance.

#rb Francis.Hurteau, Devin.Doucette, Andrew.Grant

[CL 14268431 by danny couture in ue5-main branch]
2020-09-08 10:50:47 -04:00
Marc Audy
a7c9001a94 Merging //UE5/Release-Engine-Staging to Main (//UE5/Main) @ 14075166
#rb
#rnx

[CL 14075271 by Marc Audy in ue5-main branch]
2020-08-11 01:36:57 -04:00
Marc Audy
11f5b21210 Merging //UE5/Release-Engine-Staging @ 13752110 to Main (//UE5/Main)
#rnx

[CL 13753156 by Marc Audy in ue5-main branch]
2020-06-23 18:40:00 -04:00
Matt Kuhlenschmidt
603a4119c5 Fix HAL/PlatformFileManager.h non-portable casing CIS issues
#rb none

[CL 13214257 by Matt Kuhlenschmidt in ue5-main branch]
2020-05-06 17:58:18 -04:00
andrew grant
c580697c24 Made the pop-up warning at startup about the shared DDC path being missing optional.
- The engine ships by default with a shared entry in the graph that doesn't exist so this prevents it showing for projects that haven't yet set it up.
- The value in this prompt is situational. It's beneficial on Fortnite where a huge amount of DDC content is needed to get into the game and WFH (and Mac) users may be missing a mounted drive, but less important on smaller projects.

Also documented the "UE-EnvVar=None" trick in the ini file where the DDC behavior is described.


#tests Ran with PromptIfMissing=true and verified the prompt occurred when the path didn't exist. Ran with PromptIfMissing=false and verified there was no prompt but still a warning.

[REVIEW] [at]brandon.schaefer [at]devin.doucette
[FYI] stefan.boberg


#ROBOMERGE-SOURCE: CL 13187183 via CL 13187231 via CL 13187280
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v688-13145358)

[CL 13187317 by andrew grant in Main branch]
2020-05-05 15:50:08 -04:00
andrew grant
7c2df5fb70 Editgrate fix for warning about -ddc=default not being a valid backend
[FYI] marc.audy

#ROBOMERGE-SOURCE: CL 13043962 via CL 13043963 via CL 13043964
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v683-13008971)

[CL 13043965 by andrew grant in Main branch]
2020-04-24 20:11:55 -04:00