- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.
Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.
Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.
#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18984126 by devin doucette in ue5-main branch]
Requests to Put and Get now specify the policy independently for each record or key, but the implementations have not yet been updated to take this into account.
The existing Put and Get API has been maintained temporarily to allow the conversion to happen separately, and will be removed after callers have been converted to use the new API.
#rb Zousar.Shaker
#rnx
#preflight 61b2e550ee0de9822e2e84dc
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18443052 in //UE5/Release-5.0/... via CL 18445037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18445201 by devin doucette in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.
#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454
#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.
The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.
#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94
#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)
[CL 17060649 by devin doucette in ue5-release-engine-test branch]
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.
The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.
The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.
The HTTP, S3, and pak file backends are not implemented for the new interface.
The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.
The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.
#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517
[CL 16148296 by Devin Doucette in ue5-main branch]
The async put backend now uses this new status to avoid calling CachedDataProbablyExists after every put, which has a high cost for certain backends.
#rb Zousar.Shaker
#fyi Joakim.Lindqvist, Stefan.Boberg
[CL 15000972 by Devin Doucette in ue5-main branch]
E.g.
'-ddc-local-missrate=50' simulates a 50% miss rate of keys on the local backend,
'-ddc-shared-misstypes=AnimSeq' simulates misses for any AnimSeq key. Use + to combine keys. E.g. AnimSeq+Audio
Memory DDC, FileSystemDDC and S3DDC support these options as do the classes that wrap other backends (e.g. AsyncPut).
#tests lots of warm and cold DDC
[at]ben.marsh [at]josh.engebretson [at]devin.doucette
#ROBOMERGE-SOURCE: CL 12458693 via CL 12462167 via CL 12464485 via CL 12464606
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12464717 by andrew grant in Main branch]
If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN
Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).
Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats
Interface Changes
- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use
Behavior Changes
- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.
[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option
#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)
[CL 12396757 by andrew grant in Release-Engine-Staging branch]
Updating the ArrayView with new comparison operators to have greater replacement for what TArray provides.
#rb Marc.Audy, Steve.Robb
#ROBOMERGE-OWNER: nick.darnell
#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-SOURCE: CL 11843381 via CL 11843463 via CL 11843492
#ROBOMERGE-BOT: (v656-11643781)
[CL 11879995 by nick darnell in Main branch]
#rnx
#rb none
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870584 by ryan durand in Main branch]
#lockdown Nick.Penwarden
#rb none
==========================
MAJOR FEATURES + CHANGES
==========================
Change 3209340 on 2016/11/23 by Ben.Marsh
Convert UE4 codebase to an "include what you use" model - where every header just includes the dependencies it needs, rather than every source file including large monolithic headers like Engine.h and UnrealEd.h.
Measured full rebuild times around 2x faster using XGE on Windows, and improvements of 25% or more for incremental builds and full rebuilds on most other platforms.
* Every header now includes everything it needs to compile.
* There's a CoreMinimal.h header that gets you a set of ubiquitous types from Core (eg. FString, FName, TArray, FVector, etc...). Most headers now include this first.
* There's a CoreTypes.h header that sets up primitive UE4 types and build macros (int32, PLATFORM_WIN64, etc...). All headers in Core include this first, as does CoreMinimal.h.
* Every .cpp file includes its matching .h file first.
* This helps validate that each header is including everything it needs to compile.
* No engine code includes a monolithic header such as Engine.h or UnrealEd.h any more.
* You will get a warning if you try to include one of these from the engine. They still exist for compatibility with game projects and do not produce warnings when included there.
* There have only been minor changes to our internal games down to accommodate these changes. The intent is for this to be as seamless as possible.
* No engine code explicitly includes a precompiled header any more.
* We still use PCHs, but they're force-included on the compiler command line by UnrealBuildTool instead. This lets us tune what they contain without breaking any existing include dependencies.
* PCHs are generated by a tool to get a statistical amount of coverage for the source files using it, and I've seeded the new shared PCHs to contain any header included by > 15% of source files.
Tool used to generate this transform is at Engine\Source\Programs\IncludeTool.
[CL 3209342 by Ben Marsh in Main branch]
#lockdown Nick.Penwarden
==========================
MAJOR FEATURES + CHANGES
==========================
Change 2870336 on 2016/02/17 by Marc.Audy
Continued splitting up Orion Build
* Restructure from platform based MakeBuild steps in to a PS4, Server, and Windows Client MakeBuild
* Cook server data only once for both Windows and Linux (windows reuses Linux server data)
* Split compilation of Win64 Client and Server such that MakeBuild_Server only builds Server and MakeBuild_WindowsClient only builds Client
#jira UEB-580
#rb Ben.Marsh
#tests Preflight and generated Windows Client and Server work to play game
Change 2870026 on 2016/02/17 by Wes.Hunt
Don't allow array shrinking when removing the corruption wrapper trailer.
#rb none
Updating CIS Counter
Change 2869725 on 2016/02/17 by Dmitry.Rekman
More analytics and QoS stats added for 0.19.
#rb none
#tests Ran Windows client and Linux server on compatible content.
Change 2869705 on 2016/02/16 by Ryan.Gerleve
Fix replicated properties and call RepNotifies of startup actors when scrubbing in replays.
This is the engine support for fixing OR-6817, towers not respawning when rewinding replays.
#rb john.pollard
#tests golden path, replays, ps4 nomcp
Change 2869644 on 2016/02/16 by Jason.Bestimt
#ORION_DEV - Merge MAIN (0.18) at CL# 2869635
#Tests:none
#RB:none
Change 2869586 on 2016/02/16 by Marcus.Wassmer
Fix texturestreaming RHI flushes.
#rb none
#test goldenpath
#codereview Gil.Gribb
Change 2869279 on 2016/02/16 by Lukasz.Furman
fixed minion hit reaction directions
#orion OR-13953
#rb Mieszko.Zielinski
#tests PIE: hit minions with various abilities from different angles, checked velocity of death particles when killed by abilities and towers
#codereview Dan.Youhon
Change 2869277 on 2016/02/16 by Wes.Hunt
During cook, when a package is not ready to save, actually early out of the saving code. Saves somewhere in the 130s to 200s range for cooks.
#rb daniel.lamb
#tests local windows cooks, preflight PS4 cooks
Change 2869132 on 2016/02/16 by Mieszko.Zielinski
Added a function to AISenseConfig allowing native-code MaxAge configuration #UE4
#rb Lukasz.Furman
#test none required
Change 2868981 on 2016/02/16 by Wes.Hunt
remove -LogCookStats cmdline check, always log cook stats. -SendCookAnalytics flag is still used.
This was requested by NickP.
#rb none
#tests local windows cooks
Change 2868975 on 2016/02/16 by Wes.Hunt
Don't submit DDC usage stats for zero-sized events.
#rb none
#tests local windows cook
Change 2868956 on 2016/02/16 by Jason.Bestimt
#ORION_DEV - Merge MAIN (0.18) at CL# 2868926
#RB:none
#Tests:none
Change 2868889 on 2016/02/16 by Max.Chen
Sequencer: Only allow transport control binding when editing level editor sequencers.
#rb none
#tests none
Change 2868663 on 2016/02/16 by David.Ratti
downgrade warning to display
#rb none
#tests compile
Change 2868624 on 2016/02/16 by Marcus.Wassmer
Re-Enable Defrag validation for devgeneral
#rb none
#test none
Change 2868493 on 2016/02/16 by Benn.Gallagher
Added a few more stats to morph target updates to try and narrow down hitches
#rb Bruce.Nesbit
#tests pie, -game Win64
Change 2868445 on 2016/02/16 by Dmitry.Rekman
Linux: report crashes due to stack overflow (OR-14519).
- Reserve memory for alternative stack for signal handlers. Adds about 128KB memory per thread.
- Force process spawning to use vfork() when no pipes are needed.
- Ignore all signals except explicitly handled.
- Prevent signals from being raised while another one is handled.
- Added "debug threadrecurse" and "debug threadstackoverflow" to test that.
[CL 2873763 by Andrew Grant in Main branch]
