- Parallelize package reading in AssetGatherer
- Implement scheduling ParallelFor as background prio
- Saving the cache is now based on the amount of time it would take to redo the work
so it scales better on many cores systems
- Tested on AMD TR 3970X 128GB RAM with Crucial MP600 2TB
- 33m43s to 3m45s (Cold Cache + No AV Exclusions)
- 6m25s to 38.9s (Cold Cache + AV Exclusion on .uasset)
- 1m25s to 20.8s (Warm Cache + No AV Exclusions)
- All AssetCache .bin files deleted between runs
- Cold cache = Cleared standby pages, mostly equivalent to a reboot
- Warm cache = 2nd run of 2 consecutive runs
#rnx
#rb Jamie.Dale, Chris.Gagnon, Sebastien.Lussier, Steve.Robb
#ROBOMERGE-SOURCE: CL 12620239 via CL 12620242 via CL 12620244
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12620246 by danny couture in Main branch]
The Taskgpraph needs to be shutdown and recreated entirely by the forked process for the switch to occur.
The functionnality can be disabled via "TaskGraph.EnableForkedMultithreading 0" if needed.
#rb Ryan.Gerleve, John.Barrett, Andriy.Tylychko
#ROBOMERGE-SOURCE: CL 12466340 via CL 12468404 via CL 12468502 via CL 12468606
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v672-12450963)
[CL 12468689 by louisphilippe seguin in Main branch]
#rnx
#rb none
#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869210 via CL 10869511 via CL 10869900
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870549 by ryan durand in Main branch]
[REVIEW] ben.woodhouse, david.harvey
#ROBOMERGE-SOURCE: CL 10497628 via CL 10497636 via CL 10497638 via CL 10497681
#ROBOMERGE-BOT: (v606-10482310)
[CL 10497689 by dan phillips in Main branch]
Change 8553543 by Wes.Fudala
We now set a fixed value of 24 to CEF screenInfo colorDepth when off screen rendering is enabled.
Change 8235770 by Wes.Fudala
Fix for CEFJsScripting memory stomp and memory alignment errors called out by the stomp memory allocator when running with -stompmalloc on the commandline.
Change 8065597 by Leigh.Swift
BuildPatchServices: Improving ChunkBuildDirectory behaviour when dealing with empty build or builds only containing empty files.
This is now fully supported as part of the generation flow rather than an early detected edge case.
Change 7641628 by Leigh.Swift
BuildPatchServices: Tweaks to serialisation safety.
BuildPatchServices: DiskChunkStore fix for crash when chunkdump serialisation fails due to disk space.
Change 7436869 by Leigh.Swift
BuildPatchServices: Fix regression in BPT package chunks for cloud save improvements now require message pumping.
Change 7326553 by Wes.Fudala
BuildPatchServices: Attempt to restore functionality of the -SkipBuildPatchPrereq commandline. This stopped functioning in CL# 6655502 with the BPS DLC related refactors.
The issue was reported by a number of users that were relying on this commandline as a last resort workaround for prereq install issues.
Change 7323945 by Leigh.Swift
BuildPatchServices: BuildPatchTool: Install time coefficient values exposed by DiffManifests.
This gives an indicative install duration.
The time is not necessarily accurate, but the simulation is a constant and so the value is highly comparable between different builds.
Change 7310352 by Antony.Carter
BuildPatchServices: Adding support for overriding http path for chunk requests. This allows the ability to support signed urls when downloading patch data.
Change 7095282 by Leigh.Swift
BuildPatchServices: Fix regression with manifests that have no core files.
Change 7092198 by leigh.swift
BuildPatchServices: Reuse existing code from FBuildPatchAppManifest::GetChunkShaHash in FBuildPatchManifestSet::GetChunkShaHash.
This fixes an issue where older manifest files that did not ship with chunk sha values in them, can skip sha validation of chunks like pre-DLC launcher did.
Change 6959115 by Wes.Fudala
Added functionality that will optionally expose embedded browser console logs to the client.
Change 6835841 by Leigh.Swift
BuildPatchServices: Rearranging manifest save logic to avoid unnecessary seeking forwards, which avoids an assert when undetected write failures occur.
Change 6684994 by Leigh.Swift
BuildPatchServices: Don't clean empty directories if staging only.
Change 6655502 by Mike.Erickson, Leigh.Swift, Wes.Fudala
BuildPatchServices: Restructuring how installers are configured and make use of manifest files in order to combine multiple actions on an installation directory into one installer.
This resembles a feature set for a better DLC installation experience.
Change 6404031 by Richard.Fawcett
BuildPatchTool: Only append ".manifest" to output filename if output filename has been specified on the command line.
This was causing a manifest file called literally ".manifest" to be output to the clouddir if -OutputFilename was not specified.
Change 6077240 by Wes.Fudala
Execution of browser resource load complete delegate now happens on the main thread.
Change 6076171 by Leigh.Swift
BuildPatchTool: PatchGeneration: ChunkDeltaOptimise: PackageChunks: Improved corrupt output protection against ill timed taskkill, by serialising to temp filename, and then rename on success.
BuildPatchTool: PatchGeneration: Manifest file extension added if not provided, fixing an oversight and inconsistency with other mode behaviours.
BuildPatchTool: Compactify: Only warn when failing to get a file size, if the file still exists. Otherwise log instead.
Change 6049003 by Leigh.Swift
BuildPatchServices: Adding ProcessRequiredDiskSpace to Launcher.Install.Stats which represents how much disk space the install/update process needed to complete.
BuildPatchServices: Adding ProcessAvailableDiskSpace to Launcher.Install.Stats which represents how much disk space was available at the time of checking required disk space.
Change 5915157 by Leigh.Swift
BuildPatchTool: Adding a statistic to diffmanifests for temporary disk space requirement to apply the patch.
Change 5934838 by Leigh.Swift
BuildPatchTool: PackageChunks: Adding support to provide a tagset for the previous build manifest when producing chunkdbs. This allows expanding the chunks saved out to cover tagsets not installed in the previous build.
Change 5838666 by wes.fudala
Browser can now bubble up the state of completed web resource loads.
Change 5689493 by Leigh.Swift
Adding new x86 and x64 MS VC141 CRT redist, version 14.16.27012
Change 5689462 by Leigh.Swift
Fixing process handle leaks on windows.
Core was leaking for getting an application name.
Change 5500917 by Leigh.Swift
BuildPatchTool: Adding new arg DiffAbortThreshold to ChunkDeltaOptimise mode which allows skipping of the operation if the original delta is so large that it would take too long to process, and likely have little benefit.
BuildPatchTool: Switching some Log output to use Display so that it will appear in EC and CMD windows.
Change 5337482 by Leigh.Swift
BuildPatchTool: Fix for DiffManifests mode not accurately representing delta size for tagged install sets.
Change 5261246 by Leigh.Swift
BuildPatchServices: Fix for file download needing to mock response codes for higher layer statistics code which tracks data sizes and speeds.
This is a regression from previous change to correct download failure vs corruption statistics.
Change 5224725 by Leigh.Swift
BuildPatchServices: Fix for delta download of more than 0 bytes when no update is necessary.
BuildPatchServices: Skip requesting delta metafile if no file changes are actually required for a patch.
BuildPatchTool: Reduce unnecessary data produced by BPT ChunkDeltaOptimise mode.
Change 5010941 by Mike.Erickson
BuildPatchServices: Add download scaling based on average speed per request, maximum count, and download health.
Change 5010845 by Wes.Fudala
BuildPatchServices: IDownload refactored to have specific request and response success functions, to make it clearer that a successful request does not mean the response was also good.
BuildPatchServices: Fixed issues with download failures reporting as corruptions.
Change 5000643 by Wes.Hunt
Remove HttpServiceTracker from UE4.
Change 4884381 by Leigh.Swift
BuildPatchTool: Fix for Package Chunks mode hanging when no chunks were required.
Change 4848675 by Justin.Sargent, Leigh.Swift
Speculative fixes for graphics device lost related crash, by adding additional d3d api result checks.
Improved logging for graphics device lost handling.
Improved logging for tracking down common font loading failure resulting in an ensure.
Change 4831134 by Leigh.Swift
BuildPatchTool: Fix for crash in patchgeneration when fast-forward path replays no match.
Change 4801714 by Wes.Fudala
Fix for CEF issue encountered when building using Mac Mojave + XCode10.
Change 4719149 by Leigh.Swift
BuildPatchTool: PatchGeneration mode cyclic data optimisation, reduces SHA calculation requirement counts for cyclic data.
BuildPatchTool: PatchGeneration mode fix for a bug causing non-optimal match insertion idx searching when there are 10k+ matches per scanner.
Change 4680963 by Leigh.Swift
BuildPatchTool: ChunkDeltaOptimise mode is now FeatureLevel upgrade / downgrade aware.
Change 4680947 by Leigh.Swift
BuildPatchTool: Compactify speed improvements for massive network cloud directories.
Change 4656991 by Leigh.Swift
BuildPatchServices: Make sure chunk writer robustly discovers if a chunk fails to save out.
Change 4647815 by Leigh.Swift
Upping the minimum wait time for UdpMessageBeacon thread so that it will not always wait 0ms when network sends are failing, reducing disconnect CPU usage.
Adding configurable tick rate logic to XmppConnectionJingle thread. It will now default to 100Hz max.
Change 4627355 by Michael.Trepka
Fixed a problem with CEF being unable to find locale pak files on Mac for certain language/region combinations
Change 4620800 by Leigh.Swift
Fix for CEF crash when disabling a web window that has not yet got a parent window. There's no need to worry about focus in this case.
Change 4590207 by Leigh.Swift
BuildPatchTool: PackageChunks mode now supports FeatureLevel arg
Change 4590103 by leigh.swift
BuildPatchTool: Adding new mode ChunkDeltaOptimise which reducing the download size when patching between two specific builds in a specific direction.
BuildPatchTool: Updated Enumeration, DiffManifests, Compactify, PackageChunks, and VerifyChunks modes to take account of new delta data.
BuildPatchServices: Installers now have a single shared memory chunk store, which reduces the requirement for booting
Change 4590089 by Leigh.Swift
BuildPatchTool: Adding new mode ChunkDeltaOptimise which reducing the download size when patching between two specific builds in a specific direction.
BuildPatchTool: Updated Enumeration, DiffManifests, Compactify, PackageChunks, and VerifyChunks modes to take account of new delta data.
BuildPatchServices: Installers now have a single shared memory chunk store, which reduces the requirement for booting
Change 4341076 by Leigh.Swift
BuildPatchServices: Making FBuildPatchAppManifest::GetRemovableFiles more robust to handle directories with or without trailing slash.
Change 4331754 by Leigh.Swift
BuildPatchTool: Added support for selecting ChunkWindowSize when generating patches.
BuildPatchTool: Added support for providing the FeatureLevel command-line argument to indicate the data version that should be saved out by patch generation. This warns about defaulting to LatestJson if not provided.
BuildPatchTool: Added support for generating patches with recognition for any chunks with any ChunkWindowSize found in the provided CloudDir.
BuildPatchTool: Added command-line -IgnoreOtherWindowSizes param which if provided, the generation code will only accept chunk matches that are the same as ChunkWindowSize.
BuildPatchServices: Fixes for supporting installations that use any ChunkWindowSize.
BuildPatchServices: New manifest file format to reduce file size, this is now raw compressed binary data.
#lockdown Nick.Penwarden
#rb none
[CL 8675597 by Leigh Swift in Main branch]
in my test replay run, notable items were:
~6MB Shaders
~15MB UObject
~2MB StreamingManager
#rb anthony.bills
#ROBOMERGE-SOURCE: CL 5630087 via CL 5630202 via CL 5630219 via CL 5630235 via CL 5632395
[CL 5632701 by david harvey in Main branch]
- Added memory mapped IO concept to the engine
- Implemented and tested on Windows and IOS
- Not in use by anything yet, however, as the use cases are narrow and special (bulk data that does not need massaging after load - so it can be swapped out and back in silently by OS)
[FYI] gil.gribb,chris.babcock
#ROBOMERGE-SOURCE: CL 4842051 via CL 4845281
[CL 4845299 by josh adams in Main branch]
