See review https://p4-swarm.epicgames.net/reviews/16249378
#ROBOMERGE-SOURCE: CL 16314153 via CL 16325443 via CL 16325726 via CL 16325760
#ROBOMERGE-BOT: (v804-16311228)
[CL 16331307 by tim tillotson in ue5-release-engine-staging branch]
#ROBOMERGE-SOURCE: CL 16330779 in //UE5/Main/... via CL 16330781
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16330785 by ben marsh in ue5-release-engine-staging branch]
#ROBOMERGE-SOURCE: CL 16330779 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16330781 by ben marsh in ue5-release-engine-test branch]
#ROBOMERGE-SOURCE: CL 16329778 in //UE5/Main/... via CL 16329800
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16329804 by ben marsh in ue5-release-engine-staging branch]
#ROBOMERGE-SOURCE: CL 16329778 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16329800 by ben marsh in ue5-release-engine-test branch]
#rb Trivial.
#ROBOMERGE-SOURCE: CL 16328917 in //UE5/Main/... via CL 16328921
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16328923 by patrick laflamme in ue5-release-engine-staging branch]
Elimination of some quick to resolve type system lookups.
#rnx
#rb jonathan.adamczewski
#preflight 609e5a1dcbf9a40001fe9778
#ROBOMERGE-SOURCE: CL 16328413 in //UE5/Main/... via CL 16328416
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16328419 by tim smith in ue5-release-engine-staging branch]
Elimination of some quick to resolve type system lookups.
#rnx
#rb jonathan.adamczewski
#preflight 609e5a1dcbf9a40001fe9778
#ROBOMERGE-SOURCE: CL 16328413 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16328416 by tim smith in ue5-release-engine-test branch]
[FYI] Patrick.Laflamme
#ROBOMERGE-SOURCE: CL 16326683 in //UE5/Main/... via CL 16326693
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16326695 by jeff farris in ue5-release-engine-staging branch]
Engine/Editor changes:
- Split the Editor summary session in two, one summary for the Engine properties and one for the Editor specific properties. Made it easy to extend the Engine summary to create other summaries.
- Made the summary sender as agnostics as possible of the keys it sends.
- Fixed the system wide lock contention between the process when persisting a session. (On problem caused by the lock is UE-114315).
- Fixed concurrent issue when saving the summary sessions on Linux/Mac
- Fixed performance issue when saving the summary session on Linux/Mac. This enable saving at higher frequency.
- Fixed cases where the same session summary is sent more than once.
- Fixed Windows registry key overflow that could happens if we accumulated too many sessions (in theory, this can happen)
- Made adding new properties to the summary easy and private to the implementation.
- Brought the Linux/Mac implementation closer to Windows implementation.
- Reduced memory allocation, especially when the session records a crash.
- Improved chances to send the summary non-delayed by allowing the Editor to send the reports if CRC died unexpectedly.
- Generalized the support to collect and aggregate analytics from helper processes. For example, CRC already collects analytics that is merged with the Editor summary as information supplement
- Reserved the disk space required to store the summary ahead of time to prevent failing later.
- Increased frequency at which the summary is persisted because saving the summary is more efficient. (About every 10 seconds rather than every minutes).
- Added unit tests
CrashReportClient changes:
- Created a 'session summary' from the CRC point of view to merge with the Editor summary.
- Moved analytics collection in a separated class to make the crash reporting code leaner and less noisy with all the analytics
- Merged the CRC diagnostic logger in the class collecting CRC analytics summary and make the diagnostic log a property in the summary.
- Collected analytics (on behalf of Editor) in a background thread because CRC main thread can be blocked collecting a crash, so it doesn't pay attention to other things
- Added MonitorBatteryLevel and MonitorOnACPower summary properties on Windows. Collected on CRC background thread (never blocked, so we reduce changes to miss the battery running out)
- Added MonitorSessionDuration summary property to track now long CRC ran.
- Added MonitorQuitSignalRecv summary property to detect when CRC is soft killed like: taskkill /PID 1234
- Added MonitorIsReportingCrash summary property to track when CRC dies reporting a crash.
- Added MonitorIsCollectingCrash summary property to track when CRC dies collecting a crash artifacts.
- Added IsProcessingCrash summary property to track when CRC dies processing a crash.
- Added MonitorCrashed summary property to track when CRC exception handler was triggered.
- Added MonitorWasShutdown summary property to track when CRC summary was shutdown
- Added MonitorLoggingOut summary property to track when CRC died because the user was logging out (or as result of shutting down or restarting the computer).
- More accurate value for DeathTimestamp summary property because this is now captured in CRC background thread (which cannot be busy handling a crash)
- Added crash processing timing to CRC diagnostic logs (how long it takes to collect/process a crash).
#rb Jamie.Dale, Wes.Hunt, Johan.Berg
#jira UETOOL-3500
#jira UE-114315
#ROBOMERGE-SOURCE: CL 16324612 in //UE5/Main/... via CL 16324626
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16324633 by patrick laflamme in ue5-release-engine-staging branch]
Engine/Editor changes:
- Split the Editor summary session in two, one summary for the Engine properties and one for the Editor specific properties. Made it easy to extend the Engine summary to create other summaries.
- Made the summary sender as agnostics as possible of the keys it sends.
- Fixed the system wide lock contention between the process when persisting a session. (On problem caused by the lock is UE-114315).
- Fixed concurrent issue when saving the summary sessions on Linux/Mac
- Fixed performance issue when saving the summary session on Linux/Mac. This enable saving at higher frequency.
- Fixed cases where the same session summary is sent more than once.
- Fixed Windows registry key overflow that could happens if we accumulated too many sessions (in theory, this can happen)
- Made adding new properties to the summary easy and private to the implementation.
- Brought the Linux/Mac implementation closer to Windows implementation.
- Reduced memory allocation, especially when the session records a crash.
- Improved chances to send the summary non-delayed by allowing the Editor to send the reports if CRC died unexpectedly.
- Generalized the support to collect and aggregate analytics from helper processes. For example, CRC already collects analytics that is merged with the Editor summary as information supplement
- Reserved the disk space required to store the summary ahead of time to prevent failing later.
- Increased frequency at which the summary is persisted because saving the summary is more efficient. (About every 10 seconds rather than every minutes).
- Added unit tests
CrashReportClient changes:
- Created a 'session summary' from the CRC point of view to merge with the Editor summary.
- Moved analytics collection in a separated class to make the crash reporting code leaner and less noisy with all the analytics
- Merged the CRC diagnostic logger in the class collecting CRC analytics summary and make the diagnostic log a property in the summary.
- Collected analytics (on behalf of Editor) in a background thread because CRC main thread can be blocked collecting a crash, so it doesn't pay attention to other things
- Added MonitorBatteryLevel and MonitorOnACPower summary properties on Windows. Collected on CRC background thread (never blocked, so we reduce changes to miss the battery running out)
- Added MonitorSessionDuration summary property to track now long CRC ran.
- Added MonitorQuitSignalRecv summary property to detect when CRC is soft killed like: taskkill /PID 1234
- Added MonitorIsReportingCrash summary property to track when CRC dies reporting a crash.
- Added MonitorIsCollectingCrash summary property to track when CRC dies collecting a crash artifacts.
- Added IsProcessingCrash summary property to track when CRC dies processing a crash.
- Added MonitorCrashed summary property to track when CRC exception handler was triggered.
- Added MonitorWasShutdown summary property to track when CRC summary was shutdown
- Added MonitorLoggingOut summary property to track when CRC died because the user was logging out (or as result of shutting down or restarting the computer).
- More accurate value for DeathTimestamp summary property because this is now captured in CRC background thread (which cannot be busy handling a crash)
- Added crash processing timing to CRC diagnostic logs (how long it takes to collect/process a crash).
#rb Jamie.Dale, Wes.Hunt, Johan.Berg
#jira UETOOL-3500
#jira UE-114315
#ROBOMERGE-SOURCE: CL 16324612 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16324626 by patrick laflamme in ue5-release-engine-test branch]
#ROBOMERGE-SOURCE: CL 16323617 in //UE5/Main/... via CL 16323633
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16323637 by ben marsh in ue5-release-engine-staging branch]
#ROBOMERGE-SOURCE: CL 16323617 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16323633 by ben marsh in ue5-release-engine-test branch]
* Filter out numerous source files that match */Source/ThirdParty/* that should have already been filtered, this does not include Module.Build.cs or .tps files, or if bGatherThirdPartySource is enabled
* Filter out source files that match */third_party/* by default
* Add option bIncludeDotNetPrograms to remove all ..NET projects, doing so saved me around 400MB. Note if you disable the .NET projects UnrealBuildTool won't build before compiling (but GenerateProjectFiles.bat will still build it)
Attempted changes that did not reduce memory usage by a significant amount:
* Filtering out project configurations
* Filtering out Engine/Shaders or Engine/Content/Editor/Templates
* Increasing the shared include size, even making it large enough to contain every include path didn't really make a difference. Also it can't really be increased anyway because that entire property is added to the process environment when starting a build and there's a max environment size of around 32k
Other Fixes:
* Fix vs2019 ToolVersionString 15.0 -> 16.0
* Add VCProjectVersion to Project globals
* Update UniqueIdentifier GUIDs in projects to be stable by hashing the directory path and using that hash as the GUID
* Don't write ProjectConfiguration if filtered out (Didn't affect memory usage)
* Add optional configuration filter for Debug, DebugGame, & Development
#jira UE-111822
#rb none
#ROBOMERGE-SOURCE: CL 16319735 in //UE5/Main/... via CL 16319758
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Release-Engine-Staging) (v804-16311228)
[CL 16319765 by joe kirchoff in ue5-release-engine-staging branch]
* Filter out numerous source files that match */Source/ThirdParty/* that should have already been filtered, this does not include Module.Build.cs or .tps files, or if bGatherThirdPartySource is enabled
* Filter out source files that match */third_party/* by default
* Add option bIncludeDotNetPrograms to remove all ..NET projects, doing so saved me around 400MB. Note if you disable the .NET projects UnrealBuildTool won't build before compiling (but GenerateProjectFiles.bat will still build it)
Attempted changes that did not reduce memory usage by a significant amount:
* Filtering out project configurations
* Filtering out Engine/Shaders or Engine/Content/Editor/Templates
* Increasing the shared include size, even making it large enough to contain every include path didn't really make a difference. Also it can't really be increased anyway because that entire property is added to the process environment when starting a build and there's a max environment size of around 32k
Other Fixes:
* Fix vs2019 ToolVersionString 15.0 -> 16.0
* Add VCProjectVersion to Project globals
* Update UniqueIdentifier GUIDs in projects to be stable by hashing the directory path and using that hash as the GUID
* Don't write ProjectConfiguration if filtered out (Didn't affect memory usage)
* Add optional configuration filter for Debug, DebugGame, & Development
#jira UE-111822
#rb none
#ROBOMERGE-SOURCE: CL 16319735 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)
[CL 16319758 by joe kirchoff in ue5-release-engine-test branch]
