- Add trace provider, module, & analyzer and start piping data through it as proof-of-concept
#rb max.hayes
#jira UE-169605
#rnx
#preflight 63753d471d25fe8b933fa452
[CL 23160520 by Rob Gay in ue5-main branch]
Previously any events traced while Trace allocates memory to trace events into were dropped because there was nowhere to write these events. This change
splits memory allocation into two functions, one that will capture and send events when safe to do so, and one that doesn't capture events at all. For that latter, the capture is moved out to where allocating trace buffers happens so that captured events can be added to the buffer just allocated. This ensure order is maintained and that serials are ascending on a thread.
Partial buffers; prior to this change, aux-data that was larger than available space in a buffer would be fragmented across multiple buffers. Buffers that started in the middle of aux-data were marked partial as there was no way for analysis to interpret the stream from this point. As it is now possible that allocation-traced events will be written into the buffer before it's made available, each fragment is preceded with a aux-data header. This makes the stream recoverable from any buffer and negates the need for having partial buffers at all. As aux-data can now be constructed of two or more aux-data events for a single field of an event, analysis needs to be aware of this and defragment these events.
Late connect; analysis of a stream can now start with aux-data/terminal events with no corresponding owning event. This is fine; they get skipped over in DispatchEvents().
It is a deliberate choice to have a limited amount of redirected-buffer space to capture allocation-traced events. There needs to be enough space remaining in the new buffer to trace the outer event. Some platforms also have a limited amount of stack per thread.
The Writer_AddPageToPool() in Writer_InitializePool() didn't do anything and was removed.
GInitialized was set to early and subvert the reliance on static initialisation guarantees
#rnx
#jira UE-157559
#jira UE-161280
#preflight 632057f3bc40358fa290f403
[CL 21980603 by Martin Ridgers in ue5-main branch]
Headers are updated to contain any missing #includes needed to compile and #includes are sorted. Nothing is removed.
#ushell-cherrypick of 21064294 by bryan.sefcik
#jira
#preflight 62d5c2111062f2e63015e598
#ROBOMERGE-OWNER: bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21155249 via CL 21158121 via CL 21161259
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21182053 by bryan sefcik in ue5-main branch]
#ushell-cherrypick of 20644780 by Johan.Berg
#rb martin.ridgers
#rnx
#preflight 62c8260af671e8d2faccafd7
[CL 21005759 by Martin Ridgers in ue5-main branch]
Unreal Trace Server now writes full UTF8 strings to the CBOR objects, change the corresponding string views to `FUtf8StringView` to support correct conversion to TCHAR based strings.
#rb ionut.matasaru
#jira UE-155288
#robomerge EngineMerge
#preflight 62bedf2eb024a2608c81d044
[CL 20913058 by Johan Berg in ue5-main branch]
### Features
This change sets up new interface APIs and moves concrete APIs into those interfaces to allow for providers to be implemented external to TraceServices. These Provider interfaces are given to new construction APIs in ExternalAnalysis.h that construct the appropriate Analyzer with the specified Providers.
This change creates a new and standalone Analyzer for Bookmarks spinning logic directly out of the MiscAnalyzer. This is necessary to prevent bookmark processing necessitating additional Providers that the client may not want to implement if they just want access to Bookmark data.
SummarizeTraceCommandlet implements these provider interfaces and migrates it's state tracking implementation from raw event decoding to simply forwarding on the calls from the Provider interfaces. The internal SummarizeTraceCommandlet Analyzer API in the Commandlet are mostly left unchanged as a middleman between Trace and the Commandlet existed prior to this change.
### Implementation
Since Providers can now implement multiple interfaces, we need to upcast them to the appropriate interface at registration time. This in turn requires moving the Provider memory model into IAnalysisSession to TSharedPtr/MakeShared to prevent double deletes at destruction time.
### Performance
Opening a large trace in Insights yields this performance change:
Pre: Analyzed in 8.29s at 122.8X speed.
Post: Analyzed in 8.36s at 121.8X speed.
Commandlet runtime is comparable (I didn't measure, less sensitive than Insights), and memory pressure remains more or less steady in the low teens of gigabytes.
### Testing
A/B testing and spot checking the data shows that nearly all data is either more correct or withing many-decimal-places rounding error.
UnrealInsights and Editor builds without error in NoPCH, DisableUnity configurations
### Hashtags
#rb ionut.matasaru
#jira UE-120810
#preflight 62bc9d717f31fc8c31fd6d18
#robomerge EngineMerge
[CL 20893535 by Francis Hurteau in ue5-main branch]
Adds runtime events and analysis for generic meta data scopes. Any scope emitted on the "Metadata" writer will be analyzed and accessible in the metadata provider. Currently one built-in metadata type is provided: "Asset", which mimics LLMs implementation of asset tracking.
#rb ionut.matasaru
#robomerge EngineMerge
#preflight 62b1b35d827ccccb2cd7da0d
[CL 20755918 by Johan Berg in ue5-main branch]
- "TraceAnalysis" -- captures all mem allocated by the FAnalysisProcessor.
- "Trace/CpuProfile" -- captures allocation of per-thread buffers used by the cpu profiler trace.
#jira UE-147172 (partially)
#rb Johan.Berg
#preflight 6281fc9fd819c085e0ad0fe5
[CL 20222034 by ionut matasaru in ue5-main branch]
Adds a new field type for trace events which contains a reference to a previously emitted event. In order to create a reference, the referenced event type needs to be declared with a new "definition" flag. The definition flag also defines type of id that is used for the reference, 8, 16, 32 and 64 bit values are supported.
#rb ionut.matasaru, martin.ridgers
#preflight 627393ea4e24dbee173d42c5
[CL 20055730 by Johan Berg in ue5-main branch]
Some rota items cannot make progress because their next event type is not known yet. They get moved to the end of the rota and the active count is decremented. There are two cases where the rota might be reprocessed from the beginning again; when an initial serial is set, and when the rota shows it has the next event. It is possible that on a subsequent pass through the rota, items that could not make progress previously might be able to this time. However, because the active count was left over from before they were excluded and OnData() would not do all the work it could possibly do.
All this only applies to old P2 traces.
#rb jb
#rnx
#preflight 620cd9a36202a22eeddb71ff
[CL 19014434 by Martin Ridgers in ue5-main branch]
Newer protocol traces consume as much data as they can before returning from OnData(). Older protocol support however can return will data still to process that will be continued on the next OnData() call. However, if the end of input stream has already been met the subsequent OnData() call is not made. The bug's always been present but hidden by a small input buffer.
#rb jb
#rnx
#preflight 620a5b00803d9066e67bf24d
[CL 18978509 by Martin Ridgers in ue5-main branch]
