Commit Graph

6 Commits

Author SHA1 Message Date
Unrealbot
3f463d80d4 Branch snapshot for CL 2959226
[CL 2959226 in Main branch]
2016-04-28 13:50:05 -04:00
Robert Manuszewski
1ff335d148 Copying //UE4/Dev-Core to //UE4/Main
==========================
MAJOR FEATURES + CHANGES
==========================

Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore

	Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output

Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore

	Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work

Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore

	PC builds now log engine initialization time with the load time tracker

Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore

	Optimization for FSignedArchiveReader
	 - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread

Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore

	Extra stats for PakFile group
	 - Time reading from pak handles
	 - Number of open pak handles

Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore

	Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster

Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Close disregard for GC pool when disabling it to leave it in a good state.

Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core

	FlushCurrentThreadCache calls forwarded on from FMalloc wrappers.

Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core

	Stats for MallocBinned2.

Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash.

Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors.

Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Send thread heartbeats during potentially slow tasks.

Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Added the ability to query the name of a thread given its TLS id.

Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core

	MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2.
	Switch back to MallocBinned by default.

Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core

	Shrunken GMallocBinned2MemSizeToIndex array.
	Improvement of some shared templated alignment-related functions.

Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore

	Refactoring of pak signing to be more compatible with patching
	 - Store signatures in a block at the end of the pak file
	 - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests.

Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	Don't compile UnrealPak with ICU

Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core

	GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking

Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core

	Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0.

Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Don't display message boxes in unattended mode when D3D device is removed.

Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore

	Disable reading of anything outside the pak file when in shipping builds

Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	A few improvements to thread heartbeat thread:
	- Don't check for heartbeat in debug builds
	- Don't spawn the thread heartbeat thread for programs
	- Ignore stalls when exiting

Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Fixing race conditions when using malloc verify

Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q.

Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core

	Fix for TQueue::Peek const-correctness.

Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Adding code to test signature keys.

Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	TBigInt Multiply is now 30x faster

Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core

	New FCString::Strspn and FCString::Strcspn functions.

[CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
Robert Manuszewski
e04fcc7c65 Copying //UE4/Dev-Core to //UE4/Main
==========================
MAJOR FEATURES + CHANGES
==========================

Change 2855309 on 2016/02/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	GC Cluster improvements:

	- Moved cluster creation from PostLoad to after the entire package has been loaded to make sure all referenced objects are accounted for
	- Added better support for objects that can't be added to a cluster for some reason (lazy loaded referenced etc)
	- Added -gcnoclusters option to 'obj list' to list all GC objects excluding objects in clusters
	- Reference chain search will now print info on clusters

Change 2854937 on 2016/02/03 by Steve.Robb@Dev-Core

	Pool headers now contain the small block size and lookups are done using that.
	Size->table array is now one-based indexed to allow allocations of the maximum block size (32768) to be binned as expected.
	Misc refactors.

Change 2851056 on 2016/02/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	Disregard for GC improvements: added the ability to re-open disregard for GC set at any time and keep adding objects to it.

	+ Maybe fixed startup packages loading code: load startup packages AFTER initial moduels have been loaded, don't use the seek free flag.

Change 2848788 on 2016/01/29 by Steve.Robb@Dev-Core

	ALIGNOF now supports abstract types.

Change 2852323 on 2016/02/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	GC Clusters: Making sure VerifyClusterAssumptions catches all cases:

	- Invalid objects in GC
	- Objects from other clusters that don't have their parent cluster referenced by the source cluster

Change 2846860 on 2016/01/28 by Chris.Wood@Chris.Wood.StreamB

	Added extra debug info to error messages sent back in the response from the AddCrash method
	[OPP-4995] - Crash reports are not appearing for Portal Main branch

Change 2846864 on 2016/01/28 by Chris.Wood@Chris.Wood.StreamB

	Added retry to AddCrash web requests due to unexplained missing payloads in some crashes
	[OPP-4995] - Crash reports are not appearing for Portal Main branch

Change 2846875 on 2016/01/28 by Richard.Fawcett@Richard.Fawcett_G5766_Main

	Ensure that streams are properly closed when sending and receiving data in SimpleWebRequest

	This is to resolve an intermittent issue whereby CrashReporterWebsite sometimes receives no payload on an HTTP POST request from CrashReportProcessor. This could be consistent with the request stream not being closed and flushed before the HTTP request is sent.  MSDN documentation for GetRequestStream states "You must call the Stream.Close method to close the stream and release the connection for reuse. Failure to close the stream causes your application to run out of connections."  Note that Close() is automatically called on a Stream during disposal, and Dispose() is automatically called when exiting the scope of a using statement.
	https://msdn.microsoft.com/en-us/library/d4cek6cc(v=vs.110).aspx

	#jira OPP-4995 - Crash reports are not appearing for Portal Main branch

Change 2847012 on 2016/01/28 by Steve.Robb@Dev-Core

	Allocations with alignment > OSPageSize are now illegal.
	Removal of trailing pool support.

Change 2847106 on 2016/01/28 by Steve.Robb@Dev-Core

	AllocateBlockFromPool inlined and redundancy removed.
	Misc tidy-ups.

Change 2849028 on 2016/01/29 by Steve.Robb@Dev-Core

	TIsClass fix for union types.

Change 2849039 on 2016/01/29 by Steve.Robb@Dev-Core

	Basic custom deleter support for TUniquePtr.

Change 2850853 on 2016/02/01 by Steven.Hutton@steven.hutton-CrashReporter

	Added per branch filtering for Reports output page

Change 2851035 on 2016/02/01 by Chris.Wood@Chris.Wood.StreamB

	Generating Platform and Platform Variant strings in the Crash Report Process to pass to the MDD
	[OPP-4792] - Crash reports not appearing for launcher on Windows 32-bit OS

Change 2853124 on 2016/02/02 by Joe.Graf@joe.graf_T6736_Dev-Core

	Changed crash reporter to respect command line parameter filtering when that is enabled via the project define (UECORE-318)

Change 2853960 on 2016/02/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Limiting the size of UObject Array for programs.

Change 2855582 on 2016/02/04 by Steve.Robb@Dev-Core

	Revert to MallocBinned for non-Windows and non-PS4 platforms, as it requires a 64K page size.

Change 2856837 on 2016/02/05 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	Changing FUObjectItem::ThisThreadAtomicallyClearedFlag to better support changes to multiple different flags from different threads. Made sure NoStrongReference is also cleared atomically.

Change 2857113 on 2016/02/05 by Steve.Robb@Dev-Core

	Exhausted check moved out of Allocate function.

Change 2858600 on 2016/02/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Preventing async loading suspended log spam in the editor.

	#jira UE-26486

Change 2858700 on 2016/02/08 by Steve.Robb@Dev-Core

	Code for short block support.

Change 2859112 on 2016/02/08 by Steve.Robb@Dev-Core

	Array of pool table pointers changed to array of indices.

Change 2859138 on 2016/02/08 by Steve.Robb@Dev-Core

	Index map moved into a global.

#lockdown Nick.Penwarden

[CL 2860061 by Robert Manuszewski in Main branch]
2016-02-09 09:44:54 -05:00
Robert Manuszewski
18e2561ceb Copying //UE4/Dev-Core to //UE4/Main
==========================
MAJOR FEATURES + CHANGES
==========================

Change 2836261 on 2016/01/20 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Flush FAsyncPackage cache after pre-load to reduce peak memory usage when async loading (5.5-10x).

Change 2828630 on 2016/01/14 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Adding debug code to catch memory stomps in the async loading buffers that's independent from mallocstomp allocator. Changed the signature of PageProtect functions to be able to read-only protect memory.

Change 2816129 on 2016/01/05 by Steve.Robb@Dev-Core

	Fixes for Realloc and alignment logic which caused redundant reallocations and incorrect binning.

Change 2821054 on 2016/01/08 by Steve.Robb@Dev-Core

	Further Realloc savings when realigning within a block.

Change 2806820 on 2015/12/17 by Steve.Robb@Dev-Core

	New AlignDown function, like Align, but which rounds a value/pointer down to the next alignment instead of up.

Change 2806816 on 2015/12/17 by Steve.Robb@Dev-Core

	Sort UHT modules by type to improve iteration times in conjunction with makefiles.

Change 2823235 on 2016/01/11 by Steve.Robb@Dev-Core

	UHT error messages about missing GENERATED_BODY() macros updated to represent intended use.

Change 2806815 on 2015/12/17 by Steve.Robb@Dev-Core

	Module types split into Game and Engine runtime versions.

Change 2833809 on 2016/01/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	StaticLoadObject will now respect LOAD_NoRedirects flag.

Change 2811194 on 2015/12/22 by Bob.Tellez@Z2434_DevCore

	#UE4 Prevent loading packages that have a newer LegacyFileVersion since serialization for FCustomVersion is not forward compatible. UE-24443

Change 2806818 on 2015/12/17 by Steve.Robb@Dev-Core

	Removal of stats from MallocBinned2, to be readded later.

Change 2807069 on 2015/12/17 by Steve.Robb@Dev-Core

	Clarification of some bucket hashing terminology.

Change 2815117 on 2016/01/04 by Steve.Robb@Dev-Core

	Fix for a missing root build path on game modules.

Change 2815673 on 2016/01/05 by Steve.Robb@Dev-Core

	Move FMalloc verification into a proxy object.

Change 2822873 on 2016/01/11 by Steve.Robb@Dev-Core

	Fixes to off-by-one errors and removal of BinnedSizeLimit (assumed to be the same as MAX_POOLED_ALLOCATION_SIZE after OBO fix).

Change 2822923 on 2016/01/11 by Steve.Robb@Dev-Core

	Simplification of MemSizeToPoolTable indexing.

Change 2824974 on 2016/01/12 by Steve.Robb@Dev-Core

	Assert fixed.
	AllocateBlockFromPool's return value made debuggable.

Change 2825241 on 2016/01/12 by Steve.Robb@Dev-Core

	UHT now returns an error code on a warning when -warningsaserrors is specified.

Change 2825291 on 2016/01/12 by Steve.Robb@Dev-Core

	WarningsAsErrors enabled on UHT, after disabling the hardcoded behavior in CL# 2825241.

Change 2829846 on 2016/01/15 by Steve.Robb@Dev-Core

	GitHub #1938 - wrong Max value of enum is used during net serialization

Change 2829914 on 2016/01/15 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Reduce the amount of memory allocated for async cache buffers when guarding against memory stomps.

Change 2829988 on 2016/01/15 by Steve.Robb@Dev-Core

	Generalized large pool allocations.
	More redundancy removed.

Change 2831935 on 2016/01/18 by Chris.Wood@Chris.Wood.StreamB

	Added UserActivity property to crash description in CRP and CR website.
	[OR-12043] - Phone Home where crashes occur - pass context info to Crash Reporter

	DB column added to db-09 by ColinR matching this change.
	Published to server on Jan 18th 2016

Change 2834003 on 2016/01/19 by Chris.Wood@Chris.Wood.StreamB

	Added Linux to normal callstack parsing code on CR website
	[UE-25527] - Linux CrashReporter is missing information

	Published to server on Jan 19th 2016

Change 2835466 on 2016/01/20 by Joe.Conley@Joe.Conley_EGJWD5708_Dev-Core-Minimal

	Fix issue for cancelling package loads when there are still packages queued.  Call their PackageLoadedDelegate with a "Cancelled" result.

	Should solve remaining issue with UE-24062 - "Calling CancelAsyncLoading triggers an assert in FAsyncPackage::DetachLinker()"
	- (ULevelStreaming::AsyncLevelLoadComplete was not being called if packages were still queued when cancel was issued)

Change 2836803 on 2016/01/20 by Chris.Wood@Chris.Wood.StreamB

	CrashReportWebsite - fix exception thrown when parsing certain callstack formats

Change 2837952 on 2016/01/21 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	Changing FAsyncIORequest to be stored as reference when cancelling IO requests to improve performance.

Change 2838289 on 2016/01/21 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

[CL 2845588 by Robert Manuszewski in Main branch]
2016-01-27 12:09:53 -05:00
Matthew Griffin
bb70b349ce Merging CL 2804086 from //UE4/Release-4.11 to Dev-Main (//UE4/Dev-Main) to isolate copyright update
#lockdown Nick.Penwarden

[CL 2819020 by Matthew Griffin in Main branch]
2016-01-07 08:17:16 -05:00
Robert Manuszewski
dfc39a6ca5 Copying //UE4/Dev-Core to //UE4/Main
==========================
MAJOR FEATURES + CHANGES
==========================

Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures).

	- Saves about 25ms in reachability analysis (58ms -> 33ms)
	- UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override.
	- Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them.
	- Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code
	- Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms)

Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core

	Multicast script delegate check for existing bindings replaced with ensure.
	Multicast native delegate no longer checks for existing bindings.
	Removal of old delegate code.
	Some FORCEINLINEing to improve debugging experience of stepping into delegate code.

Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster

	Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded
	Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function
	Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds.
	Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code

Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster

	Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist.

Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec

	CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII

Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core

	FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected.

Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec

	Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster

Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	Fixing potential dead lock when suspending and resuming async loading multiple times

Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2

	Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef

Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core

	Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path.
	New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs.

Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core

	FMath::IsPowerOfTwo is now templated to take any type.

Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core

	GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself.

Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core

	GitHub #1793 - File write flags argument

Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core

	Fix for FArchiveProxy::operator<< overloads.

Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec

	#jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams)

Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec

	Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes

	#jira UECORE-170 - Improve profiler loading performance (wip)

Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec

	Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism,  should be around 2x times faster (4x since the optimization pass)

	#jira UECORE-170 - Improve profiler loading performance (wip)

Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core

	TCachedOSPageAllocator abstracted from MallocBinned2.
	Misc tidy-ups.

Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec

	Profiler - Better indication of the loading progress, should no longer freeze without a progress bar

	#jira UECORE-170 - Improve profiler loading performance (wip)

Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core

	Warn when calling delegates' Create* functions when they're not assigned to anything.

	#codereview robert.manuszewski

Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	#UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values.

	Asset registry fixes by Bob Tellez. Possible fix for UE-23783.

Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1

	FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing.

Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core

	UObject serial number array initialized for debug visualization.

Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core

	Critical sections are no longer copyable.

Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00