Files
UnrealEngineUWP/Engine/Source/Programs/CrashReporter/CompressionHelper/Compression.cpp

99 lines
3.9 KiB
C++
Raw Normal View History

// Copyright 1998-2017 Epic Games, Inc. All Rights Reserved.
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3228803) #lockdown Nick.Penwarden #rb none ========================== MAJOR FEATURES + CHANGES ========================== Change 3209807 on 2016/11/24 by Chris.Wood CRP now has improved handling of lost comms with website/DB (CRP v.1.2.9) [UE-38397] - CrashReportProcess should not lose crashes when the website goes down. Change 3209936 on 2016/11/24 by Steven.Hutton Reconciled offline work. Fixed a number of display bugs with the reports page. Change 3209938 on 2016/11/24 by Steven.Hutton Reconciled offline work Adding JQuery UI packages Change 3210736 on 2016/11/28 by Steve.Robb Unset() made protected, which is an implementation details of TFunction and not the way to unbind one (may leak memory). Fixes to existing calls. https://answers.unrealengine.com/questions/494947/proper-way-to-deal-with-destroying-a-tfunction.html Change 3211181 on 2016/11/28 by Steve.Robb Improved error message when binding a delegate to a pending kill object. #jira UE-5232 Change 3211214 on 2016/11/28 by Steve.Robb PR #2978: fixed typo in FMallocLeakDetection (Contributed by finap) #jira UE-39049 Change 3211301 on 2016/11/28 by Steve.Robb PR #2892: Copy bCustomVersionsAreReset when creating an Archive from another Ar. (Contributed by surakin) #jira UE-37941 Change 3213387 on 2016/11/29 by Steven.Hutton Test of a release note parsed from the changelist Change 3213553 on 2016/11/29 by Gil.Gribb UE4 - Rework dependencies and UStruct creation so that we can create UBlueprintGeneratedClass's without needing the parent class serialized. Change 3214800 on 2016/11/30 by Robert.Manuszewski ModuleManager will now use a critical section instead of FMultiReaderSingleWriterGT to make it really thread safe. - removed FMultiReaderSingleWriterGT because it was not fully thread safe Change 3214926 on 2016/11/30 by Robert.Manuszewski Merging using Dev-Core_To_Dev-LoadTimes (reversed) Change 3214981 on 2016/11/30 by Gil.Gribb UE4 - Make sure that subobjects of CDOs are exported even if they don't have any direct references; they might be archetypes. Change 3215392 on 2016/11/30 by Steve.Robb Error out on editor builds when reading a non-boolean value in an archive. Change 3215674 on 2016/11/30 by Steve.Robb Replacement of a custom scope-exit setup with our standard macro. Change 3215720 on 2016/11/30 by Steve.Robb Default constructor for TTuple which value-initializes the elements. Change 3216777 on 2016/12/01 by Graeme.Thornton Add NoRedist and NotForLicensees folders in the game folder to the C# version of config file hierarchy. Change 3216858 on 2016/12/01 by Graeme.Thornton Improvements to pak signing and encryption - Remove compile time defines for encryption key and signing keys - Embed keys in executable by including in the UBT generated UELinkerFixups.cpp file - Add key access core delegate for passing these keys through to pak platform file Change 3216860 on 2016/12/01 by Graeme.Thornton Re-enable pak signing and encryption in ShooterGame Change 3216861 on 2016/12/01 by Graeme.Thornton Re-enable pak signing and encryption in Paragon Change 3217076 on 2016/12/01 by Gil.Gribb UE4 - replaced !GIsIntialLoad with EVENT_DRIVEN_ASYNC_LOAD_ACTIVE_AT_RUNTIME as a first step toward boot time EDL Change 3221139 on 2016/12/05 by Ben.Woodhouse Dummy integrate of CL 3221131 (versioning workarounds) from Dev-LoadTimes4.14 with accept target to prevent those workarounds being merged out of dev-loadtimes4.14. Command: p4 integrate //UE4/Dev-LoadTimes4.14/...@3221131,3221131 w/ resolve/accept-target Change 3221410 on 2016/12/05 by Steve.Robb Allow Algo::IsSorted to work on C arrays and initializer_lists. Change 3221673 on 2016/12/05 by Gil.Gribb set up an intentional merge conflict so we can do the right thing when dev-BP and dev-core come together Change 3223182 on 2016/12/06 by Chris.Wood Add EngineModeEx to CrashReportCommon [UE-39258] - Update CrashReporter to include whether Engine crashes are from vanilla build or not Change 3224646 on 2016/12/07 by Robert.Manuszewski It's no longer necessary to rebuild the cooked exe to enable the event driven loader. EDL and the new async IO are now controlled with ini settings only. Change 3224647 on 2016/12/07 by Robert.Manuszewski Event Driven Loader is now enabled by default. Change 3224669 on 2016/12/07 by Chris.Wood Compressing some crash files output from CRP to S3 (CRP v1.2.11) [UE-37564] - Add compression support for CRP crash files in S3 Add EngineModeEx to support to CRP (CRP v1.2.10) [UE-39258] - Update CrashReporter to include whether Engine crashes are from vanilla build or not Change 3224873 on 2016/12/07 by Robert.Manuszewski A few fixes to the EDL macro refactor. Change 3224933 on 2016/12/07 by Robert.Manuszewski Async Loading settings will now be logged on startup in cooked games Change 3224984 on 2016/12/07 by Robert.Manuszewski Small fix to EDL and new async IO global var init. Change 3225027 on 2016/12/07 by Robert.Manuszewski Re-enabling EDL in ShooterGame since it no longer requires the cooked exe to be re-compiled (and EDL is on by default anyway). Change 3226702 on 2016/12/08 by Steve.Robb UHT can now skip over macro-like identifiers while parsing. Correctly handles FTypefaceEntry::Name parsing, which was broken by the workaround in CL# 3219332. Syntax workaround reverted in FTypefaceEntry. #jira UE-38231 Change 3226756 on 2016/12/08 by Steve.Robb LexicalConversion renamed to Lex after #core discussion. Lex::ToString overloads added for FString itself. Change 3226766 on 2016/12/08 by Steve.Robb FCookStatsManager::ToString replaced with Lex::ToString, as it now supports the required functionality. Change 3226949 on 2016/12/08 by Robert.Manuszewski Fixing static analysis warnings Change 3228566 on 2016/12/09 by Robert.Manuszewski Making UBT not think it needs to use separate temporary target for non source code projects when EDL settings don't match default. Change 3228601 on 2016/12/09 by Steve.Robb Fix for TSparseArray visualization of types smaller than 8 bytes. Change 3228654 on 2016/12/09 by Gil.Gribb UE4 - Fix very old bug with lock free lists, probably not relevant to anything we currently use. Change 3228697 on 2016/12/09 by Graeme.Thornton Rebuilt UnrealPak [CL 3228932 by Robert Manuszewski in Main branch]
2016-12-09 11:36:14 -05:00
#include "../../../ThirdParty/zlib/v1.2.8/include/Win64/VS2013/zlib.h"
#pragma comment( lib, "../../../ThirdParty/zlib/v1.2.8/lib/Win64/VS2013/zlibstatic.lib" )
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
// Taken from GenericPlatform.h
// Unsigned base types.
typedef unsigned char uint8; // 8-bit unsigned.
typedef unsigned short int uint16; // 16-bit unsigned.
typedef unsigned int uint32; // 32-bit unsigned.
typedef unsigned long long uint64; // 64-bit unsigned.
// Signed base types.
typedef signed char int8; // 8-bit signed.
typedef signed short int int16; // 16-bit signed.
typedef signed int int32; // 32-bit signed.
typedef signed long long int64; // 64-bit signed.
// Taken from Compression.cpp
extern "C"
{
Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3228803) #lockdown Nick.Penwarden #rb none ========================== MAJOR FEATURES + CHANGES ========================== Change 3209807 on 2016/11/24 by Chris.Wood CRP now has improved handling of lost comms with website/DB (CRP v.1.2.9) [UE-38397] - CrashReportProcess should not lose crashes when the website goes down. Change 3209936 on 2016/11/24 by Steven.Hutton Reconciled offline work. Fixed a number of display bugs with the reports page. Change 3209938 on 2016/11/24 by Steven.Hutton Reconciled offline work Adding JQuery UI packages Change 3210736 on 2016/11/28 by Steve.Robb Unset() made protected, which is an implementation details of TFunction and not the way to unbind one (may leak memory). Fixes to existing calls. https://answers.unrealengine.com/questions/494947/proper-way-to-deal-with-destroying-a-tfunction.html Change 3211181 on 2016/11/28 by Steve.Robb Improved error message when binding a delegate to a pending kill object. #jira UE-5232 Change 3211214 on 2016/11/28 by Steve.Robb PR #2978: fixed typo in FMallocLeakDetection (Contributed by finap) #jira UE-39049 Change 3211301 on 2016/11/28 by Steve.Robb PR #2892: Copy bCustomVersionsAreReset when creating an Archive from another Ar. (Contributed by surakin) #jira UE-37941 Change 3213387 on 2016/11/29 by Steven.Hutton Test of a release note parsed from the changelist Change 3213553 on 2016/11/29 by Gil.Gribb UE4 - Rework dependencies and UStruct creation so that we can create UBlueprintGeneratedClass's without needing the parent class serialized. Change 3214800 on 2016/11/30 by Robert.Manuszewski ModuleManager will now use a critical section instead of FMultiReaderSingleWriterGT to make it really thread safe. - removed FMultiReaderSingleWriterGT because it was not fully thread safe Change 3214926 on 2016/11/30 by Robert.Manuszewski Merging using Dev-Core_To_Dev-LoadTimes (reversed) Change 3214981 on 2016/11/30 by Gil.Gribb UE4 - Make sure that subobjects of CDOs are exported even if they don't have any direct references; they might be archetypes. Change 3215392 on 2016/11/30 by Steve.Robb Error out on editor builds when reading a non-boolean value in an archive. Change 3215674 on 2016/11/30 by Steve.Robb Replacement of a custom scope-exit setup with our standard macro. Change 3215720 on 2016/11/30 by Steve.Robb Default constructor for TTuple which value-initializes the elements. Change 3216777 on 2016/12/01 by Graeme.Thornton Add NoRedist and NotForLicensees folders in the game folder to the C# version of config file hierarchy. Change 3216858 on 2016/12/01 by Graeme.Thornton Improvements to pak signing and encryption - Remove compile time defines for encryption key and signing keys - Embed keys in executable by including in the UBT generated UELinkerFixups.cpp file - Add key access core delegate for passing these keys through to pak platform file Change 3216860 on 2016/12/01 by Graeme.Thornton Re-enable pak signing and encryption in ShooterGame Change 3216861 on 2016/12/01 by Graeme.Thornton Re-enable pak signing and encryption in Paragon Change 3217076 on 2016/12/01 by Gil.Gribb UE4 - replaced !GIsIntialLoad with EVENT_DRIVEN_ASYNC_LOAD_ACTIVE_AT_RUNTIME as a first step toward boot time EDL Change 3221139 on 2016/12/05 by Ben.Woodhouse Dummy integrate of CL 3221131 (versioning workarounds) from Dev-LoadTimes4.14 with accept target to prevent those workarounds being merged out of dev-loadtimes4.14. Command: p4 integrate //UE4/Dev-LoadTimes4.14/...@3221131,3221131 w/ resolve/accept-target Change 3221410 on 2016/12/05 by Steve.Robb Allow Algo::IsSorted to work on C arrays and initializer_lists. Change 3221673 on 2016/12/05 by Gil.Gribb set up an intentional merge conflict so we can do the right thing when dev-BP and dev-core come together Change 3223182 on 2016/12/06 by Chris.Wood Add EngineModeEx to CrashReportCommon [UE-39258] - Update CrashReporter to include whether Engine crashes are from vanilla build or not Change 3224646 on 2016/12/07 by Robert.Manuszewski It's no longer necessary to rebuild the cooked exe to enable the event driven loader. EDL and the new async IO are now controlled with ini settings only. Change 3224647 on 2016/12/07 by Robert.Manuszewski Event Driven Loader is now enabled by default. Change 3224669 on 2016/12/07 by Chris.Wood Compressing some crash files output from CRP to S3 (CRP v1.2.11) [UE-37564] - Add compression support for CRP crash files in S3 Add EngineModeEx to support to CRP (CRP v1.2.10) [UE-39258] - Update CrashReporter to include whether Engine crashes are from vanilla build or not Change 3224873 on 2016/12/07 by Robert.Manuszewski A few fixes to the EDL macro refactor. Change 3224933 on 2016/12/07 by Robert.Manuszewski Async Loading settings will now be logged on startup in cooked games Change 3224984 on 2016/12/07 by Robert.Manuszewski Small fix to EDL and new async IO global var init. Change 3225027 on 2016/12/07 by Robert.Manuszewski Re-enabling EDL in ShooterGame since it no longer requires the cooked exe to be re-compiled (and EDL is on by default anyway). Change 3226702 on 2016/12/08 by Steve.Robb UHT can now skip over macro-like identifiers while parsing. Correctly handles FTypefaceEntry::Name parsing, which was broken by the workaround in CL# 3219332. Syntax workaround reverted in FTypefaceEntry. #jira UE-38231 Change 3226756 on 2016/12/08 by Steve.Robb LexicalConversion renamed to Lex after #core discussion. Lex::ToString overloads added for FString itself. Change 3226766 on 2016/12/08 by Steve.Robb FCookStatsManager::ToString replaced with Lex::ToString, as it now supports the required functionality. Change 3226949 on 2016/12/08 by Robert.Manuszewski Fixing static analysis warnings Change 3228566 on 2016/12/09 by Robert.Manuszewski Making UBT not think it needs to use separate temporary target for non source code projects when EDL settings don't match default. Change 3228601 on 2016/12/09 by Steve.Robb Fix for TSparseArray visualization of types smaller than 8 bytes. Change 3228654 on 2016/12/09 by Gil.Gribb UE4 - Fix very old bug with lock free lists, probably not relevant to anything we currently use. Change 3228697 on 2016/12/09 by Graeme.Thornton Rebuilt UnrealPak [CL 3228932 by Robert Manuszewski in Main branch]
2016-12-09 11:36:14 -05:00
/**
* Thread-safe abstract decompression routine. Decompresses memory from compressed buffer and writes it to uncompressed
* buffer. Returns actual uncompressed data size or a negative error code.
*
* @param UncompressedBuffer Buffer containing uncompressed data
* @param UncompressedSize Size of uncompressed data in bytes
* @param CompressedBuffer Buffer compressed data is going to be read from
* @param CompressedSize Size of CompressedBuffer data in bytes
* @return Less than zero values are error codes if the uncompress fails, greater than zero is the uncompressed size in bytes
*/
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
__declspec(dllexport)
int32 __cdecl UE4UncompressMemoryZLIB(void* UncompressedBuffer, int32 UncompressedSize, const void* CompressedBuffer, int32 CompressedSize)
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
{
// Zlib wants to use unsigned long.
unsigned long ZCompressedSize = CompressedSize;
unsigned long ZUncompressedSize = UncompressedSize;
// Uncompress data.
int32 Result = uncompress((uint8*)UncompressedBuffer, &ZUncompressedSize, (const uint8*)CompressedBuffer, ZCompressedSize);
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
return (Result == Z_OK) ? (int32)ZUncompressedSize : Result;
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
}
Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3228803) #lockdown Nick.Penwarden #rb none ========================== MAJOR FEATURES + CHANGES ========================== Change 3209807 on 2016/11/24 by Chris.Wood CRP now has improved handling of lost comms with website/DB (CRP v.1.2.9) [UE-38397] - CrashReportProcess should not lose crashes when the website goes down. Change 3209936 on 2016/11/24 by Steven.Hutton Reconciled offline work. Fixed a number of display bugs with the reports page. Change 3209938 on 2016/11/24 by Steven.Hutton Reconciled offline work Adding JQuery UI packages Change 3210736 on 2016/11/28 by Steve.Robb Unset() made protected, which is an implementation details of TFunction and not the way to unbind one (may leak memory). Fixes to existing calls. https://answers.unrealengine.com/questions/494947/proper-way-to-deal-with-destroying-a-tfunction.html Change 3211181 on 2016/11/28 by Steve.Robb Improved error message when binding a delegate to a pending kill object. #jira UE-5232 Change 3211214 on 2016/11/28 by Steve.Robb PR #2978: fixed typo in FMallocLeakDetection (Contributed by finap) #jira UE-39049 Change 3211301 on 2016/11/28 by Steve.Robb PR #2892: Copy bCustomVersionsAreReset when creating an Archive from another Ar. (Contributed by surakin) #jira UE-37941 Change 3213387 on 2016/11/29 by Steven.Hutton Test of a release note parsed from the changelist Change 3213553 on 2016/11/29 by Gil.Gribb UE4 - Rework dependencies and UStruct creation so that we can create UBlueprintGeneratedClass's without needing the parent class serialized. Change 3214800 on 2016/11/30 by Robert.Manuszewski ModuleManager will now use a critical section instead of FMultiReaderSingleWriterGT to make it really thread safe. - removed FMultiReaderSingleWriterGT because it was not fully thread safe Change 3214926 on 2016/11/30 by Robert.Manuszewski Merging using Dev-Core_To_Dev-LoadTimes (reversed) Change 3214981 on 2016/11/30 by Gil.Gribb UE4 - Make sure that subobjects of CDOs are exported even if they don't have any direct references; they might be archetypes. Change 3215392 on 2016/11/30 by Steve.Robb Error out on editor builds when reading a non-boolean value in an archive. Change 3215674 on 2016/11/30 by Steve.Robb Replacement of a custom scope-exit setup with our standard macro. Change 3215720 on 2016/11/30 by Steve.Robb Default constructor for TTuple which value-initializes the elements. Change 3216777 on 2016/12/01 by Graeme.Thornton Add NoRedist and NotForLicensees folders in the game folder to the C# version of config file hierarchy. Change 3216858 on 2016/12/01 by Graeme.Thornton Improvements to pak signing and encryption - Remove compile time defines for encryption key and signing keys - Embed keys in executable by including in the UBT generated UELinkerFixups.cpp file - Add key access core delegate for passing these keys through to pak platform file Change 3216860 on 2016/12/01 by Graeme.Thornton Re-enable pak signing and encryption in ShooterGame Change 3216861 on 2016/12/01 by Graeme.Thornton Re-enable pak signing and encryption in Paragon Change 3217076 on 2016/12/01 by Gil.Gribb UE4 - replaced !GIsIntialLoad with EVENT_DRIVEN_ASYNC_LOAD_ACTIVE_AT_RUNTIME as a first step toward boot time EDL Change 3221139 on 2016/12/05 by Ben.Woodhouse Dummy integrate of CL 3221131 (versioning workarounds) from Dev-LoadTimes4.14 with accept target to prevent those workarounds being merged out of dev-loadtimes4.14. Command: p4 integrate //UE4/Dev-LoadTimes4.14/...@3221131,3221131 w/ resolve/accept-target Change 3221410 on 2016/12/05 by Steve.Robb Allow Algo::IsSorted to work on C arrays and initializer_lists. Change 3221673 on 2016/12/05 by Gil.Gribb set up an intentional merge conflict so we can do the right thing when dev-BP and dev-core come together Change 3223182 on 2016/12/06 by Chris.Wood Add EngineModeEx to CrashReportCommon [UE-39258] - Update CrashReporter to include whether Engine crashes are from vanilla build or not Change 3224646 on 2016/12/07 by Robert.Manuszewski It's no longer necessary to rebuild the cooked exe to enable the event driven loader. EDL and the new async IO are now controlled with ini settings only. Change 3224647 on 2016/12/07 by Robert.Manuszewski Event Driven Loader is now enabled by default. Change 3224669 on 2016/12/07 by Chris.Wood Compressing some crash files output from CRP to S3 (CRP v1.2.11) [UE-37564] - Add compression support for CRP crash files in S3 Add EngineModeEx to support to CRP (CRP v1.2.10) [UE-39258] - Update CrashReporter to include whether Engine crashes are from vanilla build or not Change 3224873 on 2016/12/07 by Robert.Manuszewski A few fixes to the EDL macro refactor. Change 3224933 on 2016/12/07 by Robert.Manuszewski Async Loading settings will now be logged on startup in cooked games Change 3224984 on 2016/12/07 by Robert.Manuszewski Small fix to EDL and new async IO global var init. Change 3225027 on 2016/12/07 by Robert.Manuszewski Re-enabling EDL in ShooterGame since it no longer requires the cooked exe to be re-compiled (and EDL is on by default anyway). Change 3226702 on 2016/12/08 by Steve.Robb UHT can now skip over macro-like identifiers while parsing. Correctly handles FTypefaceEntry::Name parsing, which was broken by the workaround in CL# 3219332. Syntax workaround reverted in FTypefaceEntry. #jira UE-38231 Change 3226756 on 2016/12/08 by Steve.Robb LexicalConversion renamed to Lex after #core discussion. Lex::ToString overloads added for FString itself. Change 3226766 on 2016/12/08 by Steve.Robb FCookStatsManager::ToString replaced with Lex::ToString, as it now supports the required functionality. Change 3226949 on 2016/12/08 by Robert.Manuszewski Fixing static analysis warnings Change 3228566 on 2016/12/09 by Robert.Manuszewski Making UBT not think it needs to use separate temporary target for non source code projects when EDL settings don't match default. Change 3228601 on 2016/12/09 by Steve.Robb Fix for TSparseArray visualization of types smaller than 8 bytes. Change 3228654 on 2016/12/09 by Gil.Gribb UE4 - Fix very old bug with lock free lists, probably not relevant to anything we currently use. Change 3228697 on 2016/12/09 by Graeme.Thornton Rebuilt UnrealPak [CL 3228932 by Robert Manuszewski in Main branch]
2016-12-09 11:36:14 -05:00
/**
* Thread-safe abstract compression routine. Compresses memory from uncompressed buffer and writes it to compressed
* buffer. Returns actual compressed data size or a negative error code.
*
* @param CompressedBuffer Buffer compressed data
* @param CompressedSize Size of CompressedBuffer data in bytes
* @param UncompressedBuffer Buffer containing uncompressed data is going to be read from
* @param UncompressedSize Size of uncompressed data in bytes
* @return Less than zero values are error codes if the compress fails, greater than zero is the compressed size in bytes
*/
__declspec(dllexport)
int32 __cdecl UE4CompressMemoryZLIB(void* CompressedBuffer, int32 CompressedSize, const void* UncompressedBuffer, int32 UncompressedSize)
{
// Zlib wants to use unsigned long.
unsigned long ZCompressedSize = CompressedSize;
unsigned long ZUncompressedSize = UncompressedSize;
// Uncompress data.
int32 Result = compress2((uint8*)CompressedBuffer, &ZCompressedSize, (const uint8*)UncompressedBuffer, ZUncompressedSize, Z_DEFAULT_COMPRESSION);
return (Result == Z_OK) ? (int32)ZCompressedSize : Result;
}
/**
* Thread-safe file compression routine. Compresses memory from uncompressed buffer and writes it to compressed
* gzip file.
*
* @param Path File path to write a new compressed gzip file
* @param UncompressedBuffer Buffer containing uncompressed data is going to be read from
* @param UncompressedSize Size of uncompressed data in bytes
* @return Less than zero values are error codes if the compress fails, Z_OK if succeeded.
*/
__declspec(dllexport)
int32 __cdecl UE4CompressFileGZIP(const char* Path, const void* UncompressedBuffer, int32 UncompressedSize)
{
// Zlib wants to use unsigned long.
unsigned ZUncompressedSize = UncompressedSize;
gzFile FilePtr = gzopen(Path, "wb");
if (FilePtr == nullptr)
{
return Z_ERRNO;
}
int ZCompressedSize = gzwrite(FilePtr, UncompressedBuffer, ZUncompressedSize);
if (ZCompressedSize == 0)
{
return Z_ERRNO;
}
return gzclose(FilePtr);
}
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2783106 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Introduced GC UObject clusters. GC clusters provide means to create disregard for GC subsets at load time (e.g. Materials with material expressions and their textures). - Saves about 25ms in reachability analysis (58ms -> 33ms) - UObject classes/instances can now be marked as cluster root objects with CanBeClusterRoot() function override. - Cluster creation is automatic. Clusters don't require any manual handling for GC to collect them when nothing is referencing them. - Moved token stream processing to a new class FFastReferenceFinder to make it more generic and re-usable by other code - Removed REFERENCE_INFO macro from GC code and replaced it with a local variable (saves about ~1.9ms: 33.2ms -> 31.3ms) Change 2773094 on 2015/11/19 by Steve.Robb@Dev-Core Multicast script delegate check for existing bindings replaced with ensure. Multicast native delegate no longer checks for existing bindings. Removal of old delegate code. Some FORCEINLINEing to improve debugging experience of stepping into delegate code. Change 2782180 on 2015/11/27 by Graeme.Thornton@GThornton_DesktopMaster Make scoped seconds timer class available outside of stats build. Normal usage macros still remain guarded Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE which only times during the outmost instance of a recursive function Added SCOPE_SECONDS_COUNTER_RECURSION_SAFE_BASE and SCOPE_SECONDS_COUNTER_BASE which are defined in all build types, for easy temporary timing in Test/Shipping builds. Added a boolean parameter to the timer class which can be used to disable it without having to mess around with scoping the calling code Change 2782635 on 2015/11/30 by Graeme.Thornton@GThornton_DesktopMaster Added GetTimeStampPair() to the filemanager and platformfile interfaces. Requests timestamps for a pair of files where we assume that both files would always exist at the same wrapper level. Allows us to skip file system queries for localization package lookups where the native file is in a pak but the localized file doesn't exist. Change 2775153 on 2015/11/20 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec CrashReportServer moved out of the not for licencees, a few fixes, removed RegisterPII Change 2775560 on 2015/11/20 by Steve.Robb@Dev-Core FDelegateBase::GetDelegateInstance deprecated and replaced with FDelegateBase::GetDelegateInstanceProtected. Change 2781138 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Converted is using a new stats reader, a few more optimizations, should be 10x times faster Change 2772990 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Fixing potential dead lock when suspending and resuming async loading multiple times Change 2773023 on 2015/11/19 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Support for references added through AddReferencedObjects in FArchiveReplaceObjectRef Change 2781055 on 2015/11/25 by Steve.Robb@Dev-Core Changes to IDelegateInstance reverted to allow licensees an easier time when upgrading their use of the now-deprecated GetDelegateInstance() code path. New TryGetBoundFunctionName() to aid the debugging of delegate bindings in non-shipping configs. Change 2773114 on 2015/11/19 by Steve.Robb@Dev-Core FMath::IsPowerOfTwo is now templated to take any type. Change 2773643 on 2015/11/19 by Steve.Robb@Dev-Core GetDelegateInstance() calls replaced - delegate instances never compare equal unless you are comparing two unbound delegates (both null) or comparing a delegate with itself. Change 2777686 on 2015/11/23 by Steve.Robb@Dev-Core GitHub #1793 - File write flags argument Change 2780590 on 2015/11/25 by Steve.Robb@Dev-Core Fix for FArchiveProxy::operator<< overloads. Change 2780845 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec #jira UE-23358 - MDD relies on hard-coded P4 depot paths (fixed source context for streams) Change 2780962 on 2015/11/25 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Stats - Added FStatsWriteStream for basic saving stat messages into a stream, initial support for reading a regular stats file, minor performance optimization, coding standard fixes #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781887 on 2015/11/26 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler/ProfilerClient - Removed unneeded synchronization points, replaces with task graph SendTo jobs, removed PROFILER_THREADED_LOAD, replaced with new stats loading mechanism, should be around 2x times faster (4x since the optimization pass) #jira UECORE-170 - Improve profiler loading performance (wip) Change 2781893 on 2015/11/26 by Steve.Robb@Dev-Core TCachedOSPageAllocator abstracted from MallocBinned2. Misc tidy-ups. Change 2782198 on 2015/11/27 by Jaroslaw.Surowiec@Stream.1.JarekSurowiec Profiler - Better indication of the loading progress, should no longer freeze without a progress bar #jira UECORE-170 - Improve profiler loading performance (wip) Change 2782446 on 2015/11/29 by Steve.Robb@Dev-Core Warn when calling delegates' Create* functions when they're not assigned to anything. #codereview robert.manuszewski Change 2782538 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 #UE4 Removed DiskCachedAssetDataBuffer as it was not strictly necessary and was triggering a crash when loading the cached registry from disk. This data is now stored directly in DiskCachedAssetDataMap. It was already true that this map does not change outside of SerializeCache but now it is critical since NewCachedAssetDataMap keeps pointers directly to its values. Asset registry fixes by Bob Tellez. Possible fix for UE-23783. Change 2782564 on 2015/11/30 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 FReferenceCollector::AddReferenceObjects performance improvements for TArrays. ARO will no longer call HandleObjectReference multiple times but instead will call HandleObjectReferences just once (currently only implemented for FGCCollector). Reduces the number of virtual function calls while GC'ing. Change 2782716 on 2015/11/30 by Steve.Robb@Dev-Core UObject serial number array initialized for debug visualization. Change 2782933 on 2015/11/30 by Steve.Robb@Dev-Core Critical sections are no longer copyable. Change 2783061 on 2015/11/30 by Steve.Robb@Dev-Core
2015-12-03 14:21:29 -05:00
}