Files
UnrealEngineUWP/Engine/Source/Programs/UnrealPak/Private/SignedArchiveWriter.cpp

162 lines
4.3 KiB
C++
Raw Normal View History

// Copyright 1998-2017 Epic Games, Inc. All Rights Reserved.
Copying //UE4/Dev-Build to //UE4/Dev-Main (Source: //UE4/Dev-Build @ 3209340) #lockdown Nick.Penwarden #rb none ========================== MAJOR FEATURES + CHANGES ========================== Change 3209340 on 2016/11/23 by Ben.Marsh Convert UE4 codebase to an "include what you use" model - where every header just includes the dependencies it needs, rather than every source file including large monolithic headers like Engine.h and UnrealEd.h. Measured full rebuild times around 2x faster using XGE on Windows, and improvements of 25% or more for incremental builds and full rebuilds on most other platforms. * Every header now includes everything it needs to compile. * There's a CoreMinimal.h header that gets you a set of ubiquitous types from Core (eg. FString, FName, TArray, FVector, etc...). Most headers now include this first. * There's a CoreTypes.h header that sets up primitive UE4 types and build macros (int32, PLATFORM_WIN64, etc...). All headers in Core include this first, as does CoreMinimal.h. * Every .cpp file includes its matching .h file first. * This helps validate that each header is including everything it needs to compile. * No engine code includes a monolithic header such as Engine.h or UnrealEd.h any more. * You will get a warning if you try to include one of these from the engine. They still exist for compatibility with game projects and do not produce warnings when included there. * There have only been minor changes to our internal games down to accommodate these changes. The intent is for this to be as seamless as possible. * No engine code explicitly includes a precompiled header any more. * We still use PCHs, but they're force-included on the compiler command line by UnrealBuildTool instead. This lets us tune what they contain without breaking any existing include dependencies. * PCHs are generated by a tool to get a statistical amount of coverage for the source files using it, and I've seeded the new shared PCHs to contain any header included by > 15% of source files. Tool used to generate this transform is at Engine\Source\Programs\IncludeTool. [CL 3209342 by Ben Marsh in Main branch]
2016-11-23 15:48:37 -05:00
#include "SignedArchiveWriter.h"
#include "IPlatformFilePak.h"
#include "SecureHash.h"
Copying //UE4/Dev-Build to //UE4/Dev-Main (Source: //UE4/Dev-Build @ 3209340) #lockdown Nick.Penwarden #rb none ========================== MAJOR FEATURES + CHANGES ========================== Change 3209340 on 2016/11/23 by Ben.Marsh Convert UE4 codebase to an "include what you use" model - where every header just includes the dependencies it needs, rather than every source file including large monolithic headers like Engine.h and UnrealEd.h. Measured full rebuild times around 2x faster using XGE on Windows, and improvements of 25% or more for incremental builds and full rebuilds on most other platforms. * Every header now includes everything it needs to compile. * There's a CoreMinimal.h header that gets you a set of ubiquitous types from Core (eg. FString, FName, TArray, FVector, etc...). Most headers now include this first. * There's a CoreTypes.h header that sets up primitive UE4 types and build macros (int32, PLATFORM_WIN64, etc...). All headers in Core include this first, as does CoreMinimal.h. * Every .cpp file includes its matching .h file first. * This helps validate that each header is including everything it needs to compile. * No engine code includes a monolithic header such as Engine.h or UnrealEd.h any more. * You will get a warning if you try to include one of these from the engine. They still exist for compatibility with game projects and do not produce warnings when included there. * There have only been minor changes to our internal games down to accommodate these changes. The intent is for this to be as seamless as possible. * No engine code explicitly includes a precompiled header any more. * We still use PCHs, but they're force-included on the compiler command line by UnrealBuildTool instead. This lets us tune what they contain without breaking any existing include dependencies. * PCHs are generated by a tool to get a statistical amount of coverage for the source files using it, and I've seeded the new shared PCHs to contain any header included by > 15% of source files. Tool used to generate this transform is at Engine\Source\Programs\IncludeTool. [CL 3209342 by Ben Marsh in Main branch]
2016-11-23 15:48:37 -05:00
#include "HAL/FileManager.h"
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
FSignedArchiveWriter::FSignedArchiveWriter(FArchive& InPak, const FString& InPakFilename, const FEncryptionKey& InPublicKey, const FEncryptionKey& InPrivateKey)
: BufferArchive(Buffer)
, PakWriter(InPak)
, PakSignaturesFilename(FPaths::ChangeExtension(InPakFilename, TEXT("sig")))
, SizeOnDisk(0)
, PakSize(0)
, PublicKey(InPublicKey)
, PrivateKey(InPrivateKey)
{
Buffer.Reserve(FPakInfo::MaxChunkDataSize);
}
FSignedArchiveWriter::~FSignedArchiveWriter()
{
if (BufferArchive.Tell() > 0)
{
SerializeBufferAndSign();
}
delete &PakWriter;
}
void FSignedArchiveWriter::SerializeBufferAndSign()
{
Copying //UE4/Orion-Staging to //UE4/Main (Origin: //Orion/Dev-General @ 2904087) ========================== MAJOR FEATURES + CHANGES ========================== #lockdown Nick.Penwarden Change 2903938 on 2016/03/10 by Frank.Gigliotti Added an instance ID to FAnimMontageInstance #CodeReview Laurent.Delayen #RB Laurent.Delayen #Tests PIE Change 2903745 on 2016/03/10 by Wes.Hunt Update Oodle TPS #rb none #tests none #codereview:john.pollard Change 2903689 on 2016/03/10 by Uriel.Doyon New "LogHeroMaterials" console command, displaying the current state of materials and textures on the character hero. #rb marcus.wasmer #codereview marcus.wassmer #tests editor, playing PC games, trying the new command Change 2903669 on 2016/03/10 by Aaron.McLeran OR-17180 Make stat soundcues and stat soundwaves NOT display zero volume sounds - Change only effects debug stat commands for audio guys #rb none #tests played paragon with new debug stat commands, confirms doesn't show zero-volume sounds Change 2903625 on 2016/03/10 by John.Pollard XB1 Oodle SDK #rb none #tests none #codereview Jeff.Campeau Change 2903577 on 2016/03/10 by Ben.Marsh Remaking latest build scripts from //UE4/Main @ 2900980. Change 2903560 on 2016/03/10 by Ben.Marsh Initial version of BuildGraph scripts - used to create build processes in UE4 which can be run locally or in parallel across a build farm (assuming synchronization and resource allocation implemented by a separate system). Intended to supersede GUBP. Build graphs are declared using an XML script using syntax similar to MSBuild, ANT or NAnt, and consist of the following components: * Tasks: Building blocks which can be executed as part of the build process. Many predefined tasks are provided (<Cook>, <Compile>, <Copy>, <Stage>, <Log>, <PakFile>, etc...), and additional tasks may be added be declaring classes derived from AutomationTool.CustomTask in other UAT modules. * Nodes: A named sequence of tasks which is executed to produce outputs. Nodes may have input dependencies on other nodes before they can be executed. Declared with the <Node> element in scripts. * Agent Groups: A set of nodes nodes which is executed on the same machine if running as part of a build system. Has no effect when building locally. Declared with the <Group> element in scripts. * Triggers: Container for groups which should only be executed when explicitly triggered (using the -Trigger=<Name> or -SkipTriggers command line argument). Declared with the <Trigger> element in scripts. * Notifiers: Specifies email recipients for failures in one or more nodes, whether they should receive notifications on warnings, and so on. Properties can be passed in to a script on the command line, or set procedurally with the <Property Name="Foo" Value="Bar"/> syntax. Properties referenced with the $(Property Name) notation are valid within all strings, and will be expanded as macros when the script is read. If a property name is not set explicitly, it defaults to the contents of an environment variable with the same name. Local properties, which only affect the scope of the containing XML element (node, group, etc...) are declared with the <Local Name="Foo" Value="Bar"/> element, and will override a similarly named global property for the local property's scope. Any elements can be conditionally defined via the "If" attribute, and are largely identical to MSBuild conditions. Literals in conditions may be quoted with single (') or double (") quotes, or an unquoted sequence of letters, digits and underscore characters. All literals are considered identical regardless of how they are declared, and are considered case-insensitive for comparisons (so true equals 'True', equals "TRUE"). Available operators are "==", "!=", "And", "Or", "!", "(...)", "Exists(...)" and "HasTrailingSlash(...)". A full grammar is written up in Condition.cs. File manipulation is done using wildcards and tags. Any attribute that accepts a list of files may consist of: a Perforce-style wildcard (matching any number of "...", "*" and "?" patterns in any location), a full path name, or a reference to a tagged collection of files, denoted by prefixing with a '#' character. Files may be added to a tag set using the <Tag> Task, which also allows performing set union/difference style operations. Each node can declare multiple outputs in the form of a list of named tags, which other nodes can then depend on. Build graphs may be executed in parallel as part build system. To do so, the initial graph configuration is generated by running with the -Export=<Filename> argument (producing a JSON file listing the nodes and dependencies to execute). Each participating agent should be synced to the same changelist, and UAT should be re-run with the appropriate -Node=<Name> argument. Outputs from different nodes are transferred between agents via shared storage, typically a network share, the path to which can be specified on the command line using the -SharedStorageDir=<Path> argument. Note that the allocation of machines, and coordination between them, is assumed to be managed by an external system. A schema for the known set of tasks can be generated by running UAT with the "-Schema=<FileName>" option. Generating a schema and referencing it from a BuildGraph script allows Visual Studio to validate and auto-complete elements as you type. #rb none #codereview Marc.Audy, Wes.Hunt, Matthew.Griffin, Richard.Fawcett #tests local only so far, but not part of any build process yet Change 2903539 on 2016/03/10 by John.Pollard Improve replay playback debugging of character movement #rb none #tests replays Change 2903526 on 2016/03/10 by Ben.Marsh Remake changes from //UE4/Main without integration history, to add support for BuildGraph tasks. #rb none #tests none Change 2903512 on 2016/03/10 by Dan.Youhon Modify minimum Duration values for JumpForce and MoveToForce ability tasks so that having minimum Duration values doesn't trigger check()s #rb None #tests Compiles Change 2903474 on 2016/03/10 by Marc.Audy Fix crash if ChildActor is null #rb None #tests None Change 2903314 on 2016/03/10 by Marc.Audy Fix ParentComponent not being persisted and fixup content that was saved in the window it was broken #rb James.Golding #tests Selection of child actors works as expected #jira UE-28201 Change 2903298 on 2016/03/10 by Simon.Tovey Disabling the trails optimization. #tests none #rb none #codereview Olaf.Piesche Change 2903124 on 2016/03/10 by Robert.Manuszewski Small refactor to pak signing to help with exe protection #rb none #tests none [CL 2907678 by Andrew Grant in Main branch]
2016-03-13 18:53:13 -04:00
// Compute a hash for this buffer data
Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3208226) #lockdown Nick.Penwarden #rb None ========================== MAJOR FEATURES + CHANGES ========================== Change 3173153 on 2016/10/25 by Graeme.Thornton Pak signing changes - Integrated into EDL loader - Changed to not encrypt each CRC in the sig file, rather just store a single encryped signature of the entire sig file. Removes need to decrypt thousands of signatures at startup. Change 3173531 on 2016/10/25 by Steven.Hutton Removing unused j query packages. Change 3174743 on 2016/10/26 by Gil.Gribb UE4 - fixed COTF with EDL Change 3177896 on 2016/10/28 by Steve.Robb TSharedPtr and TSharedRef aliasing constructors. Removal of static_asserts for TSharedPtr<UObject>. Change 3180343 on 2016/10/31 by Steve.Robb Reimplementation of changes from CL#s 3050329 and 3105715 that were lost in merges 3094597 and 3105741. Change 3181382 on 2016/11/01 by Steve.Robb Visual Studio debugger visualizers for delegates. Change 3182738 on 2016/11/02 by Graeme.Thornton Re-enable signed archive reader so non-pakpreacher based reads still get signature checked Change 3183420 on 2016/11/02 by Steve.Robb Fix to TIsZeroConstructType for TScriptDelegate. Change 3184872 on 2016/11/03 by Robert.Manuszewski Fixing memory stomps in SSL certificate initialization (found with mallocstomp) Change 3184873 on 2016/11/03 by Robert.Manuszewski Adding thread safety checks to async loading code Change 3185535 on 2016/11/03 by Ben.Zeigler Fix it so calling CreateDefaultSubobject with bTransient = true sets the object transient flag. This fixes EDL Crashes involving components. Change 3186636 on 2016/11/04 by Graeme.Thornton AES encryption integrated into EDL system Pak signing and AES encryption now configurable by ini files rather than magical text files Change 3186637 on 2016/11/04 by Graeme.Thornton Configured pak signing and encryption in ShooterGame for reference Change 3186639 on 2016/11/04 by Graeme.Thornton Encryption changes for Orion * Move pak signing keys into new INI format * Add AES key and enable INI file encryption Change 3186661 on 2016/11/04 by Graeme.Thornton Change unrealpak command line params to accept AES key as a separete parameter Change 3186670 on 2016/11/04 by Robert.Manuszewski Adding a null check before using a package pointer in Linker code #jira UE-38237 Change 3186775 on 2016/11/04 by Graeme.Thornton Fix UBT defines that come in as quoted strings, losing the quotes when passed to the compiler - PS4 and Mac fixes. Other platforms might need fixing too! Change 3186823 on 2016/11/04 by Graeme.Thornton Fixed an incorrect size check in the EDL pak signing code Change 3186925 on 2016/11/04 by Graeme.Thornton Allow UnrealPak to read encryption settings from project ini files Change 3189885 on 2016/11/08 by Graeme.Thornton Static analysis warning fix Change 3190015 on 2016/11/08 by Robert.Manuszewski Thread safety fix for UBlueprintGeneratedClass::PostLoadDefaultObject while UBlueprintGeneratedClass::SerializeDefaultObject runs on the async loading thread Change 3190253 on 2016/11/08 by Chris.Wood Improved MDD performance for on the CR server. [UE-37566] - Improve MDD performance on CR server Blocked MDD init'ing the crash handling code as it isn't desirable on the server. Removed redundant call to SetSymbolPathsFromModules() from CrashDebugHelper. Change 3192993 on 2016/11/10 by Robert.Manuszewski Thread Heartbeat will no longer report the same hang multiple times. Change 3193111 on 2016/11/10 by Robert.Manuszewski Minor change in the condition that detects the same hangs - allow the same callstacks from different threads Change 3193168 on 2016/11/10 by Steve.Robb TSparseArray now reserves space in reverse so that new elements get added to the front of the allocation rather than the back, which is better for memory traversal and meets expectations more closely. Change 3193171 on 2016/11/10 by Steve.Robb Easier debugging of FPendingRegistrantInfo map. Change 3193188 on 2016/11/10 by Steve.Robb TAutoPointer deprecated. Change 3193796 on 2016/11/10 by Graeme.Thornton Fix pak creation failure when no pak signing keys are supplied Change 3194524 on 2016/11/11 by Graeme.Thornton Another static analysis warning fix Change 3195119 on 2016/11/11 by Steve.Robb TAutoPtr deprecated. Fixes to use of TAutoPtr with incompatible memory deallocations (TAutoPtr with FMemory::Malloc and new[]). Some large headers moved into .cpp files. Change 3196582 on 2016/11/14 by Gil.Gribb UE4 - Changed a check to a warning related to detaching linekrs twice. Seen in nativized BP version of platformer game. Change 3196878 on 2016/11/14 by Steve.Robb TScopedPointer deprecated. Change 3198061 on 2016/11/15 by Steve.Robb Class array is no longer regenerated when saving UClasses. Change 3198065 on 2016/11/15 by Robert.Manuszewski Making AssembleReferenceTokenStream thread safe for blueprints loaded on the async loading thread. Change 3198199 on 2016/11/15 by Robert.Manuszewski Pak platform file will now only be used if pak files exist regardless of command line paraks like -pak, -singedpak and -signed. Change 3199954 on 2016/11/16 by Graeme.Thornton Removing USING_SIGNED_CONTENT Change 3200221 on 2016/11/16 by Chris.Wood CrashReportProcess code cleanup - removing unused using directives Change 3200232 on 2016/11/16 by Chris.Wood Multiple CrashReportProcess updates and improvements (CRP v1.2.6) UE-36248 - CRP scalability: All bulk storage or shared data to S3 or suitable network drives InvalidCrashReports now saved to S3 instead of local folder Removed option tosync MinidumpDiagnostics from Perforce Moved MinidumpDiagnostics from old Perforce synched location to its own folder in E:\Services (makes more sense with manual publishing) Added improved logging to Slack with option to monitor MDD performance Added hourly log folders to MDD logs Added support for types of crashes we don't want to symbolicate (using it to skip callstack gen for hang detected ensures) Change 3200382 on 2016/11/16 by Robert.Manuszewski Async Loading code will now detach the linker when resetting async package loader to avoid situations when loading the same asset multiple times results in the following load request finding the old linker after the package has been loading but the async package hasn't been deleted yet (async package for the old request in limbo state but linker exists). Change 3200562 on 2016/11/16 by Gil.Gribb UE4 - Fixed rare issue with reloading nativized blueprints with the EDL and a minor simplication. Change 3201093 on 2016/11/16 by Ben.Zeigler #UE 38654 Fix EDL cooking to correctly search components created directly by UBlueprints, as well as the CDO components it already covered. Also explicitly mark subobject templates as editor only. Fix issue where the AssetImportData associated with Blueprint-owned Curves was ending up in the cooked subobject template list. Stopped it from creating those objects, and mark the class editor only. Change 3201736 on 2016/11/17 by Steve.Robb Strtoi64 platform and TCString functions. #fyi robert.manuszewski Change 3201938 on 2016/11/17 by Ben.Woodhouse Dummy integrate of the Square render version workaround (CL 3201913) with _accept target_ to prevent it being integrated to dev-core in future. Commandline: p4 integrate //Tasks/UE4/Dev-LoadTimes/Engine/Source/Runtime/CoreUObject/Private/UObject/LinkerLoad.cpp@3201913,3201913 //UE4/Dev-Core/Engine/Source/Runtime/CoreUObject/Private/UObject/LinkerLoad.cpp #fyi robert.manuszewski Change 3203757 on 2016/11/18 by Robert.Manuszewski Removing debug code from async loading code. Change 3203927 on 2016/11/18 by Robert.Manuszewski Fixing comments in the async loading code. Change 3204851 on 2016/11/18 by Steve.Robb Metafunction for testing if a particular operator<< overload exists, e.g. THasInserterOperator<FArchive&, FMyType&>::Value. Change 3204854 on 2016/11/18 by Steve.Robb UEnumProperty. Change 3205027 on 2016/11/18 by Ben.Zeigler Add useful functions to FAssetPtr and TAssetSubclassOf that already existed on TAssetPtr Add Get() to TSubclassOf so it matches our other wrappers Fix TSubclassOf and TAssetSubclassOf to use the more efficient template method of checking class compatibility Comment and template cleanups for AssetPtr, StringAssetReference, LazyPtr, and SubclassOf Change 3206334 on 2016/11/21 by Ben.Zeigler #UE-38773: Fix it so non-component template subobjects of CDOs are not included as creation dependencies for BP classes, also clean up GetPreloadDependencies as it was adding redundant and null entries #UE-38799: Fix it so WidgetTrees don't get picked up as subobjects, and add ensure at cook time to find null outers that would crash at runtime. Make sure the instanced widget trees are transient. Cook finishes but game is still crashing in some cases, so I might adjust this after other testing Change 3206353 on 2016/11/21 by Ben.Zeigler Fix EnumProperty to handle EDL preload dependencies properly Change 3206625 on 2016/11/21 by Ben.Zeigler Fix enum property crash at runtime by copying what array property does and making sure inner property is not transient Change 3206937 on 2016/11/21 by Ben.Zeigler #jira UE-38905 Fix it so enums inside arrays are migrated properly, the enum tag is lost so use the current one Disable other nested enum migrations as they are unlikely to work. Array property tags need to be refactored to be safer Correctly save enum tag for enum properties, it was being set but not serialized Change 3207002 on 2016/11/21 by Ben.Zeigler #jira UE-38799 Fix it so per-widget copy of widget tree and all widgets inside are properly transient, they were being cooked before but never accessed. Fix case where non ClientOnly public objects nested instead ClientOnly objects would cook but fail to load, and add ensure to catch these cases in the future. If the full outer chain isn't available, it can't be loaded anyway, and this finds issues at cook time instead of load time. We should generally outlaw non-transient objects with transient outers, it does not do what people expect. Change 3207032 on 2016/11/21 by Ben.Zeigler #jira UE-38654 Re-Fix EDL cooking with SCS-added components. They used to have the DefaultSubObject flag but no longer do [CL 3208270 by Ben Zeigler in Main branch]
2016-11-22 18:45:44 -05:00
ChunkHashes.Add(ComputePakChunkHash(&Buffer[0], Buffer.Num()));
// Flush the buffer
PakWriter.Serialize(&Buffer[0], Buffer.Num());
BufferArchive.Seek(0);
Buffer.Empty(FPakInfo::MaxChunkDataSize);
}
bool FSignedArchiveWriter::Close()
{
if (BufferArchive.Tell() > 0)
{
SerializeBufferAndSign();
}
Copying //UE4/Orion-Staging to //UE4/Main (Origin: //Orion/Dev-General @ 2904087) ========================== MAJOR FEATURES + CHANGES ========================== #lockdown Nick.Penwarden Change 2903938 on 2016/03/10 by Frank.Gigliotti Added an instance ID to FAnimMontageInstance #CodeReview Laurent.Delayen #RB Laurent.Delayen #Tests PIE Change 2903745 on 2016/03/10 by Wes.Hunt Update Oodle TPS #rb none #tests none #codereview:john.pollard Change 2903689 on 2016/03/10 by Uriel.Doyon New "LogHeroMaterials" console command, displaying the current state of materials and textures on the character hero. #rb marcus.wasmer #codereview marcus.wassmer #tests editor, playing PC games, trying the new command Change 2903669 on 2016/03/10 by Aaron.McLeran OR-17180 Make stat soundcues and stat soundwaves NOT display zero volume sounds - Change only effects debug stat commands for audio guys #rb none #tests played paragon with new debug stat commands, confirms doesn't show zero-volume sounds Change 2903625 on 2016/03/10 by John.Pollard XB1 Oodle SDK #rb none #tests none #codereview Jeff.Campeau Change 2903577 on 2016/03/10 by Ben.Marsh Remaking latest build scripts from //UE4/Main @ 2900980. Change 2903560 on 2016/03/10 by Ben.Marsh Initial version of BuildGraph scripts - used to create build processes in UE4 which can be run locally or in parallel across a build farm (assuming synchronization and resource allocation implemented by a separate system). Intended to supersede GUBP. Build graphs are declared using an XML script using syntax similar to MSBuild, ANT or NAnt, and consist of the following components: * Tasks: Building blocks which can be executed as part of the build process. Many predefined tasks are provided (<Cook>, <Compile>, <Copy>, <Stage>, <Log>, <PakFile>, etc...), and additional tasks may be added be declaring classes derived from AutomationTool.CustomTask in other UAT modules. * Nodes: A named sequence of tasks which is executed to produce outputs. Nodes may have input dependencies on other nodes before they can be executed. Declared with the <Node> element in scripts. * Agent Groups: A set of nodes nodes which is executed on the same machine if running as part of a build system. Has no effect when building locally. Declared with the <Group> element in scripts. * Triggers: Container for groups which should only be executed when explicitly triggered (using the -Trigger=<Name> or -SkipTriggers command line argument). Declared with the <Trigger> element in scripts. * Notifiers: Specifies email recipients for failures in one or more nodes, whether they should receive notifications on warnings, and so on. Properties can be passed in to a script on the command line, or set procedurally with the <Property Name="Foo" Value="Bar"/> syntax. Properties referenced with the $(Property Name) notation are valid within all strings, and will be expanded as macros when the script is read. If a property name is not set explicitly, it defaults to the contents of an environment variable with the same name. Local properties, which only affect the scope of the containing XML element (node, group, etc...) are declared with the <Local Name="Foo" Value="Bar"/> element, and will override a similarly named global property for the local property's scope. Any elements can be conditionally defined via the "If" attribute, and are largely identical to MSBuild conditions. Literals in conditions may be quoted with single (') or double (") quotes, or an unquoted sequence of letters, digits and underscore characters. All literals are considered identical regardless of how they are declared, and are considered case-insensitive for comparisons (so true equals 'True', equals "TRUE"). Available operators are "==", "!=", "And", "Or", "!", "(...)", "Exists(...)" and "HasTrailingSlash(...)". A full grammar is written up in Condition.cs. File manipulation is done using wildcards and tags. Any attribute that accepts a list of files may consist of: a Perforce-style wildcard (matching any number of "...", "*" and "?" patterns in any location), a full path name, or a reference to a tagged collection of files, denoted by prefixing with a '#' character. Files may be added to a tag set using the <Tag> Task, which also allows performing set union/difference style operations. Each node can declare multiple outputs in the form of a list of named tags, which other nodes can then depend on. Build graphs may be executed in parallel as part build system. To do so, the initial graph configuration is generated by running with the -Export=<Filename> argument (producing a JSON file listing the nodes and dependencies to execute). Each participating agent should be synced to the same changelist, and UAT should be re-run with the appropriate -Node=<Name> argument. Outputs from different nodes are transferred between agents via shared storage, typically a network share, the path to which can be specified on the command line using the -SharedStorageDir=<Path> argument. Note that the allocation of machines, and coordination between them, is assumed to be managed by an external system. A schema for the known set of tasks can be generated by running UAT with the "-Schema=<FileName>" option. Generating a schema and referencing it from a BuildGraph script allows Visual Studio to validate and auto-complete elements as you type. #rb none #codereview Marc.Audy, Wes.Hunt, Matthew.Griffin, Richard.Fawcett #tests local only so far, but not part of any build process yet Change 2903539 on 2016/03/10 by John.Pollard Improve replay playback debugging of character movement #rb none #tests replays Change 2903526 on 2016/03/10 by Ben.Marsh Remake changes from //UE4/Main without integration history, to add support for BuildGraph tasks. #rb none #tests none Change 2903512 on 2016/03/10 by Dan.Youhon Modify minimum Duration values for JumpForce and MoveToForce ability tasks so that having minimum Duration values doesn't trigger check()s #rb None #tests Compiles Change 2903474 on 2016/03/10 by Marc.Audy Fix crash if ChildActor is null #rb None #tests None Change 2903314 on 2016/03/10 by Marc.Audy Fix ParentComponent not being persisted and fixup content that was saved in the window it was broken #rb James.Golding #tests Selection of child actors works as expected #jira UE-28201 Change 2903298 on 2016/03/10 by Simon.Tovey Disabling the trails optimization. #tests none #rb none #codereview Olaf.Piesche Change 2903124 on 2016/03/10 by Robert.Manuszewski Small refactor to pak signing to help with exe protection #rb none #tests none [CL 2907678 by Andrew Grant in Main branch]
2016-03-13 18:53:13 -04:00
Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3208226) #lockdown Nick.Penwarden #rb None ========================== MAJOR FEATURES + CHANGES ========================== Change 3173153 on 2016/10/25 by Graeme.Thornton Pak signing changes - Integrated into EDL loader - Changed to not encrypt each CRC in the sig file, rather just store a single encryped signature of the entire sig file. Removes need to decrypt thousands of signatures at startup. Change 3173531 on 2016/10/25 by Steven.Hutton Removing unused j query packages. Change 3174743 on 2016/10/26 by Gil.Gribb UE4 - fixed COTF with EDL Change 3177896 on 2016/10/28 by Steve.Robb TSharedPtr and TSharedRef aliasing constructors. Removal of static_asserts for TSharedPtr<UObject>. Change 3180343 on 2016/10/31 by Steve.Robb Reimplementation of changes from CL#s 3050329 and 3105715 that were lost in merges 3094597 and 3105741. Change 3181382 on 2016/11/01 by Steve.Robb Visual Studio debugger visualizers for delegates. Change 3182738 on 2016/11/02 by Graeme.Thornton Re-enable signed archive reader so non-pakpreacher based reads still get signature checked Change 3183420 on 2016/11/02 by Steve.Robb Fix to TIsZeroConstructType for TScriptDelegate. Change 3184872 on 2016/11/03 by Robert.Manuszewski Fixing memory stomps in SSL certificate initialization (found with mallocstomp) Change 3184873 on 2016/11/03 by Robert.Manuszewski Adding thread safety checks to async loading code Change 3185535 on 2016/11/03 by Ben.Zeigler Fix it so calling CreateDefaultSubobject with bTransient = true sets the object transient flag. This fixes EDL Crashes involving components. Change 3186636 on 2016/11/04 by Graeme.Thornton AES encryption integrated into EDL system Pak signing and AES encryption now configurable by ini files rather than magical text files Change 3186637 on 2016/11/04 by Graeme.Thornton Configured pak signing and encryption in ShooterGame for reference Change 3186639 on 2016/11/04 by Graeme.Thornton Encryption changes for Orion * Move pak signing keys into new INI format * Add AES key and enable INI file encryption Change 3186661 on 2016/11/04 by Graeme.Thornton Change unrealpak command line params to accept AES key as a separete parameter Change 3186670 on 2016/11/04 by Robert.Manuszewski Adding a null check before using a package pointer in Linker code #jira UE-38237 Change 3186775 on 2016/11/04 by Graeme.Thornton Fix UBT defines that come in as quoted strings, losing the quotes when passed to the compiler - PS4 and Mac fixes. Other platforms might need fixing too! Change 3186823 on 2016/11/04 by Graeme.Thornton Fixed an incorrect size check in the EDL pak signing code Change 3186925 on 2016/11/04 by Graeme.Thornton Allow UnrealPak to read encryption settings from project ini files Change 3189885 on 2016/11/08 by Graeme.Thornton Static analysis warning fix Change 3190015 on 2016/11/08 by Robert.Manuszewski Thread safety fix for UBlueprintGeneratedClass::PostLoadDefaultObject while UBlueprintGeneratedClass::SerializeDefaultObject runs on the async loading thread Change 3190253 on 2016/11/08 by Chris.Wood Improved MDD performance for on the CR server. [UE-37566] - Improve MDD performance on CR server Blocked MDD init'ing the crash handling code as it isn't desirable on the server. Removed redundant call to SetSymbolPathsFromModules() from CrashDebugHelper. Change 3192993 on 2016/11/10 by Robert.Manuszewski Thread Heartbeat will no longer report the same hang multiple times. Change 3193111 on 2016/11/10 by Robert.Manuszewski Minor change in the condition that detects the same hangs - allow the same callstacks from different threads Change 3193168 on 2016/11/10 by Steve.Robb TSparseArray now reserves space in reverse so that new elements get added to the front of the allocation rather than the back, which is better for memory traversal and meets expectations more closely. Change 3193171 on 2016/11/10 by Steve.Robb Easier debugging of FPendingRegistrantInfo map. Change 3193188 on 2016/11/10 by Steve.Robb TAutoPointer deprecated. Change 3193796 on 2016/11/10 by Graeme.Thornton Fix pak creation failure when no pak signing keys are supplied Change 3194524 on 2016/11/11 by Graeme.Thornton Another static analysis warning fix Change 3195119 on 2016/11/11 by Steve.Robb TAutoPtr deprecated. Fixes to use of TAutoPtr with incompatible memory deallocations (TAutoPtr with FMemory::Malloc and new[]). Some large headers moved into .cpp files. Change 3196582 on 2016/11/14 by Gil.Gribb UE4 - Changed a check to a warning related to detaching linekrs twice. Seen in nativized BP version of platformer game. Change 3196878 on 2016/11/14 by Steve.Robb TScopedPointer deprecated. Change 3198061 on 2016/11/15 by Steve.Robb Class array is no longer regenerated when saving UClasses. Change 3198065 on 2016/11/15 by Robert.Manuszewski Making AssembleReferenceTokenStream thread safe for blueprints loaded on the async loading thread. Change 3198199 on 2016/11/15 by Robert.Manuszewski Pak platform file will now only be used if pak files exist regardless of command line paraks like -pak, -singedpak and -signed. Change 3199954 on 2016/11/16 by Graeme.Thornton Removing USING_SIGNED_CONTENT Change 3200221 on 2016/11/16 by Chris.Wood CrashReportProcess code cleanup - removing unused using directives Change 3200232 on 2016/11/16 by Chris.Wood Multiple CrashReportProcess updates and improvements (CRP v1.2.6) UE-36248 - CRP scalability: All bulk storage or shared data to S3 or suitable network drives InvalidCrashReports now saved to S3 instead of local folder Removed option tosync MinidumpDiagnostics from Perforce Moved MinidumpDiagnostics from old Perforce synched location to its own folder in E:\Services (makes more sense with manual publishing) Added improved logging to Slack with option to monitor MDD performance Added hourly log folders to MDD logs Added support for types of crashes we don't want to symbolicate (using it to skip callstack gen for hang detected ensures) Change 3200382 on 2016/11/16 by Robert.Manuszewski Async Loading code will now detach the linker when resetting async package loader to avoid situations when loading the same asset multiple times results in the following load request finding the old linker after the package has been loading but the async package hasn't been deleted yet (async package for the old request in limbo state but linker exists). Change 3200562 on 2016/11/16 by Gil.Gribb UE4 - Fixed rare issue with reloading nativized blueprints with the EDL and a minor simplication. Change 3201093 on 2016/11/16 by Ben.Zeigler #UE 38654 Fix EDL cooking to correctly search components created directly by UBlueprints, as well as the CDO components it already covered. Also explicitly mark subobject templates as editor only. Fix issue where the AssetImportData associated with Blueprint-owned Curves was ending up in the cooked subobject template list. Stopped it from creating those objects, and mark the class editor only. Change 3201736 on 2016/11/17 by Steve.Robb Strtoi64 platform and TCString functions. #fyi robert.manuszewski Change 3201938 on 2016/11/17 by Ben.Woodhouse Dummy integrate of the Square render version workaround (CL 3201913) with _accept target_ to prevent it being integrated to dev-core in future. Commandline: p4 integrate //Tasks/UE4/Dev-LoadTimes/Engine/Source/Runtime/CoreUObject/Private/UObject/LinkerLoad.cpp@3201913,3201913 //UE4/Dev-Core/Engine/Source/Runtime/CoreUObject/Private/UObject/LinkerLoad.cpp #fyi robert.manuszewski Change 3203757 on 2016/11/18 by Robert.Manuszewski Removing debug code from async loading code. Change 3203927 on 2016/11/18 by Robert.Manuszewski Fixing comments in the async loading code. Change 3204851 on 2016/11/18 by Steve.Robb Metafunction for testing if a particular operator<< overload exists, e.g. THasInserterOperator<FArchive&, FMyType&>::Value. Change 3204854 on 2016/11/18 by Steve.Robb UEnumProperty. Change 3205027 on 2016/11/18 by Ben.Zeigler Add useful functions to FAssetPtr and TAssetSubclassOf that already existed on TAssetPtr Add Get() to TSubclassOf so it matches our other wrappers Fix TSubclassOf and TAssetSubclassOf to use the more efficient template method of checking class compatibility Comment and template cleanups for AssetPtr, StringAssetReference, LazyPtr, and SubclassOf Change 3206334 on 2016/11/21 by Ben.Zeigler #UE-38773: Fix it so non-component template subobjects of CDOs are not included as creation dependencies for BP classes, also clean up GetPreloadDependencies as it was adding redundant and null entries #UE-38799: Fix it so WidgetTrees don't get picked up as subobjects, and add ensure at cook time to find null outers that would crash at runtime. Make sure the instanced widget trees are transient. Cook finishes but game is still crashing in some cases, so I might adjust this after other testing Change 3206353 on 2016/11/21 by Ben.Zeigler Fix EnumProperty to handle EDL preload dependencies properly Change 3206625 on 2016/11/21 by Ben.Zeigler Fix enum property crash at runtime by copying what array property does and making sure inner property is not transient Change 3206937 on 2016/11/21 by Ben.Zeigler #jira UE-38905 Fix it so enums inside arrays are migrated properly, the enum tag is lost so use the current one Disable other nested enum migrations as they are unlikely to work. Array property tags need to be refactored to be safer Correctly save enum tag for enum properties, it was being set but not serialized Change 3207002 on 2016/11/21 by Ben.Zeigler #jira UE-38799 Fix it so per-widget copy of widget tree and all widgets inside are properly transient, they were being cooked before but never accessed. Fix case where non ClientOnly public objects nested instead ClientOnly objects would cook but fail to load, and add ensure to catch these cases in the future. If the full outer chain isn't available, it can't be loaded anyway, and this finds issues at cook time instead of load time. We should generally outlaw non-transient objects with transient outers, it does not do what people expect. Change 3207032 on 2016/11/21 by Ben.Zeigler #jira UE-38654 Re-Fix EDL cooking with SCS-added components. They used to have the DefaultSubObject flag but no longer do [CL 3208270 by Ben Zeigler in Main branch]
2016-11-22 18:45:44 -05:00
FEncryptedSignature EncryptedMasterHash;
FDecryptedSignature DecryptedMasterHash;
DecryptedMasterHash.Data = ComputePakChunkHash((const uint8*)&ChunkHashes[0], ChunkHashes.Num() * sizeof(TPakChunkHash));
FEncryption::EncryptSignature(DecryptedMasterHash, EncryptedMasterHash, PrivateKey);
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
FArchive* SignatureWriter = IFileManager::Get().CreateFileWriter(*PakSignaturesFilename);
Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3208226) #lockdown Nick.Penwarden #rb None ========================== MAJOR FEATURES + CHANGES ========================== Change 3173153 on 2016/10/25 by Graeme.Thornton Pak signing changes - Integrated into EDL loader - Changed to not encrypt each CRC in the sig file, rather just store a single encryped signature of the entire sig file. Removes need to decrypt thousands of signatures at startup. Change 3173531 on 2016/10/25 by Steven.Hutton Removing unused j query packages. Change 3174743 on 2016/10/26 by Gil.Gribb UE4 - fixed COTF with EDL Change 3177896 on 2016/10/28 by Steve.Robb TSharedPtr and TSharedRef aliasing constructors. Removal of static_asserts for TSharedPtr<UObject>. Change 3180343 on 2016/10/31 by Steve.Robb Reimplementation of changes from CL#s 3050329 and 3105715 that were lost in merges 3094597 and 3105741. Change 3181382 on 2016/11/01 by Steve.Robb Visual Studio debugger visualizers for delegates. Change 3182738 on 2016/11/02 by Graeme.Thornton Re-enable signed archive reader so non-pakpreacher based reads still get signature checked Change 3183420 on 2016/11/02 by Steve.Robb Fix to TIsZeroConstructType for TScriptDelegate. Change 3184872 on 2016/11/03 by Robert.Manuszewski Fixing memory stomps in SSL certificate initialization (found with mallocstomp) Change 3184873 on 2016/11/03 by Robert.Manuszewski Adding thread safety checks to async loading code Change 3185535 on 2016/11/03 by Ben.Zeigler Fix it so calling CreateDefaultSubobject with bTransient = true sets the object transient flag. This fixes EDL Crashes involving components. Change 3186636 on 2016/11/04 by Graeme.Thornton AES encryption integrated into EDL system Pak signing and AES encryption now configurable by ini files rather than magical text files Change 3186637 on 2016/11/04 by Graeme.Thornton Configured pak signing and encryption in ShooterGame for reference Change 3186639 on 2016/11/04 by Graeme.Thornton Encryption changes for Orion * Move pak signing keys into new INI format * Add AES key and enable INI file encryption Change 3186661 on 2016/11/04 by Graeme.Thornton Change unrealpak command line params to accept AES key as a separete parameter Change 3186670 on 2016/11/04 by Robert.Manuszewski Adding a null check before using a package pointer in Linker code #jira UE-38237 Change 3186775 on 2016/11/04 by Graeme.Thornton Fix UBT defines that come in as quoted strings, losing the quotes when passed to the compiler - PS4 and Mac fixes. Other platforms might need fixing too! Change 3186823 on 2016/11/04 by Graeme.Thornton Fixed an incorrect size check in the EDL pak signing code Change 3186925 on 2016/11/04 by Graeme.Thornton Allow UnrealPak to read encryption settings from project ini files Change 3189885 on 2016/11/08 by Graeme.Thornton Static analysis warning fix Change 3190015 on 2016/11/08 by Robert.Manuszewski Thread safety fix for UBlueprintGeneratedClass::PostLoadDefaultObject while UBlueprintGeneratedClass::SerializeDefaultObject runs on the async loading thread Change 3190253 on 2016/11/08 by Chris.Wood Improved MDD performance for on the CR server. [UE-37566] - Improve MDD performance on CR server Blocked MDD init'ing the crash handling code as it isn't desirable on the server. Removed redundant call to SetSymbolPathsFromModules() from CrashDebugHelper. Change 3192993 on 2016/11/10 by Robert.Manuszewski Thread Heartbeat will no longer report the same hang multiple times. Change 3193111 on 2016/11/10 by Robert.Manuszewski Minor change in the condition that detects the same hangs - allow the same callstacks from different threads Change 3193168 on 2016/11/10 by Steve.Robb TSparseArray now reserves space in reverse so that new elements get added to the front of the allocation rather than the back, which is better for memory traversal and meets expectations more closely. Change 3193171 on 2016/11/10 by Steve.Robb Easier debugging of FPendingRegistrantInfo map. Change 3193188 on 2016/11/10 by Steve.Robb TAutoPointer deprecated. Change 3193796 on 2016/11/10 by Graeme.Thornton Fix pak creation failure when no pak signing keys are supplied Change 3194524 on 2016/11/11 by Graeme.Thornton Another static analysis warning fix Change 3195119 on 2016/11/11 by Steve.Robb TAutoPtr deprecated. Fixes to use of TAutoPtr with incompatible memory deallocations (TAutoPtr with FMemory::Malloc and new[]). Some large headers moved into .cpp files. Change 3196582 on 2016/11/14 by Gil.Gribb UE4 - Changed a check to a warning related to detaching linekrs twice. Seen in nativized BP version of platformer game. Change 3196878 on 2016/11/14 by Steve.Robb TScopedPointer deprecated. Change 3198061 on 2016/11/15 by Steve.Robb Class array is no longer regenerated when saving UClasses. Change 3198065 on 2016/11/15 by Robert.Manuszewski Making AssembleReferenceTokenStream thread safe for blueprints loaded on the async loading thread. Change 3198199 on 2016/11/15 by Robert.Manuszewski Pak platform file will now only be used if pak files exist regardless of command line paraks like -pak, -singedpak and -signed. Change 3199954 on 2016/11/16 by Graeme.Thornton Removing USING_SIGNED_CONTENT Change 3200221 on 2016/11/16 by Chris.Wood CrashReportProcess code cleanup - removing unused using directives Change 3200232 on 2016/11/16 by Chris.Wood Multiple CrashReportProcess updates and improvements (CRP v1.2.6) UE-36248 - CRP scalability: All bulk storage or shared data to S3 or suitable network drives InvalidCrashReports now saved to S3 instead of local folder Removed option tosync MinidumpDiagnostics from Perforce Moved MinidumpDiagnostics from old Perforce synched location to its own folder in E:\Services (makes more sense with manual publishing) Added improved logging to Slack with option to monitor MDD performance Added hourly log folders to MDD logs Added support for types of crashes we don't want to symbolicate (using it to skip callstack gen for hang detected ensures) Change 3200382 on 2016/11/16 by Robert.Manuszewski Async Loading code will now detach the linker when resetting async package loader to avoid situations when loading the same asset multiple times results in the following load request finding the old linker after the package has been loading but the async package hasn't been deleted yet (async package for the old request in limbo state but linker exists). Change 3200562 on 2016/11/16 by Gil.Gribb UE4 - Fixed rare issue with reloading nativized blueprints with the EDL and a minor simplication. Change 3201093 on 2016/11/16 by Ben.Zeigler #UE 38654 Fix EDL cooking to correctly search components created directly by UBlueprints, as well as the CDO components it already covered. Also explicitly mark subobject templates as editor only. Fix issue where the AssetImportData associated with Blueprint-owned Curves was ending up in the cooked subobject template list. Stopped it from creating those objects, and mark the class editor only. Change 3201736 on 2016/11/17 by Steve.Robb Strtoi64 platform and TCString functions. #fyi robert.manuszewski Change 3201938 on 2016/11/17 by Ben.Woodhouse Dummy integrate of the Square render version workaround (CL 3201913) with _accept target_ to prevent it being integrated to dev-core in future. Commandline: p4 integrate //Tasks/UE4/Dev-LoadTimes/Engine/Source/Runtime/CoreUObject/Private/UObject/LinkerLoad.cpp@3201913,3201913 //UE4/Dev-Core/Engine/Source/Runtime/CoreUObject/Private/UObject/LinkerLoad.cpp #fyi robert.manuszewski Change 3203757 on 2016/11/18 by Robert.Manuszewski Removing debug code from async loading code. Change 3203927 on 2016/11/18 by Robert.Manuszewski Fixing comments in the async loading code. Change 3204851 on 2016/11/18 by Steve.Robb Metafunction for testing if a particular operator<< overload exists, e.g. THasInserterOperator<FArchive&, FMyType&>::Value. Change 3204854 on 2016/11/18 by Steve.Robb UEnumProperty. Change 3205027 on 2016/11/18 by Ben.Zeigler Add useful functions to FAssetPtr and TAssetSubclassOf that already existed on TAssetPtr Add Get() to TSubclassOf so it matches our other wrappers Fix TSubclassOf and TAssetSubclassOf to use the more efficient template method of checking class compatibility Comment and template cleanups for AssetPtr, StringAssetReference, LazyPtr, and SubclassOf Change 3206334 on 2016/11/21 by Ben.Zeigler #UE-38773: Fix it so non-component template subobjects of CDOs are not included as creation dependencies for BP classes, also clean up GetPreloadDependencies as it was adding redundant and null entries #UE-38799: Fix it so WidgetTrees don't get picked up as subobjects, and add ensure at cook time to find null outers that would crash at runtime. Make sure the instanced widget trees are transient. Cook finishes but game is still crashing in some cases, so I might adjust this after other testing Change 3206353 on 2016/11/21 by Ben.Zeigler Fix EnumProperty to handle EDL preload dependencies properly Change 3206625 on 2016/11/21 by Ben.Zeigler Fix enum property crash at runtime by copying what array property does and making sure inner property is not transient Change 3206937 on 2016/11/21 by Ben.Zeigler #jira UE-38905 Fix it so enums inside arrays are migrated properly, the enum tag is lost so use the current one Disable other nested enum migrations as they are unlikely to work. Array property tags need to be refactored to be safer Correctly save enum tag for enum properties, it was being set but not serialized Change 3207002 on 2016/11/21 by Ben.Zeigler #jira UE-38799 Fix it so per-widget copy of widget tree and all widgets inside are properly transient, they were being cooked before but never accessed. Fix case where non ClientOnly public objects nested instead ClientOnly objects would cook but fail to load, and add ensure to catch these cases in the future. If the full outer chain isn't available, it can't be loaded anyway, and this finds issues at cook time instead of load time. We should generally outlaw non-transient objects with transient outers, it does not do what people expect. Change 3207032 on 2016/11/21 by Ben.Zeigler #jira UE-38654 Re-Fix EDL cooking with SCS-added components. They used to have the DefaultSubObject flag but no longer do [CL 3208270 by Ben Zeigler in Main branch]
2016-11-22 18:45:44 -05:00
*SignatureWriter << EncryptedMasterHash;
*SignatureWriter << ChunkHashes;
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
delete SignatureWriter;
Copying //UE4/Orion-Staging to //UE4/Main (Origin: //Orion/Dev-General @ 2904087) ========================== MAJOR FEATURES + CHANGES ========================== #lockdown Nick.Penwarden Change 2903938 on 2016/03/10 by Frank.Gigliotti Added an instance ID to FAnimMontageInstance #CodeReview Laurent.Delayen #RB Laurent.Delayen #Tests PIE Change 2903745 on 2016/03/10 by Wes.Hunt Update Oodle TPS #rb none #tests none #codereview:john.pollard Change 2903689 on 2016/03/10 by Uriel.Doyon New "LogHeroMaterials" console command, displaying the current state of materials and textures on the character hero. #rb marcus.wasmer #codereview marcus.wassmer #tests editor, playing PC games, trying the new command Change 2903669 on 2016/03/10 by Aaron.McLeran OR-17180 Make stat soundcues and stat soundwaves NOT display zero volume sounds - Change only effects debug stat commands for audio guys #rb none #tests played paragon with new debug stat commands, confirms doesn't show zero-volume sounds Change 2903625 on 2016/03/10 by John.Pollard XB1 Oodle SDK #rb none #tests none #codereview Jeff.Campeau Change 2903577 on 2016/03/10 by Ben.Marsh Remaking latest build scripts from //UE4/Main @ 2900980. Change 2903560 on 2016/03/10 by Ben.Marsh Initial version of BuildGraph scripts - used to create build processes in UE4 which can be run locally or in parallel across a build farm (assuming synchronization and resource allocation implemented by a separate system). Intended to supersede GUBP. Build graphs are declared using an XML script using syntax similar to MSBuild, ANT or NAnt, and consist of the following components: * Tasks: Building blocks which can be executed as part of the build process. Many predefined tasks are provided (<Cook>, <Compile>, <Copy>, <Stage>, <Log>, <PakFile>, etc...), and additional tasks may be added be declaring classes derived from AutomationTool.CustomTask in other UAT modules. * Nodes: A named sequence of tasks which is executed to produce outputs. Nodes may have input dependencies on other nodes before they can be executed. Declared with the <Node> element in scripts. * Agent Groups: A set of nodes nodes which is executed on the same machine if running as part of a build system. Has no effect when building locally. Declared with the <Group> element in scripts. * Triggers: Container for groups which should only be executed when explicitly triggered (using the -Trigger=<Name> or -SkipTriggers command line argument). Declared with the <Trigger> element in scripts. * Notifiers: Specifies email recipients for failures in one or more nodes, whether they should receive notifications on warnings, and so on. Properties can be passed in to a script on the command line, or set procedurally with the <Property Name="Foo" Value="Bar"/> syntax. Properties referenced with the $(Property Name) notation are valid within all strings, and will be expanded as macros when the script is read. If a property name is not set explicitly, it defaults to the contents of an environment variable with the same name. Local properties, which only affect the scope of the containing XML element (node, group, etc...) are declared with the <Local Name="Foo" Value="Bar"/> element, and will override a similarly named global property for the local property's scope. Any elements can be conditionally defined via the "If" attribute, and are largely identical to MSBuild conditions. Literals in conditions may be quoted with single (') or double (") quotes, or an unquoted sequence of letters, digits and underscore characters. All literals are considered identical regardless of how they are declared, and are considered case-insensitive for comparisons (so true equals 'True', equals "TRUE"). Available operators are "==", "!=", "And", "Or", "!", "(...)", "Exists(...)" and "HasTrailingSlash(...)". A full grammar is written up in Condition.cs. File manipulation is done using wildcards and tags. Any attribute that accepts a list of files may consist of: a Perforce-style wildcard (matching any number of "...", "*" and "?" patterns in any location), a full path name, or a reference to a tagged collection of files, denoted by prefixing with a '#' character. Files may be added to a tag set using the <Tag> Task, which also allows performing set union/difference style operations. Each node can declare multiple outputs in the form of a list of named tags, which other nodes can then depend on. Build graphs may be executed in parallel as part build system. To do so, the initial graph configuration is generated by running with the -Export=<Filename> argument (producing a JSON file listing the nodes and dependencies to execute). Each participating agent should be synced to the same changelist, and UAT should be re-run with the appropriate -Node=<Name> argument. Outputs from different nodes are transferred between agents via shared storage, typically a network share, the path to which can be specified on the command line using the -SharedStorageDir=<Path> argument. Note that the allocation of machines, and coordination between them, is assumed to be managed by an external system. A schema for the known set of tasks can be generated by running UAT with the "-Schema=<FileName>" option. Generating a schema and referencing it from a BuildGraph script allows Visual Studio to validate and auto-complete elements as you type. #rb none #codereview Marc.Audy, Wes.Hunt, Matthew.Griffin, Richard.Fawcett #tests local only so far, but not part of any build process yet Change 2903539 on 2016/03/10 by John.Pollard Improve replay playback debugging of character movement #rb none #tests replays Change 2903526 on 2016/03/10 by Ben.Marsh Remake changes from //UE4/Main without integration history, to add support for BuildGraph tasks. #rb none #tests none Change 2903512 on 2016/03/10 by Dan.Youhon Modify minimum Duration values for JumpForce and MoveToForce ability tasks so that having minimum Duration values doesn't trigger check()s #rb None #tests Compiles Change 2903474 on 2016/03/10 by Marc.Audy Fix crash if ChildActor is null #rb None #tests None Change 2903314 on 2016/03/10 by Marc.Audy Fix ParentComponent not being persisted and fixup content that was saved in the window it was broken #rb James.Golding #tests Selection of child actors works as expected #jira UE-28201 Change 2903298 on 2016/03/10 by Simon.Tovey Disabling the trails optimization. #tests none #rb none #codereview Olaf.Piesche Change 2903124 on 2016/03/10 by Robert.Manuszewski Small refactor to pak signing to help with exe protection #rb none #tests none [CL 2907678 by Andrew Grant in Main branch]
2016-03-13 18:53:13 -04:00
return FArchive::Close();
}
void FSignedArchiveWriter::Serialize(void* Data, int64 Length)
{
// Serialize data to a buffer. When the max buffer size is reached, the buffer is signed and
// serialized to disk with its signature
uint8* DataToWrite = (uint8*)Data;
int64 RemainingSize = Length;
while (RemainingSize > 0)
{
int64 BufferPos = BufferArchive.Tell();
int64 SizeToWrite = RemainingSize;
if (BufferPos + SizeToWrite > FPakInfo::MaxChunkDataSize)
{
SizeToWrite = FPakInfo::MaxChunkDataSize - BufferPos;
}
BufferArchive.Serialize(DataToWrite, SizeToWrite);
if (BufferArchive.Tell() == FPakInfo::MaxChunkDataSize)
{
SerializeBufferAndSign();
}
SizeOnDisk += SizeToWrite;
PakSize += SizeToWrite;
RemainingSize -= SizeToWrite;
DataToWrite += SizeToWrite;
}
}
int64 FSignedArchiveWriter::Tell()
{
return PakSize;
}
int64 FSignedArchiveWriter::TotalSize()
{
return PakSize;
}
void FSignedArchiveWriter::Seek(int64 InPos)
{
UE_LOG(LogPakFile, Fatal, TEXT("Seek is not supported in FSignedArchiveWriter."));
}
/**
* Useful code for testing the encryption methods
*/
void TestEncryption()
{
FEncryptionKey PublicKey;
FEncryptionKey PrivateKey;
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
TEncryptionInt P(TEXT("0x21443BD2DD63E995403"));
TEncryptionInt Q(TEXT("0x28CBB6E5749AC65749"));
FEncryption::GenerateKeyPair(P, Q, PublicKey, PrivateKey);
// Generate random data
const int32 DataSize = 1024;
uint8* Data = (uint8*)FMemory::Malloc(DataSize);
for (int32 Index = 0; Index < DataSize; ++Index)
{
Data[Index] = (uint8)(Index % 255);
}
// Generate signature
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
FDecryptedSignature OriginalSignature;
FEncryptedSignature EncryptedSignature;
FDecryptedSignature DecryptedSignature;
OriginalSignature.Data = FCrc::MemCrc32(Data, DataSize);
// Encrypt signature with the private key
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
FEncryption::EncryptSignature(OriginalSignature, EncryptedSignature, PrivateKey);
FEncryption::DecryptSignature(EncryptedSignature, DecryptedSignature, PublicKey);
// Check
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
if (OriginalSignature == DecryptedSignature)
{
UE_LOG(LogPakFile, Display, TEXT("Keys match"));
}
else
{
UE_LOG(LogPakFile, Fatal, TEXT("Keys mismatched!"));
}
Copying //UE4/Dev-Core to //UE4/Main ========================== MAJOR FEATURES + CHANGES ========================== Change 2877442 on 2016/02/23 by Graeme.Thornton@GThornton_DevCore Allow 'dumpnonframe' stat command to also take an optional group name which will filter the output Change 2879289 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Removed "-" in switch string for load time tracker. Makes -NoLoadTrackClear cmd line param actually work Change 2879323 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore PC builds now log engine initialization time with the load time tracker Change 2879418 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Optimization for FSignedArchiveReader - Compute incoming data hash in parallel on another thread while we serialize and decrypt the signature on the chunk cache worker thread Change 2879419 on 2016/02/24 by Graeme.Thornton@GThornton_DevCore Extra stats for PakFile group - Time reading from pak handles - Number of open pak handles Change 2881199 on 2016/02/25 by Graeme.Thornton@GThornton_DevCore Use 32 bit CRCs to hash chunk contents rather than SHA1. Much faster to compute than SHA1, and the decrypting 4 bytes rather than 20 is also a lot faster Change 2885687 on 2016/02/29 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Close disregard for GC pool when disabling it to leave it in a good state. Change 2887358 on 2016/03/01 by Steve.Robb@Dev-Core FlushCurrentThreadCache calls forwarded on from FMalloc wrappers. Change 2887398 on 2016/03/01 by Steve.Robb@Dev-Core Stats for MallocBinned2. Change 2887682 on 2016/03/01 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added code to catch game hangs. If game thread is stalled for more than 5 seconds we will catch that, report its callstack and crash. Change 2889573 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Renamed the new StackWalkAndDump to ThreadStackWalkAndDump to avoid name collisions and build errors. Change 2889687 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Send thread heartbeats during potentially slow tasks. Change 2889963 on 2016/03/02 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Added the ability to query the name of a thread given its TLS id. Change 2890439 on 2016/03/02 by Steve.Robb@Dev-Core MallocBinned redirection header, to allow easy switching between MallocBinned and MallocBinned2. Switch back to MallocBinned by default. Change 2891799 on 2016/03/03 by Steve.Robb@Dev-Core Shrunken GMallocBinned2MemSizeToIndex array. Improvement of some shared templated alignment-related functions. Change 2891812 on 2016/03/03 by Graeme.Thornton@GThornton_DevCore Refactoring of pak signing to be more compatible with patching - Store signatures in a block at the end of the pak file - Optimize decryption of signatures by serializing them all when the pak is opened, and decrypting on the chunk cache worker thread between read requests. Change 2891813 on 2016/03/03 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream2 Don't compile UnrealPak with ICU Change 2892386 on 2016/03/03 by Steve.Robb@Dev-Core GitHub #2070 - Fix incorrect FMallocDebug::Malloc alignment checking Change 2894189 on 2016/03/04 by Steve.Robb@Dev-Core Memswap fixes for sizes of 3, 5, 6 and 7, and early out on 0. Change 2894427 on 2016/03/04 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Don't display message boxes in unattended mode when D3D device is removed. Change 2896912 on 2016/03/07 by Graeme.Thornton@GThornton_DevCore Disable reading of anything outside the pak file when in shipping builds Change 2897053 on 2016/03/07 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 A few improvements to thread heartbeat thread: - Don't check for heartbeat in debug builds - Don't spawn the thread heartbeat thread for programs - Ignore stalls when exiting Change 2898752 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Fixing race conditions when using malloc verify Change 2899075 on 2016/03/08 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 UnrealPak will now accept hex values for p and q. Added the command line param to trust p and q. Change 2899091 on 2016/03/08 by Steve.Robb@Dev-Core Fix for TQueue::Peek const-correctness. Change 2902775 on 2016/03/10 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 Adding code to test signature keys. Change 2905040 on 2016/03/11 by Robert.Manuszewski@Robert.Manuszewski_NCL_Stream1 TBigInt Multiply is now 30x faster Change 2905218 on 2016/03/11 by Steve.Robb@Dev-Core New FCString::Strspn and FCString::Strcspn functions. [CL 2921423 by Robert Manuszewski in Main branch]
2016-03-24 13:53:55 -04:00
double OverallTime = 0.0;
double OverallNumTests = 0.0;
for (int32 TestIndex = 0; TestIndex < 10; ++TestIndex)
{
static const int64 NumTests = 500;
double Timer = FPlatformTime::Seconds();
{
for (int64 i = 0; i < NumTests; ++i)
{
FEncryption::DecryptSignature(EncryptedSignature, DecryptedSignature, PublicKey);
}
}
Timer = FPlatformTime::Seconds() - Timer;
OverallTime += Timer;
OverallNumTests += (double)NumTests;
UE_LOG(LogPakFile, Display, TEXT("%i signatures decrypted in %.4fs, Avg = %.4fs, OverallAvg = %.4fs"), NumTests, Timer, Timer / (double)NumTests, OverallTime / OverallNumTests);
}
FMemory::Free(Data);
}