Files
UnrealEngineUWP/Engine/Source/Developer/HotReload/Private/HotReload.cpp
Ben Marsh cfc9f4775f Copying //UE4/Dev-Core to //UE4/Dev-Main (Source: //UE4/Dev-Core @ 3805092)
#lockdown Nick.Penwarden
#rb none

============================
  MAJOR FEATURES & CHANGES
============================

Change 3623004 by Ben.Marsh

	Fix RemoteExecutor not taking the remote machine specs into account.

Change 3623172 by Ben.Marsh

	UGS: Fix "More Info..." button not using P4 server override.

Change 3628820 by Ben.Marsh

	PR #3979: Get working directory from task element, not tool node (Contributed by nullbus)


Change 3630424 by Graeme.Thornton

	Make the AES key parameter const in FAES::EncryptData()

Change 3632786 by Steve.Robb

	FString constructor fixed to not take an ignored void* parameter, which can be misleading.

Change 3639534 by Ben.Marsh

	Remove old P4.NET library. Doesn't seem to be used by anything.

Change 3640536 by Steve.Robb

	GitHub #4007 : Delete unnecessary specialization of MakeArrayView

	#jira UE-49617

Change 3641155 by Gil.Gribb

	UE4 - Speculative fix for problem with summary reading in FAsyncArchive2.

Change 3643932 by Ben.Marsh

	Add an example build script for updating the version number, then compiling and staging the editor and tools to an output directory. Optionally submits at the end (requires -Submit argument).

Change 3644825 by Ben.Marsh

	Use VSWHERE to find the location of MsBuild.exe, if available.

	https://github.com/EpicGames/UnrealEngine/pull/3879#issuecomment-329688645

Change 3647395 by Ben.Marsh

	Allow compiling of monolithic binaries from BuildEditorAndTools.xml, using the -set:GameTarget=FooGame -set:TargetPlatforms=Win32;Win64 options.

Change 3650300 by Ben.Marsh

	UAT: Remove code that deletes cooked data on a failed cook. The engine should write packages out transactionally now (by writing to a temporary file and moving into place), and deleting the cooked data just prevents post-mortem analysis.

Change 3650856 by Robert.Manuszewski

	Adding checks to prevent FlushAsyncLoading and LoadObject/LoadPackage from being called from any threads other than the game thread

Change 3651022 by Gil.Gribb

	UE4 - Possible fix for mysterious ensure indicating problematic recursion in the pak precacher.

Change 3658331 by Steve.Robb

	Fix for the parsing of large integer values.

Change 3661958 by Gil.Gribb

	UE4 - Fixed rare hang in task graph.

Change 3664021 by Robert.Manuszewski

	Fix for a potential GC crash caused by stale pointer in AnimInstanceProxy

	See https://udn.unrealengine.com/questions/392432/gc-issue-uaniminstancemontageinstances-empty-but-u.html

Change 3664254 by Steve.Robb

	Use ANSI allocator when thread sanitizer is enabled.  This allows the generation of more accurate and meaningful reports.

Change 3664436 by Steve.Robb

	Use TUniquePtr instead of a thread-unsafe TSharedPtr to move data between threads.

Change 3666461 by Graeme.Thornton

	Improvements to signing/encryption key embedding and runtime access
	 - Changed method of embedding key into the executable to make it more secure
	 - Added FAESKey class to wrap a 32 byte key

Change 3666462 by Graeme.Thornton

	Cut ShooterGame AES key down to 32 characters

Change 3677560 by Ben.Marsh

	PR #4074: UBT: Add include and library-related fields to module JSON output (Contributed by adamrehn)


Change 3683534 by Steve.Robb

	Refactoring of enum/struct lookup during hot reload.

Change 3683754 by Steve.Robb

	Alignment fixes to allow int64 on 32-bit platforms
	Support for integral types in IsAligned.
	Static asserts so that alignment functions will no longer be called with non-intergal, non-pointer types.
	Some fixes to existing code.

Change 3686670 by Steve.Robb

	Fix for thread-unsafe modification of static array in FString::ParseIntoArrayWS.

Change 3687540 by Ben.Marsh

	Fix all UBT/UAT output going to stderr rather than stdout.

Change 3688931 by Gil.Gribb

	UE4 - Critical fix for a rare race condition in the pak file async IO layer.

Change 3690000 by Graeme.Thornton

	Manual copy of 4.18 CL 3687869

	Make UBT include the destination INI file for a given hierarchy if it exists
	Renamed VSCode enum value to VisualStudioCode, so it matches the source accessor plugin name

Change 3690030 by Graeme.Thornton

	VSCode fixes
	 - Source Code Accessor plugin changes. Add new interface method to open a solution at a given path
	 - GameProjectUtils now uses the source navigation API to open solutions rather than hardcoding which solution file types to look for
	 - Various fixes for vscode project file generation

	#jira UE-50554

Change 3690885 by Steve.Robb

	Atomic reads in FReferenceControllerOps<ESPMode::ThreadSafe>.

Change 3691052 by Steve.Robb

	Free stats thread on shutdown.

Change 3695138 by Steve.Robb

	AsConst helper function added.

Change 3696627 by James.Hopkin

	Changed player controller iterator typedefs to use TWeakObjectPtr rather than the deprecated TAutoWeakObjectPtr

	(review-3606695)

Change 3697099 by Steve.Robb

	GitHub #4105 : Removed redundant class access modifier

Change 3697154 by Steve.Robb

	Removal of deprecated functions in delegates.
	Mutable lambdas to can now be bound to delegates.

Change 3697180 by Steve.Robb

	GitHub #4115 : Incorrect CPPMacroType used for USoftClassProperty

Change 3697239 by Steve.Robb

	Allow TArray::Insert to take an array with any allocator type.

Change 3697269 by Steve.Robb

	RelocateConstructItems instead of MoveConstructItems.

Change 3697558 by Steve.Robb

	New _GetRef functions for TArray, which return a reference to the newly-added element.
	Unit tests for these functions.

Change 3699776 by Steve.Robb

	TSAN warning suppression around IAsyncReadRequest::bCompleteAndCallbackCalled.

Change 3702397 by Steve.Robb

	TIsTrivial type trait.

Change 3702569 by Steve.Robb

	Allow a TGuardValue to be assigned to a different type from the one being guarded.

Change 3706644 by Robert.Manuszewski

	Different stack ingore count for development builds for FArchiveStackTrace

Change 3709272 by Steve.Robb

	Removal of redundant UpdateVertices, which causes a race condition on the renderer thread.

Change 3709452 by Robert.Manuszewski

	Fixed a bug where with async time limit set to a low value the async loading could hang because the linker would keep reloading the preload dependencies

Change 3709454 by Robert.Manuszewski

	Added command line option -NOEDL to disable EDL

Change 3709487 by Steve.Robb

	Remove use of PLATFORM_HAS_64BIT_ATOMICS, which is always 1.

Change 3709645 by Ben.Marsh

	Fix race condition between multiple instances of UBT trying to write out the XML config cache.

Change 3711193 by Ben.Marsh

	Add an editor setting for the shared DDC location to use.

	#jira UE-51487

Change 3713811 by Steve.Robb

	Update .modules files after a hot reload.
	Don't check for directory timestamp changes as a way of detecting new files if hot reloading with a makefile, as this is already done during makefile invalidation checks.
	Pass hotreload flags around in UBT instead of relying on global state.

	This fixes the hot reload iteration speed regression without also regressing the fix to UE-42205.

	#jira UE-51472

Change 3715654 by Steve.Robb

	GitHub #4156 : Fixed not compiling template function Algo::UpperBoundBy.

Change 3718782 by Steve.Robb

	TSharedPtr, TSharedRef and TWeakPtr assignment are now implemented as copy-and-swap to avoid an invalid smart pointer state being visible to any destructors being called.

Change 3720830 by Steve.Robb

	Initial import of TAtomic object wrapper, which guarantees atomic access to an object.

Change 3720881 by Steve.Robb

	FCompression ThreadSanitizer data race fixes.

Change 3722640 by Graeme.Thornton

	Guard network platform file heartbeat function with the socket critical section. Stop heartbeat from causing a crash when firing during async loading.

	#jira UE-51463

Change 3722655 by Steve.Robb

	Don't null name table because it's already zeroed at startup.
	Some tidy-ups.

Change 3722754 by Steve.Robb

	Thread sanitizer fix.
	Small typo fix.

Change 3722849 by Graeme.Thornton

	Improve "caching file" message in networkplatformfile so it says "Requesting file..." and is only output when we actually request the file from the server

Change 3723081 by Steve.Robb

	TAtomic is now aligned to the underlying integer type.
	TAtomic will now static assert with a better error message when given an unsupported type.
	Define added for the maximum platform-supported atomic type, and used instead of a (wrong) hardcoded number.
	Misc renames.

Change 3723270 by Ben.Marsh

	Include /d2cgsummary argument when running UBT with -Timing.

Change 3723683 by Ben.Marsh

	Do not include documentation in the generated project files by default. Suspect that the 30,000 UDN files that get added to the solution take up memory and degrate performance.

Change 3725422 by Robert.Manuszewski

	When serializing compressed archive with multithreaded compression enabled, wait for the oldest async task instead of spinning.

Change 3725735 by Robert.Manuszewski

	Making all CheckDefaultSubobjects related functions const

Change 3726167 by Steve.Robb

	FMinimalName::IsNone added.

Change 3726458 by Steve.Robb

	TAtomic will no longer instantiate for types which are not exactly a size supported by the platform layer.

Change 3726542 by Ben.Marsh

	UGS: Always include the project filename in the editor build command. The project may not be in one of the .uprojectdirs paths.

Change 3726595 by Ben.Marsh

	Allow building multiple game targets in the example BuildEditorAndTools.xml script.

Change 3726724 by Ben.Marsh

	Fix ambiguities in calculating root directory. (GitHub #4172)

Change 3726959 by Ben.Marsh

	Make sure that AutomationTool uses the same list of preprocessor definitions when compiling *.target.cs files as UnrealBuildTool does.

Change 3728437 by Steve.Robb

	VisitTupleElements now supports invocation of a functor taking arguments from multiple tuples in parallel.
	Some improved documentation.

	NOTE: This is a backward-incompatible change to VisitTupleElements.  Any existing calls will need their arguments swapping.

Change 3732262 by Gil.Gribb

	UE4 - Fixed rare hangs in the task graph.

Change 3732755 by Steve.Robb

	Stats TSAN fixes.
	Optimizations to FCycleCounter::Start() to only read the stat name once.

Change 3735000 by Robert.Manuszewski

	Always preload the AssetRegistry module on startup. even if EDL is disabled.

	Even without EDL, if the async loading thread is enabled the AssetRegistryModule will otherwise be loaded from the ASL thread and that will assert.

Change 3735292 by Robert.Manuszewski

	Made sure component visualizer is removed from VisualizersForSelection when UnregisterComponentVisualizer() is called otherwise it may cause crashes when the engine terminates.

Change 3735332 by Steve.Robb

	Refactoring of UDelegateProperty::Identical() to clarify logic.
	Fixed UMulticastDelegateProperty::Identical() to compare the bound function names.
	PPF_DeltaComparison removed, as it doesn't seem useful.

Change 3737960 by Graeme.Thornton

	VSCode - Add launch task for generating project files for the given folder

Change 3738398 by Graeme.Thornton

	Make Visual Studio source code accessor's module hotreload handler pass the 'save all files' message to the current accesor, rather than direct to the visual studio accessor

	#jira UE-51451

Change 3738405 by Graeme.Thornton

	VSCode: Format c/cpp settings strings using comment path formatting function

Change 3738928 by Steve.Robb

	Fix for lack of null conditional operators in some older Monos. (replicated from CL# 3729574 in Release-4.18)

	#jira UE-51842

Change 3739135 by Ben.Marsh

	Fix being unable to package projects in a folder called "Wolf". This is only a restricted folder for Epic's Perforce history.

	#jira UE-51855

Change 3739360 by Ben.Marsh

	UAT: Fix issue with P4PORT setting not being parsed correctly.

Change 3745959 by James.Hopkin

	#core Added ImplicitConv for safe upcasts to a specific required type, e.g. deduced delegate payload types

Change 3746125 by Steve.Robb

	FName ThreadSanitizer fixes.

Change 3747274 by Steve.Robb

	TSAN fix for FMediaTicker::Stopping.

Change 3747618 by Steve.Robb

	ThreadSanitizer data race fix for FShaderCompileThreadRunnableBase::bForceFinish.

Change 3747720 by Steve.Robb

	ThreadSanitizer fix for FMessageRouter::Stopping.

Change 3749207 by Graeme.Thornton

	First pass of CryptoKeys plugin. Allows creation/editing/cycling of AES/RSA keys.

Change 3749323 by Graeme.Thornton

	Fix UAT crash when only -targetplatform is specifiied

Change 3749349 by Steve.Robb

	TSAN_SAFE guards around LockFreeList to silence ThreadSanitizer.

Change 3749617 by Steve.Robb

	Logf static_assert for formatting string enabled.

Change 3749897 by Steve.Robb

	FDebug::LogAssertFailedMessage static assert for formatting string enabled.

Change 3754011 by Steve.Robb

	Static asserts that the allocator supports move.
	Move-enabled our allocators which don't support move.

Change 3754227 by Ben.Marsh

	Fix build command line in generated projects missing a space before the compiler version override.

	#jira UE-52226

Change 3754562 by Ben.Marsh

	PR #4206: Replace deprecated wsprintf with secure swprintf for Bootstrap executable (Contributed by jessicafalk)


Change 3755616 by Graeme.Thornton

	Runtime code for using the new crypto ini files to define signing/encryption keys

	#jira UE-46580

Change 3755666 by James.Hopkin

	Used ImplicitConv to remove Casts being used for up-casts

	#review-3745965

Change 3755671 by Graeme.Thornton

	Add log message in unrealpak to say which config file system it is using for crypto keys

Change 3755672 by Graeme.Thornton

	Updating ShooterGame with new CryptoKeys based security setup

Change 3756778 by Ben.Marsh

	Add support for running multiple jobs simultaneously on a single builder.

	When running job or agent setup, the --num-slots=X parameter defines the number of steps that can run simultaneously (EC procedures pass in the resource step limit). A lock file is created under the workspace root (D:\Build) and a reservation file is created for the first slot that can be allocated (slot-1, slot-2, etc...). The slot number is used to define the workspace name that should be used.

Change 3758498 by Ben.Marsh

	Re-throw exceptions when a file cannot be deleted when cleaning a target.

Change 3758921 by Steve.Robb

	ThreadSanitizer fix to FThreadSafeStaticStatBase::HighPerformanceEnable to do a relaxed atomic load on access.
	DoSetup() now returns the newly-allocated pointer, instead of reloading it from memory.

Change 3760599 by Graeme.Thornton

	Added missing epic header comment to some new source files

Change 3760642 by Steve.Robb

	ThreadSanitizer fix for concurrent access to GMainThreadBlockedOnRenderThread.

Change 3760669 by Graeme.Thornton

	Improvement to OpenSSL based signing key generator. Generate a full RSA key then steal the primes from it, rather than generating the primes manually.

	Added a test mode to the cryptokeys commandlet to test signing key generation

Change 3760711 by Steve.Robb

	ThreadSanitizer fixes to GIsRenderingThreadSuspended.

Change 3760739 by Steve.Robb

	ThreadSanitizer fix for FQueuedThread::TimeToDie.

Change 3760763 by Steve.Robb

	ThreadSanitizer fix for GRunRenderingThreadHeartbeat.
	Removal of unnecessary/dangerous initializer for GMainThreadBlockedOnRenderThread.

Change 3760793 by Steve.Robb

	Some simple refactoring to remove some volatile reads of BufferStartPos and BufferEndPos.

Change 3760817 by Steve.Robb

	ThreadSanitizer fixes for FAsyncWriter::BufferStartPos and BufferEndPos.

Change 3761331 by Josh.Engebretson

	UnrealBuildTool enforcement of Development and Debug configurations in existing .csproj

	#jira UE-52416

Change 3761521 by Steve.Robb

	ThreadSanitizer fixes for FEvent::EventStartCycles and EventUniqueId.

Change 3763117 by Graeme.Thornton

	PR #3722: Optimising FPaths::IsRelative() (Contributed by jovisgCL)


Change 3763358 by Graeme.Thornton

	Ensure that all branches within FGenericPlatformMisc::RootDir() produce an absolute path with no duplicate slashes
	Remove relative->abs conversion of root dir from FPaths::MakeStandardFilename(), now that we know RootDir() always returns an absolute path

	Derived from the content of this PR:
	PR #3742: Treat RootDirectory the same way as Standardized (Contributed by TroutZhang)


Change 3764058 by Graeme.Thornton

	Generate a .code-workspace file for the current workspace. Allows foreign projects to "mount" the UE4 folder so that the engine tasks are avaible, and all engine source is visible to VSCode for searching purposes

	#jira UE-52359

Change 3764705 by Steve.Robb

	Better handling of whitespace in ImportText_Internal() for set and map properties.
	Containers are now emptied upon import failure, to avoid leaving bad container states (unhashed, partial data).
	Fix to USetProperty's temp buffer size to avoid buffer overruns.
	Duplicate map keys are now skipped during import, same as USetProperty's behavior.

Change 3764731 by Steve.Robb

	Don't re-run UHT if only source files have changed in the same folder as headers.  This was already done for hot reload, but there's no reason why it should be limited to that.

Change 3765923 by Graeme.Thornton

	VSCode - "taskName" -> "label" for C# build tasks

Change 3766018 by Steve.Robb

	constexpr constructor for TAtomic.

Change 3766037 by Steve.Robb

	Misc tidyings in HotReload.cpp.

Change 3766046 by Steve.Robb

	ThreadSanitizer fixes to ENamedThreads::RenderThread and ENamedThreads::ENamedThreads_Local.

Change 3766288 by Steve.Robb

	Improved efficiency of adding/removing elements to UGCObjectReferencer::ReferencedObjects.

Change 3766374 by Josh.Engebretson

	Fix issue with ini quoted value comparison

	#jira UE-52066

Change 3766532 by Josh.Engebretson

	PR #3680: Added NetSerialize to FDateTime fixing UE-22533 (Contributed by druhasu)
	#jira UE-46156

Change 3766740 by Steve.Robb

	TMultiMap::Append added.

Change 3767523 by Steve.Robb

	ThreadSanitizer fix for UE4Delegates_Private::GNextID.

Change 3767601 by Steve.Robb

	ThreadSanitizer fix for FStats::GameThreadStatsFrame.

Change 3770567 by Ben.Marsh

	Add a FAnnotatedArchiveFormatter interface which allows querying structural type information that may not be in binary archives.

Change 3770826 by Ben.Marsh

	Move StructuredArchive implementation into Core, so primitive types can implement serialization overloads for it.

Change 3770875 by Steve.Robb

	Redundant UScriptStruct::PostLoad removed, which was causing a race condition in async loading.  This was re-establishing the CppStructOps, but that is unnecessary because native classes cannot change as a result of a load - only BP structs can, and they don't have CppStructOps.

Change 3772167 by Ben.Marsh

	Add a context-free binary formatter that can serialize tagged data. This functions as a lower-overhead binary intermediate format for JSON data.

Change 3772248 by Steve.Robb

	ThreadSanitizer fixes to FMalloc call counters.

Change 3772383 by Ben.Marsh

	Separate archive metadata from FArchive into FArchiveContext, so it can be safely exposed to consumers of FStructuredArchive.

Change 3772906 by Graeme.Thornton

	TextAssetCommandlet - Utility commandlet for testing/converting to text asset format

Change 3772932 by Ben.Marsh

	Fix "String:" prefix not being stripped from escaped string values.

Change 3772942 by Graeme.Thornton

	Add experimental setting to enable in-editor text asset format functionality
	Add "export to text" option into the content browser asset actions context menu

Change 3772955 by Ben.Marsh

	Add a new "stream" compound type to FStructuredArchive, which allows serializing a sequence of elements similarly to an array, but without serializing an explicit size. Allows passing through data to an underlying binary archive without breaking compatibility.

Change 3772963 by Ben.Marsh

	Allow querying record keys and stream lengths from annotated archive formatters, since these archives have markup for field boundaries.

Change 3773010 by Graeme.Thornton

	Added CORE_API to FArchiveFromStructuredArchive
	Gave text asset format experimental option a slightly less random tooltip comment

Change 3773057 by Ben.Marsh

	Add a flag to FArchive to determine whether the archive is text (IsTextFormat()).

	Add support for seeking within FArchiveFromStructuredArchive. For text formats, data is serialized to an in-memory buffer, with names and objects serialized as indices into an array. For non-text formats, data is serialized directly to the underlying archive.

	Also rename FStructuredArchive::TryEnterSlot() to TryEnterField().

Change 3773118 by Steve.Robb

	TSignedIntType and TUnsignedIntType type traits for getting an integer type of a given size.

Change 3773122 by Steve.Robb

	TAtomic fixes for pointer arithmetic.
	TSignedIntType used instead of reimplementing its own trait.

Change 3773123 by Steve.Robb

	Unit tests for TAtomic.

Change 3773138 by Steve.Robb

	Run numeric tests on integer types instead of basic tests.
	Fix for compiler warnings when subtracting from unsigned atomics.

Change 3773166 by Steve.Robb

	Refactoring of arithmetic operations into its own class, then basing the pointer and integral versions on that.

Change 3774216 by Gil.Gribb

	UE4 - Fix rare crash in the pak precacher immediately after unmounting a pak file.

Change 3774426 by Ben.Marsh

	Copy all C# tools to a staging directory before compiling them. This prevents access violations when compiling tools like iPhonePackager that reference DotNETCommon, and ensures we strip NotForLicensees folders out of them all.

	See: https://answers.unrealengine.com/questions/726010/418-will-not-build-from-source.html

Change 3774658 by Ben.Marsh

	Improve error reporting while generating intellisense for project files. Include the name of the target being compiled, and allow project file generation to continue without it.

Change 3775141 by Ben.Marsh

	Always output HTML5 diagnostics at "information" verbosity, to avoid every line being prefixed with "WARNING:" and screwing up the EC postprocessor.

Change 3775459 by Ben.Marsh

	Removing .NET Framework Perforce DLL as runtime dependency of engine third party library. The actual library is linked statically.

Change 3775522 by Ben.Marsh

	UGS: Treat .uproject and .uplugin files as code changes.

Change 3775597 by Ben.Marsh

	Fix post-build steps for plugins not being executed.

	#jira UE-52754

Change 3777895 by Graeme.Thornton

	StructuredArchiveFromArchive - An adapter class for wrapping an existing FArchive with a structured archive

Change 3777931 by Graeme.Thornton

	Refactored FArchiveUObjects serialization code into some static helpers
	Added FArchiveUObjectFromStructuredArchive which allows the adaption of a structured archive into an FArchive that supports the extra UObect serialization functions for weak/soft pointers

Change 3777942 by Graeme.Thornton

	Added missing CORE_API to FStructuredArchive::FStream
	Added FStructuredArchive::FSlot insertion operator for char
	Added specialization of TArray<uint8> serializer for structured archives which serializes the contents as one value

Change 3778084 by Graeme.Thornton

	Adding FPackageName::GetTextAssetPackageExtension() to access the file extension we use for text asset files

Change 3778096 by Graeme.Thornton

	Add a constructor to FArchiveUObjectFromStructuredArchive that takes a slot and passes it to the base class

Change 3778389 by Josh.Engebretson

	Fix an optimization issue with CPU benchmarking
	Add better support for debugging/testing local rocket builds

	UDN Link: https://udn.unrealengine.com/questions/400909/command-scalability-auto-gives-inaccurate-cpu-benc.html

	#jira UE-52192

Change 3778701 by Josh.Engebretson

	Ensure plugin content folders are mounted consistently.  Fixes TryConvertFilenameToLongPackageName failing to work on plugin assets

	UDN Link: https://udn.unrealengine.com/questions/276386/tryconvertfilenametolongpackagename-fails-for-plug.html

	#jira UE-40317

Change 3778832 by Chad.Garyet

	Adding enterprise path support for PCB's for UGS

Change 3780258 by Graeme.Thornton

	TextAssetCommandlet - Accumulate timings for loading packages and saving packages

Change 3780463 by Graeme.Thornton

	CryptoKeys improvements
	 - Enable CryptoKeys plugin by default
	 - Attempt to inherit settings from the old system by default
	 - Hide ini/index encryption settings from packaging settings and just inherit previous values into new system

	Minor UBT change to remove a trailing comma from the end of encryption/signing key binary strings

Change 3780557 by Ben.Marsh

	Fix LoginFlow module not being precompiled for the binary release.

Change 3780846 by Josh.Engebretson

	Improve filename to long package name resolution when provided a relative path

Change 3780863 by Ben.Marsh

	UAT: Add a better error message when a C# project has an invalid reference.

Change 3780911 by Ben.Marsh

	Update the BuildEditorAndTools.xml script to allow submitting archived binaries to Perforce.

	The "Submit To Perforce For UGS" node creates a zip of all the binaries that have been built, and submits it to the stream specified by the 'ArchiveStream' argument.

Change 3780956 by Josh.Engebretson

	Add support for ! (RemoveKey) config command to UBT

	UDN Link: https://udn.unrealengine.com/questions/397267/index.html

	#jira UE-52033

Change 3782957 by Robert.Manuszewski

	UE4 - Fixed a linear search in EDL that caused performance problems for very large maps.

Change 3784503 by Ben.Marsh

	Optimizations for FStructuredArchive:

	* Store the depth explicitly in element objects, to avoid having to loop through the scope stack to find it.
	* Prevent shrinking of arrays when removing elements.
	* Add an inline allocator to the scope and container stacks.

Change 3784700 by Ben.Marsh

	Remove the inline allocator from FStructuredArchive; checking whether the inline or backup allocator is being used is slower than just allocating up-front.

Change 3784989 by Ben.Marsh

	Compile out all the FStructuredArchive validation code when WITH_TEXT_ARCHIVE_SUPPORT = 0.

Change 3786860 by Gil.Gribb

	UE4 - Remove no buffering flag from windows async IO because it disabled the disk cache entirely.

Change 3787159 by Ben.Marsh

	Guard against UE4.0 backwards compatibility path when determining if an engine is a source distribution.

Change 3787493 by Josh.Engebretson

	Parallel pak generation now uses MaxDegreeOfParallelism option which is now set to the number of CPU cores
	Moved cryptography settings parsing out of threaded CreatePak method to avoid concurrency issue in ConfigCache.TryReadFile
	Fix for multiple threads parsing ini keys (PR 3995)

	#PR 3995
	#jira 52913
	#jira 49503

Change 3787773 by Steve.Robb

	Fix for missing final values from FOREACH_ENUM_ macros.

Change 3788287 by Ben.Marsh

	TBA: Add checks in debug builds that key names in maps and records for FStructuredArchive are unique.

Change 3788678 by Ben.Marsh

	Fix compile error due to inability to instantiate TArray<> of forward declared struct. Convert set of key names to an array to avoid including Set.h in public header for FStructuredArchive.

Change 3789353 by Graeme.Thornton

	Removed unused/rotten modes from TextAsset commandlet.
	Used existing "-iterations=n" switch to control a global iteration over the given command. Useful for performance testing.

Change 3789396 by Ben.Marsh

	Move code to validate container keys/sizes into DO_GUARD_SLOW checks, and allocate container metadata instances dynamically to fix problems with references to things not declared in headers that can't be included from StructuredArchive.h

Change 3789772 by Ben.Marsh

	Always strip trailing slashes from the end of paths specified by .build.cs files; they can cause quoted paths to be escaped on the command line.

Change 3790003 by Ben.Marsh

	TBA: Rename FStructuredArchive::EElementType::Object to FStructuredArchive::EElementType::Record.

Change 3790051 by Steve.Robb

	PIE is disabled during a hot reload.
	Hot reload in editor is disabled during PIE.
	Hot reload from IDE is deferred until after PIE is exited.
	Compiling multiple times before a hot reload (e.g. compiling multiple times in PIE) will now load the most recent change.

	#jira UE-20357
	#jira UE-52137

Change 3790709 by Steve.Robb

	Better move support for TVariant.
	EVariantTypes switched over to using an enum class to aid debugger visualization.

Change 3791422 by Ben.Marsh

	TBA: Return the type of a field from an annotated archive formatter at the point that we enter it, rather than querying all the time.

Change 3791489 by Graeme.Thornton

	TBA: Change StructuredArchiveFromArchive adapter to use the archive.Open() result directly, now that it's a slot and not a record

Change 3792344 by Ben.Marsh

	Improvements to base64 encoding library.

	* Now supports encoding and decoding with ANSICHAR and WIDECHAR implementations.
	* Added support for decoding base-64 blobs without padding marks.
	* Added support for decoding into pre-allocated buffer.
	* Added constexpr functions for determining the encoded and maximum decoded size of an input buffer.
	* Prevent writes past the end of allocated buffer (no longer need to manually remove padding bytes).

Change 3792949 by Ben.Marsh

	TBA: Rename FAnnotatedArchiveFormatter to FAnnotatedStructuredArchiveFormatter.

Change 3794078 by Robert.Manuszewski

	Fixing a crash that could happen when FGCObjects were constructed and destructed when shutting down the engine

	#jira UE-52392

Change 3794413 by Ben.Marsh

	TBA: Remove the element type parameter to SetScope(). It isn't really needed; we can just assume the element ID correctly identifies the item on the stack.

Change 3794731 by Ben.Marsh

	TBA: Optimize creation of stack elements for empty slots in FStructuredArchive. This saves a lot of bookkeeping when serializing a large number of individual fields. Since only one slot can be active at a time (and it only exists temporarily, until we write into it), we can just store the element ID assigned to it in a member variable.

Change 3795081 by Ben.Marsh

	UBT: Move LinuxCommon.cs into Platform/Linux folder.

Change 3795137 by Ben.Marsh

	UBT: Allow modules to specify private compiler definitions from the build.cs file, only visible within that module (via the "PrivateDefinitions" property).

Change 3795247 by Ben.Marsh

	Fix missing header when creating a new interface from the editor new code wizard.

	#jira UE-53174

Change 3796025 by Graeme.Thornton

	Fixed some deprecated "Definitions" warnings in OpenCV build files

Change 3796103 by Graeme.Thornton

	Disable experimental text asset option - it does nothing useful yet.

Change 3796157 by Graeme.Thornton

	Fix path type mismatch in visual studio source code accessor meaning that the DTE comms wouldn't identify a running instance of VS as having the current solution open.

	#jira UE-53206

Change 3796315 by Ben.Marsh

	Move Formatter to the correct position for initializer.

	#jira UE-53208

Change 3797082 by Ben.Marsh

	UAT: Work around for exception thrown by launching cook with "-platform=Android_ETC1 -targetplatform=Android -cookflavor=ETC1". Anrdoid_ETC1 is not a valid platform (it's a cook platform), and can't be parsed by UAT.

	#jira UE-53232

Change 3799050 by Ben.Marsh

	Make UnrealPak.version files writable for Mac and Linux.

Change 3801012 by Graeme.Thornton

	VSCode - Update source accessor to use code workspace as it's target, rather than just the project directory

Change 3801214 by Gil.Gribb

	UE4 - Remove assert to work around minor problem with lock free lists.

	#jira UE-49600

Change 3801219 by Steve.Robb

	WeakObjectPtrs now warn when casting away const.

Change 3801299 by Graeme.Thornton

	Fix quote issue with foreign project build tasks on PC

Change 3803292 by Graeme.Thornton

	Fix crash on startup when using cook-on-the-side. Force a flush of the asset registry background scanning when creating the cook-on-the-side platform registries

Change 3803559 by Steve.Robb

	TSAN fix for FMalloc::MaxSingleAlloc.

Change 3803735 by Graeme.Thornton

	Last set of cryptokeys changes
	 - Added some comments for editor exposed settings
	 - Split "encrypt assets" option into "encrypt uassets" and "encrypt all assets"

Change 3803929 by Ben.Marsh

	UGS: Show an in-place error panel when a project fails to open, allowing the user to retry and have their tabs saved instead of creating a modal dialog.

Change 3624590 by Steve.Robb

	AddReferencedObjects now generates a compile error with containers of UObject*s where the UObjectType is forward-declared, as these which won't be added to the reference collector.
	Tidy-up of existing calls to AddReferencedObjects.

Change 3629473 by Ben.Marsh

	Build: Rename the option for embedding source server information in PDB files for installed engine builds.

Change 3632894 by Steve.Robb

	VARARG* macros deprecated and usage replaced with variadic templates.

Change 3640704 by Steve.Robb

	MakeWeakObjectPtr added, which deduces a TWeakObjectPtr type from a raw pointer type.
	Fix to TWeakObjectPtr's constructor which implicitly removed const.
	Fixes to everything which didn't compile as a result.

Change 3650813 by Graeme.Thornton

	Removed FStartupPackages and associated code

Change 3651000 by Ben.Marsh

	Return the stack size from FPlatformStackWalk::CaptureStackBacktrace() rather than checking for the first null pointer, to prevent truncated callstacks if parts of the stack are zeroed out.

	#jira UE-49980

Change 3690842 by Steve.Robb

	FPlatformAtomics::AtomicRead added - needs optimizing.
	AtomicRead() used in FThreadSafeCounter::GetValue().

Change 3699416 by Steve.Robb

	Fix to debugger visualization of TArray with a TInlineAllocator or TFixedAllocator.
	Improved readability of TSparseArray visualization.

Change 3720812 by Steve.Robb

	Atomic functions for 8-bit and 16-bit.
	Android, Linux and Switch implementations now just use the Clang implementation.
	AtomicRead64 deprecated in favor of the int64* AtomicRead overload.

Change 3722698 by Steve.Robb

	VS debugger visualizers for TAtomic.

Change 3732270 by Steve.Robb

	Relaxed stores and loads.

Change 3749315 by Graeme.Thornton

	If UAT is invoked with platforms in both the -platform and -targetplatform command line switches, build using all of them rather than just the ones in -targetplatform

	#jira UE-52034

Change 3750657 by Josh.Engebretson

	Fixed issue when debugging editor cook/package and project launch operations

	#jira UE-52207

Change 3758514 by Steve.Robb

	Fixes to FString::Printf having non-literals being passed as its formatting string.

Change 3763356 by Steve.Robb

	ENamedThreads::RenderThread and ENamedThreads::RenderThread_Local encapsulated by getters and setters.

Change 3770549 by Steve.Robb

	Removal of obsolete PLATFORM_COMPILER_HAS_DEFAULTED_FUNCTIONS and PLATFORM_COMPILER_HAS_AUTO_RETURN_TYPES.
	Tidy up of existing code which uses it.

Change 3770553 by Ben.Marsh

	Adding structured serialization API to Core/CoreUObject for use with text-based assets.

	* FStructuredArchive abstracts an archive which is made up of compound types (records, arrays, and maps). Values are stored in slots within these types.
	* Records are string -> value dictionaries where the key names can be compiled out in non-editor builds or when WITH_TEXT_ARCHIVE_SUPPORT = 0.
	* Maps are string -> value dictionaries where the key names are present regardless of the build type.
	* Proxy objects are defined to express the context for serialization (FStructuredArchive::FRecord, FStructuredArchive::FArray, FStructuredArchive::FMap, FStructuredArchive::FSlot) which allows basic validation through static typing. These objects act as lightweight handles, and can be cheaply constructed and passed around on the stack. Most serialization to and from the archive is done through these objects.
	* Runtime checks perform additional validation to ensure that serialized data is well formed and written in a forward-only manner, regardless of the underlying archive type.
	* The actual input/output format is determined by a separate interface (FArchiveFormatter). Context validation (always causing matching LeaveArray for every EnterArray, etc...) is done by FStructuredArchive, so implementing these classes is fairly trivial. FArchiveFormatter can be de-virtualized in non-editor builds, where WITH_TEXT_ARCHIVE_SUPPORT = 0.
	* Includes implementations of FArchiveFormatter for binary and JSON formats.

Change 3771105 by Steve.Robb

	Deprecation warnings for PLATFORM_COMPILER_HAS_AUTO_RETURN_TYPES and PLATFORM_COMPILER_HAS_DEFAULTED_FUNCTIONS.
	Fix for incorrect warning formatting on Clang platforms.

Change 3771520 by Steve.Robb

	Start moving Clang-using platforms' pre-setup stuff into a Clang-specific header.

Change 3771564 by Steve.Robb

	More common macros moved to the Clang pre-setup header.

Change 3771613 by Steve.Robb

	EMIT_CUSTOM_WARNING_AT_LINE moved to ClangPlatformCompilerPreSetup.h.

Change 3772881 by Ben.Marsh

	Add support for serializing FName and UObject through FStructuredArchive.

	In order to allow custom linker behavior when serializing objects:

	* The constructor to JSON input formatter now takes a delegate to convert a string object name into a UObject pointer.
	* The constructor to tagged binary formatter takes a delegate to serialize a UObject pointer into any form it chooses (likely an integer index into the import table)

	Object and name types are stored as strings in JSON, using an "Object:" or "Name:" prefix to differentiate them from regular strings. Any strings that already contain one of these prefixes are prepended with a "String:" prefix (as is any string that already has a "String:" prefix).

Change 3772941 by Graeme.Thornton

	Make build work when including StructuredArchive.h from core container types
	Added standard header to new files
	Add structured archive serializer for TArray
	Fix bug in structured archive where containers weren't being popped from the scope stack

Change 3772972 by Ben.Marsh

	Add an adapter which presents a legacy FArchive interface to a FStructuredArchive slot.

	Data is serialized into this slot as a stream of elements; raw data is buffered up into fixed size chunks, names and objects are serialized separately.

	When used with FBinaryArchiveFormatter, this should result in all data being passed through to the underlying archive in a backwards compatible way, wiith no additional bookkeeping fields.

Change 3773006 by Ben.Marsh

	Rename FStructuredArchive::FRecord::EnterSlot() to EnterField().

Change 3773013 by Steve.Robb

	bUseInlining target rule added to UnrealBuildTool, which defaults to true, to allow inlining to be disabled for debugging purposes.

Change 3774499 by Ben.Marsh

	Minor fixes for FStructuredArchive related classes:

	* Text-based archive formats are now compiled out when WITH_TEXT_ARCHIVE_SUPPORT = 0.
	* Fixed issue with FTaggedBinaryArchiveFormatter state becoming corrupted when looking ahead at field types.
	* FArchiveFieldName constructor is now explicit, to fix cases where strings were being passed directly to serialize functions.

Change 3774600 by Ben.Marsh

	Add CopyFormattedData() function, which can copy data from one formatter to another. Add a test case to SerializationAPI that converts from data -> JSON -> binary -> JSON -> data.

	This function can be used to implement a generic visitor pattern, by implementing a FArchiveFormatter which receives the deserialized data.

Change 3789721 by Ben.Marsh

	TBA: Split FTaggedBinaryArchiveFormatter into separate classes for reading and writing.

Change 3789920 by Ben.Marsh

	TBA: Support automatic coercion between any numeric types in tagged binary archives. Also report the smallest type that can contain a value, rather than just in32/double.

	#jira UECORE-364

Change 3789982 by Ben.Marsh

	TBA: Change FStructuredArchive::Open() to return a slot, rather than a record, to make it easier to implement a raw FArchive adapter.

Change 3792466 by Ben.Marsh

	TBA: Better handling of raw data in text based assets. Short sequences of binary data are Base64 encoded as a single string. Longer sequences are stored as an array of Base64 encoded lines, push a SHA1 hash to detect cases where the data was merged incorrectly.

	In order to allow inference of the correct type for a field, other fields called "Base64" will be escaped to "_Base64", and any field beginning with "_" will have an additional underscore inserted. Reading files back in reverses these transformations.

Change 3792935 by Ben.Marsh

	TBA: Rename FArchiveFormatter to FStructuredArchiveFormatter for consistency with FStructuredArchive.

Change 3795100 by Ben.Marsh

	UBT: Rename the ModuleRules Definitions property to PublicDefinitions, to make its semantics clearer.

Change 3795106 by Ben.Marsh

	Replace all internal usages of ModuleRules.Definitions, and replace it with ModuleRules.PublicDefinitions.

Change 3796275 by Ben.Marsh

	Fix paths to Version.h includes from resource files.

Change 3800683 by Josh.Engebretson

	Remove WER from Mac and Linux crash reports in favor of unified runtime-xml format
	#jira UE-50073

Change 3803545 by Steve.Robb

	TWeakObjPtr const-dropping assignment fix.
	Fixes to change.

[CL 3805231 by Ben Marsh in Main branch]
2017-12-12 18:32:45 -05:00

2094 lines
70 KiB
C++

// Copyright 1998-2017 Epic Games, Inc. All Rights Reserved.
#include "CoreMinimal.h"
#include "HAL/PlatformProcess.h"
#include "GenericPlatform/GenericPlatformFile.h"
#include "HAL/FileManager.h"
#include "Misc/CoreMisc.h"
#include "Misc/Paths.h"
#include "Misc/QueuedThreadPool.h"
#include "Misc/OutputDeviceNull.h"
#include "Stats/Stats.h"
#include "Async/AsyncWork.h"
#include "Containers/Ticker.h"
#include "Misc/ConfigCacheIni.h"
#include "Misc/FeedbackContext.h"
#include "Misc/ScopedSlowTask.h"
#include "Misc/App.h"
#include "Modules/ModuleManager.h"
#include "UObject/ObjectMacros.h"
#include "UObject/UObjectGlobals.h"
#include "Serialization/ArchiveUObject.h"
#include "UObject/GarbageCollection.h"
#include "UObject/Class.h"
#include "UObject/UObjectIterator.h"
#include "UObject/UnrealType.h"
#include "Misc/PackageName.h"
#include "IHotReload.h"
#include "IDirectoryWatcher.h"
#include "DirectoryWatcherModule.h"
#include "HotReloadLog.h"
#include "AnalyticsEventAttribute.h"
#include "Interfaces/IAnalyticsProvider.h"
#include "ProfilingDebugging/ScopedTimers.h"
#include "IPluginManager.h"
#include "DesktopPlatformModule.h"
#if WITH_ENGINE
#include "Engine/Engine.h"
#include "Kismet2/KismetReinstanceUtilities.h"
#include "HotReloadClassReinstancer.h"
#include "EngineAnalytics.h"
#endif
#include "Misc/ScopeExit.h"
#if WITH_EDITOR
#include "Editor.h"
#endif
DEFINE_LOG_CATEGORY(LogHotReload);
#define LOCTEXT_NAMESPACE "HotReload"
namespace EThreeStateBool
{
enum Type
{
False,
True,
Unknown
};
static bool ToBool(EThreeStateBool::Type Value)
{
switch (Value)
{
case EThreeStateBool::False:
return false;
case EThreeStateBool::True:
return true;
default:
UE_LOG(LogHotReload, Fatal, TEXT("Can't convert EThreeStateBool to bool value because it's Unknown"));
break;
}
return false;
}
static EThreeStateBool::Type FromBool(bool Value)
{
return Value ? EThreeStateBool::True : EThreeStateBool::False;
}
};
/**
* Module for HotReload support
*/
class FHotReloadModule : public IHotReloadModule, FSelfRegisteringExec
{
public:
FHotReloadModule()
{
ModuleCompileReadPipe = nullptr;
bRequestCancelCompilation = false;
bIsAnyGameModuleLoaded = EThreeStateBool::Unknown;
bDirectoryWatcherInitialized = false;
}
/** IModuleInterface implementation */
virtual void StartupModule() override;
virtual void ShutdownModule() override;
/** FSelfRegisteringExec implementation */
virtual bool Exec( UWorld* Inworld, const TCHAR* Cmd, FOutputDevice& Ar ) override;
/** IHotReloadInterface implementation */
virtual void SaveConfig() override;
virtual bool RecompileModule(const FName InModuleName, const bool bReloadAfterRecompile, FOutputDevice &Ar, bool bFailIfGeneratedCodeChanges = true, bool bForceCodeProject = false) override;
virtual bool IsCurrentlyCompiling() const override { return ModuleCompileProcessHandle.IsValid(); }
virtual void RequestStopCompilation() override { bRequestCancelCompilation = true; }
virtual void AddHotReloadFunctionRemap(FNativeFuncPtr NewFunctionPointer, FNativeFuncPtr OldFunctionPointer) override;
virtual ECompilationResult::Type RebindPackages(const TArray<UPackage*>& Packages, EHotReloadFlags Flags, FOutputDevice &Ar) override;
virtual ECompilationResult::Type DoHotReloadFromEditor(EHotReloadFlags Flags) override;
virtual FHotReloadEvent& OnHotReload() override { return HotReloadEvent; }
virtual FModuleCompilerStartedEvent& OnModuleCompilerStarted() override { return ModuleCompilerStartedEvent; }
virtual FModuleCompilerFinishedEvent& OnModuleCompilerFinished() override { return ModuleCompilerFinishedEvent; }
virtual FString GetModuleCompileMethod(FName InModuleName) override;
virtual bool IsAnyGameModuleLoaded() override;
private:
/**
* Enumerates compilation methods for modules.
*/
enum class EModuleCompileMethod
{
Runtime,
External,
Unknown
};
/**
* Helper structure to hold on to module state while asynchronously recompiling DLLs
*/
struct FModuleToRecompile
{
/** Name of the module */
FString ModuleName;
/** Desired module file name suffix, or empty string if not needed */
FString ModuleFileSuffix;
/** The module file name to use after a compilation succeeds, or an empty string if not changing */
FString NewModuleFilename;
};
/**
* Helper structure to store the compile time and method for a module
*/
struct FModuleCompilationData
{
/** Has a timestamp been set for the .dll file */
bool bHasFileTimeStamp;
/** Last known timestamp for the .dll file */
FDateTime FileTimeStamp;
/** Last known compilation method of the .dll file */
EModuleCompileMethod CompileMethod;
FModuleCompilationData()
: bHasFileTimeStamp(false)
, CompileMethod(EModuleCompileMethod::Unknown)
{ }
};
/**
* Adds a callback to directory watcher for the game binaries folder.
*/
void RefreshHotReloadWatcher();
/**
* Adds a directory watch on the binaries directory under the given folder.
*/
void AddHotReloadDirectory(IDirectoryWatcher* DirectoryWatcher, const FString& BaseDir);
/**
* Removes a directory watcher callback
*/
void ShutdownHotReloadWatcher();
/**
* Performs hot-reload from IDE (when game DLLs change)
*/
void DoHotReloadFromIDE(const TMap<FString, FString>& NewModules);
/**
* Performs internal module recompilation
*/
ECompilationResult::Type RebindPackagesInternal(const TArray<UPackage*>& Packages, const TArray<FName>& DependentModules, EHotReloadFlags Flags, FOutputDevice& Ar);
/**
* Does the actual hot-reload, unloads old modules, loads new ones
*/
ECompilationResult::Type DoHotReloadInternal(const TMap<FString, FString>& ChangedModuleNames, const TArray<UPackage*>& Packages, const TArray<FName>& InDependentModules, FOutputDevice& HotReloadAr);
/**
* Finds all references to old CDOs and replaces them with the new ones.
* Skipping UBlueprintGeneratedClass::OverridenArchetypeForCDO as it's the
* only one needed.
*/
void ReplaceReferencesToReconstructedCDOs();
#if WITH_ENGINE
void RegisterForReinstancing(UClass* OldClass, UClass* NewClass);
void ReinstanceClasses();
/**
* Called from CoreUObject to re-instance hot-reloaded classes
*/
void ReinstanceClass(UClass* OldClass, UClass* NewClass, const TMap<UClass*, UClass*>& OldToNewClassesMap);
#endif
/**
* Tick function for FTicker: checks for re-loaded modules and does hot-reload from IDE
*/
bool Tick(float DeltaTime);
/**
* Directory watcher callback
*/
void OnHotReloadBinariesChanged(const TArray<struct FFileChangeData>& FileChanges);
/**
* Strips hot-reload suffix from module filename.
*/
static void StripModuleSuffixFromFilename(FString& InOutModuleFilename, const FString& ModuleName);
/**
* Sends analytics event about the re-load
*/
static void RecordAnalyticsEvent(const TCHAR* ReloadFrom, ECompilationResult::Type Result, double Duration, int32 PackageCount, int32 DependentModulesCount);
/**
* Declares a function type that is executed after a module recompile has finished.
*
* ChangedModules: A map between the names of the modules that have changed and their filenames.
* bRecompileFinished: Signals whether compilation has finished.
* CompilationResult: Shows whether compilation was successful or not.
*/
typedef TFunction<void(const TMap<FString, FString>& ChangedModules, bool bRecompileFinished, ECompilationResult::Type CompilationResult)> FRecompileModulesCallback;
/** Called for successfully re-complied module */
void OnModuleCompileSucceeded(FName ModuleName, const FString& NewModuleFilename);
/** Returns arguments to pass to UnrealBuildTool when compiling modules */
static FString MakeUBTArgumentsForModuleCompiling();
#if WITH_HOT_RELOAD
/**
* Starts compiling DLL files for one or more modules.
*
* @param GameName The name of the game.
* @param ModuleNames The list of modules to compile.
* @param InRecompileModulesCallback Callback function to make when module recompiles.
* @param Ar
* @param bInFailIfGeneratedCodeChanges If true, fail the compilation if generated headers change.
* @param InAdditionalCmdLineArgs Additional arguments to pass to UBT.
* @param bForceCodeProject Compile as code-based project even if there's no game modules loaded
* @return true if successful, false otherwise.
*/
bool StartCompilingModuleDLLs(const FString& GameName, const TArray< FModuleToRecompile >& ModuleNames,
FRecompileModulesCallback&& InRecompileModulesCallback, FOutputDevice& Ar, bool bInFailIfGeneratedCodeChanges,
const FString& InAdditionalCmdLineArgs, bool bForceCodeProject);
#endif
/** Launches UnrealBuildTool with the specified command line parameters */
bool InvokeUnrealBuildToolForCompile(const FString& InCmdLineParams, FOutputDevice &Ar);
/** Checks to see if a pending compilation action has completed and optionally waits for it to finish. If completed, fires any appropriate callbacks and reports status provided bFireEvents is true. */
void CheckForFinishedModuleDLLCompile(EHotReloadFlags Flags, bool& bCompileStillInProgress, bool& bCompileSucceeded, FOutputDevice& Ar, bool bFireEvents = true);
/** Called when the compile data for a module need to be update in memory and written to config */
void UpdateModuleCompileData(FName ModuleName);
/** Called when a new module is added to the manager to get the saved compile data from config */
static void ReadModuleCompilationInfoFromConfig(FName ModuleName, FModuleCompilationData& CompileData);
/** Saves the module's compile data to config */
static void WriteModuleCompilationInfoToConfig(FName ModuleName, const FModuleCompilationData& CompileData);
/** Access the module's file and read the timestamp from the file system. Returns true if the timestamp was read successfully. */
bool GetModuleFileTimeStamp(FName ModuleName, FDateTime& OutFileTimeStamp) const;
/** Checks if the specified array of modules to recompile contains only game modules */
bool ContainsOnlyGameModules(const TArray< FModuleToRecompile >& ModuleNames) const;
/** Callback registered with ModuleManager to know if any new modules have been loaded */
void ModulesChangedCallback(FName ModuleName, EModuleChangeReason ReasonForChange);
/** Callback registered with PluginManager to know if any new plugins have been created */
void PluginMountedCallback(IPlugin& Plugin);
/** FTicker delegate (hot-reload from IDE) */
FTickerDelegate TickerDelegate;
/** Handle to the registered TickerDelegate */
FDelegateHandle TickerDelegateHandle;
/** Handle to the registered delegate above */
TMap<FString, FDelegateHandle> BinariesFolderChangedDelegateHandles;
/** True if currently hot-reloading from editor (suppresses hot-reload from IDE) */
bool bIsHotReloadingFromEditor;
/** New module DLLs detected by the directory watcher */
TMap<FString, FString> DetectedNewModules;
/** Moduels that have been recently recompiled from the editor **/
TSet<FString> ModulesRecentlyCompiledInTheEditor;
/** Delegate broadcast when a module has been hot-reloaded */
FHotReloadEvent HotReloadEvent;
/** Array of modules that we're currently recompiling */
TArray< FModuleToRecompile > ModulesBeingCompiled;
/** Array of modules that we're going to recompile */
TArray< FModuleToRecompile > ModulesThatWereBeingRecompiled;
/** Last known compilation data for each module */
TMap<FName, TSharedRef<FModuleCompilationData>> ModuleCompileData;
/** Multicast delegate which will broadcast a notification when the compiler starts */
FModuleCompilerStartedEvent ModuleCompilerStartedEvent;
/** Multicast delegate which will broadcast a notification when the compiler finishes */
FModuleCompilerFinishedEvent ModuleCompilerFinishedEvent;
/** When compiling a module using an external application, stores the handle to the process that is running */
FProcHandle ModuleCompileProcessHandle;
/** When compiling a module using an external application, this is the process read pipe handle */
void* ModuleCompileReadPipe;
/** When compiling a module using an external application, this is the text that was read from the read pipe handle */
FString ModuleCompileReadPipeText;
/** Callback to execute after an asynchronous recompile has completed (whether successful or not.) */
FRecompileModulesCallback RecompileModulesCallback;
/** true if we should attempt to cancel the current async compilation */
bool bRequestCancelCompilation;
/** Tracks the validity of the game module existence */
EThreeStateBool::Type bIsAnyGameModuleLoaded;
/** True if the directory watcher has been successfully initialized */
bool bDirectoryWatcherInitialized;
/** Reconstructed CDOs map during hot-reload. */
TMap<UObject*, UObject*> ReconstructedCDOsMap;
/** Keeps record of hot-reload session starting time. */
double HotReloadStartTime;
};
namespace HotReloadDefs
{
const static FString CompilationInfoConfigSection("ModuleFileTracking");
// These strings should match the values of the enum EModuleCompileMethod in ModuleManager.h
// and should be handled in ReadModuleCompilationInfoFromConfig() & WriteModuleCompilationInfoToConfig() below
const static FString CompileMethodRuntime("Runtime");
const static FString CompileMethodExternal("External");
const static FString CompileMethodUnknown("Unknown");
// Add one minute epsilon to timestamp comparision
const static FTimespan TimeStampEpsilon(0, 1, 0);
}
IMPLEMENT_MODULE(FHotReloadModule, HotReload);
namespace UE4HotReload_Private
{
/**
* Gets editor runs directory.
*/
FString GetEditorRunsDir()
{
FString TempDir = FPaths::EngineIntermediateDir();
return FPaths::Combine(*TempDir, TEXT("EditorRuns"));
}
/**
* Creates a file that informs UBT that the editor is currently running.
*/
void CreateFileThatIndicatesEditorRunIfNeeded()
{
#if WITH_EDITOR
IPlatformFile& FS = IPlatformFile::GetPlatformPhysical();
FString EditorRunsDir = GetEditorRunsDir();
FString FileName = FPaths::Combine(*EditorRunsDir, *FString::Printf(TEXT("%d"), FPlatformProcess::GetCurrentProcessId()));
if (FS.FileExists(*FileName))
{
if (!GIsEditor)
{
FS.DeleteFile(*FileName);
}
}
else
{
if (GIsEditor)
{
if (!FS.CreateDirectory(*EditorRunsDir))
{
return;
}
delete FS.OpenWrite(*FileName); // Touch file.
}
}
#endif // WITH_EDITOR
}
/**
* Deletes file left by CreateFileThatIndicatesEditorRunIfNeeded function.
*/
void DeleteFileThatIndicatesEditorRunIfNeeded()
{
#if WITH_EDITOR
IPlatformFile& FS = IPlatformFile::GetPlatformPhysical();
FString EditorRunsDir = GetEditorRunsDir();
FString FileName = FPaths::Combine(*EditorRunsDir, *FString::Printf(TEXT("%d"), FPlatformProcess::GetCurrentProcessId()));
if (FS.FileExists(*FileName))
{
FS.DeleteFile(*FileName);
}
#endif // WITH_EDITOR
}
/**
* Gets all currently loaded game module names and optionally, the file names for those modules
*/
TArray<FString> GetGameModuleNames(const FModuleManager& ModuleManager)
{
TArray<FString> Result;
// Ask the module manager for a list of currently-loaded gameplay modules
TArray<FModuleStatus> ModuleStatuses;
ModuleManager.QueryModules(ModuleStatuses);
for (FModuleStatus& ModuleStatus : ModuleStatuses)
{
// We only care about game modules that are currently loaded
if (ModuleStatus.bIsLoaded && ModuleStatus.bIsGameModule)
{
Result.Add(MoveTemp(ModuleStatus.Name));
}
}
return Result;
}
/**
* Gets all currently loaded game module names and optionally, the file names for those modules
*/
TMap<FString, FString> GetGameModuleFilenames(const FModuleManager& ModuleManager)
{
TMap<FString, FString> Result;
// Ask the module manager for a list of currently-loaded gameplay modules
TArray< FModuleStatus > ModuleStatuses;
ModuleManager.QueryModules(ModuleStatuses);
for (FModuleStatus& ModuleStatus : ModuleStatuses)
{
// We only care about game modules that are currently loaded
if (ModuleStatus.bIsLoaded && ModuleStatus.bIsGameModule)
{
Result.Add(MoveTemp(ModuleStatus.Name), MoveTemp(ModuleStatus.FilePath));
}
}
return Result;
}
struct FPackagesAndDependentNames
{
TArray<UPackage*> Packages;
TArray<FName> DependentNames;
};
/**
* Gets named packages and the names dependents.
*/
FPackagesAndDependentNames SplitByPackagesAndDependentNames(const TArray<FString>& ModuleNames)
{
FPackagesAndDependentNames Result;
for (const FString& ModuleName : ModuleNames)
{
FString PackagePath = TEXT("/Script/") + ModuleName;
if (UPackage* Package = FindPackage(nullptr, *PackagePath))
{
Result.Packages.Add(Package);
}
else
{
Result.DependentNames.Add(*ModuleName);
}
}
return Result;
}
}
void FHotReloadModule::StartupModule()
{
UE4HotReload_Private::CreateFileThatIndicatesEditorRunIfNeeded();
bIsHotReloadingFromEditor = false;
#if WITH_ENGINE
// Register re-instancing delegate (Core)
FCoreUObjectDelegates::RegisterClassForHotReloadReinstancingDelegate.AddRaw(this, &FHotReloadModule::RegisterForReinstancing);
FCoreUObjectDelegates::ReinstanceHotReloadedClassesDelegate.AddRaw(this, &FHotReloadModule::ReinstanceClasses);
#endif
// Register directory watcher delegate
RefreshHotReloadWatcher();
// Register hot-reload from IDE ticker
TickerDelegate = FTickerDelegate::CreateRaw(this, &FHotReloadModule::Tick);
TickerDelegateHandle = FTicker::GetCoreTicker().AddTicker(TickerDelegate);
FModuleManager::Get().OnModulesChanged().AddRaw(this, &FHotReloadModule::ModulesChangedCallback);
IPluginManager::Get().OnNewPluginMounted().AddRaw(this, &FHotReloadModule::PluginMountedCallback);
}
void FHotReloadModule::ShutdownModule()
{
FTicker::GetCoreTicker().RemoveTicker(TickerDelegateHandle);
ShutdownHotReloadWatcher();
UE4HotReload_Private::DeleteFileThatIndicatesEditorRunIfNeeded();
}
bool FHotReloadModule::Exec( UWorld* Inworld, const TCHAR* Cmd, FOutputDevice& Ar )
{
#if !UE_BUILD_SHIPPING
if ( FParse::Command( &Cmd, TEXT( "Module" ) ) )
{
#if WITH_HOT_RELOAD
// Recompile <ModuleName>
if( FParse::Command( &Cmd, TEXT( "Recompile" ) ) )
{
const FString ModuleNameStr = FParse::Token( Cmd, 0 );
if( !ModuleNameStr.IsEmpty() )
{
const FName ModuleName( *ModuleNameStr );
const bool bReloadAfterRecompile = true;
const bool bForceCodeProject = false;
const bool bFailIfGeneratedCodeChanges = true;
RecompileModule( ModuleName, bReloadAfterRecompile, Ar, bFailIfGeneratedCodeChanges, bForceCodeProject);
}
return true;
}
#endif // WITH_HOT_RELOAD
}
#endif // !UE_BUILD_SHIPPING
return false;
}
void FHotReloadModule::SaveConfig()
{
// Find all the modules
TArray<FModuleStatus> Modules;
FModuleManager::Get().QueryModules(Modules);
// Update the compile data for each one
for( const FModuleStatus &Module : Modules )
{
UpdateModuleCompileData(*Module.Name);
}
}
FString FHotReloadModule::GetModuleCompileMethod(FName InModuleName)
{
if (!ModuleCompileData.Contains(InModuleName))
{
UpdateModuleCompileData(InModuleName);
}
switch(ModuleCompileData.FindChecked(InModuleName).Get().CompileMethod)
{
case EModuleCompileMethod::External:
return HotReloadDefs::CompileMethodExternal;
case EModuleCompileMethod::Runtime:
return HotReloadDefs::CompileMethodRuntime;
default:
return HotReloadDefs::CompileMethodUnknown;
}
}
bool FHotReloadModule::RecompileModule(const FName InModuleName, const bool bReloadAfterRecompile, FOutputDevice &Ar, bool bFailIfGeneratedCodeChanges, bool bForceCodeProject)
{
#if WITH_HOT_RELOAD
UE_LOG(LogHotReload, Log, TEXT("Recompiling module %s..."), *InModuleName.ToString());
// This is an internal request for hot-reload (not from IDE)
bIsHotReloadingFromEditor = true;
// A list of modules that have been recompiled in the editor is going to prevent false
// hot-reload from IDE events as this call is blocking any potential callbacks coming from the filesystem
// and bIsHotReloadingFromEditor may not be enough to prevent those from being treated as actual hot-reload from IDE modules
ModulesRecentlyCompiledInTheEditor.Empty();
FFormatNamedArguments Args;
Args.Add( TEXT("CodeModuleName"), FText::FromName( InModuleName ) );
const FText StatusUpdate = FText::Format( NSLOCTEXT("ModuleManager", "Recompile_SlowTaskName", "Compiling {CodeModuleName}..."), Args );
FScopedSlowTask SlowTask(2, StatusUpdate);
SlowTask.MakeDialog();
ModuleCompilerStartedEvent.Broadcast(false); // we never perform an async compile
FModuleManager& ModuleManager = FModuleManager::Get();
// Update our set of known modules, in case we don't already know about this module
ModuleManager.AddModule( InModuleName );
// Only use rolling module names if the module was already loaded into memory. This allows us to try compiling
// the module without actually having to unload it first.
const bool bWasModuleLoaded = ModuleManager.IsModuleLoaded( InModuleName );
const bool bUseRollingModuleNames = bWasModuleLoaded;
SlowTask.EnterProgressFrame();
/**
* Tries to recompile the specified DLL using UBT. Does not interact with modules. This is a low level routine.
*
* @param ModuleNames List of modules to recompile, including the module name and optional file suffix.
* @param Ar Output device for logging compilation status.
* @param bForceCodeProject Even if it's a non-code project, treat it as code-based project
*/
auto RecompileModuleDLLs = [this, &Ar, bFailIfGeneratedCodeChanges, bForceCodeProject](const TArray< FModuleToRecompile >& ModuleNames)
{
bool bCompileSucceeded = false;
const FString AdditionalArguments = MakeUBTArgumentsForModuleCompiling();
if (StartCompilingModuleDLLs(FApp::GetProjectName(), ModuleNames, nullptr, Ar, bFailIfGeneratedCodeChanges, AdditionalArguments, bForceCodeProject))
{
bool bCompileStillInProgress = false;
CheckForFinishedModuleDLLCompile( EHotReloadFlags::WaitForCompletion, bCompileStillInProgress, bCompileSucceeded, Ar );
}
return bCompileSucceeded;
};
bool bWasSuccessful = true;
if( bUseRollingModuleNames )
{
// First, try to compile the module. If the module is already loaded, we won't unload it quite yet. Instead
// make sure that it compiles successfully.
// Find a unique file name for the module
FString UniqueSuffix;
FString UniqueModuleFileName;
ModuleManager.MakeUniqueModuleFilename( InModuleName, UniqueSuffix, UniqueModuleFileName );
TArray< FModuleToRecompile > ModulesToRecompile;
FModuleToRecompile ModuleToRecompile;
ModuleToRecompile.ModuleName = InModuleName.ToString();
ModuleToRecompile.ModuleFileSuffix = UniqueSuffix;
ModuleToRecompile.NewModuleFilename = UniqueModuleFileName;
ModulesToRecompile.Add( ModuleToRecompile );
ModulesRecentlyCompiledInTheEditor.Add(FPaths::ConvertRelativePathToFull(UniqueModuleFileName));
bWasSuccessful = RecompileModuleDLLs(ModulesToRecompile);
}
SlowTask.EnterProgressFrame();
if( bWasSuccessful )
{
// Shutdown the module if it's already running
if( bWasModuleLoaded )
{
Ar.Logf( TEXT( "Unloading module before compile." ) );
ModuleManager.UnloadOrAbandonModuleWithCallback( InModuleName, Ar );
}
if( !bUseRollingModuleNames )
{
// Try to recompile the DLL
TArray< FModuleToRecompile > ModulesToRecompile;
FModuleToRecompile ModuleToRecompile;
ModuleToRecompile.ModuleName = InModuleName.ToString();
if (ModuleManager.IsModuleLoaded(InModuleName))
{
ModulesRecentlyCompiledInTheEditor.Add(FPaths::ConvertRelativePathToFull(ModuleManager.GetModuleFilename(InModuleName)));
}
else
{
ModuleToRecompile.NewModuleFilename = ModuleManager.GetGameBinariesDirectory() / FModuleManager::GetCleanModuleFilename(InModuleName, true);
ModulesRecentlyCompiledInTheEditor.Add(FPaths::ConvertRelativePathToFull(ModuleToRecompile.NewModuleFilename));
}
ModulesToRecompile.Add( ModuleToRecompile );
bWasSuccessful = RecompileModuleDLLs(ModulesToRecompile);
}
// Reload the module if it was loaded before we recompiled
if( bWasSuccessful && (bWasModuleLoaded || bForceCodeProject) && bReloadAfterRecompile )
{
TGuardValue<bool> GuardIsHotReload(GIsHotReload, true);
Ar.Logf( TEXT( "Reloading module %s after successful compile." ), *InModuleName.ToString() );
bWasSuccessful = ModuleManager.LoadModuleWithCallback( InModuleName, Ar );
CollectGarbage(GARBAGE_COLLECTION_KEEPFLAGS);
}
}
if (bForceCodeProject && bWasSuccessful)
{
HotReloadEvent.Broadcast( false );
}
bIsHotReloadingFromEditor = false;
return bWasSuccessful;
#else
return false;
#endif // WITH_HOT_RELOAD
}
/** Type hash for a UObject Function Pointer, maybe not a great choice, but it should be sufficient for the needs here. **/
inline uint32 GetTypeHash(FNativeFuncPtr A)
{
return *(uint32*)&A;
}
/** Map from old function pointer to new function pointer for hot reload. */
static TMap<FNativeFuncPtr, FNativeFuncPtr> HotReloadFunctionRemap;
static TSet<UBlueprint*> HotReloadBPSetToRecompile;
static TSet<UBlueprint*> HotReloadBPSetToRecompileBytecodeOnly;
/** Adds and entry for the UFunction native pointer remap table */
void FHotReloadModule::AddHotReloadFunctionRemap(FNativeFuncPtr NewFunctionPointer, FNativeFuncPtr OldFunctionPointer)
{
FNativeFuncPtr OtherNewFunction = HotReloadFunctionRemap.FindRef(OldFunctionPointer);
check(!OtherNewFunction || OtherNewFunction == NewFunctionPointer);
check(NewFunctionPointer);
check(OldFunctionPointer);
HotReloadFunctionRemap.Add(OldFunctionPointer, NewFunctionPointer);
}
ECompilationResult::Type FHotReloadModule::DoHotReloadFromEditor(EHotReloadFlags Flags)
{
// Get all game modules we want to compile
const FModuleManager& ModuleManager = FModuleManager::Get();
TArray<FString> GameModuleNames = UE4HotReload_Private::GetGameModuleNames(ModuleManager);
ECompilationResult::Type Result = ECompilationResult::Unsupported;
UE4HotReload_Private::FPackagesAndDependentNames PackagesAndDependentNames = UE4HotReload_Private::SplitByPackagesAndDependentNames(GameModuleNames);
// Analytics
double Duration = 0.0;
{
FScopedDurationTimer Timer(Duration);
Result = RebindPackagesInternal(PackagesAndDependentNames.Packages, PackagesAndDependentNames.DependentNames, Flags, *GLog);
}
RecordAnalyticsEvent(TEXT("Editor"), Result, Duration, PackagesAndDependentNames.Packages.Num(), PackagesAndDependentNames.DependentNames.Num());
return Result;
}
ECompilationResult::Type FHotReloadModule::DoHotReloadInternal(const TMap<FString, FString>& ChangedModules, const TArray<UPackage*>& Packages, const TArray<FName>& InDependentModules, FOutputDevice& HotReloadAr)
{
#if WITH_HOT_RELOAD
FModuleManager& ModuleManager = FModuleManager::Get();
ModuleManager.ResetModulePathsCache();
FFeedbackContext& ErrorsFC = UClass::GetDefaultPropertiesFeedbackContext();
ErrorsFC.ClearWarningsAndErrors();
// Rebind the hot reload DLL
TGuardValue<bool> GuardIsHotReload(GIsHotReload, true);
TGuardValue<bool> GuardIsInitialLoad(GIsInitialLoad, true);
HotReloadFunctionRemap.Empty(); // redundant
CollectGarbage(GARBAGE_COLLECTION_KEEPFLAGS); // we create a new CDO in the transient package...this needs to go away before we try again.
// Load the new modules up
bool bReloadSucceeded = false;
ECompilationResult::Type Result = ECompilationResult::Unsupported;
for (UPackage* Package : Packages)
{
FString PackageName = Package->GetName();
FString ShortPackageName = FPackageName::GetShortName(PackageName);
if (!ChangedModules.Contains(ShortPackageName))
{
continue;
}
FName ShortPackageFName = *ShortPackageName;
// Abandon the old module. We can't unload it because various data structures may be living
// that have vtables pointing to code that would become invalidated.
ModuleManager.AbandonModuleWithCallback(ShortPackageFName);
// Load the newly-recompiled module up (it will actually have a different DLL file name at this point.)
bReloadSucceeded = ModuleManager.LoadModule(ShortPackageFName) != nullptr;
if (!bReloadSucceeded)
{
HotReloadAr.Logf(ELogVerbosity::Warning, TEXT("HotReload failed, reload failed %s."), *PackageName);
Result = ECompilationResult::OtherCompilationError;
break;
}
}
// Load dependent modules.
for (FName ModuleName : InDependentModules)
{
FString ModuleNameStr = ModuleName.ToString();
if (!ChangedModules.Contains(ModuleNameStr))
{
continue;
}
ModuleManager.UnloadOrAbandonModuleWithCallback(ModuleName, HotReloadAr);
const bool bLoaded = ModuleManager.LoadModuleWithCallback(ModuleName, HotReloadAr);
if (!bLoaded)
{
HotReloadAr.Logf(ELogVerbosity::Warning, TEXT("Unable to reload module %s"), *ModuleName.GetPlainNameString());
}
}
if (ErrorsFC.GetNumErrors() || ErrorsFC.GetNumWarnings())
{
TArray<FString> AllErrorsAndWarnings;
ErrorsFC.GetErrorsAndWarningsAndEmpty(AllErrorsAndWarnings);
FString AllInOne;
for (const FString& ErrorOrWarning : AllErrorsAndWarnings)
{
AllInOne += ErrorOrWarning;
AllInOne += TEXT("\n");
}
HotReloadAr.Logf(ELogVerbosity::Warning, TEXT("Some classes could not be reloaded:\n%s"), *AllInOne);
}
if (bReloadSucceeded)
{
int32 NumFunctionsRemapped = 0;
// Remap all native functions (and gather scriptstructs)
TArray<UScriptStruct*> ScriptStructs;
for (FRawObjectIterator It; It; ++It)
{
if (UFunction* Function = Cast<UFunction>(static_cast<UObject*>(It->Object)))
{
if (FNativeFuncPtr NewFunction = HotReloadFunctionRemap.FindRef(Function->GetNativeFunc()))
{
++NumFunctionsRemapped;
Function->SetNativeFunc(NewFunction);
}
}
if (UScriptStruct* ScriptStruct = Cast<UScriptStruct>(static_cast<UObject*>(It->Object)))
{
if (!ScriptStruct->HasAnyFlags(RF_ClassDefaultObject) && ScriptStruct->GetCppStructOps() && Packages.ContainsByPredicate([=](UPackage* Package) { return ScriptStruct->IsIn(Package); }))
{
ScriptStructs.Add(ScriptStruct);
}
}
}
// now let's set up the script structs...this relies on super behavior, so null them all, then set them all up. Internally this sets them up hierarchically.
for (UScriptStruct* Script : ScriptStructs)
{
Script->ClearCppStructOps();
}
for (UScriptStruct* Script : ScriptStructs)
{
Script->PrepareCppStructOps();
check(Script->GetCppStructOps());
}
// Make sure new classes have the token stream assembled
UClass::AssembleReferenceTokenStreams();
HotReloadAr.Logf(ELogVerbosity::Display, TEXT("HotReload successful (%d functions remapped %d scriptstructs remapped)"), NumFunctionsRemapped, ScriptStructs.Num());
HotReloadFunctionRemap.Empty();
ReplaceReferencesToReconstructedCDOs();
// Force GC to collect reinstanced objects
CollectGarbage(GARBAGE_COLLECTION_KEEPFLAGS, true);
Result = ECompilationResult::Succeeded;
}
HotReloadEvent.Broadcast( !bIsHotReloadingFromEditor );
HotReloadAr.Logf(ELogVerbosity::Display, TEXT("HotReload took %4.1fs."), FPlatformTime::Seconds() - HotReloadStartTime);
bIsHotReloadingFromEditor = false;
return Result;
#else
bIsHotReloadingFromEditor = false;
return ECompilationResult::Unsupported;
#endif
}
void FHotReloadModule::ReplaceReferencesToReconstructedCDOs()
{
if (ReconstructedCDOsMap.Num() == 0)
{
return;
}
// Thread pool manager. We need new thread pool with increased
// amount of stack size. Standard GThreadPool was encountering
// stack overflow error during serialization.
static struct FReplaceReferencesThreadPool
{
FReplaceReferencesThreadPool()
{
Pool = FQueuedThreadPool::Allocate();
int32 NumThreadsInThreadPool = FPlatformMisc::NumberOfWorkerThreadsToSpawn();
verify(Pool->Create(NumThreadsInThreadPool, 256 * 1024));
}
~FReplaceReferencesThreadPool()
{
Pool->Destroy();
}
FQueuedThreadPool* GetPool() { return Pool; }
private:
FQueuedThreadPool* Pool;
} ThreadPoolManager;
// Async task to enable multithreaded CDOs reference search.
class FFindRefTask : public FNonAbandonableTask
{
public:
explicit FFindRefTask(const TMap<UObject*, UObject*>& InReconstructedCDOsMap, int32 ReserveElements)
: ReconstructedCDOsMap(InReconstructedCDOsMap)
{
ObjectsArray.Reserve(ReserveElements);
}
void DoWork()
{
for (UObject* Object : ObjectsArray)
{
class FReplaceCDOReferencesArchive : public FArchiveUObject
{
public:
FReplaceCDOReferencesArchive(UObject* InPotentialReferencer, const TMap<UObject*, UObject*>& InReconstructedCDOsMap)
: ReconstructedCDOsMap(InReconstructedCDOsMap)
, PotentialReferencer(InPotentialReferencer)
{
ArIsObjectReferenceCollector = true;
ArIgnoreOuterRef = true;
}
virtual FString GetArchiveName() const override
{
return TEXT("FReplaceCDOReferencesArchive");
}
FArchive& operator<<(UObject*& ObjRef)
{
UObject* Obj = ObjRef;
if (Obj && Obj != PotentialReferencer)
{
if (UObject* const* FoundObj = ReconstructedCDOsMap.Find(Obj))
{
ObjRef = *FoundObj;
}
}
return *this;
}
const TMap<UObject*, UObject*>& ReconstructedCDOsMap;
UObject* PotentialReferencer;
};
FReplaceCDOReferencesArchive FindRefsArchive(Object, ReconstructedCDOsMap);
Object->Serialize(FindRefsArchive);
}
}
FORCEINLINE TStatId GetStatId() const
{
RETURN_QUICK_DECLARE_CYCLE_STAT(FFindRefTask, STATGROUP_ThreadPoolAsyncTasks);
}
TArray<UObject*> ObjectsArray;
private:
const TMap<UObject*, UObject*>& ReconstructedCDOsMap;
};
const int32 NumberOfThreads = FPlatformMisc::NumberOfWorkerThreadsToSpawn();
const int32 NumObjects = GUObjectArray.GetObjectArrayNum();
const int32 ObjectsPerTask = FMath::CeilToInt((float)NumObjects / NumberOfThreads);
// Create tasks.
TArray<FAsyncTask<FFindRefTask>> Tasks;
Tasks.Reserve(NumberOfThreads);
for (int32 TaskId = 0; TaskId < NumberOfThreads; ++TaskId)
{
Tasks.Emplace(ReconstructedCDOsMap, ObjectsPerTask);
}
// Distribute objects uniformly between tasks.
int32 CurrentTaskId = 0;
for (FObjectIterator ObjIter; ObjIter; ++ObjIter)
{
UObject* CurObject = *ObjIter;
if (CurObject->IsPendingKill())
{
continue;
}
Tasks[CurrentTaskId].GetTask().ObjectsArray.Add(CurObject);
CurrentTaskId = (CurrentTaskId + 1) % NumberOfThreads;
}
// Run async tasks in worker threads.
for (FAsyncTask<FFindRefTask>& Task : Tasks)
{
Task.StartBackgroundTask(ThreadPoolManager.GetPool());
}
// Wait until tasks are finished
for (FAsyncTask<FFindRefTask>& AsyncTask : Tasks)
{
AsyncTask.EnsureCompletion();
}
ReconstructedCDOsMap.Empty();
}
ECompilationResult::Type FHotReloadModule::RebindPackages(const TArray<UPackage*>& InPackages, EHotReloadFlags Flags, FOutputDevice &Ar)
{
ECompilationResult::Type Result = ECompilationResult::Unknown;
// Get game packages
const FModuleManager& ModuleManager = FModuleManager::Get();
TArray<FString> GameModuleNames = UE4HotReload_Private::GetGameModuleNames(ModuleManager);
UE4HotReload_Private::FPackagesAndDependentNames PackagesAndDependentNames = UE4HotReload_Private::SplitByPackagesAndDependentNames(GameModuleNames);
// Get a set of source packages combined with game packages
TSet<UPackage*> PackagesIncludingGame(InPackages);
int32 NumInPackages = PackagesIncludingGame.Num();
PackagesIncludingGame.Append(PackagesAndDependentNames.Packages);
// Check if there was any overlap
bool bInPackagesIncludeGame = PackagesIncludingGame.Num() < NumInPackages + PackagesAndDependentNames.Packages.Num();
// If any of those modules were game modules, we'll compile those too
TArray<UPackage*> Packages;
TArray<FName> Dependencies;
if (bInPackagesIncludeGame)
{
Packages = PackagesIncludingGame.Array();
Dependencies = MoveTemp(PackagesAndDependentNames.DependentNames);
}
else
{
Packages = InPackages;
}
double Duration = 0.0;
{
FScopedDurationTimer RebindTimer(Duration);
Result = RebindPackagesInternal(Packages, Dependencies, Flags, Ar);
}
RecordAnalyticsEvent(TEXT("Rebind"), Result, Duration, Packages.Num(), Dependencies.Num());
return Result;
}
ECompilationResult::Type FHotReloadModule::RebindPackagesInternal(const TArray<UPackage*>& InPackages, const TArray<FName>& DependentModules, EHotReloadFlags Flags, FOutputDevice& Ar)
{
#if WITH_HOT_RELOAD
if (InPackages.Num() == 0)
{
Ar.Logf(ELogVerbosity::Warning, TEXT("RebindPackages not possible (no packages specified)"));
return ECompilationResult::Unsupported;
}
// Verify that we're going to be able to rebind the specified packages
for (UPackage* Package : InPackages)
{
check(Package);
if (Package->GetOuter())
{
Ar.Logf(ELogVerbosity::Warning, TEXT("Could not rebind package for %s, package is either not bound yet or is not a DLL."), *Package->GetName());
return ECompilationResult::Unsupported;
}
}
// We can only proceed if a compile isn't already in progress
if (IsCurrentlyCompiling())
{
Ar.Logf(ELogVerbosity::Warning, TEXT("Could not rebind package because a module compile is already in progress."));
return ECompilationResult::Unsupported;
}
FModuleManager::Get().ResetModulePathsCache();
bIsHotReloadingFromEditor = true;
HotReloadStartTime = FPlatformTime::Seconds();
TArray< FName > ModuleNames;
for (UPackage* Package : InPackages)
{
// Attempt to recompile this package's module
FName ShortPackageName = FPackageName::GetShortFName(Package->GetFName());
ModuleNames.Add(ShortPackageName);
}
// Add dependent modules.
ModuleNames.Append(DependentModules);
// Start compiling modules
//
// NOTE: This method of recompiling always using a rolling file name scheme, since we never want to unload before
// we start recompiling, and we need the output DLL to be unlocked before we invoke the compiler
ModuleCompilerStartedEvent.Broadcast(!(Flags & EHotReloadFlags::WaitForCompletion)); // we perform an async compile providing we're not waiting for completion
FModuleManager& ModuleManager = FModuleManager::Get();
TArray< FModuleToRecompile > ModulesToRecompile;
for( FName CurModuleName : ModuleNames )
{
// Update our set of known modules, in case we don't already know about this module
ModuleManager.AddModule( CurModuleName );
// Find a unique file name for the module
FModuleToRecompile ModuleToRecompile;
ModuleToRecompile.ModuleName = CurModuleName.ToString();
ModuleManager.MakeUniqueModuleFilename( CurModuleName, ModuleToRecompile.ModuleFileSuffix, ModuleToRecompile.NewModuleFilename );
ModulesToRecompile.Add( ModuleToRecompile );
}
// Kick off compilation!
const FString AdditionalArguments = MakeUBTArgumentsForModuleCompiling();
bool bCompileStarted = StartCompilingModuleDLLs(
FApp::GetProjectName(),
ModulesToRecompile,
[this, InPackages, DependentModules, &Ar](const TMap<FString, FString>& ChangedModules, bool bRecompileFinished, ECompilationResult::Type CompilationResult)
{
if (ECompilationResult::Failed(CompilationResult) && bRecompileFinished)
{
Ar.Logf(ELogVerbosity::Warning, TEXT("HotReload failed, recompile failed"));
return;
}
DoHotReloadInternal(ChangedModules, InPackages, DependentModules, Ar);
},
Ar,
false, /* bFailIfGeneratedCodeChanges */
AdditionalArguments,
false /* bForceCodeProject */
);
if (!bCompileStarted)
{
Ar.Logf(ELogVerbosity::Warning, TEXT("RebindPackages failed because the compiler could not be started."));
bIsHotReloadingFromEditor = false;
return ECompilationResult::OtherCompilationError;
}
// Go ahead and check for completion right away. This is really just so that we can handle the case
// where the user asked us to wait for the compile to finish before returning.
if (!!(Flags & EHotReloadFlags::WaitForCompletion))
{
bool bCompileStillInProgress = false;
bool bCompileSucceeded = false;
FOutputDeviceNull NullOutput;
CheckForFinishedModuleDLLCompile( Flags, bCompileStillInProgress, bCompileSucceeded, NullOutput );
if( !bCompileStillInProgress && !bCompileSucceeded )
{
Ar.Logf(ELogVerbosity::Warning, TEXT("RebindPackages failed because compilation failed."));
bIsHotReloadingFromEditor = false;
return ECompilationResult::OtherCompilationError;
}
}
if (!!(Flags & EHotReloadFlags::WaitForCompletion))
{
Ar.Logf(ELogVerbosity::Warning, TEXT("HotReload operation took %4.1fs."), float(FPlatformTime::Seconds() - HotReloadStartTime));
bIsHotReloadingFromEditor = false;
}
else
{
Ar.Logf(ELogVerbosity::Warning, TEXT("Starting HotReload took %4.1fs."), float(FPlatformTime::Seconds() - HotReloadStartTime));
}
return ECompilationResult::Succeeded;
#else
Ar.Logf(ELogVerbosity::Warning, TEXT("RebindPackages not possible (hot reload not supported)"));
return ECompilationResult::Unsupported;
#endif
}
#if WITH_ENGINE
namespace {
static TArray<TPair<UClass*, UClass*> >& GetClassesToReinstance()
{
static TArray<TPair<UClass*, UClass*> > Data;
return Data;
}
}
void FHotReloadModule::RegisterForReinstancing(UClass* OldClass, UClass* NewClass)
{
TPair<UClass*, UClass*> Pair;
Pair.Key = OldClass;
Pair.Value = NewClass;
TArray<TPair<UClass*, UClass*> >& ClassesToReinstance = GetClassesToReinstance();
ClassesToReinstance.Add(MoveTemp(Pair));
}
void FHotReloadModule::ReinstanceClasses()
{
#if WITH_HOT_RELOAD
if (GIsHotReload)
{
UClass::AssembleReferenceTokenStreams();
}
#endif // WITH_HOT_RELOAD
TArray<TPair<UClass*, UClass*> >& ClassesToReinstance = GetClassesToReinstance();
TMap<UClass*, UClass*> OldToNewClassesMap;
for (const TPair<UClass*, UClass*>& Pair : ClassesToReinstance)
{
// Don't allow reinstancing of UEngine classes
if (Pair.Key->IsChildOf(UEngine::StaticClass()))
{
UE_LOG(LogHotReload, Warning, TEXT("Engine class '%s' has changed but will be ignored for hot reload"), *Pair.Key->GetName());
continue;
}
if (Pair.Value != nullptr)
{
OldToNewClassesMap.Add(Pair.Key, Pair.Value);
}
}
for (const TPair<UClass*, UClass*>& Pair : ClassesToReinstance)
{
// Don't allow reinstancing of UEngine classes
if (!Pair.Key->IsChildOf(UEngine::StaticClass()))
{
ReinstanceClass(Pair.Key, Pair.Value, OldToNewClassesMap);
}
}
ClassesToReinstance.Empty();
}
void FHotReloadModule::ReinstanceClass(UClass* OldClass, UClass* NewClass, const TMap<UClass*, UClass*>& OldToNewClassesMap)
{
TSharedPtr<FHotReloadClassReinstancer> ReinstanceHelper = FHotReloadClassReinstancer::Create(NewClass, OldClass, OldToNewClassesMap, ReconstructedCDOsMap, HotReloadBPSetToRecompile, HotReloadBPSetToRecompileBytecodeOnly);
if (ReinstanceHelper->ClassNeedsReinstancing())
{
UE_LOG(LogHotReload, Log, TEXT("Re-instancing %s after hot-reload."), NewClass ? *NewClass->GetName() : *OldClass->GetName());
ReinstanceHelper->ReinstanceObjectsAndUpdateDefaults();
}
}
#endif
void FHotReloadModule::OnHotReloadBinariesChanged(const TArray<FFileChangeData>& FileChanges)
{
if (bIsHotReloadingFromEditor)
{
// DO NOTHING, this case is handled by RebindPackages
return;
}
const FModuleManager& ModuleManager = FModuleManager::Get();
TMap<FString, FString> GameModuleFilenames = UE4HotReload_Private::GetGameModuleFilenames(ModuleManager);
if (GameModuleFilenames.Num() == 0)
{
return;
}
// Check if any of the game DLLs has been added
for (const FFileChangeData& Change : FileChanges)
{
// Ignore changes that aren't introducing a new file.
//
// On the Mac the Add event is for a temporary linker(?) file that gets immediately renamed
// to a dylib. In the future we may want to support modified event for all platforms anyway once
// shadow copying works with hot-reload.
#if PLATFORM_MAC
if (Change.Action != FFileChangeData::FCA_Modified)
#else
if (Change.Action != FFileChangeData::FCA_Added)
#endif
{
continue;
}
// Ignore files that aren't of module type
FString Filename = FPaths::GetCleanFilename(Change.Filename);
if (!Filename.EndsWith(FPlatformProcess::GetModuleExtension()))
{
continue;
}
for (const TPair<FString, FString>& NameFilename : GameModuleFilenames)
{
// Handle module files which have already been hot-reloaded.
FString BaseName = FPaths::GetBaseFilename(NameFilename.Value);
StripModuleSuffixFromFilename(BaseName, NameFilename.Key);
// Hot reload always adds a numbered suffix preceded by a hyphen, but otherwise the module name must match exactly!
if (!Filename.StartsWith(BaseName + TEXT("-")))
{
continue;
}
if (ModulesRecentlyCompiledInTheEditor.Contains(FPaths::ConvertRelativePathToFull(Change.Filename)))
{
continue;
}
// Add to queue. We do not hot-reload here as there may potentially be other modules being compiled.
DetectedNewModules.Emplace(NameFilename.Key, Change.Filename);
UE_LOG(LogHotReload, Log, TEXT("New module detected: %s"), *Filename);
}
}
}
void FHotReloadModule::StripModuleSuffixFromFilename(FString& InOutModuleFilename, const FString& ModuleName)
{
// First hyphen is where the UE4Edtior prefix ends
int32 FirstHyphenIndex = INDEX_NONE;
if (InOutModuleFilename.FindChar('-', FirstHyphenIndex))
{
// Second hyphen means we already have a hot-reloaded module or other than Development config module
int32 SecondHyphenIndex = FirstHyphenIndex;
do
{
SecondHyphenIndex = InOutModuleFilename.Find(TEXT("-"), ESearchCase::IgnoreCase, ESearchDir::FromStart, SecondHyphenIndex + 1);
if (SecondHyphenIndex != INDEX_NONE)
{
// Make sure that the section between hyphens is the expected module name. This guards against cases where module name has a hyphen inside.
FString HotReloadedModuleName = InOutModuleFilename.Mid(FirstHyphenIndex + 1, SecondHyphenIndex - FirstHyphenIndex - 1);
if (HotReloadedModuleName == ModuleName)
{
InOutModuleFilename = InOutModuleFilename.Mid(0, SecondHyphenIndex);
SecondHyphenIndex = INDEX_NONE;
}
}
} while (SecondHyphenIndex != INDEX_NONE);
}
}
void FHotReloadModule::RefreshHotReloadWatcher()
{
FDirectoryWatcherModule& DirectoryWatcherModule = FModuleManager::Get().LoadModuleChecked<FDirectoryWatcherModule>(TEXT("DirectoryWatcher"));
IDirectoryWatcher* DirectoryWatcher = DirectoryWatcherModule.Get();
if (DirectoryWatcher)
{
// Watch the game directory
AddHotReloadDirectory(DirectoryWatcher, FPaths::ProjectDir());
// Also watch all the game plugin directories
for(const TSharedRef<IPlugin>& Plugin : IPluginManager::Get().GetEnabledPlugins())
{
if (Plugin->GetLoadedFrom() == EPluginLoadedFrom::Project && Plugin->GetDescriptor().Modules.Num() > 0)
{
AddHotReloadDirectory(DirectoryWatcher, Plugin->GetBaseDir());
}
}
}
}
void FHotReloadModule::AddHotReloadDirectory(IDirectoryWatcher* DirectoryWatcher, const FString& BaseDir)
{
FString BinariesPath = FPaths::ConvertRelativePathToFull(BaseDir / TEXT("Binaries") / FPlatformProcess::GetBinariesSubdirectory());
if (FPaths::DirectoryExists(BinariesPath) && !BinariesFolderChangedDelegateHandles.Contains(BinariesPath))
{
IDirectoryWatcher::FDirectoryChanged BinariesFolderChangedDelegate = IDirectoryWatcher::FDirectoryChanged::CreateRaw(this, &FHotReloadModule::OnHotReloadBinariesChanged);
FDelegateHandle Handle;
if (DirectoryWatcher->RegisterDirectoryChangedCallback_Handle(BinariesPath, BinariesFolderChangedDelegate, Handle))
{
BinariesFolderChangedDelegateHandles.Add(BinariesPath, Handle);
}
}
}
void FHotReloadModule::ShutdownHotReloadWatcher()
{
FDirectoryWatcherModule* DirectoryWatcherModule = FModuleManager::GetModulePtr<FDirectoryWatcherModule>(TEXT("DirectoryWatcher"));
if( DirectoryWatcherModule != nullptr )
{
IDirectoryWatcher* DirectoryWatcher = DirectoryWatcherModule->Get();
if (DirectoryWatcher)
{
for (const TPair<FString, FDelegateHandle>& Pair : BinariesFolderChangedDelegateHandles)
{
DirectoryWatcher->UnregisterDirectoryChangedCallback_Handle(Pair.Key, Pair.Value);
}
}
}
}
bool FHotReloadModule::Tick(float DeltaTime)
{
// We never want to block on a pending compile when checking compilation status during Tick(). We're
// just checking so that we can fire callbacks if and when compilation has finished.
// Ignored output variables
bool bCompileStillInProgress = false;
bool bCompileSucceeded = false;
FOutputDeviceNull NullOutput;
CheckForFinishedModuleDLLCompile( EHotReloadFlags::None, bCompileStillInProgress, bCompileSucceeded, NullOutput );
if (DetectedNewModules.Num() == 0)
{
return true;
}
#if WITH_EDITOR
if (GEditor)
{
// Don't try to do an IDE reload yet if we're PIE - wait until we leave
if (GEditor->bIsPlayWorldQueued || GEditor->PlayWorld)
{
return true;
}
// Don't allow hot reloading if we're running networked PIE instances
// The reason, is it's fairly complicated to handle the re-wiring that needs to happen when we re-instance objects like player controllers, possessed pawns, etc...
const TIndirectArray<FWorldContext>& WorldContextList = GEditor->GetWorldContexts();
for (const FWorldContext& WorldContext : WorldContextList)
{
if (WorldContext.World() && WorldContext.World()->WorldType == EWorldType::PIE && WorldContext.World()->NetDriver)
{
return true; // Don't allow automatic hot reloading if we're running PIE instances
}
}
}
#endif // WITH_EDITOR
// We have new modules in the queue, but make sure UBT has finished compiling all of them
if (!FDesktopPlatformModule::Get()->IsUnrealBuildToolRunning())
{
IFileManager& FileManager = IFileManager::Get();
// Remove any modules whose files have disappeared - this can happen if a compile event has
// failed and deleted a DLL that was there previously.
for (auto It = DetectedNewModules.CreateIterator(); It; ++It)
{
if (!FileManager.FileExists(*It->Value))
{
It.RemoveCurrent();
}
}
DoHotReloadFromIDE(DetectedNewModules);
DetectedNewModules.Empty();
}
else
{
UE_LOG(LogHotReload, Verbose, TEXT("Detected %d reloaded modules but UnrealBuildTool is still running"), DetectedNewModules.Num());
}
return true;
}
void FHotReloadModule::DoHotReloadFromIDE(const TMap<FString, FString>& NewModules)
{
const FModuleManager& ModuleManager = FModuleManager::Get();
int32 NumPackagesToRebind = 0;
int32 NumDependentModules = 0;
ECompilationResult::Type Result = ECompilationResult::Unsupported;
double Duration = 0.0;
TArray<FString> GameModuleNames = UE4HotReload_Private::GetGameModuleNames(ModuleManager);
if (GameModuleNames.Num() > 0)
{
FScopedDurationTimer Timer(Duration);
if (NewModules.Num() == 0)
{
return;
}
UE_LOG(LogHotReload, Log, TEXT("Starting Hot-Reload from IDE"));
HotReloadStartTime = FPlatformTime::Seconds();
FScopedSlowTask SlowTask(100.f, LOCTEXT("CompilingGameCode", "Compiling Game Code"));
SlowTask.MakeDialog();
// Update compile data before we start compiling
for (const TPair<FString, FString>& NewModule : NewModules)
{
// Move on 10% / num items
SlowTask.EnterProgressFrame(10.f/NewModules.Num());
FName ModuleName = *NewModule.Key;
UpdateModuleCompileData(ModuleName);
OnModuleCompileSucceeded(ModuleName, NewModule.Value);
}
SlowTask.EnterProgressFrame(10);
UE4HotReload_Private::FPackagesAndDependentNames PackagesAndDependentNames = UE4HotReload_Private::SplitByPackagesAndDependentNames(GameModuleNames);
SlowTask.EnterProgressFrame(80);
NumPackagesToRebind = PackagesAndDependentNames.Packages.Num();
NumDependentModules = PackagesAndDependentNames.DependentNames.Num();
Result = DoHotReloadInternal(NewModules, PackagesAndDependentNames.Packages, PackagesAndDependentNames.DependentNames, *GLog);
}
RecordAnalyticsEvent(TEXT("IDE"), Result, Duration, NumPackagesToRebind, NumDependentModules);
}
void FHotReloadModule::RecordAnalyticsEvent(const TCHAR* ReloadFrom, ECompilationResult::Type Result, double Duration, int32 PackageCount, int32 DependentModulesCount)
{
#if WITH_ENGINE
if (FEngineAnalytics::IsAvailable())
{
TArray< FAnalyticsEventAttribute > ReloadAttribs;
ReloadAttribs.Add(FAnalyticsEventAttribute(TEXT("ReloadFrom"), ReloadFrom));
ReloadAttribs.Add(FAnalyticsEventAttribute(TEXT("Result"), ECompilationResult::ToString(Result)));
ReloadAttribs.Add(FAnalyticsEventAttribute(TEXT("Duration"), FString::Printf(TEXT("%.4lf"), Duration)));
ReloadAttribs.Add(FAnalyticsEventAttribute(TEXT("Packages"), FString::Printf(TEXT("%d"), PackageCount)));
ReloadAttribs.Add(FAnalyticsEventAttribute(TEXT("DependentModules"), FString::Printf(TEXT("%d"), DependentModulesCount)));
FEngineAnalytics::GetProvider().RecordEvent(TEXT("Editor.Usage.HotReload"), ReloadAttribs);
}
#endif
}
void FHotReloadModule::OnModuleCompileSucceeded(FName ModuleName, const FString& NewModuleFilename)
{
// If the compile succeeded, update the module info entry with the new file name for this module
FModuleManager::Get().SetModuleFilename(ModuleName, NewModuleFilename);
#if WITH_HOT_RELOAD
// UpdateModuleCompileData() should have been run before compiling so the
// data in ModuleInfo should be correct for the pre-compile dll file.
FModuleCompilationData& CompileData = ModuleCompileData.FindChecked(ModuleName).Get();
FDateTime FileTimeStamp;
bool bGotFileTimeStamp = GetModuleFileTimeStamp(ModuleName, FileTimeStamp);
CompileData.bHasFileTimeStamp = bGotFileTimeStamp;
CompileData.FileTimeStamp = FileTimeStamp;
if (CompileData.bHasFileTimeStamp)
{
CompileData.CompileMethod = EModuleCompileMethod::Runtime;
}
else
{
CompileData.CompileMethod = EModuleCompileMethod::Unknown;
}
WriteModuleCompilationInfoToConfig(ModuleName, CompileData);
#endif
}
FString FHotReloadModule::MakeUBTArgumentsForModuleCompiling()
{
FString AdditionalArguments;
if ( FPaths::IsProjectFilePathSet() )
{
// We have to pass FULL paths to UBT
FString FullProjectPath = FPaths::ConvertRelativePathToFull(FPaths::GetProjectFilePath());
// @todo projectdirs: Currently non-installed projects that exist under the UE4 root are compiled by UBT with no .uproject file
// name passed in (see bIsProjectTarget in VCProject.cs), which causes intermediate libraries to be saved to the Engine
// intermediate folder instead of the project's intermediate folder. We're emulating this behavior here for module
// recompiling, so that compiled modules will be able to find their import libraries in the original folder they were compiled.
if( FApp::IsEngineInstalled() || !FullProjectPath.StartsWith( FPaths::ConvertRelativePathToFull( FPaths::RootDir() ) ) )
{
const FString ProjectFilenameWithQuotes = FString::Printf(TEXT("\"%s\""), *FullProjectPath);
AdditionalArguments += FString::Printf(TEXT("%s "), *ProjectFilenameWithQuotes);
}
}
// Use new FastPDB option to cut down linking time. Currently disabled due to problems with missing symbols in VS2015.
// AdditionalArguments += TEXT(" -FastPDB");
return AdditionalArguments;
}
#if WITH_HOT_RELOAD
bool FHotReloadModule::StartCompilingModuleDLLs(const FString& GameName, const TArray< FModuleToRecompile >& ModuleNames,
FRecompileModulesCallback&& InRecompileModulesCallback, FOutputDevice& Ar, bool bInFailIfGeneratedCodeChanges,
const FString& InAdditionalCmdLineArgs, bool bForceCodeProject)
{
// Keep track of what we're compiling
ModulesBeingCompiled = ModuleNames;
ModulesThatWereBeingRecompiled = ModulesBeingCompiled;
const TCHAR* BuildPlatformName = FPlatformMisc::GetUBTPlatform();
const TCHAR* BuildConfigurationName = FModuleManager::GetUBTConfiguration();
RecompileModulesCallback = MoveTemp(InRecompileModulesCallback);
// Pass a module file suffix to UBT if we have one
FString ModuleArg;
if (ModuleNames.Num())
{
Ar.Logf(TEXT("Candidate modules for hot reload:"));
for( const FModuleToRecompile& Module : ModuleNames )
{
if( !Module.ModuleFileSuffix.IsEmpty() )
{
ModuleArg += FString::Printf( TEXT( " -ModuleWithSuffix %s %s" ), *Module.ModuleName, *Module.ModuleFileSuffix );
}
else
{
ModuleArg += FString::Printf( TEXT( " -Module %s" ), *Module.ModuleName );
}
Ar.Logf( TEXT( " %s" ), *Module.ModuleName );
// prepare the compile info in the FModuleInfo so that it can be compared after compiling
FName ModuleFName(*Module.ModuleName);
UpdateModuleCompileData(ModuleFName);
}
}
FString ExtraArg;
#if UE_EDITOR
// NOTE: When recompiling from the editor, we're passed the game target name, not the editor target name, but we'll
// pass "-editorrecompile" to UBT which tells UBT to figure out the editor target to use for this game, since
// we can't possibly know what the target is called from within the engine code.
ExtraArg = TEXT( "-editorrecompile " );
#endif
if (bInFailIfGeneratedCodeChanges)
{
// Additional argument to let UHT know that we can only compile the module if the generated code didn't change
ExtraArg += TEXT( "-FailIfGeneratedCodeChanges " );
}
// If there's nothing to compile, don't bother linking the DLLs as the old ones are up-to-date
ExtraArg += TEXT("-canskiplink ");
// Shared PCH does no work with hot-reloading engine/editor modules as we don't scan all modules for them.
if (!ContainsOnlyGameModules(ModuleNames))
{
ExtraArg += TEXT("-nosharedpch ");
}
FString TargetName = GameName;
#if WITH_EDITOR
// If there are no game modules loaded, then it's not a code-based project and the target
// for UBT should be the editor.
if (!bForceCodeProject && !IsAnyGameModuleLoaded())
{
TargetName = TEXT("UE4Editor");
}
#endif
FString CmdLineParams = FString::Printf( TEXT( "%s%s %s %s %s%s" ),
*TargetName, *ModuleArg,
BuildPlatformName, BuildConfigurationName,
*ExtraArg, *InAdditionalCmdLineArgs );
const bool bInvocationSuccessful = InvokeUnrealBuildToolForCompile(CmdLineParams, Ar);
if ( !bInvocationSuccessful )
{
// No longer compiling modules
ModulesBeingCompiled.Empty();
ModuleCompilerFinishedEvent.Broadcast(FString(), ECompilationResult::OtherCompilationError, false);
// Fire task completion delegate
if (RecompileModulesCallback)
{
RecompileModulesCallback( TMap<FString, FString>(), false, ECompilationResult::OtherCompilationError );
RecompileModulesCallback = nullptr;
}
}
return bInvocationSuccessful;
}
#endif
bool FHotReloadModule::InvokeUnrealBuildToolForCompile(const FString& InCmdLineParams, FOutputDevice &Ar)
{
#if WITH_HOT_RELOAD
// Make sure we're not already compiling something!
check(!IsCurrentlyCompiling());
// Setup output redirection pipes, so that we can harvest compiler output and display it ourselves
void* PipeRead = NULL;
void* PipeWrite = NULL;
verify(FPlatformProcess::CreatePipe(PipeRead, PipeWrite));
ModuleCompileReadPipeText = TEXT("");
FProcHandle ProcHandle = FDesktopPlatformModule::Get()->InvokeUnrealBuildToolAsync(InCmdLineParams, Ar, PipeRead, PipeWrite);
// We no longer need the Write pipe so close it.
// We DO need the Read pipe however...
FPlatformProcess::ClosePipe(0, PipeWrite);
if (!ProcHandle.IsValid())
{
// We're done with the process handle now
ModuleCompileProcessHandle.Reset();
ModuleCompileReadPipe = NULL;
}
else
{
ModuleCompileProcessHandle = ProcHandle;
ModuleCompileReadPipe = PipeRead;
}
return ProcHandle.IsValid();
#else
return false;
#endif // WITH_HOT_RELOAD
}
void FHotReloadModule::CheckForFinishedModuleDLLCompile(EHotReloadFlags Flags, bool& bCompileStillInProgress, bool& bCompileSucceeded, FOutputDevice& Ar, bool bFireEvents)
{
#if WITH_HOT_RELOAD
bCompileStillInProgress = false;
ECompilationResult::Type CompilationResult = ECompilationResult::OtherCompilationError;
// Is there a compilation in progress?
if( !IsCurrentlyCompiling() )
{
Ar.Logf(TEXT("Error: CheckForFinishedModuleDLLCompile: There is no compilation in progress right now"));
return;
}
bCompileStillInProgress = true;
FText StatusUpdate;
if ( ModulesBeingCompiled.Num() > 0 )
{
FFormatNamedArguments Args;
Args.Add( TEXT("CodeModuleName"), FText::FromString( ModulesBeingCompiled[0].ModuleName ) );
StatusUpdate = FText::Format( NSLOCTEXT("FModuleManager", "CompileSpecificModuleStatusMessage", "{CodeModuleName}: Compiling modules..."), Args );
}
else
{
StatusUpdate = NSLOCTEXT("FModuleManager", "CompileStatusMessage", "Compiling modules...");
}
FScopedSlowTask SlowTask(0, StatusUpdate, GIsSlowTask);
SlowTask.MakeDialog();
// Check to see if the compile has finished yet
int32 ReturnCode = -1;
while (bCompileStillInProgress)
{
// Store the return code in a temp variable for now because it still gets overwritten
// when the process is running.
int32 ProcReturnCode = -1;
if( FPlatformProcess::GetProcReturnCode( ModuleCompileProcessHandle, &ProcReturnCode ) )
{
ReturnCode = ProcReturnCode;
bCompileStillInProgress = false;
}
if (bRequestCancelCompilation)
{
FPlatformProcess::TerminateProc(ModuleCompileProcessHandle);
bCompileStillInProgress = bRequestCancelCompilation = false;
}
if( bCompileStillInProgress )
{
ModuleCompileReadPipeText += FPlatformProcess::ReadPipe(ModuleCompileReadPipe);
if (!(Flags & EHotReloadFlags::WaitForCompletion))
{
// We haven't finished compiling, but we were asked to return immediately
break;
}
SlowTask.EnterProgressFrame(0.0f);
// Give up a small timeslice if we haven't finished recompiling yet
FPlatformProcess::Sleep( 0.01f );
}
}
bRequestCancelCompilation = false;
if( bCompileStillInProgress )
{
Ar.Logf(TEXT("Error: CheckForFinishedModuleDLLCompile: Compilation is still in progress"));
return;
}
// Compilation finished, now we need to grab all of the text from the output pipe
ModuleCompileReadPipeText += FPlatformProcess::ReadPipe(ModuleCompileReadPipe);
// This includes 'canceled' (-1) and 'up-to-date' (-2)
CompilationResult = (ECompilationResult::Type)ReturnCode;
// If compilation succeeded for all modules, go back to the modules and update their module file names
// in case we recompiled the modules to a new unique file name. This is needed so that when the module
// is reloaded after the recompile, we load the new DLL file name, not the old one.
// Note that we don't want to do anything in case the build was canceled or source code has not changed.
TMap<FString, FString> ChangedModules;
if(CompilationResult == ECompilationResult::Succeeded)
{
ChangedModules.Reserve(ModulesThatWereBeingRecompiled.Num());
for( FModuleToRecompile& CurModule : ModulesThatWereBeingRecompiled )
{
// Were we asked to assign a new file name for this module?
if( CurModule.NewModuleFilename.IsEmpty() )
{
continue;
}
if (IFileManager::Get().FileSize(*CurModule.NewModuleFilename) <= 0)
{
continue;
}
// If the file doesn't exist, then assume it doesn't needs rebinding because it wasn't recompiled
FDateTime FileTimeStamp = IFileManager::Get().GetTimeStamp(*CurModule.NewModuleFilename);
if (FileTimeStamp == FDateTime::MinValue())
{
continue;
}
FName ModuleName = *CurModule.ModuleName;
// If the file is the same as what we remembered it was then assume it doesn't needs rebinding because it wasn't recompiled
TSharedRef<FModuleCompilationData>* CompileDataPtr = ModuleCompileData.Find(ModuleName);
if (CompileDataPtr && (*CompileDataPtr)->FileTimeStamp == FileTimeStamp)
{
continue;
}
// If the compile succeeded, update the module info entry with the new file name for this module
OnModuleCompileSucceeded(ModuleName, CurModule.NewModuleFilename);
// Move modules
ChangedModules.Emplace(MoveTemp(CurModule.ModuleName), MoveTemp(CurModule.NewModuleFilename));
}
}
ModulesThatWereBeingRecompiled.Empty();
// We're done with the process handle now
FPlatformProcess::CloseProc(ModuleCompileProcessHandle);
ModuleCompileProcessHandle.Reset();
FPlatformProcess::ClosePipe(ModuleCompileReadPipe, 0);
Ar.Log(*ModuleCompileReadPipeText);
const FString FinalOutput = ModuleCompileReadPipeText;
ModuleCompileReadPipe = NULL;
ModuleCompileReadPipeText = TEXT("");
// No longer compiling modules
ModulesBeingCompiled.Empty();
bCompileSucceeded = !ECompilationResult::Failed(CompilationResult);
if ( bFireEvents )
{
const bool bShowLogOnSuccess = false;
ModuleCompilerFinishedEvent.Broadcast(FinalOutput, CompilationResult, !bCompileSucceeded || bShowLogOnSuccess);
// Fire task completion delegate
if (RecompileModulesCallback)
{
RecompileModulesCallback( ChangedModules, true, CompilationResult );
RecompileModulesCallback = nullptr;
}
}
#endif // WITH_HOT_RELOAD
}
void FHotReloadModule::UpdateModuleCompileData(FName ModuleName)
{
// Find or create a compile data object for this module
TSharedRef<FModuleCompilationData>* CompileDataPtr = ModuleCompileData.Find(ModuleName);
if(CompileDataPtr == nullptr)
{
CompileDataPtr = &ModuleCompileData.Add(ModuleName, TSharedRef<FModuleCompilationData>(new FModuleCompilationData()));
}
// reset the compile data before updating it
FModuleCompilationData& CompileData = CompileDataPtr->Get();
CompileData.bHasFileTimeStamp = false;
CompileData.FileTimeStamp = FDateTime(0);
CompileData.CompileMethod = EModuleCompileMethod::Unknown;
#if WITH_HOT_RELOAD
ReadModuleCompilationInfoFromConfig(ModuleName, CompileData);
FDateTime FileTimeStamp;
bool bGotFileTimeStamp = GetModuleFileTimeStamp(ModuleName, FileTimeStamp);
if (!bGotFileTimeStamp)
{
// File missing? Reset the cached timestamp and method to defaults and save them.
CompileData.bHasFileTimeStamp = false;
CompileData.FileTimeStamp = FDateTime(0);
CompileData.CompileMethod = EModuleCompileMethod::Unknown;
WriteModuleCompilationInfoToConfig(ModuleName, CompileData);
}
else
{
if (CompileData.bHasFileTimeStamp)
{
if (FileTimeStamp > CompileData.FileTimeStamp + HotReloadDefs::TimeStampEpsilon)
{
// The file is newer than the cached timestamp
// The file must have been compiled externally
CompileData.FileTimeStamp = FileTimeStamp;
CompileData.CompileMethod = EModuleCompileMethod::External;
WriteModuleCompilationInfoToConfig(ModuleName, CompileData);
}
}
else
{
// The cached timestamp and method are default value so this file has no history yet
// We can only set its timestamp and save
CompileData.bHasFileTimeStamp = true;
CompileData.FileTimeStamp = FileTimeStamp;
WriteModuleCompilationInfoToConfig(ModuleName, CompileData);
}
}
#endif
}
void FHotReloadModule::ReadModuleCompilationInfoFromConfig(FName ModuleName, FModuleCompilationData& CompileData)
{
FString DateTimeString;
if (GConfig->GetString(*HotReloadDefs::CompilationInfoConfigSection, *FString::Printf(TEXT("%s.TimeStamp"), *ModuleName.ToString()), DateTimeString, GEditorPerProjectIni))
{
FDateTime TimeStamp;
if (!DateTimeString.IsEmpty() && FDateTime::Parse(DateTimeString, TimeStamp))
{
CompileData.bHasFileTimeStamp = true;
CompileData.FileTimeStamp = TimeStamp;
FString CompileMethodString;
if (GConfig->GetString(*HotReloadDefs::CompilationInfoConfigSection, *FString::Printf(TEXT("%s.LastCompileMethod"), *ModuleName.ToString()), CompileMethodString, GEditorPerProjectIni))
{
if (CompileMethodString.Equals(HotReloadDefs::CompileMethodRuntime, ESearchCase::IgnoreCase))
{
CompileData.CompileMethod = EModuleCompileMethod::Runtime;
}
else if (CompileMethodString.Equals(HotReloadDefs::CompileMethodExternal, ESearchCase::IgnoreCase))
{
CompileData.CompileMethod = EModuleCompileMethod::External;
}
}
}
}
}
void FHotReloadModule::WriteModuleCompilationInfoToConfig(FName ModuleName, const FModuleCompilationData& CompileData)
{
FString DateTimeString;
if (CompileData.bHasFileTimeStamp)
{
DateTimeString = CompileData.FileTimeStamp.ToString();
}
GConfig->SetString(*HotReloadDefs::CompilationInfoConfigSection, *FString::Printf(TEXT("%s.TimeStamp"), *ModuleName.ToString()), *DateTimeString, GEditorPerProjectIni);
FString CompileMethodString = HotReloadDefs::CompileMethodUnknown;
if (CompileData.CompileMethod == EModuleCompileMethod::Runtime)
{
CompileMethodString = HotReloadDefs::CompileMethodRuntime;
}
else if (CompileData.CompileMethod == EModuleCompileMethod::External)
{
CompileMethodString = HotReloadDefs::CompileMethodExternal;
}
GConfig->SetString(*HotReloadDefs::CompilationInfoConfigSection, *FString::Printf(TEXT("%s.LastCompileMethod"), *ModuleName.ToString()), *CompileMethodString, GEditorPerProjectIni);
}
bool FHotReloadModule::GetModuleFileTimeStamp(FName ModuleName, FDateTime& OutFileTimeStamp) const
{
FString Filename = FModuleManager::Get().GetModuleFilename(ModuleName);
if (IFileManager::Get().FileSize(*Filename) > 0)
{
OutFileTimeStamp = FDateTime(IFileManager::Get().GetTimeStamp(*Filename));
return true;
}
return false;
}
bool FHotReloadModule::IsAnyGameModuleLoaded()
{
if (bIsAnyGameModuleLoaded == EThreeStateBool::Unknown)
{
bool bGameModuleFound = false;
// Ask the module manager for a list of currently-loaded gameplay modules
TArray< FModuleStatus > ModuleStatuses;
FModuleManager::Get().QueryModules(ModuleStatuses);
for (auto ModuleStatusIt = ModuleStatuses.CreateConstIterator(); ModuleStatusIt; ++ModuleStatusIt)
{
const FModuleStatus& ModuleStatus = *ModuleStatusIt;
// We only care about game modules that are currently loaded
if (ModuleStatus.bIsLoaded && ModuleStatus.bIsGameModule)
{
// There is at least one loaded game module.
bGameModuleFound = true;
break;
}
}
bIsAnyGameModuleLoaded = EThreeStateBool::FromBool(bGameModuleFound);
}
return EThreeStateBool::ToBool(bIsAnyGameModuleLoaded);
}
bool FHotReloadModule::ContainsOnlyGameModules(const TArray<FModuleToRecompile>& ModulesToCompile) const
{
const FString AbsoluteProjectDir(FPaths::ConvertRelativePathToFull(FPaths::ProjectDir()));
bool bOnlyGameModules = true;
for (auto& ModuleToCompile : ModulesToCompile)
{
const FString FullModulePath(FPaths::ConvertRelativePathToFull(ModuleToCompile.NewModuleFilename));
if (!FullModulePath.StartsWith(AbsoluteProjectDir))
{
bOnlyGameModules = false;
break;
}
}
return bOnlyGameModules;
}
void FHotReloadModule::ModulesChangedCallback(FName ModuleName, EModuleChangeReason ReasonForChange)
{
// Force update game modules state on the next call to IsAnyGameModuleLoaded
bIsAnyGameModuleLoaded = EThreeStateBool::Unknown;
// If the hot reload directory watcher hasn't been initialized yet (because the binaries directory did not exist) try to initialize it now
if (!bDirectoryWatcherInitialized)
{
RefreshHotReloadWatcher();
bDirectoryWatcherInitialized = true;
}
}
void FHotReloadModule::PluginMountedCallback(IPlugin& Plugin)
{
FDirectoryWatcherModule& DirectoryWatcherModule = FModuleManager::Get().LoadModuleChecked<FDirectoryWatcherModule>(TEXT("DirectoryWatcher"));
IDirectoryWatcher* DirectoryWatcher = DirectoryWatcherModule.Get();
if (DirectoryWatcher)
{
if (Plugin.GetLoadedFrom() == EPluginLoadedFrom::Project && Plugin.GetDescriptor().Modules.Num() > 0)
{
AddHotReloadDirectory(DirectoryWatcher, Plugin.GetBaseDir());
}
}
}
#undef LOCTEXT_NAMESPACE