Commit Graph

119 Commits

Author SHA1 Message Date
pj kack
3ad849a688 UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Always run UnrealPak itself with -nopak to avoid picking up and mounting any existing pak files in the project directory.

Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.

Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.

#rb per.larsson
#tests identical binary output

[CL 34266589 by pj kack in ue5-main branch]
2024-06-11 03:37:13 -04:00
graeme thornton
6ccbb2644f [Backout] - CL34089536
[FYI] PJ.Kack
Original CL Desc
-----------------------------------------------------------------
UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.

Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.

Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.

#rb per.larsson
#tests identical binary output

[CL 34107564 by graeme thornton in ue5-main branch]
2024-06-04 17:23:55 -04:00
pj kack
9e01741303 UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.

Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.

Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.

#rb per.larsson
#tests identical binary output

[CL 34089539 by pj kack in ue5-main branch]
2024-06-04 06:12:40 -04:00
josh adams
c0f2905ce7 - Added a method for Shared build environment builds to have certain modules marked as caring about SDK version overrides for some platforms - if any of those platforms are overridden, the DLL will be written to the project Binaries dir instead of Engine (set bAllowSDKOverrideModulesWithSharedEnvironment = true in a Target.cs)
- Added ability to optionally skip SDK validation when making a Target instance even if we don't want to skip Target validation (used when we make the base UnrealEditor target to compare to project's Editor target - we don't need to manage SDK for that temporary UnrealEditor target).
- ShaderCompileWorker is now BuildEnvironment.UniqueIfNeeded, which will build into the project binaries dir if needed (based on per-project SDK override)
- UnrealPak is marked as not caring about an SDK versions so it can be built along with SCW that does care
- Added $(BinaryDir) variable to UEBuildTarget, etc, for the location of the first Binary in the output binaries
#rb David.Harvey, Joe.Kirchoff

[CL 31661939 by josh adams in ue5-main branch]
2024-02-20 18:39:20 -05:00
matt peters
f6fc4928d4 IoStoreUtilities: Add TargetPlatform module to UnrealPak so that GenerateZenFileSystemManifest and other commands needing TargetPlatform will work.
#rb PJ.Kack
#rnx

[CL 27237093 by matt peters in ue5-main branch]
2023-08-21 09:20:06 -04:00
robert millar
2ac24cb374 Add target rules option for enabling trace.
#rb ionut.matasaru

[CL 27011549 by robert millar in ue5-main branch]
2023-08-10 17:10:14 -04:00
bryan sefcik
91c57d395e Removed redundant module includes.
#preflight 645d4bf3aa3c584c0b5b3a67

[CL 25435653 by bryan sefcik in ue5-main branch]
2023-05-11 16:48:21 -04:00
ionut matasaru
3d7c0e8440 UBT: Removed deprecated TargetInfo::bUseMallocProfiler toggle in Source/Programs.
#jira UE-141779
#rb Joe.Kirchoff
#rb Johan.Berg
#preflight 64380aea7a00790bfd116430

[CL 25101875 by ionut matasaru in ue5-main branch]
2023-04-19 04:15:59 -04:00
christopher waters
6a4206d490 Removing bad Launch include paths from programs.
[CL 24328631 by christopher waters in ue5-main branch]
2023-02-20 17:39:13 -05:00
pj kack
4023ee1dde UnrealPak: Optimizations for small workloads.
Add -fastexit parameter to terminate process instead of shutting down engine.
Turn off build target features to build less code (total dll size reduced from 56 MB to 46 MB).
Disable external profiler supports (skips loading of some dlls).
Don't spawn any reserve workers (by adding a DefaultEngine.ini for UnrealPak).
Make UnrealPAk compile in Test config (by fixing compile error in PerforceConnection.cpp).

#rb carlmagnus.nordin
#rnx
#preflight 63e5eaf66473b534f14e377b

[CL 24118943 by pj kack in ue5-main branch]
2023-02-10 10:13:31 -05:00
paul chipchase
2c1e11b225 UnrealPak now supports rehydrating virtualized files as they are placed into pak files
#rb CarlMagnus.Nordin, Per.Larsson
#rnx
#preflight 63496fcf1f6054a99fe8bd0c

- When parsing the paklist, any file with a -rehydrate flag will be considered for rehydration

- Add the virtualization module to the PakFileUtilities module
- Add the source control module to UnrealPak (the virtualization module should be taking care of this)
- While parsing the files to be included in the pak file, we will record if any of the files require rehydration, if so this will be noted in FPakCommandLineParameters::bRequiresRehydration and used to initialize the virtualization system.
-- We only need to initialize the system once, even if we detect that multiple pak files have files that need rehydration.
-- If no pak file needs rehydration we do not initialize the system and the virtualization module is never loaded.
- FOutputPakFileEntry::CompressedFileBuffer was made private with accessors to make the refactor easier.
- Now when we start to compress a file, we always load it entirely into memory (along with any potential padding needed), if we then detect that the file doesn't require padding then we just use the in memory buffer as the output and if we do need compression we compress the buffer.
-- This means that there is only one place we load the file from disk, meaing only one place we need to insert the rehydration code to. This load is where the rehydration occurs.
-- In the previous code we would load the file into memory and then retain this copy during the compression pass, so no additional memory should be used.
- At the moment when we get the buffer back fromt he rehydration pass it will be in the form of a FSharedBuffer and so for now we need to memcpy to FCompressedFileBuffer::UncompressedBuffer, which is a waste of cpu cycles.
-- In a future change we should change UncompressedBuffer from TArray to FSharedBuffer to avoid this.

[CL 22595708 by paul chipchase in ue5-main branch]
2022-10-18 05:23:58 -04:00
rune stubbe
67f98f295b Fix for issue where running UnrealPak with -WriteBackMetadataToAssetRegistry=OriginalFile would crash because UObjectRedirector was not properly initialized
#rb matt.peters, carlmagnus.nordin
[FYI] dan.thompson, graham.wihlidal
#preflight 6304e4265366f61a426ab19b

[CL 21524991 by rune stubbe in ue5-main branch]
2022-08-23 19:22:37 -04:00
christopher waters
22de0b360d Adding a "deprecation" path for engine includes to allow types to be moved without breaking licensee module builds.
- Non-engine modules/targets will have to specify the "version" of includes via IncludeOrderVersion in TargetRules or ModuleRules.
- This setting will control the value of UE_ENABLE_INCLUDE_ORDER_DEPRECATED_IN_XXX where XXX is the version of the engine.
- When moving types out of a header, users will need to include the new location of the type in the header it was removed from but only if UE_ENABLE_INCLUDE_ORDER_DEPRECATED_IN_XXX is set.
- If a target does not change its IncludeOrderVersion to the latest version, UBT will print out a message telling users how to upgrade.
- This change introduces a new set of SharedPCH permutations to make sure modules with older versions get the PCH with UE_ENABLE_INCLUDE_ORDER_DEPRECATED_IN_XXX set correctly.

#jira none
#rb jonathan.adamczewski, joe.kirchoff
#preflight 623e1d3d196f3ae80b4c37ee

[CL 19518359 by christopher waters in ue5-main branch]
2022-03-25 16:16:30 -04:00
pj kack
884f412263 Enable trace for the UnrealPak program
#jira none
#rb carlmagnus.nordin
#rnx
#preflight 61e520e2a2616066f675e9ef

#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18631137 in //UE5/Release-5.0/... via CL 18631146 via CL 18631149
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18631161 by pj kack in ue5-main branch]
2022-01-17 03:13:58 -05:00
carlmagnus nordin
341637ba37 IoStore: Changes to allow IoStore to run as part of UnrealPak instead of as a cmdlet
#rb per.larsson
#rnx

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17517167 via CL 17517189 via CL 17517191 via CL 17519703 via CL 17519714
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v870-17433530)

[CL 17519729 by carlmagnus nordin in ue5-main branch]
2021-09-15 10:40:05 -04:00
charles bloom
e676538c73 intentional nop touch of UnrealPak.cpp so it gets rebuilt with Oodle plugin
#rb none

[CL 15994514 by charles bloom in ue5-main branch]
2021-04-13 16:49:41 -04:00
Marc Audy
01b7c9f4f5 Merge UE5/RES @ 15958325 to UE5/Main
This represents UE4/Main @ 15913390 and Dev-PerfTest @ 15913304

[CL 15958515 by Marc Audy in ue5-main branch]
2021-04-08 14:32:07 -04:00
SuperLanceur
869067cfa7 #jira UE-106397
#rb trivial

PR #7676: Misplaced parenthesis (Contributed by SuperLanceur)

[CL 15134734 by SuperLanceur in ue5-main branch]
2021-01-19 05:59:56 -04:00
Marcus Wassmer
3b81cf8201 Merging using //UE5/Main_to_//UE5/Release-Engine-Staging @14384769
autoresolved files
#rb none

[CL 14384911 by Marcus Wassmer in ue5-main branch]
2020-09-24 00:43:27 -04:00
Arne Schober
3bf935d1ce Explicitly tagging IsInRenderingThread, IsInGameThread etc. functions to be able to move them between theads in the future. The old behaviour is still in packe to verify the new system, but will be removed in the future.
#RB Andriy.Tylychko, Jonas.Meyer

[CL 13825935 by Arne Schober in ue5-main branch]
2020-07-02 13:21:15 -04:00
Marc Audy
11f5b21210 Merging //UE5/Release-Engine-Staging @ 13752110 to Main (//UE5/Main)
#rnx

[CL 13753156 by Marc Audy in ue5-main branch]
2020-06-23 18:40:00 -04:00
Ryan Durand
74c879d5f3 Updating copyrights for Engine Programs.
#rnx
#rb none
#jira none

#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869242 in //Fortnite/Release-12.00/... via CL 10869536
#ROBOMERGE-BOT: FORTNITE (Main -> Dev-EngineMerge) (v613-10869866)

[CL 10870960 by Ryan Durand in Main branch]
2019-12-26 23:06:02 -05:00
graeme thornton
9505003326 Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5447485 by graeme thornton in Main branch]
2019-03-19 07:53:22 -04:00
Josh Adams
d0bf843c9c - Merging Dev-Kairos/Engine/... to Main/Engine/...
- Brings over the necessary engine changes for embedding UE4 mobile as a dylib/so in native mobile app
- Various changes for facial animation, screen recording, others
- ARKit and ARCore plugins were removed, as deemed "not ready"
#rb many people


#ROBOMERGE-OWNER: josh.adams
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 5201138 via CL 5203024

[CL 5226277 by Josh Adams in Main branch]
2019-02-27 11:57:17 -05:00
James Hopkin
a9f7efc154 Copying //UE4/Dev-Build to Dev-Main (//UE4/Dev-Main)
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 4718806

[CL 4732416 by James Hopkin in Dev-Core branch]
2019-01-16 10:11:46 -05:00