[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34498668 by zousar shaker in ue5-main branch]
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.
#rb PJ.Kack
[CL 34481636 by zousar shaker in ue5-main branch]
- restored zlib source for cross platform builds in the UE stream
- added ninjabuild support to UBT cmake build target
- some minor catch2 and libpng changes to better support cross platform builds.
#ushell-cherrypick of 34416030 by David.Harvey
[CL 34445726 by michael wanderson in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Always run UnrealPak itself with -nopak to avoid picking up and mounting any existing pak files in the project directory.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34266589 by pj kack in ue5-main branch]
[FYI] PJ.Kack
Original CL Desc
-----------------------------------------------------------------
UnrealPak: When the existing optional .uproject argument is provided first on the commandline, then let UnrealPak use that to set its project directory and override the monolithic engine program directory (Engine/Programs/UnrealPak).
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34107564 by graeme thornton in ue5-main branch]
This will make UnrealPak read from and use the project-specific ini files.
This will make UnrealPak write to and use the project-specific Saved directory, e.g. to save the UnrealPak.log file.
In the BuildCookRun/CopyBuildToStagingDirectory scripts always invoke UnrealPak with the .uproject file argument.
Motivation: When UnrealPak is run with -compressionEnableDDC it needs to use the project's ddc configuration instead of the engine's ddc configuration.
Note: When UnrealPak is invoked without the project argument it will still use the default program directory, and there is no change in behavior when e.g. manually executing inspection commands like list, extract and describe.
Note: All important packaging options are already passed on command line from BuildCookRun/CopyBuildToStagingDirectory, so from that perspective the change should be low risk, but in the future there is a possibility to reduce the number of command line options and rely on the project configuration instead.
#rb per.larsson
#tests identical binary output
[CL 34089539 by pj kack in ue5-main branch]
This log file may contain more verbose logging than the one generated by RunAndLog() based on stdout/stderr.
Route all calls to UnrealPak through the same RunUnrealPak function.
#rb dan.engelbrecht
#rnx
[CL 33920501 by pj kack in ue5-main branch]
Removing redundant copy as well, as the files are already tagged via other steps
#jira UE-212012
#rb jeremie.roy
[CL 32873716 by ryan hummer in ue5-main branch]
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Fixed issue with FPaths::IsStaged() with -pak generated builds (the StagedBuild.ini file can't be in the .pak since it needs ProjectDir() to find .pak files!)
#rb ben.hoffman
[CL 32203658 by bob tellez in ue5-main branch]
#rb Per.Larsson
#jira UE-189912
#rnx
- There were multiple areas that checked if a cooked file was of a certain extension, this code has been consolidated in a single method ::CanCookedFileBeStaged and as we were excluding .ucas/.utoc I added .uondemandtoc to be excluded too.
- There are three places with code logic that try to copy container files from one location to another but each one is subtley different so I left the code duplication in place but added support to copy any .uondemandtoc file that might exist. Unlike the .ucas/.utoc the new type is optional so that had to be taken into account.
- ::DeletePakFiles has been updated to also delete all .uondemandtoc files present. The only code path calling this does not seem to be easily accessed, I have added a code comment describing the only method I found that could actually cause it to be executed to hopefully help the next person working in this area.
- Note that in the code branch for creating a patch for chunked installed builds (line 3955) I am not confident that this works at all. All if the file copying operations use 'PatchSourceContentPath' as the source which is almost certainly wrong and the path itself seems to always contain a wildcard character which would cause it to be wrong as well. For now I have followed the current code convention and used 'PatchSourceContentPath' when copying the .uondemandtoc but looking into this code path will need to be it's own work item.
[CL 32020661 by paul chipchase in ue5-main branch]
#rb Martin.Ridgers
#rnx
- Make sure that the path to the pakfile directory and the path to the cryptokeys are properly quoted.
[CL 31983383 by paul chipchase in ue5-main branch]
#rb Martin.Ridgers
#rnx
- Instead of passing in '-Upload=<PathToPaks>' the IasTool expects first a command 'Upload' followed by '<PathToPaks>' as it's own arg.
- Rather than adding an IasTool version of ::GetUnrealPakLocation, I have opted to keep the code all together in a single function as we are likely to remove this from the general staging scripts at some point.
[CL 31902068 by paul chipchase in ue5-main branch]
