Commit Graph

212 Commits

Author SHA1 Message Date
paul chipchase
0376e1d8b2 Added command line switch '-ApplyIoStoreOnDemand' to UAT BuildCookRun that will force the IoStoreOnDemand module into the target project and force on additional settings to allow IoStoreOnDemand to work.
#rb Per.Larsson
#jira UE-218584
#rnx

- This is only intended for development and testing IoStoreOnDemand as it allows us to apply it to a project without changing the project files on disk. This should not be used to enable the feature on a project in production.

### Enabling the code module
- When forcing on IoStoreOnDemand, BuildCookRun will pass '-CompileIoStoreOnDemand' to UnrealBuildTool which will set 'bCompileIoStoreOnDemand' to true in the target rules. This in turn will add the 'IoStoreOnDemand' module to the Engine module but only for runtime targets.
- None of this should be used when enabling the feature for a project permanentl, instead just add 'IoStoreOnDemand' module to your project's *.build.cs

### Staging SSL certificates
- IoStoreOnDemand needs access to the engine/project#s SSL certificates at runtime  so we need to stage them, which is done by forcing 'bStageSSLCertificates' to be true during BuildCookRun.
- To do this properly for a project set ini:Engine:[/Script/Engine.NetworkSettings]:n.VerifyPeer=true

### Generate Chunks
- In order for IoStoreOnDemand to work we need to generate chunks (more than one container file) which can be done by forcing ProjectParams.Manifests to true.
- To do this properly for a project set ini:Game:[/Script/UnrealEd.ProjectPackagingSettings]:bGenerateChunks=true

### Pak Rules
- Once chunked builds are enabled IoStoreOnDemand will require at least one PakRule that sets  'bOnDemand' to true. So first we check to see if any rules like this for the existing project and if not we force add one via CreateIoStoreOnDemandPakRules. This rule will include all *.ubulk and *.uptnl files except ones found under a \map\ directory. This is because some level bulkdata files can get too large for IoStoreOnDemand to support so excluding anything under a map directory is a quick and easy way to try and work around that. The end result is that other than map data, pretty much all bulkdata will end up on demand, which is a good stress test for the system but not a great representation on how the system should be used.
- If you are trying to set up rules properly you'd need to consider what data you actually want to stream over the network and what data you want to remain on disk, which would greatly depend on your project set up. One popular choice is to only stream optional (.uptnl) bulkdata and leave the required bulkdata to be found on disk.

### Enabling Persistent Local Storage
- IoStoreOnDemand requires persistent local storage to be set aside for it's caching which may or may not require special set up for a project depending on the target platform. If we detect that the current project has not been set up for persistent local storage we will attempt to enable it for the project during staging.
- To set this up properly you will need to consult the documentation for your target platform and follow the instructions.

[CL 34702097 by paul chipchase in ue5-main branch]
2024-06-27 02:50:43 -04:00
stephen ma
ff0fe033ec Add -projectprogramoverride to allow BuildCookRun to override passed in uproject paramter when building programs.
[REVIEW] josh.adams
#rb Josh.Adams

[CL 34388872 by stephen ma in ue5-main branch]
2024-06-14 19:23:10 -04:00
joe kirchoff
72a8c5996d Fix various XML comment issues in AutomationTool and various scripts (non-functional change)
#rnx

[CL 34263204 by joe kirchoff in ue5-main branch]
2024-06-10 21:51:45 -04:00
josh adams
1ef3087292 - Ability to override Xcodebuild settings when staging and packaging with BuildCookRun, it is not used when building
#rb josh.adams

[CL 33128829 by josh adams in ue5-main branch]
2024-04-20 11:08:10 -04:00
zousar shaker
67fe4b2afe Fix stage/deploy functionality so that if you're using fileserver/projectstore functionality, it doesn't write a layout file that expects loose files (INIs, etc) to be on the filesystem. This required moving some pieces of functionality onto ProjectParams so that they are accessible by a platform automation layer.
#rb dan.engelbrecht, PJ.Kack

[CL 33041507 by zousar shaker in ue5-main branch]
2024-04-17 13:30:25 -04:00
ben marsh
777487df11 Horde: Only enable code signing by default when running on an Epic build machine. The IsBuildMachine flag is true for any Horde deployment, but UAT hard-codes the signing certificate name for Epic. Check for the EpicInternal.txt flag to indicate whether we're running on Epic's farm.
#jira UE-210179

[CL 32376723 by ben marsh in ue5-main branch]
2024-03-20 14:56:15 -04:00
josh adams
610ad3e065 - Added a shared GetEditorForProject() function which should be used in UAT to get the path to the binary to run, becasue UnrealEditor-Cmd.exe is not always correct (if the editor is a Unique build environment, the editor to use to cook etc is the project editor. not UnrealEditor)
#rb david.harvey

[CL 31621119 by josh adams in ue5-main branch]
2024-02-19 15:10:30 -05:00
dan thompson
15fd28a21e Reference Chunk Database:
* Optionally output a CSV of changed/new packages.
* Add an additional directory input if containers get separated off during a build process
* Catch optional segment containers and patch up the container id per the id fix in a separate CL.
bunch of different preflights due to CIS issues.
#rb fabian.giesen
#jira UE-172130

[CL 31576665 by dan thompson in ue5-main branch]
2024-02-16 15:10:30 -05:00
chris varnsverry
f12ea6b8eb - Fix UAT not suggesting -skipcook as an option when Stage is requested and no cook param is provided.
- Note Cook is true when -skipcook is set on command line so the condition itself is valid it just was missing -skipcook in the exception message

[REVIEW] [at]Alejandro.Aguilar [at]Rob.Cannaday

#rb alejandro.aguilar

[CL 30806276 by chris varnsverry in ue5-main branch]
2024-01-23 11:08:05 -05:00
josh adams
980ffe3ce7 - Allow for code to set ProjectParams.UnrealExe (not used in Main, but this restores functionality that was removed)
#rb josh.adams

[CL 30661384 by josh adams in ue5-main branch]
2024-01-17 12:15:06 -05:00
josh adams
8de5dde4df - Added support for UAT to cook with a specific project's editor .exe for projects that use a Unique BuildEnvironment (this will cook with the game's editor, so it will use the game's uniquely compiled engine modules (useful for per-project SDKs that have different targetplatform dlls)
- The target receipt already listed the proper exe to run, taking unique/shared into account, so this change delayed-looks-up the path from the receipt on first use (like when running the editor to cook)
#rb david.harvey

[CL 30658383 by josh adams in ue5-main branch]
2024-01-17 10:07:42 -05:00
sebastian schoner
a7c5f5f909 Allow to skip -fileopenlog for cooks
The file open log is a facility that automatically keeps a log of all files the process has opened and their order. This is rather expensive as it (a) also tracks failed attempts to open files and (b) it maintains a large table in memory, storing which files have already been opened (so every file is only logged once). This table can easily grow into the tens of megabytes; even on an empty project I see this taking 60MB by itself. Making this optional allows us to skip this step for UEFN cloud cooks: in these cases we're throwing that log file away anyway.

#rb rob.perren

[CL 30344591 by sebastian schoner in ue5-main branch]
2023-12-15 04:22:42 -05:00
per larsson
5d4e6f8552 IAS - added UAT option -Upload=<args> for uploading ondemand content to S3 compatible endpoint
Use -Upload=LocalZen to upload to local Zen server

#rb Paul.Chipchase

[CL 29777954 by per larsson in ue5-main branch]
2023-11-16 10:37:28 -05:00
alexander suvorov
0223b4b83d Move RetainStagedDirectory setting from launcher profile to project packaging settings.
#jira UE-185249
#rb none

[CL 28095878 by alexander suvorov in ue5-main branch]
2023-09-21 14:25:49 -04:00
david harvey
cf438cded9 CustomDeployment handler can be specified on the command line.
- still only used on Windows at the moment.

#jira UE-179187
#rnx
#rb Tomasz.Obrebski

[CL 27712186 by david harvey in ue5-main branch]
2023-09-08 11:14:48 -04:00
Matt Peters
e3be932357 Cooker: AdditionalPluginDirectories: Fix staging to look in the RemappedPlugins directory when cooking DLC plugins that are in AdditionalPluginDirectories.
Fix cooker to store all metadata in the RemappedPlugins.
Fix cooker for even normal DLCs to store all metadata in the DLC's directory rather than the project's directory.
#jira UE-192828
#rn Bugfix, Core
#rb Zousar.Shaker

[CL 27205588 by Matt Peters in ue5-main branch]
2023-08-18 12:05:51 -04:00
eric knapik
52215d80a3 #Jira: FORT-617630
Adding buildgraph option to EnablePlugins for cooking and building. Enabling a plugin in the buildgraph will enable it for cook and compilation but will not enable it in the application that must be done via other means
Updating BuildCookRun commands so they pass the EnablePlugins command through the build system

[FYI] Taylor.Robert

[CL 25881480 by eric knapik in ue5-main branch]
2023-06-08 16:23:30 -04:00
dmytro ivanov
15a6f27a3e Pass RunCommandline during launch to improve iteration times on mobile by not needing to rebuild .apk every time, because they are not passed via staged build anymore.
#jira UE-185550
#rb adam.kinge

[CL 25868747 by dmytro ivanov in ue5-main branch]
2023-06-08 08:25:43 -04:00
David Harvey
bf4d7523a2 Fixes for multiple build targets:
- do not need to specify a default server target when doing -cook
 - UFE allows launching default Game and Client targets now that a default can be specified in BuildSettings (note that it won't verify that a default has been configured - UAT handles this)


#jira UE-181924
#rb Josh.Adams
#preflight 6446823027014596f36a5411

[CL 25163212 by David Harvey in ue5-main branch]
2023-04-24 09:30:03 -04:00
dan engelbrecht
db5e568754 Game client no longer needs -zenstoreproject or -zenstorehost arguments to correctly run with a COTF server which uses Zen.
Game client now accepts only -filehostip for connection to a CTOF server, -cookonthefly is no longer necessary and is ignored.
Game client will now receive ProjectName (ProjectId), Platform (OplogId), Zen server host name and port from COTF server if running with zenstore.
Fixed issues with generating the ProjectId from a path where drive letters where lower case.

#rb zousar.shaker pj.kack
#preflight

[CL 24889513 by dan engelbrecht in ue5-main branch]
2023-04-03 01:51:40 -04:00
bryan sefcik
ab8961a2f0 Added zenstore and iterate arguments to the ProjectParams.
Updated CookProfile.

#preflight

[CL 24734295 by bryan sefcik in ue5-main branch]
2023-03-21 13:03:34 -04:00
Ben Marsh
e141cb6196 More UAT structured logging fixes.
#preflight 6408dee3b0544ef0b4bca696

[CL 24564590 by Ben Marsh in ue5-main branch]
2023-03-08 14:32:15 -05:00
Ben Marsh
235eb77726 Change to structured logging in other AutomationTool projects.
#preflight 6408c4478c0039bbf750e861

[CL 24562553 by Ben Marsh in ue5-main branch]
2023-03-08 12:43:35 -05:00
Ben Marsh
89d6613573 UAT: Refactor AutomationUtils and BuildGraph to use structured logging.
#preflight 6407c4c26e1dadfcdab25849

[CL 24553706 by Ben Marsh in ue5-main branch]
2023-03-07 21:23:47 -05:00
bryan sefcik
836b14219e * Added an interface that contains ParamHelp attributes for ProjectParam so we can reflect the parameters in other tools.
* Updated LauncherTool to traverse the class interfaces looking for attributes. Using GetCustomAttributes is not good enough because it will not find the HelpAttributes because the base attribute class(DescriptionAttribute) does not explicitly mark itself that the attribute is inherited.

#rb gary.yuan
#preflight 64028b895515f4f57b0a62c0

[CL 24511953 by bryan sefcik in ue5-main branch]
2023-03-03 20:48:42 -05:00