1026 Commits

Author SHA1 Message Date
ben marsh
e29ca7383b Add a log event matcher for running bash scripts. Improves errors in Horde server build jobs.
#rnx

[CL 37173211 by ben marsh in 5.5 branch]
2024-10-16 09:26:34 -04:00
james singer
10680d051b Make Turnkey persist UE_SDKS_ROOT on Mac and Linux
#jira UE-223174
[REVIEW] [at]josh.adams, [at]zack.neyland
#rb zack.neyland

[CL 36762504 by james singer in 5.5 branch]
2024-10-01 20:18:14 -04:00
josh adams
7e0ec5711d - Changed Hybrid project checking to always check all enabled platforms to see if any need to be hybrid. This helps address an issue where a single-platform (say IOS) plugin was causing packaging to fail because the host platform (Mac) to clean up the hybrid target files, so then packaging for IOS would fail to find the compiled hybrid targets
#rb Chris.Babcock
#jira UE-224099

[CL 36758053 by josh adams in 5.5 branch]
2024-10-01 19:35:42 -04:00
ben marsh
2af764e351 Merging latest Horde changes from Main.
[CL 36756615 by ben marsh in 5.5 branch]
2024-10-01 19:23:06 -04:00
ben marsh
595309196c Merging NET 8 upgrades from Main.
[CL 36755991 by ben marsh in 5.5 branch]
2024-10-01 19:12:37 -04:00
ryan hummer
77b26055ce Code Signing use /tr instead of /t
As we now force the algo so /tr needs to be used.

#rnx
#rb none

[CL 36752802 by ryan hummer in 5.5 branch]
2024-10-01 18:45:11 -04:00
ryan hummer
2e93d91d33 Update sha1 code signing to use /fd and /td args
Args are now required with new signtool shipped with newer win sdks

#rnx
#rb none

[CL 36749483 by ryan hummer in 5.5 branch]
2024-10-01 18:06:07 -04:00
Ryan Hummer
4af2fd066d Updating Dev-Release-5.5 from Main at CL #36144969
#okforversepublic

[CL 36146571 by Ryan Hummer in Dev-5.5 branch]
2024-09-10 10:26:02 -04:00
ryan hummer
51da608b6f Windows Code Signing Adjustments
* Fixing code so the first server is used
* removing a server as it appears to be not reliable anymore

#rnx
#jira UE-222611
#rb none

[CL 35798006 by ryan hummer in ue5-main branch]
2024-08-26 10:19:26 -04:00
ryan hummer
cf259e9255 Removing the restriction to force tvOS and iOS into serial compiles
The issue needing the restriction no longer exists, each target has a unique .app folder with modern xcode tooling

#rnx
#rb none

[CL 35519183 by ryan hummer in ue5-main branch]
2024-08-13 20:43:22 -04:00
marc audy
4543d66de1 Only call p4 client for clients where the root path is a match for the file in question
#rnx
#jira

[CL 35255025 by marc audy in ue5-main branch]
2024-08-01 15:56:42 -04:00
sebastian schoner
a254e62b57 Check AllowSpew before putting process outputs into stderr
When launching a process via UAT, you can control whether it logs its output, or whether it is just put into the output. This flag used to be respected for both stdout and stderr output, but it got lost during a merge and subsequent integration for stderr. This CL reinstates it in this codepath: Not doing this is super annoying because the stderr and stdout can otherwise overlap in really annoying ways (stderr can just come in from a background thread and interfere with whatever you write into stdout).

#nojira

#rnx

[CL 35223950 by sebastian schoner in ue5-main branch]
2024-07-31 15:48:57 -04:00
dan thompson
b7af5fc2f6 Make sure chunkdbs also don't save the chunks they serialize as compressed.
#rb jon.olick
#rnx

[CL 35074129 by dan thompson in ue5-main branch]
2024-07-24 21:28:36 -04:00
jon olick
3d1d77b9b7 Use new MinCompression stuff in BPT on iOS platform.
[CL 35070793 by jon olick in ue5-main branch]
2024-07-24 18:41:02 -04:00
joe kirchoff
106ca956d8 AutomationTool: Add KnownLogEvent code for code signing
#rnx
#jira UE-218503

[CL 35067791 by joe kirchoff in ue5-main branch]
2024-07-24 17:22:19 -04:00
joe kirchoff
e0710ab967 AutomationTool: Add KnownLogEvent code for code signing
#rnx
#jira UE-218503

[CL 35067772 by joe kirchoff in ue5-main branch]
2024-07-24 17:21:22 -04:00
zousar shaker
995dc340cb Allow staging to pull chunk manifest info from zenserver directly instead of from files on the filesystem so that zen snapshots (which contain the chunk manifests produced during cook, but not as files on the filesystem) can stage to pak correctly.
#rb per.larsson

[CL 35056945 by zousar shaker in ue5-main branch]
2024-07-24 12:23:49 -04:00
david harvey
716217de44 Add PostStagingFileCopy hook for CustomDeploymentHandler
#jira UE-189295, UE-208522
#rb dave.barrett
#rnx

[CL 34895490 by david harvey in ue5-main branch]
2024-07-18 03:48:26 -04:00
brendan lienau
50e57f8d61 [Backout] - CL34821109
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL34741902
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntet] Modify Gauntlet Device reservations to allow for an explicit device within an UnrealTargetConstraint
#jira UE-213331
#rb Josh.Engebretson

[CL 34868074 by brendan lienau in ue5-main branch]
2024-07-17 11:30:20 -04:00
bob tellez
6a466acd42 [Backout] - CL34741902
[FYI] brendan.lienau
Original CL Desc
-----------------------------------------------------------------
[Gauntet] Modify Gauntlet Device reservations to allow for an explicit device within an UnrealTargetConstraint
#jira UE-213331
#rb Josh.Engebretson

[CL 34821133 by bob tellez in ue5-main branch]
2024-07-15 14:46:52 -04:00
brendan lienau
a94d2f4070 [Gauntet] Modify Gauntlet Device reservations to allow for an explicit device within an UnrealTargetConstraint
#jira UE-213331
#rb Josh.Engebretson

[CL 34741915 by brendan lienau in ue5-main branch]
2024-06-28 12:23:21 -04:00
paul chipchase
0376e1d8b2 Added command line switch '-ApplyIoStoreOnDemand' to UAT BuildCookRun that will force the IoStoreOnDemand module into the target project and force on additional settings to allow IoStoreOnDemand to work.
#rb Per.Larsson
#jira UE-218584
#rnx

- This is only intended for development and testing IoStoreOnDemand as it allows us to apply it to a project without changing the project files on disk. This should not be used to enable the feature on a project in production.

### Enabling the code module
- When forcing on IoStoreOnDemand, BuildCookRun will pass '-CompileIoStoreOnDemand' to UnrealBuildTool which will set 'bCompileIoStoreOnDemand' to true in the target rules. This in turn will add the 'IoStoreOnDemand' module to the Engine module but only for runtime targets.
- None of this should be used when enabling the feature for a project permanentl, instead just add 'IoStoreOnDemand' module to your project's *.build.cs

### Staging SSL certificates
- IoStoreOnDemand needs access to the engine/project#s SSL certificates at runtime  so we need to stage them, which is done by forcing 'bStageSSLCertificates' to be true during BuildCookRun.
- To do this properly for a project set ini:Engine:[/Script/Engine.NetworkSettings]:n.VerifyPeer=true

### Generate Chunks
- In order for IoStoreOnDemand to work we need to generate chunks (more than one container file) which can be done by forcing ProjectParams.Manifests to true.
- To do this properly for a project set ini:Game:[/Script/UnrealEd.ProjectPackagingSettings]:bGenerateChunks=true

### Pak Rules
- Once chunked builds are enabled IoStoreOnDemand will require at least one PakRule that sets  'bOnDemand' to true. So first we check to see if any rules like this for the existing project and if not we force add one via CreateIoStoreOnDemandPakRules. This rule will include all *.ubulk and *.uptnl files except ones found under a \map\ directory. This is because some level bulkdata files can get too large for IoStoreOnDemand to support so excluding anything under a map directory is a quick and easy way to try and work around that. The end result is that other than map data, pretty much all bulkdata will end up on demand, which is a good stress test for the system but not a great representation on how the system should be used.
- If you are trying to set up rules properly you'd need to consider what data you actually want to stream over the network and what data you want to remain on disk, which would greatly depend on your project set up. One popular choice is to only stream optional (.uptnl) bulkdata and leave the required bulkdata to be found on disk.

### Enabling Persistent Local Storage
- IoStoreOnDemand requires persistent local storage to be set aside for it's caching which may or may not require special set up for a project depending on the target platform. If we detect that the current project has not been set up for persistent local storage we will attempt to enable it for the project during staging.
- To set this up properly you will need to consult the documentation for your target platform and follow the instructions.

[CL 34702097 by paul chipchase in ue5-main branch]
2024-06-27 02:50:43 -04:00
david harvey
46f5c56af0 Fix for custom deployment handlers interfering when multiple platforms are specified on the BuildCookRun command line.
#jira UE-218487
#rb tomasz.obrebski
#rnx

[CL 34674250 by david harvey in ue5-main branch]
2024-06-26 09:50:39 -04:00
zousar shaker
839606e1e9 2nd submit: Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to  be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid  chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.

#rb PJ.Kack

[CL 34565129 by zousar shaker in ue5-main branch]
2024-06-21 11:48:33 -04:00
ben marsh
5a81e8dca5 AutomationTool: Flag "RunUBT ERROR:" messages as an ignorable exit code error.
[CL 34563686 by ben marsh in ue5-main branch]
2024-06-21 11:15:54 -04:00