Commit Graph

737 Commits

Author SHA1 Message Date
jack porter
63ef117561 Perform restricted staged folder name checking before doing platform filename remapping
Back out incorrect fix from CL 20056908
#jira UE-147094
#preflight 62790b2fea2a9db9dc560710
#rb Adam.Kinge
#lockdown Mitchell.Wilson

#ROBOMERGE-AUTHOR: jack.porter
#ROBOMERGE-SOURCE: CL 20110639 in //UE5/Release-5.0/... via CL 20111734
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)

[CL 20114350 by jack porter in ue5-main branch]
2022-05-09 19:52:57 -04:00
Dan Thompson
3a6321167b Packaging Reference Chunk Database - "Lossless Patch Preventer"
Can now provide reference iostore containers to reuse compressed chunks from. If a match is found on *the decompressed data*, instead of recompressing the blocks, they are read off of disk. This allows tweaks of the compressor algorithm without introducing changes as the runtime still sees the exact same data. Additionally this allows for fairly dramatic staging speedups as nvme speeds are significantly faster than high effort compressions. This is distinct from the DDC compression because:
1) DDC compression ties in the compressor version/method
2) We are explicitly interested in chunks that are deployed to end users, not merely cached for speed.

To facilitate this, several changes were made to IoStore:

FIoStoreReader now directly reads from IFileHandles* instead of routing through the GenericPlatformFile async read system, as that system is sensitive to build #defines and can result in constant file opens under load (indeed, for anything not a .pak file, every read is an open/close).

Cold file cache read speed improvements from ~140MB/s to ~1 GB/s. Hot is more.

Additionally:
    FIoStoreReader switched to UE::Tasks from taskgraph for tasks in order to facilitate task retraction during waits as the previous ReadAsync call was trivial to deadlock when called from worker threads due to its use of TFuture<>.
    FIoStoreReader::ReadCompressed now returns the compressed blocks as they were on disk - padded to AES encryption block size.

#rb fabian.giesen
#rb jeff.roberts
#preflight 627586dcf77c9c2b543d4d8b

[CL 20086673 by Dan Thompson in ue5-main branch]
2022-05-06 18:22:44 -04:00
Josh Adams
f18143aa8b - Added missing commandline option for the optional file staging directory (-optionalfilestagingdirectory=)
#rb trivial
#preflight 627423bafd59d6606e0162f0

[CL 20063141 by Josh Adams in ue5-main branch]
2022-05-05 15:36:34 -04:00
jack porter
a5d55f3660 Use platform-remapped staging path when checking for restricted folder names
#jira UE-147094
#rb Ben Zeigler
#preflight 6273b716c04406dfea7e49b2
#lockdown Rolando.Caloca

#ROBOMERGE-AUTHOR: jack.porter
#ROBOMERGE-SOURCE: CL 20056908 in //UE5/Release-5.0/... via CL 20057485
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)

[CL 20061168 by jack porter in ue5-main branch]
2022-05-05 14:09:36 -04:00
Josh Adams
defdfa230a - Added option for IoStore to write optional files (.o.ucas) to a different directory. This is used when making a a cooked editor as DLC against a client build (the client build has to make the optional editor data, but we do not want to ship the optional data with the client). The MakeCookedEditor script will then copy the files from that location into the editor's Stage directory (under Content/Paks)
#rb carl-magnus.nordin,francis.hurteau
#preflight 6273f8fd8d32cd80d8a8ecdb

[CL 20059504 by Josh Adams in ue5-main branch]
2022-05-05 12:28:59 -04:00
PJ Kack
6b6bc48fc5 UAT: Add additional startup and BuildCookRun logging and timings
#jira none
#rb trivial
#rnx
#preflight 627379dda56dd762120e6e2e

[CL 20055418 by PJ Kack in ue5-main branch]
2022-05-05 03:40:01 -04:00
PJ Kack
65391545db UAT: Restrict the COTF -filehostip list to ip v4 host addresses and only include 127.0.0.1 on host platforms or platforms that support port redirection.
Makes connecting to cotf faster on some platforms.
Removes warnings like: LogSockets: Warning: Could not serialize fe80::edc3:3284:5b1b:1f32%7, got error code SE_NO_ERROR [0]
Makes the string shorter and more readable.

Test:
RunUAT.bat BuildCookRun -project=Samples\Games\ShooterGame\ShooterGame.uproject -platform=XYZ -config=development -build -cookonthefly -stage -run

FileHostIp example on PC & android:
Before: -filehostip=127.0.0.1+fe80::edc3:3284:5b1b:1f32%7+10.24.101.44+::1+127.0.0.1+fe80::58d4:b8b3:ff7a:4974%9+172.18.240.1
After : -filehostip=127.0.0.1+10.24.101.44+172.18.240.1

FileHostIp example on console:
Before: -filehostip=127.0.0.1+fe80::edc3:3284:5b1b:1f32%7+10.24.101.44+::1+127.0.0.1+fe80::58d4:b8b3:ff7a:4974%9+172.18.240.1
After : -filehostip=10.24.101.44+172.18.240.1

#jira none
#rb carlmagnus.nordin, per.larsson
#rnx
#preflight 62722ddafe45c124493258a8

[CL 20038819 by PJ Kack in ue5-main branch]
2022-05-04 04:18:17 -04:00
Francis Hurteau
c719a4680a Add commandline boilerplate to support optional cooking through UAT
#rb Matt.Peters
#jira UE-142231
#preflight none

[CL 19929815 by Francis Hurteau in ue5-main branch]
2022-04-26 17:52:04 -04:00
jonathan adamczewski
d4f03243e3 Remove various obsolete (deprecated) items in C# code
#jira none
#trivial
#rnx
#preflight 625a32bbde3638cefa4106cb

[CL 19786027 by jonathan adamczewski in ue5-main branch]
2022-04-18 11:27:53 -04:00
jonathan adamczewski
b6ac2aa225 BuildGraph: RunUAT via dotnet & AutomationTool.dll
#jira none
#rnx
#preflight 6259f6a7016074c111fa3d39

[CL 19778495 by jonathan adamczewski in ue5-main branch]
2022-04-15 19:03:08 -04:00
jonathan adamczewski
c12d99d854 .net6.0 is not supported in VS2019. Suppress warnings about that when using that IDE (with this change, when building UnrealBuildTool & AutomationTool)
#jira none
#rnx
#rb joe.kirchoff
#preflight 624f2c163c2aff69c54ff9eb

[CL 19675097 by jonathan adamczewski in ue5-main branch]
2022-04-07 15:06:02 -04:00
mitchell fisher
f039ee7b30 [FORT-465286] McpConfigHelper::Find() surfaces nested exception details if instance activation throws
#rb trivial
#preflight skip

#ROBOMERGE-OWNER: mitchell.fisher
#ROBOMERGE-AUTHOR: mitchell.fisher
#ROBOMERGE-SOURCE: CL 19653682 via CL 19654984 via CL 19655731 via CL 19655738
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v938-19570697)

[CL 19657590 by mitchell fisher in ue5-main branch]
2022-04-06 18:24:32 -04:00
jonathan adamczewski
c1bb6901bf Upgrade to .NET 6.0
#jira UE-119846
#preflight 624cbb2ecc0872e96b0d5181

[CL 19652696 by jonathan adamczewski in ue5-main branch]
2022-04-06 14:46:50 -04:00
PJ Kack
5f6c6d5beb UAT: Add BuildCookRun support for running both a dedicated server and cotf server.
Merge RunClientWithServer() and RunStandAloneClient() functions into one RunClient() function to resolve numerous inconsistencies between them and between unattended/non-unattended mode.
Always use SC.StageTargetPlatform.RunClient() to launch clients so that the logic works on all platforms.
When either the client, dedicated server or cook server exits, then all other processes exits too.
Hitting Ctrl-c in UAT kills all processes.

ShooterGame testing on Win64, consoles and android:
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cook -stage -pak -deploy -run
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -skipcook -skipstage -skipdeploy -run
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -skipcook -skipstage -skipdeploy -run -unattended
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cookonthefly -stage -deploy -run
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cookonthefly -stage -deploy -run -unattended
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cookonthefly -zenstore -stage -deploy -run
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cookonthefly -zenstore -stage -deploy -run -unattended

ShooterGame Win64 Desktop specific testing with lan server and multiple clients:
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cookonthefly -zenstore -server -stage -run -map=Highrise -numclients=3
BuildCookRun -project=ShooterGame -platform=Win64 -config=development -skipbuild -cookonthefly -zenstore -server -stage -run -map=Highrise -numclients=3 -unattended

EngineTest testing on Win64 and console:
BuildCookRun -project=EngineTest -platform=Win64 -config=development -skipbuild -cook -stage -pak -deploy -run -runautomationtest=Project.System
BuildCookRun -project=EngineTest -platform=Win64 -config=development -skipbuild -cookonthefly -stage -deploy -run -runautomationtest=Project.System
BuildCookRun -project=EngineTest -platform=Win64 -config=development -skipbuild -cookonthefly -zenstore -stage -deploy -run -runautomationtest=Project.System

#jira none
#rb carlmagnus.nordin
#rnx
#preflight 624d79fdbf5b974989964f4c

[CL 19645492 by PJ Kack in ue5-main branch]
2022-04-06 08:03:43 -04:00
Jerome Delattre
93a0f48bd8 Using System.Text.Json for Device reservation instead of fastJSON and making the deserialization not case sensitive
#rb Josh.Engebretson, Chris.Constantinescu
#preflight 62434b28292f228e09e53f10
#rnx
#jira none

[CL 19575272 by Jerome Delattre in ue5-main branch]
2022-03-31 13:43:17 -04:00
PJ Kack
640a176fd5 UAT: Ctrl-c exit fixes.
In ProcessManager.KillAll() make a local copy of global ActiveProcesses and to avoid calling any external code while iterating the internal global list.
In ProcessResult.StopProcess() make sure to set the ExitCode member from the killed process, else the information is lost forever since the member Proc has already been nulled out.
In CommandUtils.Run() check Result.ProcessObject for null for the case when WaitForExit() returns after StopProcess() has been called from another thread (ctrl-c handler).

#jira none
#rb carlmagnus.nordin
#rnx
#preflight 62441c4dc667881bf4e70722

[CL 19570185 by PJ Kack in ue5-main branch]
2022-03-31 03:25:13 -04:00
jonathan adamczewski
5053cae2d8 Launch UnrealBuildTool via dotnet unrealbuildtool.dll, not unrealbuildtool.exe
#jira none
#rnx
#preflight 624261bdb6084b98321e9251

[CL 19544939 by jonathan adamczewski in ue5-main branch]
2022-03-29 13:20:46 -04:00
jonathan adamczewski
0a6164bbd3 nullability annotations, rethrow fixes, net6.0 warning suppression
#jira none
#trivial
#rnx
#preflight 62425155c61d8a458f230657

[CL 19536049 by jonathan adamczewski in ue5-main branch]
2022-03-28 20:50:06 -04:00
jonathan adamczewski
01786aed54 GoogleDriveHelper: correct rethrow
#jira none
#trivial
#rnx
#preflight none

[CL 19535531 by jonathan adamczewski in ue5-main branch]
2022-03-28 19:43:45 -04:00
jonathan adamczewski
5c68d1d9ef Revert //UE5/Main/... changelist 19531725 - return to NET Core 3.1
#preflight 62422fd0470aff98e946bfce
#jira UE-119846

[CL 19534572 by jonathan adamczewski in ue5-main branch]
2022-03-28 18:25:19 -04:00
justin marcus
bab21f02ae Add MergeDirectory_NoExceptions which does not delete the contents of the targer directory.
#preflight https://horde.devtools.epicgames.com/job/623e0cb18330d27ee3740f4f

#ROBOMERGE-OWNER: justin.marcus
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 19525960 via CL 19527026 via CL 19527842 via CL 19527853
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v937-19513599)

[CL 19532718 by justin marcus in ue5-main branch]
2022-03-28 16:27:08 -04:00
jonathan adamczewski
18ac15cbb1 Upgrade to .NET 6.0
#jira UE-119846
#preflight 6241fc06470aff98e943c035

[CL 19531725 by jonathan adamczewski in ue5-main branch]
2022-03-28 15:30:56 -04:00
Josh Adams
3be42005d9 - Removed code that was accidentally checked in, that broke Android .apk names
#fyi chris.babcock
#rb trivial
#jira UE-146630
#preflight skip

[CL 19456849 by Josh Adams in ue5-main branch]
2022-03-21 15:42:23 -04:00
josh adams
73790836c2 - Added PlatformValues to DeviceInfo in UAT land. Removed the Subtype field and changed it's very limited use to PlatformValues
- Added support for copying remote debugging files to SteamDeck when deploying
- Username is now pulled out of the Device info, or put on the commandline via -deviceuser to override
#rb brandon.schaefer
#jira none
#p4v-preflight-copy 19392535
#preflight 6233ac4af101b8d0afa9abf4

[CL 19428566 by josh adams in ue5-main branch]
2022-03-17 18:23:21 -04:00
Tim Smith
603b903987 Prepare UBT and automation code type scanning for the moving of the MSBuild code. This requires a change in how we handle the load failure of Microsoft.build.framework.
#rb leigh.swift
#rnx
#preflight 62337c800820efd0945e40d0

[CL 19424430 by Tim Smith in ue5-main branch]
2022-03-17 14:44:47 -04:00