Commit Graph

109 Commits

Author SHA1 Message Date
Ben Marsh
ea34b09df3 BuildGraph: Add tasks for running Git and Docker, and add support for deleting directories from <Delete> tasks.
[CL 16144334 by Ben Marsh in ue5-main branch]
2021-04-28 11:27:19 -04:00
jonathan adamczewski
17602ab008 Support two msbuilds, for NET Framework and Core
This is a preparatory change ahead of adding the ability to build NET Core projects to the CsCompile task, and subsequent conversion of NET Framework projects to NET Core. On its own, it should have no substantive visible effect.

#jira none

[CL 16008195 by jonathan adamczewski in ue5-main branch]
2021-04-14 13:24:20 -04:00
jonathan adamczewski
70ed64e369 Fix "Build Tools CS Mac" on Mac
RunUAT.sh:
Setup Mono environment on Mac and Linux so that AutomationTool is able to launch xbuild for CsCompile tasks that have not been transitioned to NET Core.
(Linux change is untested)

CsCompileTask.cs:
Don't add the "/restore" argument to command line when building using xbuild as it does not appear to be supported.

InstalledEngineBuild.xml:
Remove CsCompile invocations for UAT projects as these are currently not able to be compiled by UAT: CsCompile does not know how use the bundled dotnet to build NET Core projects, and [our bundled] mono does not support NET Core projects.

This file is consumed by AutomationTool, which is compiled by RunUAT.sh. As such, it can generally be assumed that the UAT build is up to date when the built files are consumed (InstalledEngineBuild.xml has directives to tag the AutomationTool files to ensure they are included.)

#jira UE-112471

[CL 15902338 by jonathan adamczewski in ue5-main branch]
2021-04-02 12:03:38 -04:00
Sebastien Lussier
102ce82546 Added support for distributed World Partition HLOD generation
* Runs multiple steps on different machines
* Setup: Done on a single machine, will create/update/delete HLOD actors as required. The distributed setup step takes a "BuilderCount" argument, which will be used to write a build manifest that specify the work that needs to be done by the N builders
* Build: Performed on multiple machines, read work to be done from the build manifest
* Submit: Final step, perform on a single machine. Gather files modified in all the previous steps and submit them in a single changelist

Intermediary files that are created/modified/deleted are transfered between the jobs as build products, and submitted in the last step.

To run with UAT:
RunUAT.bat BuildGraph -Script=Collaboration\Frosty\Frosty\Build\FrostyHLOD.xml -Target="HLOD Generation" -set:MapName=YourMapName -set:HLODBuilderCount=N

To run steps individually:
* Setup (N = Builder count to use)
    YourProject YourMapName -run=WorldPartitionBuilderCommandlet -SCCProvider=Perforce -AllowCommandletRendering -Builder=WorldPartitionHLODsBuilder -SetupHLODs -DistributedBuild -BuilderCount=N
* Build (i from 0 to N-1):
    YourProject YourMapName -run=WorldPartitionBuilderCommandlet -SCCProvider=Perforce -AllowCommandletRendering -Builder=WorldPartitionHLODsBuilder -DistributedBuild -BuildHLODs -BuilderIdx=i
* Submit:
    YourProject YourMapName -run=WorldPartitionBuilderCommandlet -SCCProvider=Perforce -AllowCommandletRendering -Builder=WorldPartitionHLODsBuilder -DistributedBuild -SubmitHLODs


#fyi jeanfrancois.dube, richard.malo, vincent.beauchemin

[CL 15693777 by Sebastien Lussier in ue5-main branch]
2021-03-14 21:58:48 -04:00
ben marsh
ff498fd482 Fix incorrect hostname for PCB submit workspace.
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 15685133 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v781-15675533)

[CL 15685153 by ben marsh in ue5-main branch]
2021-03-11 23:17:52 -04:00
ben marsh
d53e84576b Include a hash of the server address when creating a workspace for submitting binaries. Should fix issues with different edge servers being selected.
#rb none
#jira UE-109866

#ROBOMERGE-SOURCE: CL 15681548 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v781-15675533)

[CL 15684115 by ben marsh in ue5-main branch]
2021-03-11 21:31:36 -04:00
Ryan Durand
168200dff5 Add support for shunting Unzip tasks to the legacy unzip logic.
#rnx
#rb none
#jira none

[CL 15649177 by Ryan Durand in ue5-main branch]
2021-03-08 20:02:46 -04:00
ben marsh
a4d4c5c295 Fix for missing file permissions on Mac.
Switches the default zip utility to be the built in NET ZipArchive class. Adds additional support for storing permission bits by P/Invoking libSystem.dylib and adding the flags from stat64() to the ZipArchiveEntry. Also restores the permissions on unzip. One less dependency on Ionic.Zip. :)

#jira UE-109089
[FYI] Jonathan.Adamczewski
#rb none

#ROBOMERGE-SOURCE: CL 15620859 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v777-15581079)

[CL 15620862 by ben marsh in ue5-main branch]
2021-03-04 22:18:07 -04:00
Marc Audy
9753392e2b Merge UE5/RES CL# 15462083 to UE5/Main
This represents UE4/Main @ 15414221

[CL 15463811 by Marc Audy in ue5-main branch]
2021-02-18 18:13:28 -04:00
benoit deschenes
08076c32d7 Integrating CL# 15321861 from //UE4/Main
#jira UE-108402
#rb none

[CL 15388736 by benoit deschenes in ue5-main branch]
2021-02-11 12:34:27 -04:00
Marc Audy
cac1fe0019 Merge UE5/Release-Engine-Staging @ CL# 15299266 to UE5/Main
This represents UE4/Main @ CL# 15277572

[CL 15299962 by Marc Audy in ue5-main branch]
2021-02-03 14:57:28 -04:00
Ryan Durand
764dded8db Fix compile errors in WaitTask.cs.
#rnx
#rb none
#jira none

[CL 15187253 by Ryan Durand in ue5-main branch]
2021-01-25 17:13:41 -04:00
Ryan Durand
852d3639dc Adding WaitTask.
#rnx
#rb none
#jira none

[CL 15183849 by Ryan Durand in ue5-main branch]
2021-01-25 15:53:37 -04:00
Marc Audy
bf80889353 UE5/Release-Engine-Staging to UE5/Main
This represents UE4/Main up to CL# 14958402

[CL 15028197 by Marc Audy in ue5-main branch]
2021-01-08 19:56:07 -04:00
Ben Marsh
068f97616c Restore packages before building projects. Should fix IncludeTool errors in CIS.
#rb none
#rnx

[CL 14964192 by Ben Marsh in ue5-main branch]
2020-12-23 11:23:34 -04:00
Ben Marsh
6f927647b1 Changing C# code over to using EpicGames.Core over DotNETCommon.
#rb none
#rnx

[CL 14962096 by Ben Marsh in ue5-main branch]
2020-12-21 23:07:37 -04:00
Ben Marsh
64651ad903 Remove references to FastJSON functionality in DotNETCommon. Use a duplicate copy in AutomationScripts called Json_DEPRECATED instead.
#rb none
#rnx

[CL 14962028 by Ben Marsh in ue5-main branch]
2020-12-21 22:05:50 -04:00
Marc Audy
ada7c144fa Merge //UE5/Release-Engine-Staging @14903491 to //UE5/Main
[CL 14906022 by Marc Audy in ue5-main branch]
2020-12-11 14:21:20 -04:00
Joakim Lindqvist
e7039d3d35 UBT and UAT now use .NET Core instead of Framework and Mono. This means that we use the same runtime on Windows, Linux and Mac. Further benefits including newer C# features and a lot of intresting features for the future around AOT and Tiered compilation.
Some behavior changes:
Output paths - Both tools are now output to a subdirectory of Binaries/Dotnet, I believe most hardcoded paths have been fixed up but there may be tools that will fail because of this.
UAT Plugin Building - As .NET Core does not support AppDomain unloading, how we build the plugins has changed quite a bit, these are now built before UAT is started rather then by UAT itself. If you just start UAT via RunUAT.bat/sh this should just continue to work.

#rb ben.marsh

[CL 14834347 by Joakim Lindqvist in ue5-main branch]
2020-12-02 06:57:13 -04:00
Joakim Lindqvist
f90e40ffb0 UAT can now build as a netcore application.
Added a NET_CORE define to allow us to have changes side by side.
The AWS S3 changes are required due to us requiring to upgrade the S3 assembly version to get net core support (which made all methods async).
The ACL checks for files are not available in the system libraries of net core, as such the api is a bit different.

AutomationToolLauncher now just spawns a subprocess when used in netcore, as netcore does not support custom AppDomains and shadow copying. We will generally need to revisit this for netcore as this whole feature of building the source for UAT in UAT is not really possible.

To enable this set environment variable "UE_USE_DOTNET=1", note that with netcore all applications change their output path so this will likely break a bit of tooling when enabled.

#rb ben.marsh

[CL 14572339 by Joakim Lindqvist in ue5-main branch]
2020-10-26 06:08:59 -04:00
Marc Audy
4c1bb11c29 Merge UE5/Release-Engine-Staging to UE5/Main @ 14548662
This represents UE4/Main @ 14525125 + cherrypicked fixes
#skipundocheck

[CL 14551026 by Marc Audy in ue5-main branch]
2020-10-22 19:19:16 -04:00
Ryan Durand
894ba25ab1 Fix P4Verbose being required on SubmitTask.
#rnx
#rb none
#jira none

[CL 14502236 by Ryan Durand in ue5-main branch]
2020-10-15 12:12:20 -04:00
Ryan Durand
e72ad24fed Adding the ability to allow verbose P4 output for SubmitTask.
#rnx
#rb none
#jira none

[CL 14502067 by Ryan Durand in ue5-main branch]
2020-10-15 11:51:57 -04:00
Ben Marsh
486408b1a9 Rename UE5Editor to UnrealEditor.
#rb none
#jira UE-97600

[CL 14292067 by Ben Marsh in ue5-main branch]
2020-09-10 15:39:00 -04:00
Ben Marsh
be7034ec27 Rename UE4Editor to UE5Editor, and generated solution/projects from UE4 -> UE5.
UE4Game/Client/Server targets left intact for the time being.

#rb none

[CL 14288076 by Ben Marsh in ue5-main branch]
2020-09-10 09:10:30 -04:00