Commit Graph

126 Commits

Author SHA1 Message Date
ben marsh
06cca0e4c1 BuildGraph: Add support for passing text files containing environment variables into AWS, Docker, Helm and Spawn tasks. Also add an Aws-AssumeRole task which can assume a role and write a new list of environment variables.
#ROBOMERGE-SOURCE: CL 16341432 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16341434 by ben marsh in ue5-release-engine-test branch]
2021-05-15 12:49:14 -04:00
ben marsh
26a3e12c4c BuildGraph: Remove --dry-run argument from Helm task.
#ROBOMERGE-SOURCE: CL 16339760 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16339779 by ben marsh in ue5-release-engine-test branch]
2021-05-14 22:37:41 -04:00
ben marsh
68ccd6b7d7 BuildGraph: Use a standard AWS config file to configure Aws/Docker/Helm tasks.
#ROBOMERGE-SOURCE: CL 16338005 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16338006 by ben marsh in ue5-release-engine-test branch]
2021-05-14 20:03:09 -04:00
ben marsh
dfe8908b37 Remove hard-coded tool location.
#ROBOMERGE-SOURCE: CL 16337227 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16337230 by ben marsh in ue5-release-engine-test branch]
2021-05-14 18:30:55 -04:00
ben marsh
33bd87712f BuildGraph: Add specific tasks for building and pushing images with Docker.
#ROBOMERGE-SOURCE: CL 16337213 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16337221 by ben marsh in ue5-release-engine-test branch]
2021-05-14 18:29:58 -04:00
ben marsh
666c442adc BuildGraph: Add a dedicated Git-Clone task for BuildGraph, to save some boilerplate code in scripts.
#ROBOMERGE-SOURCE: CL 16334501 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16334601 by ben marsh in ue5-release-engine-test branch]
2021-05-14 16:06:10 -04:00
ben marsh
6a35469bea Add an explicit accessor for getting the path to the DotNet host.
#ROBOMERGE-SOURCE: CL 16312643 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16312660 by ben marsh in ue5-release-engine-test branch]
2021-05-13 10:25:11 -04:00
ben marsh
b44514c377 Fix path to DotNET executable.
#ROBOMERGE-SOURCE: CL 16312463 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16312484 by ben marsh in ue5-release-engine-test branch]
2021-05-13 10:06:45 -04:00
Ben Marsh
f49990aaa0 BuildGraph: Add a task for executing DotNet commands.
[CL 16271288 by Ben Marsh in ue5-main branch]
2021-05-11 12:04:26 -04:00
Ben Marsh
25b44e2c2e Add a Kubectl task.
[CL 16249810 by Ben Marsh in ue5-main branch]
2021-05-10 12:03:08 -04:00
Ben Marsh
74a415cac9 Add a Helm task to allow updating releases.
[CL 16249773 by Ben Marsh in ue5-main branch]
2021-05-10 12:01:51 -04:00
Ben Marsh
f6f8beda42 Enable verbose output from Docker tasks by default.
[CL 16249754 by Ben Marsh in ue5-main branch]
2021-05-10 12:01:04 -04:00
Ben Marsh
803fa2e63d Prevent Docker password being logged to stdout.
[CL 16247915 by Ben Marsh in ue5-main branch]
2021-05-10 09:49:47 -04:00
Ryan Durand
ceacd78804 Build GitSync on Horde.
#rnx
#rb none
#jira none

[CL 16247562 by Ryan Durand in ue5-main branch]
2021-05-10 09:01:20 -04:00
Ryan Durand
604cecf460 Add a verbose option to DeleteTask to see what files and folders are being deleted.
#rnx
#rb none
#jira none

[CL 16183644 by Ryan Durand in ue5-main branch]
2021-05-03 09:45:50 -04:00
Ryan Durand
d95d6ddf00 Only try to delete folders if they exist.
#rnx
#rb none
#jira none

[CL 16161162 by Ryan Durand in ue5-main branch]
2021-04-29 15:59:44 -04:00
Ben Woodhouse
b0ec2a07ac CopyTask - optionally throw an error if files not found
#rb luke.thatcher
#jira UE-114025

[CL 16156542 by Ben Woodhouse in ue5-main branch]
2021-04-29 09:02:09 -04:00
Ben Marsh
ea34b09df3 BuildGraph: Add tasks for running Git and Docker, and add support for deleting directories from <Delete> tasks.
[CL 16144334 by Ben Marsh in ue5-main branch]
2021-04-28 11:27:19 -04:00
jonathan adamczewski
17602ab008 Support two msbuilds, for NET Framework and Core
This is a preparatory change ahead of adding the ability to build NET Core projects to the CsCompile task, and subsequent conversion of NET Framework projects to NET Core. On its own, it should have no substantive visible effect.

#jira none

[CL 16008195 by jonathan adamczewski in ue5-main branch]
2021-04-14 13:24:20 -04:00
jonathan adamczewski
70ed64e369 Fix "Build Tools CS Mac" on Mac
RunUAT.sh:
Setup Mono environment on Mac and Linux so that AutomationTool is able to launch xbuild for CsCompile tasks that have not been transitioned to NET Core.
(Linux change is untested)

CsCompileTask.cs:
Don't add the "/restore" argument to command line when building using xbuild as it does not appear to be supported.

InstalledEngineBuild.xml:
Remove CsCompile invocations for UAT projects as these are currently not able to be compiled by UAT: CsCompile does not know how use the bundled dotnet to build NET Core projects, and [our bundled] mono does not support NET Core projects.

This file is consumed by AutomationTool, which is compiled by RunUAT.sh. As such, it can generally be assumed that the UAT build is up to date when the built files are consumed (InstalledEngineBuild.xml has directives to tag the AutomationTool files to ensure they are included.)

#jira UE-112471

[CL 15902338 by jonathan adamczewski in ue5-main branch]
2021-04-02 12:03:38 -04:00
Sebastien Lussier
102ce82546 Added support for distributed World Partition HLOD generation
* Runs multiple steps on different machines
* Setup: Done on a single machine, will create/update/delete HLOD actors as required. The distributed setup step takes a "BuilderCount" argument, which will be used to write a build manifest that specify the work that needs to be done by the N builders
* Build: Performed on multiple machines, read work to be done from the build manifest
* Submit: Final step, perform on a single machine. Gather files modified in all the previous steps and submit them in a single changelist

Intermediary files that are created/modified/deleted are transfered between the jobs as build products, and submitted in the last step.

To run with UAT:
RunUAT.bat BuildGraph -Script=Collaboration\Frosty\Frosty\Build\FrostyHLOD.xml -Target="HLOD Generation" -set:MapName=YourMapName -set:HLODBuilderCount=N

To run steps individually:
* Setup (N = Builder count to use)
    YourProject YourMapName -run=WorldPartitionBuilderCommandlet -SCCProvider=Perforce -AllowCommandletRendering -Builder=WorldPartitionHLODsBuilder -SetupHLODs -DistributedBuild -BuilderCount=N
* Build (i from 0 to N-1):
    YourProject YourMapName -run=WorldPartitionBuilderCommandlet -SCCProvider=Perforce -AllowCommandletRendering -Builder=WorldPartitionHLODsBuilder -DistributedBuild -BuildHLODs -BuilderIdx=i
* Submit:
    YourProject YourMapName -run=WorldPartitionBuilderCommandlet -SCCProvider=Perforce -AllowCommandletRendering -Builder=WorldPartitionHLODsBuilder -DistributedBuild -SubmitHLODs


#fyi jeanfrancois.dube, richard.malo, vincent.beauchemin

[CL 15693777 by Sebastien Lussier in ue5-main branch]
2021-03-14 21:58:48 -04:00
ben marsh
ff498fd482 Fix incorrect hostname for PCB submit workspace.
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 15685133 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v781-15675533)

[CL 15685153 by ben marsh in ue5-main branch]
2021-03-11 23:17:52 -04:00
ben marsh
d53e84576b Include a hash of the server address when creating a workspace for submitting binaries. Should fix issues with different edge servers being selected.
#rb none
#jira UE-109866

#ROBOMERGE-SOURCE: CL 15681548 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v781-15675533)

[CL 15684115 by ben marsh in ue5-main branch]
2021-03-11 21:31:36 -04:00
Ryan Durand
168200dff5 Add support for shunting Unzip tasks to the legacy unzip logic.
#rnx
#rb none
#jira none

[CL 15649177 by Ryan Durand in ue5-main branch]
2021-03-08 20:02:46 -04:00
ben marsh
a4d4c5c295 Fix for missing file permissions on Mac.
Switches the default zip utility to be the built in NET ZipArchive class. Adds additional support for storing permission bits by P/Invoking libSystem.dylib and adding the flags from stat64() to the ZipArchiveEntry. Also restores the permissions on unzip. One less dependency on Ionic.Zip. :)

#jira UE-109089
[FYI] Jonathan.Adamczewski
#rb none

#ROBOMERGE-SOURCE: CL 15620859 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v777-15581079)

[CL 15620862 by ben marsh in ue5-main branch]
2021-03-04 22:18:07 -04:00