Commit Graph

159 Commits

Author SHA1 Message Date
ben marsh
0c5ab1392c Horde: Add REST API for compute service.
* New tasks can be queued with a POST to /compute/{channelid}
* Status updates can be dequeued with a POST to /compute/{channelid}/updates
* ChannelId can be any series of lowercase letters, numbers, and the characters _ and -.
* Request/response objects are declared in Api.cs.

[FYI] Joe.Kirchoff

#ROBOMERGE-SOURCE: CL 17226912 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17226933 by ben marsh in ue5-release-engine-test branch]
2021-08-18 21:41:39 -04:00
josh engebretson
6ac5ca68ab Horde: Fix missing copyright
#jira none
#rnx
[FYI] ben.marsh

#ROBOMERGE-SOURCE: CL 17197748 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17197795 by josh engebretson in ue5-release-engine-test branch]
2021-08-17 12:06:46 -04:00
ben marsh
8e7b0c8791 Horde: New API for remote execution.
Now allows for a stateless API and supports work-stealing between pods (via a shared task list stored in Redis). Clients now send requests for work to be executed with messages returned through a particular channel, which can be polled for outputs.

ExecuteV2 command in HordeAgent allows exercising the new code via the command line. Now mirrors the entire input directory by default, and only takes a JSON file to describe the command to perform.

#ROBOMERGE-SOURCE: CL 17171093 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17171094 by ben marsh in ue5-release-engine-test branch]
2021-08-14 17:57:26 -04:00
ben marsh
bc1d02523d Allow serializing structs to compact binary objects using reflection, and automatically instantiate converter types with the same generic arguments as the tagged class if necessary.
#ROBOMERGE-SOURCE: CL 17171079 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17171080 by ben marsh in ue5-release-engine-test branch]
2021-08-14 17:47:57 -04:00
ben marsh
6998b7c298 Allow implicit conversion from HashSet<T> to ReadOnlyHashSet<T>.
#ROBOMERGE-SOURCE: CL 17171063 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17171070 by ben marsh in ue5-release-engine-test branch]
2021-08-14 17:46:15 -04:00
ben marsh
7205f3ce65 Horde: Add additional functionality to Redis library, including support for hashes.
#ROBOMERGE-SOURCE: CL 17171062 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17171066 by ben marsh in ue5-release-engine-test branch]
2021-08-14 17:46:07 -04:00
ben marsh
4001f41dac Horde: Add a typed abstraction for Redis pub/sub channels.
#ROBOMERGE-SOURCE: CL 17162181 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17162188 by ben marsh in ue5-release-engine-test branch]
2021-08-12 21:02:24 -04:00
ben marsh
b7c5c7bece Add support for serializing int, DateTime, Nullable, and enum types to compact binary through reflection.
Also fix deserialized DateTime values being treated as local times rather than UTC.

#ROBOMERGE-SOURCE: CL 17161724 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17161731 by ben marsh in ue5-release-engine-test branch]
2021-08-12 20:16:20 -04:00
ben marsh
62cec4c2f7 Fix ListSegment excluding the first item in the collection.
#ROBOMERGE-SOURCE: CL 17161703 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17161709 by ben marsh in ue5-release-engine-test branch]
2021-08-12 20:13:04 -04:00
ben marsh
918840019c Horde: Add a library for accessing typed values in Redis.
#ROBOMERGE-SOURCE: CL 17161674 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17161677 by ben marsh in ue5-release-engine-test branch]
2021-08-12 20:08:35 -04:00
jonathan adamczewski
fa77a7ef2c update diagnostic text
#jira none
#rb none
#trivial

#ROBOMERGE-SOURCE: CL 17159330 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17159341 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-12 17:34:20 -04:00
jonathan adamczewski
2b7b866f45 Move DotNETCommon/BuildUtilities to Shared/EpicGames.Build
Also moves DotNETCommon/Metadata.cs to Shared/Metadata.cs

#jira none
#rb tim.smith

#ROBOMERGE-SOURCE: CL 17116964 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17116977 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-10 11:09:18 -04:00
andrew firth
6cb4d4d4cd [P4VUtils] - add Fast reconcile options for code + all - performs a reconcile on files in P4 that are writeable but NOT checked out
[at]Ben.Marsh [at]Brandon.Dawnson
#rb Ben.Marsh

#ROBOMERGE-SOURCE: CL 17096670 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17096673 by andrew firth in ue5-release-engine-test branch]
2021-08-07 22:51:41 -04:00
ben marsh
d86ad5954d Horde: Remove unnecessary dependency on LibGit2Sharp from EpicGames.Perforce.Managed.
#ROBOMERGE-SOURCE: CL 17096409 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17096413 by ben marsh in ue5-release-engine-test branch]
2021-08-07 19:23:01 -04:00
ben marsh
c8da790ae1 Horde: Allow slicing lists starting at a given offset.
#ROBOMERGE-SOURCE: CL 17094351 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17094356 by ben marsh in ue5-release-engine-test branch]
2021-08-07 10:43:30 -04:00
ben marsh
87b4ea8fc6 Horde: Support serialization of bool values to compact binary objects through attribute-driven serialization.
#ROBOMERGE-SOURCE: CL 17094350 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17094352 by ben marsh in ue5-release-engine-test branch]
2021-08-07 10:43:23 -04:00
ryan hummer
58994ea55e Horede: Make fileSize Optional for the Perforce PrintRecord
Perforce is sometimes omitting that field which causes pre-flights to fail, and it's not used by any code right now.

#ROBOMERGE-SOURCE: CL 17087306 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17087344 by ryan hummer in ue5-release-engine-test branch]
2021-08-06 16:12:43 -04:00
carl bystrom
9895bc78e9 Horde: Prevent exceptions being swallowed during REAPI sandbox setup
* Improve test coverage for ActionExecutor by testing missing digest scenarios
* Raise a dedicated exception for missing digests for easier detection, primarily in tests

Previous use of ParallelTask.ForEachAsync prevented exceptions from being propagated. This caused create process exceptions to occur when the binary for an action could not be found.

#ROBOMERGE-SOURCE: CL 17067435 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17067444 by carl bystrom in ue5-release-engine-test branch]
2021-08-05 09:41:46 -04:00
joe kirchoff
de1e588e8b UnrealBuildTool: Clean up some intellisense suggestions, format documents, remove and sort usings, etc.
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 17059447 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17059463 by joe kirchoff in ue5-release-engine-test branch]
2021-08-04 16:50:01 -04:00
jonathan adamczewski
2b6ba5375d Make relative, with fewer allocations
#ROBOMERGE-SOURCE: CL 17056403 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17056411 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-04 14:33:52 -04:00
jonathan adamczewski
f3c823dcaa UnrealBuildTool:
Add a -Help option that prints descriptions of global options.
(Tool mode options are not available - support for those may be added in a future CL)

Example output:

Global options:
  -Help               :  Display this help.
  -Verbose            :  Increase output verbosity
  -VeryVerbose        :  Increase output verbosity more
  -Log                :  Specify a log file location instead of the default Engine/Programs/UnrealBuildTool/Log.txt
  -Timestamps         :  Include timestamps in the log
  -FromMsBuild        :  Format messages for msbuild
  -Progress           :  Write progress messages in a format that can be parsed by other programs
  -NoMutex            :  Allow more than one instance of the program to run at once
  -WaitMutex          :  Wait for another instance to finish and then start, rather than aborting immediately
  -RemoteIni          :  Remote tool ini directory
  -Mode=              :  Select tool mode. One of the following (default tool mode is "Build"):
                           AggregateParsedTimingInfo, Build, Clean, Deploy, Execute, GenerateClangDatabase, GenerateProjectFiles,
                           IOSPostBuildSync, JsonExport, ParseMsvcTimingInfo, PVSGather, QueryTargets, SetupPlatforms,
                           ValidatePlatforms, WriteDocumentation, WriteMetadata
  -Clean              :  Clean build products. Equivalent to -Mode=Clean
  -ProjectFiles       :  Generate project files based on IDE preference. Equivalent to -Mode=GenerateProjectFiles
  -ProjectFileFormat= :  Generate project files in specified format. May be used multiple times.
  -Makefile           :  Generate Linux Makefile
  -CMakefile          :  Generate project files for CMake
  -QMakefile          :  Generate project files for QMake
  -KDevelopfile       :  Generate project files for KDevelop
  -CodeliteFiles      :  Generate project files for Codelite
  -XCodeProjectFiles  :  Generate project files for XCode
  -EddieProjectFiles  :  Generate project files for Eddie
  -VSCode             :  Generate project files for Visual Studio Code
  -VSMac              :  Generate project files for Visual Studio Mac
  -CLion              :  Generate project files for CLion
  -Rider              :  Generate project files for Rider

#jira none

#ROBOMERGE-SOURCE: CL 17018675 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17018680 by jonathan adamczewski in ue5-release-engine-test branch]
2021-08-02 14:45:58 -04:00
ben marsh
16c9fb5650 Horde: Rearrange public Horde classes and projects.
* EpicGames.Horde contains all types and functionality common to server, agent, and clients of Horde.
* EpicGames.Horde.Client contains client interfaces for calling server functionality (but not server stubs for implementing those interfaces). Together with EpicGames.Horde, this defines the public interface for Horde. Note that types are still defined within the EpicGames.Horde namespace.
* HordeCommon contains functionality common to the server and agent (eg. session/lease messaging, task data structures, etc...), but which is not exposed to clients of Horde.

EpicGames.Horde.Common contains legacy REAPI functionality and will be removed once CB-REAPI is up and running.

[FYI] Joe.Kirchoff, Carl.Bystrom

#ROBOMERGE-SOURCE: CL 17012746 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17012752 by ben marsh in ue5-release-engine-test branch]
2021-08-01 17:10:17 -04:00
ben marsh
1d62123a7e Missing files.
#ROBOMERGE-SOURCE: CL 17012673 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17012674 by ben marsh in ue5-release-engine-test branch]
2021-08-01 16:30:41 -04:00
ben marsh
f443277309 Horde: Implement new gRPC interfaces for the blob store and ref table. Currently exist alongside the older implementations.
#ROBOMERGE-SOURCE: CL 17012664 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17012668 by ben marsh in ue5-release-engine-test branch]
2021-08-01 16:28:52 -04:00
ben marsh
eb3d8b9644 Horde: Add common definitions for the ref table, blob store, and cbobject/iohash wrapper types.
#ROBOMERGE-SOURCE: CL 17012437 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17012438 by ben marsh in ue5-release-engine-test branch]
2021-08-01 14:17:16 -04:00