Commit Graph

111 Commits

Author SHA1 Message Date
ben marsh
d0e8ab9c78 Horde: Reduce peak memory usage when building workspace snapshots. Was previously keeping references to Memory<byte> ranges within a much larger buffer, preventing them from being GC'd.
#ROBOMERGE-SOURCE: CL 16867148 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16867166 by ben marsh in ue5-release-engine-test branch]
2021-07-15 16:56:49 -04:00
joe kirchoff
ce96ff9682 UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess
Process.TotalProcessorTime raises an exception on non-windows if the process has also exited, so don't do that

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16863633 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863635 by joe kirchoff in ue5-release-engine-test branch]
2021-07-15 12:25:51 -04:00
joe kirchoff
8b0febe436 [Backout] - CL16863209
[FYI] joe.kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16863370 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863374 by joe kirchoff in ue5-release-engine-test branch]
2021-07-15 12:06:17 -04:00
ben marsh
7e3cf9f70b Horde: Output additional debug info when StatsV2 file is not found.
#ROBOMERGE-SOURCE: CL 16863233 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863239 by ben marsh in ue5-release-engine-test branch]
2021-07-15 11:55:19 -04:00
joe kirchoff
2d2e57475c UnrealBuildTool: Use process directly to get wall clock execution time
Process.StartTime raises an exception on non-windows platforms if the process has already exited, so cache it immediately after starting the FrameworkProcess

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16863209 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863223 by joe kirchoff in ue5-release-engine-test branch]
2021-07-15 11:53:59 -04:00
joe kirchoff
98cc7bc5e9 [Backout] - CL16856917
[FYI] Joe.Kirchoff
Original CL Desc
-----------------------------------------------------------------
UnrealBuildTool: Use process directly to get wall clock execution time

#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16857196 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16857201 by joe kirchoff in ue5-release-engine-test branch]
2021-07-14 19:23:02 -04:00
joe kirchoff
6b5d852bae UnrealBuildTool: Use process directly to get wall clock execution time
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 16856917 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16856921 by joe kirchoff in ue5-release-engine-test branch]
2021-07-14 18:56:42 -04:00
ben marsh
491fca0c7c Fix AsBinary() returning the incorrect payload buffer for binary fields. Was causing Horde agents to deserialize incorrect hash values.
#ROBOMERGE-SOURCE: CL 16852536 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16852540 by ben marsh in ue5-release-engine-test branch]
2021-07-14 13:55:40 -04:00
aurel cordonnier
d7923f70ef P4 Blame tool for P4V (requires input of the line number)
#rb marc.audy, ben.marsh
#jira none

#ROBOMERGE-SOURCE: CL 16852242 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16852247 by aurel cordonnier in ue5-release-engine-test branch]
2021-07-14 13:24:35 -04:00
danny couture
42a7534833 Fix crash when jobs are not available (i.e. MacOS)
Only show processor time when available

#rnx
#rb Will.Damon

#ROBOMERGE-SOURCE: CL 16836540 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16836555 by danny couture in ue5-release-engine-test branch]
2021-07-13 09:25:44 -04:00
danny couture
96f9fb7e00 Add cpu time spent per compilation unit in parallel and task executor
Add top 20 report when compilation ends for parallel and task executor
Add total cpu usage across all cores when compilation finishes for parallel and task executor
Use real processor time reported by OS when possible to account for internal usage of multiple threads (i.e. cl.exe codegen uses 8 threads)
Enabled by default for Epic developers to increase awareness on most problematic compilation times

#rnx
#rb Ben.Marsh, Joe.Kirchoff
#preflight 60ec463819a0060001dc5eb2

#ROBOMERGE-SOURCE: CL 16835803 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16835812 by danny couture in ue5-release-engine-test branch]
2021-07-13 06:55:03 -04:00
ben marsh
e96c1e14e9 Missing copyright notice.
#ROBOMERGE-SOURCE: CL 16817994 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16817997 by ben marsh in ue5-release-engine-test branch]
2021-07-09 09:51:51 -04:00
ben marsh
e7a9e6bbd1 Missing file.
#ROBOMERGE-SOURCE: CL 16817970 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16817971 by ben marsh in ue5-release-engine-test branch]
2021-07-09 09:30:01 -04:00
ben marsh
bb5b6ff0b0 Add missing Md5Hash file.
#ROBOMERGE-SOURCE: CL 16817966 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16817967 by ben marsh in ue5-release-engine-test branch]
2021-07-09 09:26:38 -04:00
ben marsh
334d38e86c Horde: Write managed workspace snapshots using Compact Binary.
#ROBOMERGE-SOURCE: CL 16811525 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16811533 by ben marsh in ue5-release-engine-test branch]
2021-07-06 11:05:18 -04:00
ben marsh
5feac1c075 Rename ReadOnlyUtf8String to Utf8String. Strings are always immutable in C#.
#ROBOMERGE-SOURCE: CL 16807051 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16807055 by ben marsh in ue5-release-engine-test branch]
2021-07-02 12:17:42 -04:00
ben marsh
7bf7a6596b Fix buffer overrun in compact binary writer.
#ROBOMERGE-SOURCE: CL 16807025 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16807029 by ben marsh in ue5-release-engine-test branch]
2021-07-02 12:12:09 -04:00
ben marsh
2c41930599 Horde: Various compact binary library improvements.
CbObject/CbBinary/CbArray:
* Added proxy classes for viewing objects in the debugger. CbArray now displays as an array, CbObject displays as an array of fields, etc...
* Removed overloaded "View" methods, which are not required in C# due to GC.

CbWriter:
* Significantly fewer memory allocations/copying. Output buffer is now allocated in large chunks, with length fields only inserted when copying to the final output buffer.
* Methods for writing named/unnamed fields are now more idomatic with other C# serialization classes (ie. WriteXXX(Name, Value) and WriteXXXValue(Value)).

Other:
* Added serialization tests to Horde projects and fixed NuGet dependencies.

#ROBOMERGE-SOURCE: CL 16804192 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16804202 by ben marsh in ue5-release-engine-test branch]
2021-07-01 09:16:47 -04:00
ben marsh
71cc101ae1 Missing file for Blake3 hashes.
#ROBOMERGE-SOURCE: CL 16781033 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16781041 by ben marsh in ue5-release-engine-test branch]
2021-06-24 22:01:47 -04:00
ben marsh
67d5b5cfb7 Horde: Basic implementation of storage API.
Current limitations:

* Only supports a filesystem backend
* No garbage collection
* Refs are stored in the database, and do not automatically expire.
* Validation of refs is slow and does not cache any data.

#ROBOMERGE-SOURCE: CL 16778495 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16778515 by ben marsh in ue5-release-engine-test branch]
2021-06-24 16:58:05 -04:00
ben marsh
decd803e39 Horde: Add TypeConverter for IoHash objects, to allow passing them via MVC routes.
#ROBOMERGE-SOURCE: CL 16757331 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16757359 by ben marsh in ue5-release-engine-test branch]
2021-06-23 11:07:21 -04:00
joe kirchoff
fe6eb49434 Horde: Move base RemoteExecution classes to EpicGames.Horde.Common
Fix IoHash Digest size being set to the size of the hash rather than the size of the hashed data

#rb Ben.Marsh
#rnx

#ROBOMERGE-SOURCE: CL 16729492 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16729518 by joe kirchoff in ue5-release-engine-test branch]
2021-06-21 12:37:31 -04:00
ben marsh
1946b0059a Missing file for CB serialization.
#ROBOMERGE-SOURCE: CL 16708503 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16708558 by ben marsh in ue5-release-engine-test branch]
2021-06-17 14:42:46 -04:00
jonathan adamczewski
9a6787995b BuildUtilities:
- Add Unreal.DotnetPath
- in Unreal.FindRootDirectory(), fix assembly path case sooner (allows UBT to run even if I type 'Engine' as 'engine')

EpicGames.Core:
- Add RuntimePlatform.ExeExtension

#trivial
#jira none

#ROBOMERGE-SOURCE: CL 16704658 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16704661 by jonathan adamczewski in ue5-release-engine-test branch]
2021-06-17 10:43:00 -04:00
ben marsh
e4745ce8fc Add missing configurations for EpicGames.Serialization.Tests.
#ROBOMERGE-SOURCE: CL 16703657 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16703665 by ben marsh in ue5-release-engine-test branch]
2021-06-17 09:00:17 -04:00