Commit Graph

366 Commits

Author SHA1 Message Date
Ryan Durand
97c2bbabc5 Add support for per platform FilterPlugin inis for plugins which was originally in a code review for CL 18874143.
#rnx
#jira UE-125146
#rb ryan.hummer
#preflight 62169f2b104496cff8a2c4e8

[CL 19122049 by Ryan Durand in ue5-main branch]
2022-02-24 14:48:24 -05:00
Joakim Lindqvist
28a6cd7b1b Missing file from change 19111703 (more tests for EpicGames.Serialization)
#preflight none

[CL 19111816 by Joakim Lindqvist in ue5-main branch]
2022-02-24 06:01:24 -05:00
Joakim Lindqvist
cf2bef4b2e Moved Compact Binary formatters into a EpicGames.AspNet shared library, this shared library can be used for any common dependencies that also requires asp.net
#preflight https://horde.devtools.epicgames.com/job/621760d3647bb7debf80b277

[CL 19111741 by Joakim Lindqvist in ue5-main branch]
2022-02-24 05:48:57 -05:00
Joakim Lindqvist
6e85431bc6 EpicGames.Serialization - Added support for C# arrays (and not just lists) when serializing C# properties to compact binary
Also fixed issue were empty lists would not be serialized (field skipped)

#preflight none

[CL 19111704 by Joakim Lindqvist in ue5-main branch]
2022-02-24 05:44:28 -05:00
Joakim Lindqvist
b37024fe9b C# Compact Binary Serialization - Added more tests from Horde Storage (Jupiter Tests) and fixed issues exposed by them.
#preflight none

[CL 19111703 by Joakim Lindqvist in ue5-main branch]
2022-02-24 05:44:12 -05:00
jonathan adamczewski
2c3a702a8c Upgrade to nuget package Microsoft.Win32.Registry 5.0.0
#trivial
#jira none
#rnx
#preflight 62149c809e2201e214fe1a6c

[CL 19076594 by jonathan adamczewski in ue5-main branch]
2022-02-22 13:52:23 -05:00
Joe Kirchoff
1bea75f650 ManagedProcess: Process.StartTime & ExitTime can't be trusted on Mac
#rnx
#rb trivial
#preflight none

[CL 19045300 by Joe Kirchoff in ue5-main branch]
2022-02-17 19:06:44 -05:00
Ben Marsh
1839e4e0f4 Horde: Add commands to analyze and extract contents of tree packs, and add support for creating "pointer files" in packs to Perforce depot/revision numbers rather than actually including the file contents themselves.
#preflight none

[CL 19042497 by Ben Marsh in ue5-main branch]
2022-02-17 16:39:27 -05:00
Tim Smith
cd07867bf6 1) Adding new collection of UE types into C#
2) Enhancements to StringView and a few other helper classes
3) New UE style ignore case comparison
4) Moved shared UHT definitions into a common C# file.

#rb jonathan.adamczewski
#rnx
#preflight 620e3cdc8b6428e701a24666

[CL 19033446 by Tim Smith in ue5-main branch]
2022-02-17 07:40:56 -05:00
Ben Marsh
41d7f3c295 Horde: Simplify process for converting TreePack objects into refs, prior to adding to commit service.
#preflight 620d145101253d2e19133d94

[CL 19014499 by Ben Marsh in ue5-main branch]
2022-02-16 10:17:52 -05:00
Ben Marsh
bfbb1e7890 Horde: Use dependency injection for command discovery in CommandHost, and support other commands within the server application too. Makes it much easier to perform utility operations using standard services without needing to register a REST endpoint to do it.
#preflight 620c23d7475fb819ddd19b88

[CL 19005141 by Ben Marsh in ue5-main branch]
2022-02-15 17:20:45 -05:00
Ben Marsh
8a67c4acaf Horde: Move generic functionality for hosting different commands within an application into EpicGames.Core. Simplifies command line parsing, error handling, help output, etc...
#preflight 620c005c615db7478d393b1e

[CL 19001122 by Ben Marsh in ue5-main branch]
2022-02-15 14:41:10 -05:00
Ben Marsh
b40a918c81 Implement IComparable<> for FileReference and DirectoryReference, allowing them to be sorted in lists.
#preflight none

[CL 18995790 by Ben Marsh in ue5-main branch]
2022-02-15 10:02:06 -05:00
Ben Marsh
10021af683 UGS: Fix exception when opening a file from Perforce that doesn't exist on the local disk.
#preflight none
#jira UE-142325

[CL 18983991 by Ben Marsh in ue5-main branch]
2022-02-14 14:35:44 -05:00
Ben Marsh
18b7c9901f Horde: Missing file.
#preflight none

[CL 18983556 by Ben Marsh in ue5-main branch]
2022-02-14 14:15:45 -05:00
Ben Marsh
eb9e54cedb Horde: Add helper classes for facilitate packing merkle trees into larger blobs, exploiting temporal coherency when updating them. Supports content-aware slicing of larger files.
The "logical" tree consists of nodes prefixed by a byte indicating its type (TreePackNodeType). Supported node types are Binary (leaf data with no external references), Object (a generic compact-binary encoded object), Directory (a directory structure which may reference other files or directories by name), and Concat (a concatenated sequence of Binary objects).

Nodes are packed together into "physical" blobs for storage, which default to 10mb in size and may reference other blobs. Each blob has an import and export table listing nodes they contain and other blobs they require nodes from. The hierarchy of blob dependencies also form an acyclic graph.

Physical blobs are reused between subsequent runs, with updated/replacement data being added into new blobs at the root of the graph. Physical blobs can be discarded and re-packed along with the new data once a costing heuristic indicates it is more efficient to do so. The costing heuristic takes into account the size of useful vs wasted space in a pack, its age, and an estimate for how costly it would be for machines to download it (see TreePack.GetCostHeuristic).

HordeAgent includes two new utility commands that can be use to create pack files from a tree of files or Perforce depot path, with the following syntax:

    HordeAgent.exe archive create -source=[Local or Perforce Depot Path To Read From] -target=[Path to an output TXT file containing the hash of the root object; blobs are placed alongside it in the same directory] -base=[Path to a previous archive TXT file to generate a delta from]
    HordeAgent.exe archive extract -input=[Archive root hash TXT file] -output=[Output directory]

#fyi Carl.Bystrom
#preflight 620aa313fa3ad732bb151326

[CL 18983187 by Ben Marsh in ue5-main branch]
2022-02-14 13:58:11 -05:00
Ben Marsh
892805279c P4: Add support for intercepting I/O through native Perforce library. This allows streaming sync data directly into memory, avoiding any writes to disk.
#preflight 620a741d17632e948459cb9d

[CL 18979717 by Ben Marsh in ue5-main branch]
2022-02-14 10:37:38 -05:00
Ben Marsh
184f8d56fb Change IoHash to be a pure struct rather than wrapping a ReadOnlyMemory<byte> buffer.
#preflight 620a675317632e9484560674

[CL 18978869 by Ben Marsh in ue5-main branch]
2022-02-14 09:35:17 -05:00
Ryan Hummer
8eb68ed08a Make the type field optional for OpenRecord parsing
if -s is passed through the type field will not be present.

#rnx
#jira none
#rb josh.engebretson
#preflight skip

[CL 18843871 by Ryan Hummer in ue5-main branch]
2022-02-03 11:25:22 -05:00
Ben Marsh
a9d4363a36 Horde: Fix incorrect method called for Utf8String.LastIndexOf.
#preflight none

[CL 18830126 by Ben Marsh in ue5-main branch]
2022-02-02 13:14:21 -05:00
Josh Engebretson
89c10334de Horde: Add systemic issue handler
#jira none
#skipci
#preflight none
#rnx

[CL 18813242 by Josh Engebretson in ue5-main branch]
2022-02-01 14:51:36 -05:00
Ben Marsh
d666c0c19a Horde: Support for running standalone compute requests using HordeAgent.
- JSON job descriptor contains storage/compute server settings.
- Supports auth using OAuth2 or bearer token.
- Added static extension methods to configure Horde Storage and Horde Compute via dependency injection.

#preflight none

[CL 18788867 by Ben Marsh in ue5-main branch]
2022-01-31 08:54:35 -05:00
Ben Marsh
5e8d4f4224 Horde: Add missing JsonConverter for BucketId.
#preflight none

[CL 18786328 by Ben Marsh in ue5-main branch]
2022-01-30 16:35:48 -05:00
Ben Marsh
8f0c4a9ddd Horde: Fix test failures due to DI service lifetimes.
#preflight none

[CL 18784692 by Ben Marsh in ue5-main branch]
2022-01-29 17:56:53 -05:00
Ben Marsh
778e2ecc2a Horde: Mising file.
#preflight none

[CL 18784254 by Ben Marsh in ue5-main branch]
2022-01-29 14:55:19 -05:00