Commit Graph

645 Commits

Author SHA1 Message Date
Ben Marsh
074e08da2b BuildGraph: Fix labels not getting passed through from the aggregate into the bytecode.
#preflight none

[CL 21060760 by Ben Marsh in ue5-main branch]
2022-07-12 14:59:18 -04:00
Tim Smith
5d35876169 Added support for UE_5_1_OR_LATER style define constants for UBT/UHT C# plugins.
#rb self
#rnx
#preflight 62cd5d7b21129e2bb824626f

[CL 21054775 by Tim Smith in ue5-main branch]
2022-07-12 07:48:21 -04:00
Ben Marsh
b198fd4a3c BuildGraph: Fix test cases with recent changes to storing graph structures as BG objects.
#preflight 62cc695dad490b52dad4efc9

[CL 21044414 by Ben Marsh in ue5-main branch]
2022-07-11 14:24:04 -04:00
Ben Marsh
7d85a883eb BuildGraph: Fix compile errors in tests.
#preflight none

[CL 21039069 by Ben Marsh in ue5-main branch]
2022-07-11 09:56:06 -04:00
Ben Marsh
d769555c4a Add Analyze configuration to EpicGames.Serialization.
#preflight none

[CL 21007076 by Ben Marsh in ue5-main branch]
2022-07-08 10:44:44 -04:00
Joakim Lindqvist
6c0afc1fcb CbPackage - Fixed up some warnings in the C# CbPackage code, reducing access scope on things that should be private
Also fixed up inconsistent whitespace.

#jira UE-158872
#preflight https://horde.devtools.epicgames.com/job/62c80bc6f671e8d2fac950f7

[CL 21004958 by Joakim Lindqvist in ue5-main branch]
2022-07-08 07:04:41 -04:00
Ben Marsh
83f89d2b66 Horde: Remove legacy bundle implementation, and fix up tests to work with new implementation.
#preflight none

[CL 20989605 by Ben Marsh in ue5-main branch]
2022-07-07 15:04:04 -04:00
Ben Marsh
c8cd7f5979 Horde: Rework commit mirroring to use latest bundle implementation.
#preflight none

[CL 20987022 by Ben Marsh in ue5-main branch]
2022-07-07 13:26:07 -04:00
Joe Kirchoff
032c08b7ad Epicgames.Build: Use a Lazy FileSystemInfo to ensure info is initialized before use while still deferring the Refresh until accessed
#rnx
#rb none
#jira UE-156859
#preflight 62c631db3c5171c81722998b

[CL 20978882 by Joe Kirchoff in ue5-main branch]
2022-07-06 21:33:53 -04:00
Ben Marsh
66af5fcd70 Fix duplicated data packet when copying compressed data from Perforce response streams.
#preflight none

[CL 20972578 by Ben Marsh in ue5-main branch]
2022-07-06 16:12:45 -04:00
Ben Marsh
3259d15ff7 Fix missing XML documentation
#preflight none

[CL 20967443 by Ben Marsh in ue5-main branch]
2022-07-06 11:11:48 -04:00
Ben Marsh
bf68cf5780 Horde: Add an implementation of IBlobStore built on the IStorageBackend interface in Horde.Build.
#preflight none

[CL 20967257 by Ben Marsh in ue5-main branch]
2022-07-06 11:03:38 -04:00
Ben Marsh
1fb237117f Horde: Remove accessor methods for data/references in blobs; expose them as properties instead.
#preflight none

[CL 20965815 by Ben Marsh in ue5-main branch]
2022-07-06 09:23:28 -04:00
Ben Marsh
37bac7fbcb EpicGames.Core: Add FileTransaction class with methods for replacing files in a transactional manner.
#preflight none

[CL 20965768 by Ben Marsh in ue5-main branch]
2022-07-06 09:20:55 -04:00
Ben Marsh
30496c90b5 EpicGames.Core: Add DateTime serializers for IMemoryReader/IMemoryWriter.
#preflight none

[CL 20965756 by Ben Marsh in ue5-main branch]
2022-07-06 09:20:02 -04:00
Joakim Lindqvist
0e6faae889 Fixed failing CbPackage test, Assert.ThrowsException does not support async lambda.
#preflight none

[CL 20964595 by Joakim Lindqvist in ue5-main branch]
2022-07-06 06:46:36 -04:00
Joakim Lindqvist
c8798de32a C# implementation of CompactBinaryPackage (cbpkg)
#rb carl.bystrom
#preflight none

[CL 20964277 by Joakim Lindqvist in ue5-main branch]
2022-07-06 06:02:07 -04:00
Tim Smith
3a61b37681 Fix runtime issue where internal UHT crashes due to a fix for VS2019.
#rb self
#rnx
#jira UE-158204
#robomerge EMT
#preflight 62c2e1df7358826af8128a95

[CL 20934189 by Tim Smith in ue5-main branch]
2022-07-04 09:13:15 -04:00
justin marcus
e489d8065b Unzip optimization for archives with lots of small files
- Remove redunant exists check when creating directories.  Directory.CreateDirectory already does that.
- Use streams directly instead of calling ZipFileExtensions.ExtractToFile.


[REVIEW] [at]Mitchell.Fisher [at]Graeme.Thornton
#preflight https://horde.devtools.epicgames.com/job/62bf34b2b024a2608c89b606

#ROBOMERGE-OWNER: justin.marcus
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 20919800 via CL 20919810 via CL 20919999 via CL 20920500
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20923415 by justin marcus in ue5-main branch]
2022-07-01 19:42:38 -04:00
Ben Marsh
1097ac0d90 Horde: Rewrite bundle create/extract commands to work with new bundle structures.
#preflight none

[CL 20921192 by Ben Marsh in ue5-main branch]
2022-07-01 16:52:32 -04:00
Ben Marsh
1a1282f27f Horde: Fix corrupt offsets into compressed packet data for bundles.
#preflight none

[CL 20921160 by Ben Marsh in ue5-main branch]
2022-07-01 16:50:19 -04:00
Ben Marsh
38f3bc55ef BuildGraph: Various VM improvements.
* Added support for deserializing BgObject types directly into native classes.
* Removed opcodes for creating graph structures. These are now created in user code from BgObject types.
* Removed BgNodeSpecBuilder. BgNode objects can now be modified directly (returning a modified copy).
* Added concrete types for option parameters. The VM now keeps track of any parameters for evaluated options, allowing them to be added into the graph definition.
* Order dependencies now take nodes rather than outputs.
* Added explicit support for native thunks, whose bindings are saved to a sideband channel during compilation and referenced in bytecode as an index. This generalizes code that was previously specific to node definitions.
* Added a name table to bytecode, to optimize situations where we reference the same string mulitple times.

#preflight 62bf3c583f0d6beee2e8f4a6

[CL 20918762 by Ben Marsh in ue5-main branch]
2022-07-01 14:47:54 -04:00
Joakim Lindqvist
3ccacbb4ea OidcToken - Reworked configuration so that its built into our shared library, and allows for configuration in more traditional ways with Engine, Game and Restricted folders being supported.
Ugs - Updated ugs to read the configuration via the shared library, thus removing the support to read it from UnrealGameSync.ini. Added a ini setting to allow which providers to use though, so that we can have more providers listed in the configuration then UGS will want to care about but also to allow us to control opt in of this feature per project.

#preflight none

[CL 20911818 by Joakim Lindqvist in ue5-main branch]
2022-07-01 04:36:09 -04:00
Ben Marsh
0f7177dcbf Horde: Fix tests.
#preflight none

[CL 20893709 by Ben Marsh in ue5-main branch]
2022-06-30 09:25:56 -04:00
Ben Marsh
35e9ebc4c1 Horde: API for manipulating tree structures in storage. Designed to abstract the logical tree structure away from storage/retrieval when allow working on larger-than-memory data sets.
* ITreeStore defines an API for reading/writing named tree structures.
* ITreeStore<T> allows reading/writing trees of nodes derived from the TreeNode base class, which can be specialized by clients. TreeNode tracks whether the node has been mutated, and allows recursing through the tree with strongly typed objects. Weak references are kept to deserialized TreeNode objects to ensure synchronization with the serialized tree objects.
* Default (and only current) implementation of ITreeStore is BundleStore, which packs nodes together in an efficient binary format and supports incrementally updating bundles. Nodes are hashed and deduplicated within bundles (and the known working set of the BundleStore instance), but packed Bundles are stored using a unique id returned by IBlobStore.
* DirectoryNode and FileNode nodes allow storing file trees in bundles, and implement content-aware chunking of binary data using a rolling BuzHash over an input stream.

#fyi Joakim.Lindqvist
#preflight 62acb0a5b47403e5aef09f74

[CL 20887473 by Ben Marsh in ue5-main branch]
2022-06-29 21:14:05 -04:00