Commit Graph

258 Commits

Author SHA1 Message Date
devin doucette
1b7ea4c667 CompactBinary: Added validation functions for FCb[Type] for ease of use
#rb Zousar.Shaker
#rnx
#preflight 61b10476a746977eacc416b8

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18412013 in //UE5/Release-5.0/... via CL 18412047
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18412059 by devin doucette in ue5-release-engine-test branch]
2021-12-08 16:14:32 -05:00
devin doucette
cae603f507 CompactBinary: Added a missing call to Ar.SetError() when loading attachments from an archive
#rb Zousar.Shaker
#rnx
#preflight 61b10493a746977eacc41aef

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18409758 in //UE5/Release-5.0/... via CL 18409778
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18409786 by devin doucette in ue5-release-engine-test branch]
2021-12-08 14:34:28 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
devin doucette
6c127f2679 CompactBinary: Minor corrections to documentation and quick edits to simplify code
#rb trivial
#rnx
#preflight 61786fb37855e800019b94f9

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17938538 in //UE5/Release-5.0/... via CL 17938541
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v885-17909292)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17938556 by devin doucette in ue5-release-engine-test branch]
2021-10-26 19:44:56 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
devin doucette
17d7a4d04d CompactBinary: Updated FCbObjectId::NewObjectId() to match the structure of Oid in Zen
#rb Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17741911 in //UE5/Release-5.0/... via CL 17741924
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17742108 by devin doucette in ue5-release-engine-test branch]
2021-10-06 17:38:34 -04:00
stefan boberg
cdd8e79783 CompactBinary: Implemented FCbObjectId::NewObjectId and fixed JSON serialization of ObjectId fields
[FYI] devin.doucette

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: stefan.boberg
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 17694257 in //UE5/Release-5.0/... via CL 17694263
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17742072 by stefan boberg in ue5-release-engine-test branch]
2021-10-06 17:36:30 -04:00
ionut matasaru
2f3404b10e Fixed usage of UE_LOG (arguments for "%s" formatting).
#rb Zousar.Shaker

#ROBOMERGE-AUTHOR: ionut.matasaru
#ROBOMERGE-SOURCE: CL 17691626 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0

[CL 17691646 by ionut matasaru in ue5-release-engine-test branch]
2021-10-01 10:30:26 -04:00
carlmagnus nordin
51c9794dea PackageStore refactor
* Decouple container concept from IoDispatcher
* Decoiuple PackageStore implementation from AsyncLoading2
* Restore ucas unmount fix that got kist when merrging from UE4
* Fix packages being left in the PackageStiore even after unmounting contaiiners

#rnx
#rb pj.kack, per.larsson
#preflight 61520cc52afc2d0001146ce7

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17641845 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v874-17637634)

[CL 17642353 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-28 04:00:33 -04:00
per larsson
d8faf17ea1 CbObject: Use AsObjectId when serializing object ID's to JSON
#rb stefan.boberg
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17628163 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17628165 by per larsson in ue5-release-engine-test branch]
2021-09-27 04:39:48 -04:00
paul chipchase
ecdec9bf35 Add an UE5 specific version EUnrealEngineObjectUE5Version to be used for global changes instead of EUnrealEngineObjectUEVersion. By splitting and storing both version numbers we allow for hypothetical future UE4 changes that will not conflict when merged to UE5.
#rb CarlMagnus.Nordin
#rnx
#tests Ran overnight preflights on several platforms, opened/cooked/staged/ran the oldest version of InfiltratorDemo that can be downloaded (4.11)

### ObjectVersion
- Add a new version enum EUnrealEngineObjectUE5Version.
-- This version number starts at 1000 which leaves more than enough for for EUnrealEngineObjectUEVersion to be expanded
- Even though very few changes (if any at all) to EUnrealEngineObjectUE4Version are expected there is a static assert to make sure that EUnrealEngineObjectUEVersion::AUTOMATIC_VERSION never overtakes EUnrealEngineObjectUE5Version::INITIAL_VERSION.
- Add a struct FPackageFileVersion that wraps around the version numbers and is used to store them instead of raw int32 values which was done before. This should make it easier to add new version numbers in the future if we desire (although this will cause problems in places that serialize the struct directly)

### FPackageFileSummary
- Adding a new entry to CurrentLegacyFileVersion at value -8 which shows the UE5 version being added. This lets us make the changes without needing to submit anything to UE4 Main.
- When loading a package that does not have a UE5 version, it will remain at 0.
- Added ::IsFileVersionTooOld and ::IsFileVersionTooNew to replace hardcoded tests in the code base for version validity. This will make it easier to make changes in the future.
- A few months ago most of the accessors of the version number were deprecated in favour of a version that did not contain the Engine number (ie UE4Ver -> UEVer in Archive) but to work with these changes the renamed methods now will return or accept the version as FPackageFileVersion rather than int32.  The old UE4 methods will remain deprecated and direct licensees to use the new methods.

### Archive
- Now stores the version as a FPackageFileVersion rather than int32

### LinkerLoad
- Reports the larger version number if we detect a higher version number than we support. Note that this could cause an issue if the UE4 version is ever raised but helps keep the code simple.

### AssetData
- Need to add a new version here to manage existing data that only has the UE4 version

### EditorDomain
- We do not need to version the format, we can just invalidate existing editor domain entries via EditorDomainVersion

### EditorServer
- When reporting that a package is too old we report the UE4 version as that is the only version that can be older than VER_UE4_OLDEST_LOADABLE_PACKAGE
- When reporting that a package is too new it can be either the UE4 or the UE5 version so we print them together "UE4Ver|UE5Ver"

### ContentCommandlets
- The min and max resave versions have been kept as a single value, you will not be able to resave against different UE4 and UE5 versions at the same time. It doesn't seem like a useful feature and would greatly increase the complexity of the code.
- We will also only report the file version as a single value.

### ManifestUObject
- This class was setting an older obsolete version on purpose to try and maintain compatibility with older clients so we need to provide a way to create an older UE4 only version that will leave the UE5 version as unset.

### NetworkPlatformFile
- I was unable to test the code path in FNetworkPlatformFile::ProcessServerCachedFilesResponse as I am unsure how to run the game in a mode that will actually use it.
- When reading an older "CookedVersion.txt" that was saved with a single version, the reads will fail and this will count as a version change in the code so that all of the existing files will be deleted. The existing code would not give the user a log message when this happens and given the very small time window where this might happen caused by this change I have opted to leave this alone and not add any additional logging.
- If we do detect a version mismatch we will still only log the version number as a single version.

### CookOnTheFlyServer
- We now add each version number to the IniVersionMap rather than merge the version and license version as a key/value pair. This allows us to a) use both the UE4 and UE5 version numbers b) we now log a warning that the version values don't match when it is changed, previously since it was a key value we would log a warning about an additional setting instead.
-- I also added "vs" to the log message when values are mismatched to make the space between the two values being printed clearer.

#ROBOMERGE-OWNER: paul.chipchase
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17549459 via CL 17550236 via CL 17550238 via CL 17550582
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17550583 by paul chipchase in ue5-release-engine-test branch]
2021-09-17 07:04:55 -04:00
devin doucette
ec3fa9680f CompositeBuffer: Renamed Flatten() to ToShared()
#rb none
#rnx
[FYI] Zousar.Shaker
#preflight 613f872b3bbb48000114081d

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17495384 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17495398 by devin doucette in ue5-release-engine-test branch]
2021-09-13 15:16:06 -04:00
devin doucette
2eb0cd18ea CompactBinary: Fixed attachment constructors missing calls to MakeOwned, and attachment loading that was too permissive
#rb none
#rnx
#preflight 613a7d7780b8e70001767bcd

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17478507 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17478513 by devin doucette in ue5-release-engine-test branch]
2021-09-09 19:40:22 -04:00
zousar shaker
8c80e5a866 Change Compact Binary Package API to represent binary attachments as compressed buffers identified by their raw hash.
Change Compact Binary Package serialization for binary attachments, compressed binary attachments and objects.

#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17450533 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17450587 by zousar shaker in ue5-release-engine-test branch]
2021-09-07 18:07:00 -04:00
devin doucette
5a57d677bf CompactBinary: Added FIoHashBuilder and switched CompactBinary to use it
#rb Zousar.Shaker
#rnx
#preflight 612fb7306256b100011931e9

#ROBOMERGE-SOURCE: CL 17390911 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17390961 by devin doucette in ue5-release-engine-test branch]
2021-09-01 15:05:03 -04:00
devin doucette
50067a1ce1 CompactBinary: Added CompactBinaryToJson
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17243503 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17243507 by devin doucette in ue5-release-engine-test branch]
2021-08-19 23:26:08 -04:00
devin doucette
8d2516231f CompactBinary: Added FCbValue for low-level access to field values
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17236058 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17236067 by devin doucette in ue5-release-engine-test branch]
2021-08-19 12:56:55 -04:00
devin doucette
765d482c95 CompactBinary: Rename payload to value to prepare for a public value type
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17235752 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17235792 by devin doucette in ue5-release-engine-test branch]
2021-08-19 12:40:52 -04:00
martin ridgers
e39cf135f4 Load and platform-file tracing migrated away from attachments.
#rb cm
#rnx

#ROBOMERGE-SOURCE: CL 17048386 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17049897 by martin ridgers in ue5-release-engine-test branch]
2021-08-04 09:33:37 -04:00
charles bloom
494b3de14e allow SerializeCompressedNew to read less than requested
fix FArchiveLoadCompressedProxy reading past end on last partial chunk

#rb devin.doucette

#ROBOMERGE-SOURCE: CL 17038478 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v848-17036448)

[CL 17038565 by charles bloom in ue5-release-engine-test branch]
2021-08-03 16:55:49 -04:00
danny couture
3d73ddfb07 Fix game-thread starvation under high load by better distributing the work among threads
#rnx
#rb Arciel.Rekman

#ROBOMERGE-SOURCE: CL 17033153 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v846-17029325)

[CL 17035589 by danny couture in ue5-release-engine-test branch]
2021-08-03 14:44:15 -04:00
aurel cordonnier
dc4bf61540 Merge from Release-Engine-Staging @ 17030559 to Release-Engine-Test
This represents UE4/Main @ 17030256 and Dev-PerfTest @ 17030553

[CL 17031509 by aurel cordonnier in ue5-release-engine-test branch]
2021-08-03 11:56:47 -04:00
charles bloom
1ecb02670e Archive support CompressionFormat other than NAME_Zlib
v2 header tag to indicate format is serialized in data
writes new data, continues to load old data

#rb devin.doucette

#ROBOMERGE-SOURCE: CL 17017219 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17017233 by charles bloom in ue5-release-engine-test branch]
2021-08-02 12:28:00 -04:00
steve robb
fdd30297da FNames can now be constructed with UTF8CHAR strings.
UTF-8 support added to FString::AppendChars() and made able to correctly handle variably-encoded source strings (both wider and narrower than TCHAR).
FUTF8ToTCHAR now accepts either UTF8CHAR* or ANSICHAR* for better compatibility.
FUtf8StringView now binds to UTF8CHAR* or any UTF8CHAR range, and can be constructed via a _U8SV string literal suffix.
CompactBinary now supports UTF-8.
Miscellaneous other UTF8CHAR fixes.

#rb devin.doucette

#ROBOMERGE-SOURCE: CL 17017066 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17017071 by steve robb in ue5-release-engine-test branch]
2021-08-02 12:15:42 -04:00