This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
#jira UE-132939
LargeMemoryArchive::ReleaseOwnership() sets TotalSize() to 0, so TotalSize must be called before calling ReleaseOwnership. Calling them both as arguments into a function was succeeding on some compilers and failing on others due to compiler optimization differences.
#rb Zousar.Shaker
#rnx
#[fyi] Francis.Hurteau
#ushell-cherrypick of 17947411 by Matt.Peters
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18008712 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)
[CL 18008728 by matt peters in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
FCoreUObjectDelegates::OnEndLoadPackage:
Fix the implementation in AsyncLoading1 to handle recursive loads; FAsyncPackage->PackageObjLoaded includes UObjects loaded from packages that were import dependencies of the package, and we need to send OnEndLoadPackage for those as well.
Also filter out the transient package and script and memory packages from all OnEndLoadPackage calls; this was affecting all three of AsyncLoading1, AsyncLoading2, and UObjectGlobals.
Did some comment improvement and removal of dead code in AsyncLoading1 while I was working in the area.
#ushell-cherrypick of 17686607 by jamie.dale
#ushell-cherrypick of 17734361 by Matt.Peters
#rb None, cherrypick
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17735351 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v879-17706426)
[CL 17735376 by matt peters in ue5-release-engine-test branch]
#rb CarlMagnus.Nordin
#jira none
#rnx
#preflight 6156eabd7b2a62000169d451
### Current Changes
- By storing the location of the toc in the package file summary we can access the toc by reading the summary and then jumping to it. It should also be possible for external scripts to be able to do this as well.
- We currently store the identifier, the offset into the package file and the uncompressed size of payload, which will be stored in FCompressedBuffer format.
-- We don't need all this info right now but it means that the structure will be compatible with the experimental sidecar feature. It would also (hypothetically) allow us to move the offset/size info from the bulkdata objects themselves and simply rely on the toc, in which case the loading from within the package file or a sidecar file will be the same.
- When saving a package's exports we will gather a reference to each virtualized bulkdata serialized. The toc should appear before the payloads in the file but we do not know the correct info for the toc until after the payloads have been appended to the package file. So we need to write out a dummy toc that will reserve the space in the file and then after we have appended the payloads, seek back in the file and write out the correct info.
### Future Work
- Although this CL does have a number of code changes to allow this to work with text based assets there are some problems with how that format is currently saved which will prevent it from working (and currently prevents VBD from working with it as well). This will be fixed as a separate issue.
- The experimental sidecar feature uses FTocEntry but serializes a TArray to disk. This should be updated to use FPayloadToc in a later submit.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17690578 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17690585 by paul chipchase in ue5-release-engine-test branch]
* Strip names that are not used when cooking for UPS
* Strip names that are not used by the loader when staging for IoStore
#preflight 615424b7f4d2a4000107be68, 615450a875fae600010d4820
#rb pj.kack, francis.hurteau
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17674004 in //UE5/Release-5.0/... via CL 17674013
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17674014 by carlmagnus nordin in ue5-release-engine-test branch]
#rb Danny Couture
#rnx
### Problem
- The crash would occur when the texture being since contains a reference to another texture (via composite) which is also in need of syncing on the users machine.
- After the files on disk have been updated the current packages are reloaded but the original version of the texture will still be in memory when the composite texture notifies that it has been reloaded causing a texture compilation job with the older now invalid texture.
-- VirtualizedBulkData relied on out of date packages not trying to access their payloads on disk once the file has been changed, but with the above scenario it was possible. There could also be other paths causing this (even if the access of out of date package payloads at this point is a waste of time)
### Fix
- The virtualized bulkdata objects now reference themselves with the FLinkerLoader if the payload is on disk, just as the old bulkdata system did. When file changing operations are invoked, the FLinkerLoader will inform it's referenced bulkdata to load the payloads off disk and hold them in memory.
-- Note that virtualized bulkdata was trying to avoid this memory bloat by design, but if we cannot be certain that the bulkdata will not try to access the payload after the file changes (as was thought) then we must take the payload into memory.
- Added a log message to VBD if we detect that the loaded payload does not match the VBD members expectations. This log message can either be an error or a fatal error depending on the define 'VBD_CORRUPTED_PAYLOAD_IS_FATAL' (defaulted to on)
- The registration to FLinkerLoader can be easily disabled by setting the define 'VBD_ALLOW_LINKERLOADER_ATTACHMENT' to 0
### Outstanding Issues
- Cloned bulkdata objects (via assignment) do not currently register themselves to the FLinkerLoader (the old bulkdata system did not) if we were to add this then we need to make sure that the FLinkerLoader registration is thread safe. This should be done as a separate work item so that we can unblock the current workflows that this changelist addresses.
-- Note that if we did not update cloned bulkdata to work with this system then we could drop the registration and just iterate over the package's bulkdata and force them to load into memory when required. However that would leave the cloned virtual bulkdata in an uncertain state and is not desirable.
- When saving a package, the virtualized bulkdata objects that it contains will change their members so that future payload access can come from the newly saved files, where as the old bulkdata system would force the payload to remain in memory for the rest of the editor session. If we want to keep this then we will need to try and make sure that the newly updated bulkdata can be registered so that it receives future notifications about the file being changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17648118 via CL 17648147 via CL 17648161 via CL 17648163 via CL 17648175
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17648184 by paul chipchase in ue5-release-engine-test branch]
#jira UE-124537
#rb jamie.dale
#ushell-cherrypick of 17407178 by jason.walter
#ROBOMERGE-AUTHOR: jason.walter
#ROBOMERGE-SOURCE: CL 17573843 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)
[CL 17573844 by jason walter in ue5-release-engine-test branch]
#rb CarlMagnus.Nordin
#rnx
#tests Ran overnight preflights on several platforms, opened/cooked/staged/ran the oldest version of InfiltratorDemo that can be downloaded (4.11)
### ObjectVersion
- Add a new version enum EUnrealEngineObjectUE5Version.
-- This version number starts at 1000 which leaves more than enough for for EUnrealEngineObjectUEVersion to be expanded
- Even though very few changes (if any at all) to EUnrealEngineObjectUE4Version are expected there is a static assert to make sure that EUnrealEngineObjectUEVersion::AUTOMATIC_VERSION never overtakes EUnrealEngineObjectUE5Version::INITIAL_VERSION.
- Add a struct FPackageFileVersion that wraps around the version numbers and is used to store them instead of raw int32 values which was done before. This should make it easier to add new version numbers in the future if we desire (although this will cause problems in places that serialize the struct directly)
### FPackageFileSummary
- Adding a new entry to CurrentLegacyFileVersion at value -8 which shows the UE5 version being added. This lets us make the changes without needing to submit anything to UE4 Main.
- When loading a package that does not have a UE5 version, it will remain at 0.
- Added ::IsFileVersionTooOld and ::IsFileVersionTooNew to replace hardcoded tests in the code base for version validity. This will make it easier to make changes in the future.
- A few months ago most of the accessors of the version number were deprecated in favour of a version that did not contain the Engine number (ie UE4Ver -> UEVer in Archive) but to work with these changes the renamed methods now will return or accept the version as FPackageFileVersion rather than int32. The old UE4 methods will remain deprecated and direct licensees to use the new methods.
### Archive
- Now stores the version as a FPackageFileVersion rather than int32
### LinkerLoad
- Reports the larger version number if we detect a higher version number than we support. Note that this could cause an issue if the UE4 version is ever raised but helps keep the code simple.
### AssetData
- Need to add a new version here to manage existing data that only has the UE4 version
### EditorDomain
- We do not need to version the format, we can just invalidate existing editor domain entries via EditorDomainVersion
### EditorServer
- When reporting that a package is too old we report the UE4 version as that is the only version that can be older than VER_UE4_OLDEST_LOADABLE_PACKAGE
- When reporting that a package is too new it can be either the UE4 or the UE5 version so we print them together "UE4Ver|UE5Ver"
### ContentCommandlets
- The min and max resave versions have been kept as a single value, you will not be able to resave against different UE4 and UE5 versions at the same time. It doesn't seem like a useful feature and would greatly increase the complexity of the code.
- We will also only report the file version as a single value.
### ManifestUObject
- This class was setting an older obsolete version on purpose to try and maintain compatibility with older clients so we need to provide a way to create an older UE4 only version that will leave the UE5 version as unset.
### NetworkPlatformFile
- I was unable to test the code path in FNetworkPlatformFile::ProcessServerCachedFilesResponse as I am unsure how to run the game in a mode that will actually use it.
- When reading an older "CookedVersion.txt" that was saved with a single version, the reads will fail and this will count as a version change in the code so that all of the existing files will be deleted. The existing code would not give the user a log message when this happens and given the very small time window where this might happen caused by this change I have opted to leave this alone and not add any additional logging.
- If we do detect a version mismatch we will still only log the version number as a single version.
### CookOnTheFlyServer
- We now add each version number to the IniVersionMap rather than merge the version and license version as a key/value pair. This allows us to a) use both the UE4 and UE5 version numbers b) we now log a warning that the version values don't match when it is changed, previously since it was a key value we would log a warning about an additional setting instead.
-- I also added "vs" to the log message when values are mismatched to make the space between the two values being printed clearer.
#ROBOMERGE-OWNER: paul.chipchase
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17549459 via CL 17550236 via CL 17550238 via CL 17550582
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17550583 by paul chipchase in ue5-release-engine-test branch]
1) Functions added for LiveCoding have been removed and the data is now just a static.
2) Consolidated defered registration into a single routine. This includes CppStructOps registration where every routine was over 100 bytes in length.
3) Removed the extra argument in function registration to avoid 7 bytes on windows
4) Fixed a typo in a macro name.
#b none
#rnx
#ROBOMERGE-AUTHOR: tim.smith
#ROBOMERGE-SOURCE: CL 17490728 via CL 17490835 via CL 17490836 via CL 17490837
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17490838 by tim smith in ue5-release-engine-test branch]
