Commit Graph

743 Commits

Author SHA1 Message Date
paul chipchase
21c2777e84 Add an UE5 specific version EUnrealEngineObjectUE5Version to be used for global changes instead of EUnrealEngineObjectUEVersion. By splitting and storing both version numbers we allow for hypothetical future UE4 changes that will not conflict when merged to UE5.
#rb CarlMagnus.Nordin
#rnx
#tests Ran overnight preflights on several platforms, opened/cooked/staged/ran the oldest version of InfiltratorDemo that can be downloaded (4.11)

### ObjectVersion
- Add a new version enum EUnrealEngineObjectUE5Version.
-- This version number starts at 1000 which leaves more than enough for for EUnrealEngineObjectUEVersion to be expanded
- Even though very few changes (if any at all) to EUnrealEngineObjectUE4Version are expected there is a static assert to make sure that EUnrealEngineObjectUEVersion::AUTOMATIC_VERSION never overtakes EUnrealEngineObjectUE5Version::INITIAL_VERSION.
- Add a struct FPackageFileVersion that wraps around the version numbers and is used to store them instead of raw int32 values which was done before. This should make it easier to add new version numbers in the future if we desire (although this will cause problems in places that serialize the struct directly)

### FPackageFileSummary
- Adding a new entry to CurrentLegacyFileVersion at value -8 which shows the UE5 version being added. This lets us make the changes without needing to submit anything to UE4 Main.
- When loading a package that does not have a UE5 version, it will remain at 0.
- Added ::IsFileVersionTooOld and ::IsFileVersionTooNew to replace hardcoded tests in the code base for version validity. This will make it easier to make changes in the future.
- A few months ago most of the accessors of the version number were deprecated in favour of a version that did not contain the Engine number (ie UE4Ver -> UEVer in Archive) but to work with these changes the renamed methods now will return or accept the version as FPackageFileVersion rather than int32.  The old UE4 methods will remain deprecated and direct licensees to use the new methods.

### Archive
- Now stores the version as a FPackageFileVersion rather than int32

### LinkerLoad
- Reports the larger version number if we detect a higher version number than we support. Note that this could cause an issue if the UE4 version is ever raised but helps keep the code simple.

### AssetData
- Need to add a new version here to manage existing data that only has the UE4 version

### EditorDomain
- We do not need to version the format, we can just invalidate existing editor domain entries via EditorDomainVersion

### EditorServer
- When reporting that a package is too old we report the UE4 version as that is the only version that can be older than VER_UE4_OLDEST_LOADABLE_PACKAGE
- When reporting that a package is too new it can be either the UE4 or the UE5 version so we print them together "UE4Ver|UE5Ver"

### ContentCommandlets
- The min and max resave versions have been kept as a single value, you will not be able to resave against different UE4 and UE5 versions at the same time. It doesn't seem like a useful feature and would greatly increase the complexity of the code.
- We will also only report the file version as a single value.

### ManifestUObject
- This class was setting an older obsolete version on purpose to try and maintain compatibility with older clients so we need to provide a way to create an older UE4 only version that will leave the UE5 version as unset.

### NetworkPlatformFile
- I was unable to test the code path in FNetworkPlatformFile::ProcessServerCachedFilesResponse as I am unsure how to run the game in a mode that will actually use it.
- When reading an older "CookedVersion.txt" that was saved with a single version, the reads will fail and this will count as a version change in the code so that all of the existing files will be deleted. The existing code would not give the user a log message when this happens and given the very small time window where this might happen caused by this change I have opted to leave this alone and not add any additional logging.
- If we do detect a version mismatch we will still only log the version number as a single version.

### CookOnTheFlyServer
- We now add each version number to the IniVersionMap rather than merge the version and license version as a key/value pair. This allows us to a) use both the UE4 and UE5 version numbers b) we now log a warning that the version values don't match when it is changed, previously since it was a key value we would log a warning about an additional setting instead.
-- I also added "vs" to the log message when values are mismatched to make the space between the two values being printed clearer.

#ROBOMERGE-OWNER: paul.chipchase
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17549459 via CL 17550236 via CL 17550238 via CL 17550582 via CL 17550583
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v870-17433530)

[CL 17550586 by paul chipchase in ue5-main branch]
2021-09-17 07:05:39 -04:00
carlmagnus nordin
87f7fcf436 Remove PackageIdToLoad field from UPackage in favour of generating the id dynamically from the loaded path
#rnx
#jira UE-121632
#rb pj.kack

#ROBOMERGE-OWNER: carlmagnus.nordin
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17517182 via CL 17517190 via CL 17517193 via CL 17519704 via CL 17536402
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v870-17433530)

[CL 17536403 by carlmagnus nordin in ue5-main branch]
2021-09-16 06:38:50 -04:00
Per Larsson
073f273d8b COTF: Better lock scope when handling cooked packages (server) and moved subscription to cook-on-the-fly messages (client) when package store is initializing
#rb none
#jira none
#rnx

[CL 17469520 by Per Larsson in ue5-main branch]
2021-09-09 05:13:55 -04:00
aurel cordonnier
e3f7878676 Merge from Release-Engine-Test @ 17462327 to UE5/Main
This represents UE4/Main @17430120 and Dev-PerfTest @17437669

[CL 17463546 by aurel cordonnier in ue5-main branch]
2021-09-08 16:42:26 -04:00
pj kack
c82841015a AsyncLoading2: Add back missing fileopenlog generation functionality (unintentionally removed in cl 16481564)
#rb carlmagnus.nordin
#preflight none
#rnx


#ROBOMERGE-SOURCE: CL 17399712
#ROBOMERGE-BOT: (v865-17346139)

[CL 17400415 by pj kack in ue5-main branch]
2021-09-02 06:07:06 -04:00
paul chipchase
3a9cbded79 Improve the documentation for the bulkdata method ::RemoveBulkData to make it clear that calling it will prevent the bulkdata object from reloading payloads from disk in future.
#rb Per.Larsson
#rnx
#preflight 612e03f6677f0e000144b652

- Updated the comments to make the methods intent very clear (unload current payloads and prevent future payloads from being loaded from disk)
- Fix an inconsistency with BD2 running from IoStore, where calling ::RemoveBulkData was not preventing future loads in all cases. We now invalidate the PackageID when running from the IoStore in addition to unregistering the filepath token when running from pak files.
- No longer clear the flags when calling ::RemoveBulkData to be consistent with BD1.
- We might want to consider deprecating this method name and using something like ::Reset instead, but that would imply that we would reset all members and I do not see the value in making this much larger change if we do not need to do so.

[CL 17366830 by paul chipchase in ue5-main branch]
2021-08-31 08:59:42 -04:00
CarlMagnus Nordin
766351a253 Implemented a stripped down version of FPackagePath for use outside of the editor in order to save memory
#review-17269963
#rnx
#jira UE-120698

[CL 17300999 by CarlMagnus Nordin in ue5-main branch]
2021-08-25 08:01:25 -04:00
charles derousiers
073aecd9bf Fix StartAsyncLoading() with IODispatcher.
This fix was originally made by Paul Chipchase.

#rb Paul.Chipchase
#preflight 61249d24cc11eb0001ac4ef2


#ROBOMERGE-SOURCE: CL 17282377
#ROBOMERGE-BOT: (v858-17259218)

[CL 17282388 by charles derousiers in ue5-main branch]
2021-08-24 04:02:26 -04:00
CarlMagnus Nordin
932a5242b0 AsyncLoading2:
Remove limit on pending package size since request buffers are no longer allocated up front.
Rely on the iodispatcher to sort requests instead of using the global load order for each package.
No longer store export bundle size and load order in the package store.
#jira FROST-4677
#rb per.larsson
#rnx

[CL 17267090 by CarlMagnus Nordin in ue5-main branch]
2021-08-23 10:50:03 -04:00
CarlMagnus Nordin
66647c26a0 AsyncLoading2: Fixed issue with IoBuffers being kept alive longer than intended
#rnx
#jira none
#rb per.larsson

[CL 17236848 by CarlMagnus Nordin in ue5-main branch]
2021-08-19 13:54:20 -04:00
Andriy Tylychko
2a295eb685 deprecated FTicker and family and replaced by thread-safe FTSTicker
#jira UE-120090
#rb francis.hurteau

[CL 17176325 by Andriy Tylychko in ue5-main branch]
2021-08-16 11:05:18 -04:00
CarlMagnus Nordin
8c9034fa33 AsyncLoading2: Fixed memory leak in package request queue
#rb pj.kack
#rnx
#jira UE-121014

[CL 17147784 by CarlMagnus Nordin in ue5-main branch]
2021-08-12 02:36:33 -04:00
CarlMagnus Nordin
0f03d0fd62 AsyncLoading2:
Fixed edge case where imported packages failed to register as dependencies if they had serialized all their exports but not postloaded them yet
#rb pj.kack
#rnx
#jira UE-120987

[CL 17099821 by CarlMagnus Nordin in ue5-main branch]
2021-08-09 07:49:28 -04:00
Martin Ridgers
13b3376b51 Load and platform-file tracing migrated away from attachments.
#rb cm
#rnx

[CL 17048386 by Martin Ridgers in ue5-main branch]
2021-08-04 08:07:11 -04:00
CarlMagnus Nordin
ae52f30549 AsyncLoading2:
Ensure that all reachable packages have all their exports serialized before starting post load instead of relying on the serialization order. This is still required in some cases.
Apply the same logic to ensure that all reachable packages are post loaded before finishing loading

#robomerge[bot1] dev-enginemerge-test
#rb pj.kack
#rnx
#jira none

[CL 17027677 by CarlMagnus Nordin in ue5-main branch]
2021-08-03 06:03:19 -04:00
Matt Peters
26eb624010 #jira UE-120709
AsyncLoading with EditorDomain: batch up the callbacks in FAsyncLoadingThread::ProcessLoadedPackages so that the EditorDomain callback that saves the just-loaded package does not cause a stack overflow by adding 14 functions on the stack for each pending package discovered in the FlushAsyncLoading call.
#rb Francis.Hurteau
#rnx

[CL 17017812 by Matt Peters in ue5-main branch]
2021-08-02 13:22:52 -04:00
CarlMagnus Nordin
23761e1901 IoStore:
Add more flexibility to the redirects/localization functionality by moving redirect information away from the package summaries and import tables
Changed import references from global import hashes to package local export hashes

#robomerge[bot1] dev-enginemerge-test
#rb pj.kack
#rnx
#jira none

[CL 17014898 by CarlMagnus Nordin in ue5-main branch]
2021-08-02 07:21:02 -04:00
Matt Peters
686e35b862 Generalize IPackageResourceManager::OnEndLoad into FCoreUObjectDelegates::OnEndLoadPackage for use by the BulkDataRegistry, and add UPackage::bHasBeenEndLoaded for BulkDataRegistry to know when a package has finished its initial load serialization.
Add OnEndLoad calls for Async loaded packages, since the editor sometimes uses async loads.
#rb Francis.Hurteau
#rnx

[CL 16861817 by Matt Peters in ue5-main branch]
2021-07-15 09:48:56 -04:00
Matt Peters
07a0af0f5b BulkDataRegistry
Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.

#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker

[CL 16768772 by Matt Peters in ue5-main branch]
2021-06-24 00:50:53 -04:00
aurel cordonnier
d17d20ca36 Merge from Release-Engine-Test @ 16758890 to UE5/Main
This represents UE4/Main @ 16738161 and Dev-PerfTest @ 16737719 (and Release-17.00 @ 16658211)

[CL 16763350 by aurel cordonnier in ue5-main branch]
2021-06-23 17:51:32 -04:00
Per Larsson
03c092dbba Fix non unity Crash reporter compile error
#rb none
#rnx

[CL 16725383 by Per Larsson in ue5-main branch]
2021-06-21 03:37:23 -04:00
Per Larsson
c3b3499a99 Remove dependency to CookOnTheFly module from CoreUObject and Launch when compiling programs
#rb Paul.Chipchase

[CL 16725292 by Per Larsson in ue5-main branch]
2021-06-21 02:42:49 -04:00
Zousar Shaker
cf116088ae Integrating //UE5/Dev-Cooker @ 16678003 to //UE5/Main (Zousar.Shaker-YEG-0943-Quaternary)
Non-DevIteration_ShooterGame

[CL 16678907 by Zousar Shaker in ue5-main branch]
2021-06-15 16:36:57 -04:00
paul chipchase
a5173e295e Fix a bug where bulkdata that was compressed during the content cook was returning the compressed data rather than uncompressed data when the IoDispatcher was active.
- This has gone so long without being noticed as it is not expected that bulkdata will be compressed during a cook, as the pakfiles themselves will be compressed.
- InternalLoadFromIoStore now only loads via the IoDispatcher directly to the destination buffer if the data is in uncompressed format.
- If the data is going to be compressed we let the IoDispatcher allocate and return the compressed buffer and bulkdata will decompress that to the destination buffer instead.

#rb Per.Larsson, CM.Nordin
#fyi Charles.deRousiers
#preflight 60c83e2143ec380001e2ad1b

[CL 16670240 by paul chipchase in ue5-main branch]
2021-06-15 03:13:34 -04:00
Francis Hurteau
0729641dc7 Fixed crash in BulkData when detaching an already detached bulkdata
#rb Danny.Couture
#issue 313

[CL 16641407 by Francis Hurteau in ue5-main branch]
2021-06-11 10:45:03 -04:00