Commit Graph

329 Commits

Author SHA1 Message Date
paul chipchase
77c12cf814 Virtualized bulkdata is now know as EditorBulkData
#rb PJ.Kack
#rnx
#jira UE-139708
#preflight 61eaaedfc92021e535b6d1e0

- The goal is to show a clear separation between the bulkdata object and the virtualization system.
- NOTE that all of these classes were added during 5.0 development so we are just renaming and moving them, no deprecation paths.
- Renamed FVirtualizedUntypedBulkData to FEditorBulkData
-- Also removed the derived templated types, they were never actually used
- Renamed FVirtualizedBulkDataReader to FEditorBulkDataReader
- Renamed FVirtualizedBulkDataWriter to FEditorBulkDataWriter
- Moved the renamed classes to the UE::Serialization namespace from UE::Virtualization
- Renamed the files of the renamed classes where required.
- Replaced use of LogVirtualization with LogSerialization.
- Renamed defines prefixed with VBD_* to UE_ and make sure they are undefed by the end of the cpp
- Edited unit tests to show up under "System.CoreUObject.Serialization.EditorBulkData.*"

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18688778 in //UE5/Release-5.0/... via CL 18688787 via CL 18688810
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18688823 by paul chipchase in ue5-main branch]
2022-01-21 09:18:53 -05:00
marc audy
0af59ec276 AsyncLoading2:
Add support for importing uncooked source packages from cooked packages
#rnx
#jira UE-136265, FORT-426538
#rb pj.kack
#preflight 61e6ff6a3778a195debba826

#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18655480 via CL 18664834 via CL 18665656 via CL 18665916 via CL 18667722 via CL 18667764 via CL 18668174
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)

[CL 18668352 by marc audy in ue5-main branch]
2022-01-19 18:28:42 -05:00
paul chipchase
29009a6598 Add code comment to explain why FPayloadToc/FTocEntry have not yet been removed from the code base.
#rb trivial
#rnx
#preflight 61e7f90a843acf1b24204678

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18656840 in //UE5/Release-5.0/... via CL 18656845 via CL 18656859
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18656865 by paul chipchase in ue5-main branch]
2022-01-19 07:02:33 -05:00
paul chipchase
62007de5f7 Removed USE_BULKDATA_STREAMING_TOKEN from the bulkdata API
#rb CarlMagnus.Nordin, pj.kack
#rnx
#jira UE-133429
#preflight 61e7c020b875abaf33179ccc

### Token Removal
- Now that we support the IoStore in the editor there is no reason to keep the bulkdata streaming token as both the editor and runtime systems can use the same code paths.
- Existing use of USE_BULKDATA_STREAMING_TOKEN will now give deprecation warnings however code should continue to work as long as STREAMINGTOKEN_PARAM was correctly used.
- ::CreateStreamingToken has been removed from the bulkdata api, but any calls to this should've been wrapped with #if USE_BULKDATA_STREAMING_TOKEN

### API Fixes
- We can remove all code that relied on USE_BULKDATA_STREAMING_TOKEN being == 1
- LANDSCAPE_LOD_STREAMING_USE_TOKEN required USE_BULKDATA_STREAMING_TOKEN to be enabled, so we can now assume that it is always 0 and remove the associated code paths.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18656055 in //UE5/Release-5.0/... via CL 18656061 via CL 18656063
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18656065 by paul chipchase in ue5-main branch]
2022-01-19 04:40:14 -05:00
paul chipchase
73afc77555 Clean up virtualized bulkdata serialization and allow the package trailer to contain payloads with different access types.
#rb PJ.Kack
#jira UE-139348
#rnx
#preflight 61e6742a3778a195dea02199

### PackageTrailer
- We no longer store the access mode for all payloads in the trailer. Instead each payload entry in the lookup table will store the access mode for that specific payload. This allow us to store many payloads of different access types in the same trailer.
- Note that this format change is currently versioned (going from 0 to 1) and is compatible with older version.
-- To allow this, the << operator has been removed from FLookupTableEntry and replaced with a ::Serialize method that takes the version of the package trailer header as an input so that it can run it's own versioning checks.
- When loading trailers with version 0 we apply the access mode to all non virtualized payload entries in the lookup table to reproduce the single access mode for the entire trailer.
- A FPackageTrailerBuilder can now be created directly from an existing FPackageTrailer that will reference all of the trailers local payloads via the new static method ::CreateReferenceToTrailer
- Removed FPackageTrailer::TrySave as it is no longer used when creating a trailer for the editor domain.
- Renamed FPackageTrailer::LoadPayload to ::LoadLocalPayload to make it clear the type of payload that can be loaded.
-- Someday the package trailer should be able to load payloads of any type.

### VirtualizedBulkData Serialization Changes
- We now record if the bulkdata's payload is stored in a package trailer or not via the 'StoredInPackageTrailer' flag, which makes it easier to identify critical errors in serialization (the trailer is missing for example)
-- Storing this info as a flag is now possible as the design for the trailer has changed and a fully virtualized package will still contain a trailer.
- We now only write out OffsetInFile if the payload is not stored in a FPackageTrailer, if the bulkdata object does use a package trailer via the new flag then we take the offset from the trailer instead.
- We also no longer write out a path when saving a payload as referenced to the editor domain, it is assumed that the trailer will know where the workspace domain data is (namely the owning package path)
-- Due to the 'StoredInPackageTrailer' only being added now, this means that packages already saved with a package trailer will work with this code logic as those packages will have serialized the offsets to disk, but won't have the new flag and so will serialize the offsets when loading until re-saved. (worth noting that nobody should encounter this as the package trailer system is disabled by default)
- The loading code is a lot easier to read now and a lot of the branches have been removed although we still have a few extra checks for older formats that might have been saved during development.
- When loading the only real consideration we have is if the payload is in the trailer or not.
- Future Work:
-- If we are not writing to the trailer then we use the legacy serialization system which appends the bulkdata to the end of the package. This is used in two cases, 1) When the package trailer system is disabled 2) If the payload is updated in postload and we are writing out the package to the editor domain as the package trailer does not support writing out local payloads outside of the workspace domain at the moment. The trailer should handle this itself and not rely on the legacy serialization system so that we can remove it.
-- Note that if a payload is stored by reference, we are not adding it explicitly to the payload trailer and we are relying on the trailer builder already containing the reference information when it is created in SavePackage. We do however run an assert to ensure that the reference does exist in the trailer but it might be worth revisiting this.

### VirtualizedBulkData Misc
- The member 'PackageSegment' is now initialized in the constructor to EPackageSegment::Header.
- Moved ::GetPackagePathFromOwner from being a method to a utility function as it accessed no members.
- Validating the payload for saving during serialization has been moved from ::Serialize to it's own method ::TryPayloadValidationForSaving.

### PackageSubmissionChecks
- Added an error message if we detect referenced payloads in a trailer we are trying to virtualize as this condition should never occur. If we encounter it, we give a user facing error message and fail the submit process. It might be worth demoting this to an assert instead as it is not an error the user is expected to see.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637906 in //UE5/Release-5.0/... via CL 18637926 via CL 18637929
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18637932 by paul chipchase in ue5-main branch]
2022-01-18 05:41:44 -05:00
paul chipchase
4e18264103 Clean up the logging when a virtualized payload fails to load in some way to prevent multiple duplicate log entries.
#rb PJ.Kack
#jira UE-138715
#rnx
#preflight 61de7c826a076ddb53d85e76

- We now only validate the payloads contents when ::GetDataInternal is called. This prevents duplicate log messages being raised.
-- Duplicate calls to IsValid have been removed.
-- This also means we don't need ::GetCorruptedPayloadErrorMsgForLoad anymore as it is only used in a single place.
- Renamed ::IsValid to ::IsDataValid to make it clearer that we are checking the contents of the payload.
--  The utility no longer checks if the payload fails to load (size should be >0 but the payload is null) but only checks if the payload is corrupted or does not match the expected content hash.
--- This is because the payload failing to load is an error, but it is something that we expect to see happen on occasion, perhaps when the network is flaky etc. However the wrong or corrupted data being loaded is not expected and so needs to be reported as it's own specific error.
--- Splitting these types of failures up, should allow us to consider enabling VBD_CORRUPTED_PAYLOAD_IS_FATAL again. Although now we prevent the saving of packages with corrupted payloads we probably don't need to treat a corrupted payload as a reason to terminate the editor.
- Removed warning from FVirtualizedUntypedBulkData::PullData when the payload failed to pull as IVirtualizationSystem will have already logged an error and the warning was incorrect as it was displaying the payload identifier as the bulkdata guid.
- VBD_CORRUPTED_DATA_SEVERITY is now set as an Error if VBD_CORRUPTED_PAYLOAD_IS_FATAL is disabled.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18581869 in //UE5/Release-5.0/... via CL 18581873 via CL 18581874
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)

[CL 18581878 by paul chipchase in ue5-main branch]
2022-01-12 02:21:01 -05:00
paul chipchase
295add5dae The package trailer system can now work in projects with the editor domain enable as well.
#rb Matt.Peters
#rnx
#jira UE-136891
#preflight 61dbf8eeaf44b43628343f00

- Now when saving a package to the editor domain we will load the existing package trailer (if one exists) from the workspace domain and then create a new trailer that references those payloads rather than store them locally.
- VirtualizedBulkData serialization is still overly complicated and this code adds more branches there. If we stopped the VBD objects from serializing their offsets at all and rely entirely on the package trailer to know where the payloads are stored then we could simplify this, but that requires the package trailer system to be turned on so will be done as future work.

### VirtualizedBulkData
- Split bAllowVirtualizationOnSave into a second variable. The original now controls if we push the payload on save (off by default and not something we are likely to keep) and the new variable (bForceRehydrationOnSave) controls if we must pull payloads when the package is saved (on by default) which will be moved to a config value once properly supported.
-- The split was done to make the logic in ::Serialize clearer
- We now correctly Register/Unregister the bulkdata object when ::DetachFromDisk is called. This prevents the bulkdata registry from trying to access the workspace domain file after it has been invalidated.
-- In addition we now attach to the FLinkerLoader's cachable archive (if there is one) when using the ::IsReferencingByPackagePath branch of serialization.
- When saving the bulkdata as part of an editor domain save we can keep the virtualization flag as we don't have to worry about the workspace domain file changing without the editor domain version being rebuilt anyway.

### SavePackage
- If the package is part of a procedural save and not a cook we will try to load the trailer from the workspace domain and then convert it to a referenced trailer (which contains the correct info to load the payload from the workspace domain package file but does not contain the payloads itself) which is then appended to the package.

### PackageTrailer
- Removed the fatal error if a project has both the editor domain and the package trailer system enabled.
- Added a method ::CreateReference that creates a trailer that references the payloads in an existing trailer.
- Added a method ::TrySave, allowing direct writing of a trailer to an archive.
-- Now that we have TrySave and TryLoad it probably makes sense to add the << operator as the public interface and move the methods to be private.
- FFooter and FHeader have their own serialization overloads now rather than relying on the higher level structures to do the work.
- When building a new trailer we now create a new FPackageTrailer structure and then write that out in one go, rather than writing each value as it is calculated to make the code easier to read.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18557778 in //UE5/Release-5.0/... via CL 18557792
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18557812 by paul chipchase in ue5-release-engine-test branch]
2022-01-10 05:11:44 -05:00
devin doucette
35393bbb2b DDC: Split FPayload into separate FValue and FValueId types
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.

#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18531864 by devin doucette in ue5-release-engine-test branch]
2022-01-06 11:05:57 -05:00
matt peters
dbdb14e0c7 EditorDomain: Prevent saving of packages that require resaved bulk data.
#preflight 61d37a7c81634832b0e84c77

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18504368 in //UE5/Release-5.0/... via CL 18504381
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18504396 by matt peters in ue5-release-engine-test branch]
2022-01-03 17:56:37 -05:00
paul chipchase
9dd885f6bb A small clean up on how the package trailer is generated.
#rb PJ.Kack
#rnx
#preflight 61b79447f807c623046e91a5

### FPackageTrailer
- FPackageTrailerBuilder::Create now renamed to ::CreateFromTrailer to make it a bit clearer.
- Added 'AccessMode' to the file format documentation.
- FPackageTrailerBuilder no longer tracks infomation about the trailer once it is written to disk. Instead when writing to disk we will create a valid FPackageTrailer as part of the serialization. This will be passed to callbacks and can be queried by virtualized bulkdata.
-- Since the builder no longer tracks data after creating the trailer, we can remove checks from methods like ::AddPayload.
-- Removed FPackageTrailerBuilder::FindPayloadOffset and added a version to FPackageTrailer instead.
- Changed a number of methods to take FPayloadId by reference rather than by value.

### VirtualizedBulkData
- Replaced areas of code responsible for updating previously written out values with the utility function ::UpdateArchiveData to help with code readability.

### Misc
- FLinkerSave no longer creates a FPackageTrailerBuilder by default, instead one should be added to the FLinkerSave if we want to build a trailer for the package being saved.
- FLinkerLoad now considers 0 to be an invalid value for PayloadTocOffset in addition to INDEX_NONE, so we now check if the value is > than 0. This is because when FPackageFileSummary objects are created, all members are set to zero and we know that a FPackageTrailer cannot be 0 and the rest of the package still exist so we can safely consider it invalid.
- SavePackage/SavePackage2 now create the FPackageTrailerBuilder when the linker is created if the package trailer feature is enabled.
-- We now always call BuildAndWriteTrailer and then only skip the creation of the trailer if the builder is missing or conditions are not met.
-- If we do not write out the trailer, we make sure to set PayloadTocOffset in the FPackageFileSummary to INDEX_NONE as it is clearer than leaving it at the default 0.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18453324 in //UE5/Release-5.0/... via CL 18453328
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v898-18417669)

[CL 18453333 by paul chipchase in ue5-release-engine-test branch]
2021-12-14 06:30:37 -05:00
marc audy
d039fb8d42 Fix pushing the wrong object in to the replaced references map. Previously the replaced object was stored, but it should be the object that has the property for which the object is replaced.
Removed unnecessary FArchiveReplaceFieldReferences parameters.
#rb Richard.TalbotWatkin

#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 18432250 in //UE5/Release-5.0/... via CL 18435402
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18435893 by marc audy in ue5-release-engine-test branch]
2021-12-10 18:06:14 -05:00
matt peters
7db9547a14 VirtualizedBulkData: EditorDomain support for loading any format by reference from WorkspaceDomain. Previously this was only supported for legacy WorkspaceDomain packages, but we also need it for packages that have been resaved and use VirtualizedBulkData.
Also fix two other minor issues: Make Move constructor available for FIntVirtualizedBulkData, and make Register public so UObjects can notify the registry that they own a BulkData.
#rb Paul.Chipchase
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18431598 in //UE5/Release-5.0/... via CL 18435352
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18435594 by matt peters in ue5-release-engine-test branch]
2021-12-10 17:50:18 -05:00
marc audy
ca10aea2a1 Update FArchiveReplaceOrClearExternalReferences to be able to take EArchiveReplaceObjectFlags consistent with parent class
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 18427792 in //UE5/Release-5.0/... via CL 18427811
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18427827 by marc audy in ue5-release-engine-test branch]
2021-12-09 20:15:42 -05:00
paul chipchase
152887cb45 Add an experimental feature to content virtualization to allow some bulkdata objects to opt out of being virtualized.
#rb PJ.Kack
#rnx
#preflight 61a8d6d8a7179bfa550218af

- This is an experimental feature being used during testing and development, it is not intended to be used by a shippable project.
-- The code is all wrapped by a single define, UE_ENABLE_VIRTUALIZATION_TOGGLE, making it easy to disable and remove
-- The actual method called on bulkdata, ::SetVirtualizationOptOut is being submitted already deprecated to prevent it's accidental use.
- The use case is to allow projects to opt out of virtualizing static meshes as they are more fragile (i.e. hard crashes) when their payloads cannot be accessed.
- Even when ::SetVirtualizationOptOut is called the feature will not work unless  [Core.System.Experimental]AllowVirtualizationOptOut=True is set in the config file. This allows projects to opt into this.
- When a bulkdata is opting out, we simply send the payload to be stored at the end of the package like we used to, rather than send it to the package trailer. This way we do not need to store any additional state in the trailer to track if a payload is allowed to be virtualized or not. Since this feature is throw away I did not want to make any data format changes.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18352158 in //UE5/Release-5.0/... via CL 18352163
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18352180 by paul chipchase in ue5-release-engine-test branch]
2021-12-02 10:18:37 -05:00
marc audy
a598bd7bad FArchiveReplaceObjectRef (and subclasses) now pass parameters via flags enum rather than long list of bools
FArchiveReplaceObjectRefBase no longer tracks replaced references by default. GetReferencedResources will ensure if archiver was not run with TrackReplacedReferences.
#rb Michael.Noland
#preflight 61a7c89c1a368fd6038f4b79

#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 18343619 in //UE5/Release-5.0/... via CL 18343639
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18343679 by marc audy in ue5-release-engine-test branch]
2021-12-01 15:20:47 -05:00
paul chipchase
ddff3a39f5 Reworked the virtualization system so that we only virtualize a package when it is submitted to source control, which makes the overall system much less dangerous.
#rb PJ.Kack
#rnx
#preflight 61a4b235405273b2c3daa7c7

### Virtualization
- The idea is to move from our current set up, where the virtualization happens when a package is saved and the payloads are saved to a local virtualization storage system with the final push to persistent storage occurring when the package is submitted to a system where we save payloads into the package file, but move them to persistent virtualization storage when the package file is submitted.
- The main advantage is that if someone submits a package file to source control without virtualizing it, we don't have to worry that others might not be able to load the package, the worst case scenario is that data sizes get bigger until the package is virtualized again.
- This is only the first pass, in the future we can do further optimizations, like not storing payloads locally if we know that they are already in the persistent storage system etc.
- In order to keep the virtualization process simple we want to be able to do it without needing to re-save the package, so to this end we will storage the payloads at the end of the package file as before, but instead of storing them inside of the package file format we will use a new container, the FPackageTrailer that is appended to the package file instead.
- In theory this will make future sidecar work easier as the trailer can just be moved to the sidecar file as as long as we know where to find the trailer things will just work (tm)
- For now VBD will continue to serialize it's offset/size to disk and use those values to read the payload directly when not virtualizing. Ideally we'd use the trailer but this will help reduce the risk.

### Current Issues
- The system does not work with the editor domain and has minor issues with text based assets
- The trailer system is disabled by default via the config system [Core]UsePackageTrailer=False and can then be enabled on select test projects until fully functional.

### PackageTrailer
- The trailer is split into two parts FPackageTrailerBuilder and FPackageTrailer
- FPackageTrailerBuilder is used when building the trailer during package save and FPackageTrailer is the structure we load and actually use.
- When saving the trailer we try to avoid using containers and such so that we can try to keep the specification for the file format very clear so that licensees can build their own tools for the virtualization process if they so desire.
- the header file contains a breakdown of the new format.

### VirtualizedBulkData
- FPayloadToc is now only used by the sidecar experimental feature. In a future update the sidecar will be changed to basically be the package trailer but stored in it's own file.
- FindPayloadsInPackageFile has been moved to PackageTrailer.h as it has a closer association with that code.
- Added a new method, ::LoadFromPackageTrailer, to load payloads from the package trailer without needing to know where the payload is on disk. Using this is opt in with the cvar "Serialization.LoadFromTrailer" and is only provided for debugging purposes. Although once this system has matured it will likely become the preferred way to access payloads.
- We no longer allow virtualization of bulkdata payloads on save, but I am not sure if we might want to allow this as an option in the future. So for now the code branch is disabled by the global constant bAllowVirtualizationOnSave.
- Added utility functions ::GetLinkerLoadFromOwner and ::GetTrailerFromOwner for easy access to the LinkerLoad/Trailer from the owning object.
- Added a utility ::UpdateArchiveData for seeking back to a known position in an archive, writing over it, then seeking back, which is fairly common in the package saving code paths. This might be worth moving to Archive.h for general use.
- ::LoadFromPackageFile has been refactored to prefer returning error values early over nested ifs.
- We no longer serialize the EFlags::IsVirtualized flag to disk. Instead it is applied when the package is loaded if we detect that a payload is virtualized and then removed before saved to disk. This allows the virtualization process to occur without resaving the package.
- We currently support the old serialize to disk path which is enabled when the FPackageTrailer system is disabled and the newer system when it is enabled.
- We do however continue to serialize the offset in the package file to disk when saved. Strictly speaking this is not needed as we can look this up from the package trailer when the package is loaded but continuing to serialize it out to disk helps keep the old and newer data format paths working together without adding too much special case code.
-- Once the FPackageTrailer feature is no longer optional we can consider changing this.

### SavePackage/SavePackage2
- Removed the functions that would create the FPayloadToc
- Added new functions to help create the trailer.
- Moved the end of package tag out of the if to it's own scope (still won't run if a package writer exists), the reasoning is to make it clearer where the package format ends and where we can start writing the payload trailer.
- We have checks to make sure that the trailer does not try to write to disk for both text based assets and if the editor domain is enabled. Support for these features will be added later, but since the package trailer system is disabled by default this shouldn't cause problems.

### LinkerLoad
- After we parse the FPackageFileSummary we now also parse the header of the package trailer. In the workspace domain this will incur additional seek costs (which may or not actually seek and invalidate the internal file cache depending on how large the package's exports are) as the trailer is found after the package
-- In the future, the package trailer header can be stored right after the FPackageFileSummary in the editor domain which will remove this overhead.
- By parsing the header of the trailer at this point, virtualized bulkdata objects in the package can use the look up info to determine if their payloads are virtualized or not, and if not then where they reside inside the package file.

### LinkerSave
- We no longer collect a list of all virtualized bulkdata in a package while saving as we no longer generate the FPayloadToc so BulkDataInPackage has been removed.
- We now gather data to be appended to the package trailer via TrailerBuilder.

### DumpPayloadToc
- This command now dumps info based on the package trailer if one can be found.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18307893 in //UE5/Release-5.0/... via CL 18307905
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18307913 by paul chipchase in ue5-release-engine-test branch]
2021-11-29 07:02:24 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
george rolfe
e597577034 StaticClassProvider & StaticStructProvider concepts, and replaced functionality in two dependent files
#jira UE-133246
#preflight 617be019c44dc5000165e827
#rb steve.robb

#ROBOMERGE-AUTHOR: george.rolfe
#ROBOMERGE-SOURCE: CL 17991327 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)

[CL 17991335 by george rolfe in ue5-release-engine-test branch]
2021-10-29 22:22:08 -04:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
paul chipchase
e2ba2ad071 Fix potential crash when syncing textures from within the editor.
#rb Danny Couture
#rnx

### Problem
- The crash would occur when the texture being since contains a reference to another texture (via composite) which is also in need of syncing on the users machine.
- After the files on disk have been updated the current packages are reloaded but the original version of the texture will still be in memory when the composite texture notifies that it has been reloaded causing a texture compilation job with the older now invalid texture.
-- VirtualizedBulkData relied on out of date packages not trying to access their payloads on disk once the file has been changed, but with the above scenario it was possible. There could also be other paths causing this (even if the access of out of date package payloads at this point is a waste of time)

### Fix
- The virtualized bulkdata objects now reference themselves with the FLinkerLoader if the payload is on disk, just as the old bulkdata system did. When file changing operations are invoked, the FLinkerLoader will inform it's referenced bulkdata to load the payloads off disk and hold them in memory.
-- Note that virtualized bulkdata was trying to avoid this memory bloat by design, but if we cannot be certain that the bulkdata will not try to access the payload after the file changes (as was thought) then we must take the payload into memory.
- Added a log message to VBD if we detect that the loaded payload does not match the VBD members expectations. This log message can either be an error or a fatal error depending on the define 'VBD_CORRUPTED_PAYLOAD_IS_FATAL' (defaulted to on)
- The registration to FLinkerLoader can be easily disabled by setting the define 'VBD_ALLOW_LINKERLOADER_ATTACHMENT' to 0

### Outstanding Issues
- Cloned bulkdata objects (via assignment) do not currently register themselves to the FLinkerLoader (the old bulkdata system did not) if we were to add this then we need to make sure that the FLinkerLoader registration is thread safe. This should be done as a separate work item so that we can unblock the current workflows that this changelist addresses.
-- Note that if we did not update cloned bulkdata to work with this system then we could drop the registration and just iterate over the package's bulkdata and force them to load into memory when required. However that would leave the cloned virtual bulkdata in an uncertain state and is not desirable.
- When saving a package, the virtualized bulkdata objects that it contains will change their members so that future payload access can come from the newly saved files, where as the old bulkdata system would force the payload to remain in memory for the rest of the editor session. If we want to keep this then we will need to try and make sure that the newly updated bulkdata can be registered so that it receives future notifications about the file being changed.

#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17648118 via CL 17648147 via CL 17648161 via CL 17648163 via CL 17648175
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17648184 by paul chipchase in ue5-release-engine-test branch]
2021-09-28 13:05:01 -04:00
carlmagnus nordin
51c9794dea PackageStore refactor
* Decouple container concept from IoDispatcher
* Decoiuple PackageStore implementation from AsyncLoading2
* Restore ucas unmount fix that got kist when merrging from UE4
* Fix packages being left in the PackageStiore even after unmounting contaiiners

#rnx
#rb pj.kack, per.larsson
#preflight 61520cc52afc2d0001146ce7

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17641845 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v874-17637634)

[CL 17642353 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-28 04:00:33 -04:00
michael noland
0cda7da7d4 Partial pass of fixing type truncation warnings in CoreUObject
#jira UE-127233
#rb Devin.Doucett

#ROBOMERGE-OWNER: michael.noland
#ROBOMERGE-AUTHOR: michael.noland
#ROBOMERGE-SOURCE: CL 17618902 in //UE5/Release-5.0/... via CL 17618928
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v871-17566257)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17618930 by michael noland in ue5-release-engine-test branch]
2021-09-24 00:31:55 -04:00
devin doucette
7f300d6125 DerivedData: Added a prototype of FDerivedData
Serialization and the FBuildDefinition type of derived data have not been implemented yet. This version is functional in the editor when used with buffers or unstructured cache keys.

#rb Zousar.Shaker
#preflight 614cdb35c2ef060001f6425b

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17614269 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17614332 by devin doucette in ue5-release-engine-test branch]
2021-09-23 17:31:21 -04:00