Convert BulkData to use a separate file handle (not AttachedAr) when MakeSureBulkDataIsLoaded is called from non-gamethread. This prevents unguarded multithreaded access to AttachedAr.
#preflight 61dc386a5b6f9bf2e6cf6a96
#rb Paul.Chipchase
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18561037 in //UE5/Release-5.0/... via CL 18561055 via CL 18561076
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18561093 by matt peters in ue5-main branch]
Convert BulkData to use a separate file handle (not AttachedAr) when MakeSureBulkDataIsLoaded is called from non-gamethread. This prevents unguarded multithreaded access to AttachedAr.
#preflight 61dc386a5b6f9bf2e6cf6a96
#rb Paul.Chipchase
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18561037 in //UE5/Release-5.0/... via CL 18561055
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18561076 by matt peters in ue5-release-engine-test branch]
#rb Matt.Peters
#rnx
#jira UE-136891
#preflight 61dbf8eeaf44b43628343f00
- Now when saving a package to the editor domain we will load the existing package trailer (if one exists) from the workspace domain and then create a new trailer that references those payloads rather than store them locally.
- VirtualizedBulkData serialization is still overly complicated and this code adds more branches there. If we stopped the VBD objects from serializing their offsets at all and rely entirely on the package trailer to know where the payloads are stored then we could simplify this, but that requires the package trailer system to be turned on so will be done as future work.
### VirtualizedBulkData
- Split bAllowVirtualizationOnSave into a second variable. The original now controls if we push the payload on save (off by default and not something we are likely to keep) and the new variable (bForceRehydrationOnSave) controls if we must pull payloads when the package is saved (on by default) which will be moved to a config value once properly supported.
-- The split was done to make the logic in ::Serialize clearer
- We now correctly Register/Unregister the bulkdata object when ::DetachFromDisk is called. This prevents the bulkdata registry from trying to access the workspace domain file after it has been invalidated.
-- In addition we now attach to the FLinkerLoader's cachable archive (if there is one) when using the ::IsReferencingByPackagePath branch of serialization.
- When saving the bulkdata as part of an editor domain save we can keep the virtualization flag as we don't have to worry about the workspace domain file changing without the editor domain version being rebuilt anyway.
### SavePackage
- If the package is part of a procedural save and not a cook we will try to load the trailer from the workspace domain and then convert it to a referenced trailer (which contains the correct info to load the payload from the workspace domain package file but does not contain the payloads itself) which is then appended to the package.
### PackageTrailer
- Removed the fatal error if a project has both the editor domain and the package trailer system enabled.
- Added a method ::CreateReference that creates a trailer that references the payloads in an existing trailer.
- Added a method ::TrySave, allowing direct writing of a trailer to an archive.
-- Now that we have TrySave and TryLoad it probably makes sense to add the << operator as the public interface and move the methods to be private.
- FFooter and FHeader have their own serialization overloads now rather than relying on the higher level structures to do the work.
- When building a new trailer we now create a new FPackageTrailer structure and then write that out in one go, rather than writing each value as it is calculated to make the code easier to read.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18557778 in //UE5/Release-5.0/... via CL 18557792 via CL 18557812
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18557816 by paul chipchase in ue5-main branch]
#rb Matt.Peters
#rnx
#jira UE-136891
#preflight 61dbf8eeaf44b43628343f00
- Now when saving a package to the editor domain we will load the existing package trailer (if one exists) from the workspace domain and then create a new trailer that references those payloads rather than store them locally.
- VirtualizedBulkData serialization is still overly complicated and this code adds more branches there. If we stopped the VBD objects from serializing their offsets at all and rely entirely on the package trailer to know where the payloads are stored then we could simplify this, but that requires the package trailer system to be turned on so will be done as future work.
### VirtualizedBulkData
- Split bAllowVirtualizationOnSave into a second variable. The original now controls if we push the payload on save (off by default and not something we are likely to keep) and the new variable (bForceRehydrationOnSave) controls if we must pull payloads when the package is saved (on by default) which will be moved to a config value once properly supported.
-- The split was done to make the logic in ::Serialize clearer
- We now correctly Register/Unregister the bulkdata object when ::DetachFromDisk is called. This prevents the bulkdata registry from trying to access the workspace domain file after it has been invalidated.
-- In addition we now attach to the FLinkerLoader's cachable archive (if there is one) when using the ::IsReferencingByPackagePath branch of serialization.
- When saving the bulkdata as part of an editor domain save we can keep the virtualization flag as we don't have to worry about the workspace domain file changing without the editor domain version being rebuilt anyway.
### SavePackage
- If the package is part of a procedural save and not a cook we will try to load the trailer from the workspace domain and then convert it to a referenced trailer (which contains the correct info to load the payload from the workspace domain package file but does not contain the payloads itself) which is then appended to the package.
### PackageTrailer
- Removed the fatal error if a project has both the editor domain and the package trailer system enabled.
- Added a method ::CreateReference that creates a trailer that references the payloads in an existing trailer.
- Added a method ::TrySave, allowing direct writing of a trailer to an archive.
-- Now that we have TrySave and TryLoad it probably makes sense to add the << operator as the public interface and move the methods to be private.
- FFooter and FHeader have their own serialization overloads now rather than relying on the higher level structures to do the work.
- When building a new trailer we now create a new FPackageTrailer structure and then write that out in one go, rather than writing each value as it is calculated to make the code easier to read.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18557778 in //UE5/Release-5.0/... via CL 18557792
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18557812 by paul chipchase in ue5-release-engine-test branch]
Fixed issue where SavePackage2 would fail to gather custom versions for CDOs
* FPackageHarvester::ProcessExport() is using UClass::SerializeDefaultObject() which will not call CDO->Serialize()
[at]francis.hurteau
#localization none
#tests Resaved BP assets packages, child classes of AWaterBodyIsland - those now have a FWaterCustomVersion as expected
#preferred_whitelister bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 18550470 via CL 18550475 via CL 18550481 via CL 18550486 via CL 18550487 via CL 18550665 via CL 18550671 via CL 18550677
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18550685 by bob tellez in ue5-main branch]
Fixed issue where SavePackage2 would fail to gather custom versions for CDOs
* FPackageHarvester::ProcessExport() is using UClass::SerializeDefaultObject() which will not call CDO->Serialize()
[at]francis.hurteau
#localization none
#tests Resaved BP assets packages, child classes of AWaterBodyIsland - those now have a FWaterCustomVersion as expected
#preferred_whitelister bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 18550470 via CL 18550475 via CL 18550481 via CL 18550486 via CL 18550487 via CL 18550665 via CL 18550671
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18550677 by bob tellez in ue5-release-engine-test branch]
#rb trivial
#jira UE-138459
#preflight 61d888e64c252480ca397e49
- If the asyc read being waited on takes a very long time (1000ms+) then we log a warning that there was a stall. However the warning prints the size of the bulkdata payload to the warning to give the user an idea of if the wait was due to the length of the data or some other reason.
- Calling ::GetBulkDataSize cannot be done in the destructor of the bulkdata object because the method is virtual, which was causing the crash.
- We can however call ::GetBulkDataSizeOnDisk as it is not virtual and gives a more accurate picture of the length of the data being loaded, as it can be smaller than the result of ::GetBulkDataSize if compression is used.
- Switched use of %lld to use INT64_FMT instead.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18548835 in //UE5/Release-5.0/... via CL 18548858 via CL 18548885
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18548901 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-138459
#preflight 61d888e64c252480ca397e49
- If the asyc read being waited on takes a very long time (1000ms+) then we log a warning that there was a stall. However the warning prints the size of the bulkdata payload to the warning to give the user an idea of if the wait was due to the length of the data or some other reason.
- Calling ::GetBulkDataSize cannot be done in the destructor of the bulkdata object because the method is virtual, which was causing the crash.
- We can however call ::GetBulkDataSizeOnDisk as it is not virtual and gives a more accurate picture of the length of the data being loaded, as it can be smaller than the result of ::GetBulkDataSize if compression is used.
- Switched use of %lld to use INT64_FMT instead.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18548835 in //UE5/Release-5.0/... via CL 18548858
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18548885 by paul chipchase in ue5-release-engine-test branch]
Some minor deprecation cleanup
#rb louisphilippe.seguin, jon.nabozny
#ROBOMERGE-AUTHOR: brian.bekich
#ROBOMERGE-SOURCE: CL 18546386 via CL 18546409 via CL 18546424 via CL 18547591 via CL 18547656 via CL 18547712
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18547788 by brian bekich in ue5-main branch]
Some minor deprecation cleanup
#rb louisphilippe.seguin, jon.nabozny
#ROBOMERGE-AUTHOR: brian.bekich
#ROBOMERGE-SOURCE: CL 18546386 via CL 18546409 via CL 18546424 via CL 18547591 via CL 18547656
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18547712 by brian bekich in ue5-release-engine-test branch]
- When enabled, folders are actual objects : modifying a folder won't affect actors (more compliant with OFPA).
- Enabled by default for World Partition and Level Instances.
- Can be enabled on regular levels (experimental feature 'Use Actor Folder Objects').
- If Level uses OFPA, ActorFolders will also be saved in their own package (but will use __ExternalObjects__ root folder).
- In a future refactoring, external actors will also be moved in the same root folder.
- Generalized saving object in an external package (different from its outer package)
- World outliner supports old folders and new actor folder objects : Levels using the actor folder objects will show a root Level node (like Level Instances).
#rb patrick.enfedaque, jeanfrancois.dube
#preflight 61d84b356511bc498e5cb858
#ROBOMERGE-AUTHOR: richard.malo
#ROBOMERGE-SOURCE: CL 18543443 in //UE5/Release-5.0/... via CL 18543482 via CL 18543525
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18543568 by richard malo in ue5-main branch]
- When enabled, folders are actual objects : modifying a folder won't affect actors (more compliant with OFPA).
- Enabled by default for World Partition and Level Instances.
- Can be enabled on regular levels (experimental feature 'Use Actor Folder Objects').
- If Level uses OFPA, ActorFolders will also be saved in their own package (but will use __ExternalObjects__ root folder).
- In a future refactoring, external actors will also be moved in the same root folder.
- Generalized saving object in an external package (different from its outer package)
- World outliner supports old folders and new actor folder objects : Levels using the actor folder objects will show a root Level node (like Level Instances).
#rb patrick.enfedaque, jeanfrancois.dube
#preflight 61d84b356511bc498e5cb858
#ROBOMERGE-AUTHOR: richard.malo
#ROBOMERGE-SOURCE: CL 18543443 in //UE5/Release-5.0/... via CL 18543482
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18543525 by richard malo in ue5-release-engine-test branch]
Having only one Append overload with a single argument makes for clearer error messages. Otherwise, a failure to match Append(CharRangeType) leads to an error that ArgType cannot be converted to CharType.
#rb Zousar.Shaker
#preflight 61d72f3f6511bc498e54c250
#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18533638 in //UE5/Release-5.0/... via CL 18534325 via CL 18534364
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18534540 by devin doucette in ue5-main branch]
Having only one Append overload with a single argument makes for clearer error messages. Otherwise, a failure to match Append(CharRangeType) leads to an error that ArgType cannot be converted to CharType.
#rb Zousar.Shaker
#preflight 61d72f3f6511bc498e54c250
#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18533638 in //UE5/Release-5.0/... via CL 18534325
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18534364 by devin doucette in ue5-release-engine-test branch]
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.
#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856 via CL 18531864
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18531880 by devin doucette in ue5-main branch]
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.
#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18531864 by devin doucette in ue5-release-engine-test branch]
Manually merging CLs 18393342 & 17797050
Additionally patches UClass::ClearSparseClassData to clear all descendent's sparse class data and ensure that the ptr to SparseClassDataStruct is always nulled-out (previously with CL 18393342 this could be left dangling in child classes).
#preflight 61d58694c73b7e46b2662f59
#rb Matt.Peters,Phillip.Kavan
#ROBOMERGE-OWNER: thomas.sarkanen
#ROBOMERGE-AUTHOR: thomas.sarkanen
#ROBOMERGE-SOURCE: CL 18529418 via CL 18529419 via CL 18529421 via CL 18529423 via CL 18529425 via CL 18529657 via CL 18529658 via CL 18529668
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18529670 by thomas sarkanen in ue5-main branch]
Manually merging CLs 18393342 & 17797050
Additionally patches UClass::ClearSparseClassData to clear all descendent's sparse class data and ensure that the ptr to SparseClassDataStruct is always nulled-out (previously with CL 18393342 this could be left dangling in child classes).
#preflight 61d58694c73b7e46b2662f59
#rb Matt.Peters,Phillip.Kavan
#ROBOMERGE-OWNER: thomas.sarkanen
#ROBOMERGE-AUTHOR: thomas.sarkanen
#ROBOMERGE-SOURCE: CL 18529418 via CL 18529419 via CL 18529421 via CL 18529423 via CL 18529425 via CL 18529657 via CL 18529658
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18529668 by thomas sarkanen in ue5-release-engine-test branch]
