#rb PJ.Kack
#jira UE-142926
#preflight 621e1f177f2803279b575764
- Now when payloads are submitted, we will submit them from within the project's saved directory (by default) this now works because we write a .p4ignore file to the submission directory which will override any additional ignore file higher up in the directory structure.
-- A new utility function ::CreateSubmissionSessionDirectory was added to make sure that the directory we are submitting from has a p4ignore file. It is added in it's own scope as it is assumed that additional functionality will be added here later.
- If a user wishes to choose a different location to submit from, they can override the environment variable "UE-VirtualizationWorkingDir" to provide a different location. This might be useful if the user does not have a huge amount of spare space on the disk that the project is running from.
- There is also an optional entry to the ini file set up 'SubmitFromTempDir' (false by default) which when set to true will cause the system to attempt to submit the payload files from the machines temp directory instead.
-- There isn't an easy way for users to override this locally, which reduces the usefulness of the option. This might be worth improving in the future although that time would be better spent removing the need for the directory in the first place.
- We now try to find/create the root directory to submit payloads from when the backend is created. Failure at this point is not expected and will prevent the backend from working and so is considered an error.
- Added more logging to track the session guid when submitting and to be clear about where the files are being submitted from.
[CL 19197367 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-143680
#preflight 621776a4335015645d967662
#lockdown mark.lintott
- Strings inside of a LOCTEXT macro must be on a single line.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 19114310 in //UE5/Release-5.0/... via CL 19114561
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19152877 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#jira UE-143686
#rnx
#preflight 62189ed68849a7424f7d2e03
- Workspaces created to submit virtualized assets are now prefixed as "VASubmission-" rather than "MirageSubmission-"
-- Use of the term mirage is deprecated and should be replaced by VA or VirtualizedAsset(s)
- Set the description when we create the workspace to info the user that it was auto generated during the submission of virtualized payloads.
[CL 19143761 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-142748
#rnx
#preflight 620e06d53e74a28b341e72b6
#lockdown aurel.cordonnier
- Turns out that the option did work fine, it just needed to be tested properly, so removed the assert that was guarding against mixed package trailers when submitting virtualized assets.
- Improved verbose logging when a payload is filtered based on payload size.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 19073541 in //UE5/Release-5.0/... via CL 19090494
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19134901 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-134434
#rnx
#preflight 620d1a10742ffef420223b15
- Changing a number of log items from 'Log' to 'Display' so that the info shows up more easily.
- Removed a number of log items that added no useful infomation
- Edited some of the existing log items to be clearer.
[CL 19031890 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#jira UE-133472
#rnx
#preflight 620cef8f492761fc5cb53b05
- Because all of the work is blocking on the main thread adding text to ::EnterProgressFrame to better track progress can be misleading so for now we only display "Virtualizing Assets..."
[CL 19012877 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-133473
#preflight 620bf6ae483ff0ae5ec22257
### VirtualizationManager
- Added EPackageFilterMode that can be set via [Core.ContentVirtualization] in the config file to control the default virtualization behavior.
-- When set to EPackageFilterMode::OptOut(the default) all package paths will virtualize unless excluded by a pattern in UVirtualizationFilterSettings::ExcludePackagePaths.
-- When set to EPackageFilterMode::OptIn then no package path will virtualize unless included by a pattern in UVirtualizationFilterSettings::IncludePackagePaths.
- Added a TRACE_CPUPROFILER_EVENT_SCOPE to the constructor and ::MountBackends to better track the set up time costs.
- Change use of FConfigCacheIni::LoadLocalIniFile to use GConfig, there is no need to load our own.
- Improved verbose logging to show when a payload is rejected via filtering.
- We now early out if all payloads being requested in a push are rejected during validation.
- Renamed the FString overload for ::ShouldVirtualizePackage to ::ShouldVirtualize to make the difference clearer.
- Added support for UVirtualizationFilterSettings::ExcludePackagePaths when filtering
### UVirtualizationFilterSettings
- Now has a new FString array ExcludePackagePaths, which contains the paths/patterns used to force packages to be virtualized when filtering.
[CL 19010992 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-141304
#rnx
#lockdown Mark.Lintott
#preflight 61fd3cccb5092d45ad18c953
- Not all backends gain from pushing payloads in batches. In this case the backend can opt not to override the default implementation which will just iterate over the batch of payloads and then push each one individually.
- However when the default path was used, FPushRequest::Status was not updated after the push was performed, which resulted in each request returning the default (failed) status.
-- The backends that use the default path are not enabled in any of our internal projects so this was not noticed.
- So even though the payloads were pushed, the virtualization process would log and error and block any submit being performed.
-- Subsequent submits of the same packages would work because we would note much earlier that the payloads are already stored in the backend and the broken code path would never run.
- Added fall through comment as required by the coding standards
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18863318 in //UE5/Release-5.0/... via CL 18863334 via CL 18863548
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18863566 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 61fabeb8ad2ae6c3b763a2b9
- Renamed to ::QueryPayloadStatuses since the output data is in the form of an array of FPayloadStatus
- The method now returns EQueryResult rather than bool, which has 'Success' as value 0, all other values indicate an error.
-- This can be expanded in the future if we start to provide more info from backends as to what actually failed.
- FPayloadStatus has been cleaned up so that 'Partial' is not 'FoundPartial' to give a better indication that the payload was found in at least one storage backend, but not all of them.
[CL 18861036 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Devin.Doucette
#jira UE-133497
#rnx
#preflight 61f835491c5ac5523462810a
- A lot of files have been touched but most of this is changing FPayload::IsValid to !FIoHash::IsZero, the main changes are in EditorBulkData/EditorBulkDataTests
- The only difference between FPayloadId and FIoHash is that the former considered the hash of an invalid or empty buffer to be 'invalid' too where as FIoHash would return a valid hash
-- To keep behaviour the same, we only hash payloads in EditorBulkData using a utility method ::HashBuffer which will not hash empty or invalid payloads and return a default FIoHash instead.
-- Unit tests have been extended to prove that the behaviour has not changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18806362 in //UE5/Release-5.0/... via CL 18808527 via CL 18821790
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822154 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-136758
#preflight 61f3e3626b5aea38e5b4cab7
- Package filtering was disabled when the api was changed to take batches of payloads to push to the backend as the context per payload had to be changed to a raw FString rather than FPackagePath.
- We now convert each context to a FPackagePath and if it is valid test against filtering, if it is not valid we assume that the payload did not come from a package.
- Still tempted to move filtering out of virtualization manager entirely and move it to PackageSubmissionChecks, but that can always be done at a later date.
- Retested the filtering system, plus the ability to add filters to the config files for GameFeature plugins.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18776038 in //UE5/Release-5.0/... via CL 18777356 via CL 18777639
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18777650 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-140366
#preflight 61f2a6f752396dbfeeede332
- Add a way to poll the virtualization system to see if pushing to a backend storage type is possible or not.
- If we cannot push to persistent backend storage when submitting packages to source control we just log in verbose mode and return rather than giving an error. Submitting a non-virtualized package is no longer the problem it once was.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18769119 in //UE5/Release-5.0/... via CL 18769125 via CL 18769151
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18769154 by paul chipchase in ue5-main branch]
#rb none
#jira UE-140126
#rnx
#preflight 61efbb8e8e5c8d4eaabb987c
- FScopedStatsCounter can only add a single hit/size in it's destructor, so a for loop calling ::AddHit on it was just overwriting the values each time.
- Use TrackCyclesOnly so that 'Timer' only adds the time taken to the stats.
- Now the for loop adds hits to the stats manually for each successful request.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18720805 in //UE5/Release-5.0/... via CL 18720806 via CL 18720836
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18720839 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-139708
#preflight 61eaaedfc92021e535b6d1e0
- The goal is to show a clear separation between the bulkdata object and the virtualization system.
- NOTE that all of these classes were added during 5.0 development so we are just renaming and moving them, no deprecation paths.
- Renamed FVirtualizedUntypedBulkData to FEditorBulkData
-- Also removed the derived templated types, they were never actually used
- Renamed FVirtualizedBulkDataReader to FEditorBulkDataReader
- Renamed FVirtualizedBulkDataWriter to FEditorBulkDataWriter
- Moved the renamed classes to the UE::Serialization namespace from UE::Virtualization
- Renamed the files of the renamed classes where required.
- Replaced use of LogVirtualization with LogSerialization.
- Renamed defines prefixed with VBD_* to UE_ and make sure they are undefed by the end of the cpp
- Edited unit tests to show up under "System.CoreUObject.Serialization.EditorBulkData.*"
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18688778 in //UE5/Release-5.0/... via CL 18688787 via CL 18688810
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18688823 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-139348
#rnx
#preflight 61e6742a3778a195dea02199
### PackageTrailer
- We no longer store the access mode for all payloads in the trailer. Instead each payload entry in the lookup table will store the access mode for that specific payload. This allow us to store many payloads of different access types in the same trailer.
- Note that this format change is currently versioned (going from 0 to 1) and is compatible with older version.
-- To allow this, the << operator has been removed from FLookupTableEntry and replaced with a ::Serialize method that takes the version of the package trailer header as an input so that it can run it's own versioning checks.
- When loading trailers with version 0 we apply the access mode to all non virtualized payload entries in the lookup table to reproduce the single access mode for the entire trailer.
- A FPackageTrailerBuilder can now be created directly from an existing FPackageTrailer that will reference all of the trailers local payloads via the new static method ::CreateReferenceToTrailer
- Removed FPackageTrailer::TrySave as it is no longer used when creating a trailer for the editor domain.
- Renamed FPackageTrailer::LoadPayload to ::LoadLocalPayload to make it clear the type of payload that can be loaded.
-- Someday the package trailer should be able to load payloads of any type.
### VirtualizedBulkData Serialization Changes
- We now record if the bulkdata's payload is stored in a package trailer or not via the 'StoredInPackageTrailer' flag, which makes it easier to identify critical errors in serialization (the trailer is missing for example)
-- Storing this info as a flag is now possible as the design for the trailer has changed and a fully virtualized package will still contain a trailer.
- We now only write out OffsetInFile if the payload is not stored in a FPackageTrailer, if the bulkdata object does use a package trailer via the new flag then we take the offset from the trailer instead.
- We also no longer write out a path when saving a payload as referenced to the editor domain, it is assumed that the trailer will know where the workspace domain data is (namely the owning package path)
-- Due to the 'StoredInPackageTrailer' only being added now, this means that packages already saved with a package trailer will work with this code logic as those packages will have serialized the offsets to disk, but won't have the new flag and so will serialize the offsets when loading until re-saved. (worth noting that nobody should encounter this as the package trailer system is disabled by default)
- The loading code is a lot easier to read now and a lot of the branches have been removed although we still have a few extra checks for older formats that might have been saved during development.
- When loading the only real consideration we have is if the payload is in the trailer or not.
- Future Work:
-- If we are not writing to the trailer then we use the legacy serialization system which appends the bulkdata to the end of the package. This is used in two cases, 1) When the package trailer system is disabled 2) If the payload is updated in postload and we are writing out the package to the editor domain as the package trailer does not support writing out local payloads outside of the workspace domain at the moment. The trailer should handle this itself and not rely on the legacy serialization system so that we can remove it.
-- Note that if a payload is stored by reference, we are not adding it explicitly to the payload trailer and we are relying on the trailer builder already containing the reference information when it is created in SavePackage. We do however run an assert to ensure that the reference does exist in the trailer but it might be worth revisiting this.
### VirtualizedBulkData Misc
- The member 'PackageSegment' is now initialized in the constructor to EPackageSegment::Header.
- Moved ::GetPackagePathFromOwner from being a method to a utility function as it accessed no members.
- Validating the payload for saving during serialization has been moved from ::Serialize to it's own method ::TryPayloadValidationForSaving.
### PackageSubmissionChecks
- Added an error message if we detect referenced payloads in a trailer we are trying to virtualize as this condition should never occur. If we encounter it, we give a user facing error message and fail the submit process. It might be worth demoting this to an assert instead as it is not an error the user is expected to see.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637906 in //UE5/Release-5.0/... via CL 18637926 via CL 18637929
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637932 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-136612
#rnx
#preflight 61e589b7904123989a1817cc
- If a perforce connection issue is encountered when starting up the editor we will now print an error to the message log and then on the first tick of the editor display a notification to get the users attention. It will no longer result in a fatal error.
- Add a new category "Asset Virtualization" to the message log so that we can start to improve the feedback we give to end users rather than only reporting problems via UE_LOG. This also introduces a new localization text name space.
-- I will need to go over a lot of the virtualization modules error handling and move it to use the message log instead.
- When the source control backend first starts up we now specifically check if the connection works and if not try to give the user feedback on how to fix it.
-- Ideally we'd provide a login dialog for the user to fix the problem in place but due to how early in the start up process we need to be able to access the virtualization backends this is quite tricky.
- In the future we will build the concept of a backends connection into a formalized concept in the system along with better error handling. At this point FSourceControlBackend::OnConnectionError should be moved to general use for all backends. this will allow us to provide much better status info to the user.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637036 in //UE5/Release-5.0/... via CL 18637043 via CL 18637054
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637614 by paul chipchase in ue5-main branch]
