#rb mark.lintott
#jira UE-165045
#rnx
- Add a new event (GetAnalyticsRecordEvent) to the UE::Virtualization namespace. Different telemetry systems can hook into this and respond when the virtualization system tries to send a new analytics event.
-- This even accepts an enum class of flags to provide better context for the event. At the moment the only valid flag is to request that the telemetry system flushes the event to make sure that it is sent before returning control to the event. In this way we can know that events are safely recorded before terminating the current process.
- To begin with we are only adding failure events and not recording any successes
-- Send an analytics even when the initial connection to perforce fails, the event should note if the user opted to retry their connection (after entering new details) or if they wanted to skip the check and load the editor anyway.
-- Send an analytics event when a payload fails to pull, the event should note if the user opted to retry to pull the payload or quit the process.
[CL 31087065 by paul chipchase in ue5-main branch]
#rb none
### Dialog
- When the source control backend tries to connect and fails we will now show a dialog allowing the user to try new settings rather than let the editor continue and crash.
- The dialog will allow the user to enter the server address (P4PORT) and their username (P4USER).
- Once the dialog is displayed the user will have three options to continue.
-- "Reset To Defaults" which will remove all settings saved to the users local ini files and attempt to use the environment defaults to connect.
-- "Retry Connection" which will take the new address and username that the user provided and attempt to connect with those settings.
-- "Skip" no connection attempt will be made. This may cause instability later if the user needs data that they cannot access.
- If the connection succeeds after the user has reset to defaults or provided new settings, then they will be saved in the local ini file for future sessions.
- The dialog will not show if:
-- Slate is disabled for the current processor has not yet initialized
-- The error is found not on the game thread. This means that lazy init will not currently work well with this dialog. Attempts to mashal the error to the game thread have been spotty at best and there is a high chance of thread locking so attempts were abandoned.
-- If the -unattended flag is set on the process.
-- If the dialog is disabled via the config file when setting up the backend (UseRetryConnectionDialog=False)
-- If 'engine.ini:[Core.VirtualizationModule]:UseLocalIniFileSettings' is false as future sessions would not be able to use the new settings and would become tiresome. In these cases the environment settings will need to be fixed.
- If the dialog is now shown then the backend is marked as unconnected and will not work.
### Known Issues
- We currently cannot get the settings that the failed p4 connection used, nor the error message dueto how the API works. So for now we just display an empty server address and username edit box. This will be fixed when the API is changed.
- There is currently no good way to inform the user where their settings come from.
- There is no easy way to reset the settings once saved. To allow the user to change the server address to a better proxy for example.
### Initialization
- In order to allow a slate dialog to be used, the initialization of the VA system has been moved later in pre init, to the first avaliable point after slate has initialized and its shaders have been built. This in theory should not cause a a problem as engine content cannot be virtualized.
-- Added a new option 'engine.ini:[Core.ContentVirtualization]:InitPreSlate' with a default of false. When set to true the VA system will initialize in the original, earlier portion of engine pre init. This is only included in case an existing virtualized project encounters a problem with the new ordering.
[CL 26115392 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-187789
- Changed the notification callbacks (used for statistics gathering) from TMulticastDelegate to TTSMulticastDelegate
[CL 25813486 by paul chipchase in ue5-main branch]
#rb none
#jira UE-182542
#preflight 643699cdeb6cd0e63a6bd729
- Previously we were duplicating all packages being virtualized, without the recently virtualized payloads, then try to move them one at a time to replace the original packages.
- Now we do all of the prep work such as checking the packages out of revision control and checking that they can be written to first, then duplicate & move each package one at a time.
-- This way we only ever have a single additional duplicated package at a time.
- Moved the final summary logging to an exit scope so we can return when ever we want in the function.
- Added early out for error case.
- Added early out if there are no packages to remove virtualized payloads from.
- Improved the code commands in IVirtualizationSystem::TryVirtualizePackages so that the caller knows the potential failure case scenarios.
[CL 25005153 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd670a71079a8d1c0e837b
- Since the API was forcing the caller to pass in a results structure to be filled in, we might as well make it the return value.
- Added a ::WasSuccessful method to the results structures that can be used instead of checking if the result had errors or not.
- Remove the reset method from FVirtualizationResult/FRehydrationResult as they no longer need it.
- The older deprecated methods still use the results enum, so we cannot easily deprecate those enums yet.
[CL 23626072 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 639c4112012902cb8db43e13
- This allows us to provide the user with more ways to customize the virtualization and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- Previously if we virtualized a package that was not checked out in revision control we would warn the user and then skip updating the package file on disk. This means the payloads would be uploaded but the user would be left with no local changes. Since sometimes we know we don't need to check out any package (virtualizing the packages in a change list for example) we don't want to always incur the cost of polling reivision control to see which packages do need checking out. This is why we now allow the caller to request package files be checked out via the new options enum EVirtualizationOptions.
-- If the EVirtualizationOptions::Checkout flag is provided we will poll the revision control status of all package files and then check out those which need it.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- Added a new utility function to SourceControlUtilties to make it easier to check out packages. There is similar functionality elsewhere in the code base but the virtualization module is too low level to make use of it.
- Updated existing code that calls ::TryVirtualizePackages and add cases of ''using namespace UE::Virtualization' where required to improve readability.
- The UnrealVirtualizationTool now supports a new cmdline option "-checkout" that can be used when virtualizing packages. This will checkout any package that was actually virtualized so the result can be saved back out to the workspace domain. This means we no longer require the caller to have checked out the packages before running the tool.
[CL 23536832 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-171492
#rnx
#preflight 6391f02d67018b14b581c0b0
- It is possible that a project might want to have persistent storage enabled but no caching storage, it is probably a bad idea but it should work. However before this fix doing so would cause an error to be logged during the virtualization of a package and fail the process.
- During the virtualization process we now check if pushing is enabled for cache storage before trying to push there, if not then we log that the phase is being skipped.
- Even though the above fixes the problem we would still get odd results when pushing payloads to cached storage without any backends enabled.
-- We now check if the pushing process is disabled before processing requests, so we can fail faster. There is now also a proper error value for this that can be stored in each request (FPushRequest) where as previously they'd all still display "pending"
-- We also check to see if the requested push type has any associated backend before processing requests and have a specific error value for this. Note that we consider
- We also now force the visibility of the slow task progress bar, as otherwise we might not end up displaying the virtualization progress to the progress dialog (this is a work around to a long standing problem with the slow task system due to the time limit associated with updating the dialog via EnterProgressFrame, this needs to be fixed elsewhere)
#ushell-cherrypick of 23445439 by paul.chipchase
[CL 23461405 by paul chipchase in ue5-main branch]
* Fixed so all headers that are supposed to be included independently can compile (there are headers that must be included through other headers)
* Added HEADER_UNIT_SKIP to headers that must be included through others
* Added HEADER_UNIT_UNSUPPORTED to headers that are not supported by platform but is part of the include list anyway (files in Clang folder and pthread files).
* Added HEADER_UNIT_UNSUPPORTED to headers that have bad dependencies and include files from modules they don't depend on. (KeyChainUtilities.h and PackageWriter.h)
#preflight 637281309e3bea8079feb733
#rb devin.doucette
[CL 23127795 by henrik karlsson in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63526abcae33b04ec1ee4a65
- The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker.
-- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use.
-- The DDC backend should be taking full advantage of the new batching API
-- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit.
- This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc.
-- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away.
[CL 22707955 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 634805f7f93be0f634b8ed5b
- Added a new FArchive 'FFixedBufferWriterArchive' that will let me serialize to an already allocated, fixed sized memory buffer.
- Moved the code for opening and reading packages to shared utility functions so that it is easier to keep the error messages consistent.
- Moved common code to a new function ::TryRehydrateBuilder so that both rehydration methods can share it.
- Note that technically we are changing the API since now we accept a TConstArrayView for the package file paths for both the virtualization and rehydration paths, but existing code will be compatible with this change.
[CL 22511406 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160942
#rnx
#preflight 6336f2ab5c2225fe5f6c08a5
- Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead.
-- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc.
- Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown.
- Changed the backends to use the new results system.
-- Not all paths were correctly setting the state in the file system backend.
-- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do.
- Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload.
[CL 22279127 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163103
#rnx
#preflight 6318989c2b7fe03eb664e9f0
### VirtualizationSystem/VirtualizationManager
- Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request
- Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety.
### IVirtualizationBackend
- The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs.
- Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters.
-- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942
- Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing.
- File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests.
-- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not.
[CL 21907558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161296
#rnx
#preflight 62fe3ce73d3fb466b229bcc0
- There are some usecases that require the VA system to initialize the first time it is accessed (usually the first time we attempt to pull a virtualized payload) rather than be initialized in the program start up. This change provides three different methods to achieve this:
-- Setting the define 'UE_VIRTUALIZATION_SYSTEM_LAZY_INIT' to 1 in a programs .target.cs
-- Setting [Core.ContentVirtualization]LazyInit=true in the Engine ini file
-- Running with the commandline option -VA-LazyInit
- If we detect that the source control backend is being initialized on a background thread we do not try to run the FConnect operation. The backend will still work but this does reduce the potential error checking on initialization. This is done because the FConnect operation currently only works on the main thread and to change this would be a bigger work item than we can schedule at the moment.
- UE::Virtualization::Initialize functions now take a EInitializationFlags enum as a parameter. This enum allows the call to ignore all lazy init settings and force the initialization immediately. This is useful for programs like the Virtualization standalone tool which just needs to start the system when needed.
-- The call to ::Initialize in LaunchEngineLoop passes in None and does not ignore lazy initialization.
-- Calls to ::Initialize in the UnrealVirtualizationTool however all use EInitializationFlags::ForceInitialize and ignore lazy initialization settings.
- Fixed an odd bug in UE::Virtualization::Initialize where the error path (if the config file cannot be found) was using a different start up code path.
- Add asserts when assigning to GVirtualizationSystem to make sure that it is null. This is not 100% safe but should catch some potential threading issues, if any.
- Add an assert after lazy initialization (IVirtualizationSystem::Get) to make sure that GVirtualizationSystem was assigned a valid object.
- Improve how we check for legacy values in [Core.ContentVirtualization]. We now support multiple allowed values.
- Added a way to poll if a VA system has been initialize yet or not, this allows us to avoid initializing a VA system if one has not yet been created and we try to:
-- Dump VA profiling stats after cooking
-- Send VA stats to studio analytics
- Note that currently using lazy init loading will probably cause the VA statistics panel not to work, this will be fixed in future work where we will allow the panel to register for a callback when the system is initialized.
[CL 21467510 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160943, UE-151671
#rnx
#preflight 62fa3334153b17e7462954b3
### Problem
- The virtualization process first checks to see if any of the local payloads are already stored in persistent storage and only try to submit those which aren't, which would mean if we are submitting a package with a local payload that is already in the persistent storage system but not in cached storage we have no way to cache it.
- This logic also introduced problems where filtered out payloads were being virtualized which was fixed with the addition of ENABLE_FILTERING_HACK which was not a robust long term solution.
- Checking if the payloads need to be pushed or not complicated the logic of the virtualization process and makes it harder to understand and make fixes too.
### Fix
- Removed code for ENABLE_FILTERING_HACK and UE_PRECHECK_PAYLOAD_STATUS from the virtualization process
- Removed some code loops from the virtualization process but we still end up with one loop finding all of the package trailers and a second loop setting up the FWorkspaceDomainPayloadProvider
-- This second loop will be removed in a future work item to avoid trying to push duplicate payloads (unless I end up doing that work at the virtualization manager level instead)
- We make one push for cached storage and another for persistent
-- Cached storage push failure will only result in a warning, failing the persistent push will error out as before
-- We need to reset the request states after the cache push. In a future work item we will likely allow a single push to specific both storage solutions, so I am leaving this code using the same set of requests for both pushes.
#robomerge FNMain
[CL 21386503 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-156189
#rnx
#preflight 62f4c098e60c9215b9bcde08
- The orignal name implied tha the payloads would only be stored locally on the users machine, but the intent is to describe a number of backends that are a) faster than the persistent storage backends b) no guarantee that the payload will be found.
- The new name better describes this functionality.
- We still accept 'LocalStorageHierarchy' but will log a warning informing the user to update their graph but this backwards compatibility code will most likely be removed before 5.1 ships.
-- EStorageType::Local will follow the usual deprecation rules.
- Not strictly related to this change but I also cleaned up and added additional loggng during the initialization of the virtualization manager.
[CL 21333601 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-156750
#rnx
#preflight 62f212e13b773d04161ee7dd
### Problem
- The payloads stored in map files tend to change more than other assets and would cause a lot more churn in the VA system.
- Some other systems like the landscape component are not able to sensibly continue if their payloads cannot be accessed (heightmaps for exmaple) and would prefer not to allow virtualization.
- As a short term fix we need an option to disable the virtualization on all payloads in map files. Future improvements to the filtering system will allow systems to more easily opt their payloads out of virtualization. When this is functional we might want to change the default from true to false.
### Feature
- The config optionf for this filtering is "[Core.VirtualizationModule]FilterMapContent=True"
- Testing if the owning UObject for a payload is in a umap can get tricky, because we not only need to check the umap but we also need to check if it is in a "_builddata.uasset" file, which is an additional file we store next to a umap containing things like lightmaps etc.
- At the moment we check for this by finding the outermost object for the given owner and check to see if it is a ULevel, UWorld or UMapBuildDataRegistry. This is a bit of a kludge but the types we need to check against are not accessible by this module and making them accessible will pull in a lot of dependencies that we'd prefer not to add.
-- One improvement might be to tag the FLinkerSave with the info we need and passing that into the serialization process rather than trying to work it out ourselves but I am wary of making that change until we are 100% sure that we want to keep this feature.
### Refactor
- Removed IVirtualizationSystem::IsDisabledForObject and replaced it with ::FilterPayload which can return multiple reasons for preventing a payload from virtualizing. (the method was added during 5.1 development so it should be fine to just replace it without deprecation)
-- The original behaviour for FVirtualizationManager::IsDisabledForObject has been moved to FVirtualizationManager::ShouldVirtualizeAsset
- Added a new header to declare enums/types used by the various parts of the virtualization system and started by moving EPayloadFilterReason there from the package trailer header. This allows both the core API and PackageTrailer to use EPayloadFilterReason without creating overburdened header dependencies.
-- EPayloadFilterReason has moved from the UE namespace to UE::Virtualization so the package trailer code needed updating accordingly.
- EditorBulkData will ask the virtualization system for the base filter reason, then add it's own reasons if UE_ENABLE_VIRTUALIZATION_TOGGLE is enabled. This bit of code will be removed for 5.1
[CL 21283179 by paul chipchase in ue5-main branch]
#rb none
#jira UE-159596
#rnx
#preflight 62d94160110da0f944afdd32
### Problem
- When rolling our virtualization to projects we have run into a number of set up problems which has caused users to fail to submit packages from the editor as when the virtualization failed we prevented the overall submit from going through.
- Although we will eventually work through all of these problems and add better fault tolerance for now it is annoying for the content team so we need an easy way for a project to allow package submits to be completed even if the virtualization process part of the submit failed.
- Worth noting that the virtualization process failing should never affect the state of the package files, the only downside is that the packages may not be virtualized.
### Fix
- The virtualization/rehydration of packages now return an enum instead of a bool although at the moment we are only returning Success and Failed, but it will allow us to expand in the future
- Combined the code calling the virtualization process from both the source control window and the source control changelist window into a single utility function.
-- Now the two windows only diverge on how they report failure to the user.
- This new utility function now checks to see if the virtualization manager suggests if failing the virtualization process should block any package submission to source control or not.
- I am not happy with this solution, but I haven't come up with anything that feels right, this solution will at least work.
[CL 21199597 by paul chipchase in ue5-main branch]
Headers are updated to contain any missing #includes needed to compile and #includes are sorted. Nothing is removed.
#ushell-cherrypick of 21065896 by bryan.sefcik
#preflight 62d4b1a5a6141b6adfb0c892
#jira
#ROBOMERGE-OWNER: Bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21150156 via CL 21151754 via CL 21154719
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
#ROBOMERGE-CONFLICT from-shelf
[CL 21181076 by Bryan sefcik in ue5-main branch]
Things to note regarding this pass:
* No includes are currently being removed from public headers.
* Any private file that has a #if is currently not being updated.
#preflight 62c703bbd13fac04f11da948
[CL 20985655 by bryan sefcik in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
