#rb Per.Larsson
#jira UE-163103
#rnx
#preflight 6318989c2b7fe03eb664e9f0
### VirtualizationSystem/VirtualizationManager
- Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request
- Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety.
### IVirtualizationBackend
- The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs.
- Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters.
-- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942
- Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing.
- File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests.
-- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not.
[CL 21907558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163016
#rnx
#preflight
- The backend is no longer used as we access horde storage via the ddc backend.
[CL 21862840 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161973
#rnx
#preflight 63074424ae13a5a09865a281
- We could try force loading and enabling the plugin at this point but I am not a fan of the idea of overriding the set up of the target in a way that is not immediately obvious to the end user.
- The virtualization module needs to include the 'Projects' module to be able to access the plugin system directly and while editing the file I took the opportunity to clean up how the module is declaring its dependencies.
[CL 21684932 by paul chipchase in ue5-main branch]
- Set VA.DisableVirtualization CVar for CookedEditor targets
- Use UE_VIRTUALIZATION_CONNECTION_LAZY_INIT for [redacted] CookedEditor targets
- setup FLinuxDeviceProfileSelectorModule so Linux CookedEditor use the right device profile
- Fix bLazyInitConnections not being set by UE_VIRTUALIZATION_CONNECTION_LAZY_INIT and fix logging
- Add AllowVirtualAssets to automation scripts for [redacted]
[REVIEW] [at]Josh.Adams [at]Eric.Knapik [at]Paul.Chipchase
#preflight https://horde.devtools.epicgames.com/job/630664c1516bef57ffef3c24
[CL 21553248 by justin marcus in ue5-main branch]
#rb Per.Larsson
#jira UE-161685
#rnx
#preflight 6304e0dfae13a5a098e2b9c9
- Added a cvar 'VA.DisableVirtualization' and a commandline '-VA-DisableVirtualization-' that when set are equivalent to setting '[Core.VirtualizationModule].EnablePayloadVirtualization=false' and will. prevent the VA system from virtualizing any packages on submit.
[CL 21524913 by paul chipchase in ue5-main branch]
#rb Devin.Doucette
#jira UE-161599
#rnx
#preflight 6303c8d65a5d4e4624e7bf52
- There are some use cases that require the VA system to be initialized and configured correctly but would prefer that the backend connections only run if absolutely needed (usually when a payload is pulled or pushed for the first time), this change provides four different ways of doing this:
-- Setting [Core.VirtualizationModule]LazyInitConnections=true in the Engine ini file
-- Setting the define 'UE_VIRTUALIZATION_CONNECTION_LAZY_INIT' to 1 in a programs .target.cs
-- Running with the commandline option -VA-LazyInitConnections
-- Setting the cvar 'VA.LazyInitConnections' to 1 (only works if it is set before the VA system is initialized, changing it mid editor via the console does nothing)
--- Note that after the config file, each setting there only opts into lazy initializing the connections, setting the cvar to 0 for example will not prevent the cmdline from opting in etc.
- In the future we will allow the connection code to run async, so the latency can be hidden behind the editor loading, but for the current use case we are taking the minimal approach.
-- This means we only support the backend being in 3 states. No connection has been made yet, the connection is broken and the connection is working.
-- To keep things simple we only record if we have attempted to connect the backends or not. We don't check individual backends nor do we try to reconnect failed ones etc. This is all scheduled for a future work item.
- If the connections are not initialized when the VA system is, we wait until the first time someone calls one of the virtualization methods that will actually use a connection: Push/Pull/Query
-- We try connecting all of the backends at once, even if they won't be used in the call to keep things simple.
- Only the source control backend makes use of the connection system. The horde storage (http) backend could take advantage too, but it is currently unused and most likely going to just be deleted so there seemed little point updating it.
- If we try to run an operation on an unconnected backend we only log to verbose. This is to maintain existing behaviour where a failed backend would not be mounted at all. This logging will likely be revisited in a future work item.
[CL 21511855 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161408
#rnx
#preflight 62ff43f35344ca3df4962ec8
- If there was no VA activity we explicitly log that there was no profile data to show
- Only log the operations that had activity, there is no point writing out a bunch of zeros for an operation to show nothing happened.
[CL 21468200 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161296
#rnx
#preflight 62fe3ce73d3fb466b229bcc0
- There are some usecases that require the VA system to initialize the first time it is accessed (usually the first time we attempt to pull a virtualized payload) rather than be initialized in the program start up. This change provides three different methods to achieve this:
-- Setting the define 'UE_VIRTUALIZATION_SYSTEM_LAZY_INIT' to 1 in a programs .target.cs
-- Setting [Core.ContentVirtualization]LazyInit=true in the Engine ini file
-- Running with the commandline option -VA-LazyInit
- If we detect that the source control backend is being initialized on a background thread we do not try to run the FConnect operation. The backend will still work but this does reduce the potential error checking on initialization. This is done because the FConnect operation currently only works on the main thread and to change this would be a bigger work item than we can schedule at the moment.
- UE::Virtualization::Initialize functions now take a EInitializationFlags enum as a parameter. This enum allows the call to ignore all lazy init settings and force the initialization immediately. This is useful for programs like the Virtualization standalone tool which just needs to start the system when needed.
-- The call to ::Initialize in LaunchEngineLoop passes in None and does not ignore lazy initialization.
-- Calls to ::Initialize in the UnrealVirtualizationTool however all use EInitializationFlags::ForceInitialize and ignore lazy initialization settings.
- Fixed an odd bug in UE::Virtualization::Initialize where the error path (if the config file cannot be found) was using a different start up code path.
- Add asserts when assigning to GVirtualizationSystem to make sure that it is null. This is not 100% safe but should catch some potential threading issues, if any.
- Add an assert after lazy initialization (IVirtualizationSystem::Get) to make sure that GVirtualizationSystem was assigned a valid object.
- Improve how we check for legacy values in [Core.ContentVirtualization]. We now support multiple allowed values.
- Added a way to poll if a VA system has been initialize yet or not, this allows us to avoid initializing a VA system if one has not yet been created and we try to:
-- Dump VA profiling stats after cooking
-- Send VA stats to studio analytics
- Note that currently using lazy init loading will probably cause the VA statistics panel not to work, this will be fixed in future work where we will allow the panel to register for a callback when the system is initialized.
[CL 21467510 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62fcc14d1e39eb26a0851cd0
- Sometimes it is useful to be able to dump the payload info from packages not under the current project as a quick form of debugging and given the format of the package trailer there is no reason why we should allow it.
- If we fail to turn the arg into a valid FPackagePath can now also check to see if the arg is an absolute file path via the FileManager
- If both of these options fail we then log an error as before.
[CL 21422989 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160954
#rnx
#preflight 62fb9de9086f90bbc4394829
- The problem with the old logic was that we checked all of the payloads initially to see which needed to be submitted but if enough payloads were requested we would split the submission into a number of batches. If a payload was in more than once batch then it would be submitted in the first batch and then fail the MarkForAdd command in the subsequent batches. When we detected this error we would fail the virtualization process.
- In the long term we should probably eliminate duplicate requests at the highest level (FVirtualizationManager) so no backend has to worry about it, but we need this working now so it was easiest to add duplicate checks to the source control backend
-- Note that if another process submits a payload between the status check and the mark for add calls then we will also get errors, but that is true for a lot of our source control code.
- Now when pushing payloads we first iterate over the requests and create a unique list of requests to submit. This eliminates the possibility of duplicate payloads being submitted and failing the MarkForAdd command.
- We now record the push request state for each payload in a map and then only apply it to the original requests if the push succeeds.
-- We still only really have failed/success statuses and nothing recording "already in the backend" which is scheduled to be fixed elsewhere in the backlog.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 21404920 via CL 21408168 via CL 21408189 via CL 21408222
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v975-21357124)
[CL 21411940 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160943, UE-151671
#rnx
#preflight 62fa31bbae3edb54c979e53e
- By checking the state of the payloads first we can avoid loading and writing a payload to disk, just to find that it is already in the depot. If we are very lucky we might find that all of the payloads are already uploaded, in which case we can avoid creating the submission workspace as well.
- The lock to prevent too many concurrent p4 connections does cover a wider area of code than it did before, in theory we could relinquish the lock while creating the payload files for submission, but generally the time it takes to do that is much smaller than the source control operations themselves so there is probably not much point.
- Any request that we find is already uploaded we can just mark as a success. We then create an array of pointers to the requests that were not found and operate on that for the rest of submission.
- Note that we also split up the payload existence checks into batches with the same max size limit of submission, although checking if a payload exists should be much less taxing on the system I'd rather not risk overloading the perforce server where possible.
#robomerge FNMain
[CL 21386511 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160943, UE-151671
#rnx
#preflight 62fa3334153b17e7462954b3
### Problem
- The virtualization process first checks to see if any of the local payloads are already stored in persistent storage and only try to submit those which aren't, which would mean if we are submitting a package with a local payload that is already in the persistent storage system but not in cached storage we have no way to cache it.
- This logic also introduced problems where filtered out payloads were being virtualized which was fixed with the addition of ENABLE_FILTERING_HACK which was not a robust long term solution.
- Checking if the payloads need to be pushed or not complicated the logic of the virtualization process and makes it harder to understand and make fixes too.
### Fix
- Removed code for ENABLE_FILTERING_HACK and UE_PRECHECK_PAYLOAD_STATUS from the virtualization process
- Removed some code loops from the virtualization process but we still end up with one loop finding all of the package trailers and a second loop setting up the FWorkspaceDomainPayloadProvider
-- This second loop will be removed in a future work item to avoid trying to push duplicate payloads (unless I end up doing that work at the virtualization manager level instead)
- We make one push for cached storage and another for persistent
-- Cached storage push failure will only result in a warning, failing the persistent push will error out as before
-- We need to reset the request states after the cache push. In a future work item we will likely allow a single push to specific both storage solutions, so I am leaving this code using the same set of requests for both pushes.
#robomerge FNMain
[CL 21386503 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-160083
#rnx
#preflight 62f64714185b21882a875427
- When this backend was written, it shared the source control provider interface with the rest of the editor, so if we were unable to restore the workspace it would likely break the editor in interesting and hard to diagnose ways.
- Given the scope for problems and the low change of the operatation I opted to make the error fatal.
- Now that the backend owned it's own source control provider interface, the worst thing that can happen is that future virtualization attempts will also fail, but the user will not be left with a non functioning editor. With this in mind there is no longer a reason to consider this a fatal failure.
[CL 21353746 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f5f1a01e61d1ba0e6c893b
- Note that the older name 'SourceControl' will continue to work but will log a warning if used to notify the user to update their config files.
[CL 21352223 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight
- The constructor parameters should've been const, this doesn't show when compiling with MSVC as we have that warning currently disabled.
[CL 21335846 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f4f504b66d5d9313969a92
- Use the new macro UE_REGISTER_VIRTUALIZATION_BACKEND_FACTORY_LEGACY when trying to rename how backends are identified.
[CL 21334721 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-156189
#rnx
#preflight 62f4c098e60c9215b9bcde08
- The orignal name implied tha the payloads would only be stored locally on the users machine, but the intent is to describe a number of backends that are a) faster than the persistent storage backends b) no guarantee that the payload will be found.
- The new name better describes this functionality.
- We still accept 'LocalStorageHierarchy' but will log a warning informing the user to update their graph but this backwards compatibility code will most likely be removed before 5.1 ships.
-- EStorageType::Local will follow the usual deprecation rules.
- Not strictly related to this change but I also cleaned up and added additional loggng during the initialization of the virtualization manager.
[CL 21333601 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f3ab2023003b62a324f67a
- The older versions of this value ([Core.ContentVirtualization].EnablePushToBackend and [Core.VirtualizationModule].EnablePushToBackend) will continue to work but will log a warning to the user telling them to update it.
- It is very unlikely that anyone is actually using these old values and so we can remove this backwards compatibility when 5.1 ships. It is only present to allow us to fix up internal projects and to avoid interrupting people testing the system.
[CL 21317383 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-160619
#preflight 62f37c86b66d5d93133d67e3
- Based on work from Jess.Kube
### Virtualization
- The source control backend now takes an optional config option "ClientStream" which takes the name of the client stream to use.
- If a client stream is set then workspaces created for payload submission will use that and not provide a client-view mapping.
#ushell-cherrypick of 21195584 by Jess.Kube
### PerforceSourceControl
- Allow FCreateWorkspace to create workspaces with streams as well as classic workspaces cia FCreateWorkspace::SetStream.
- Add a method FCreateWorkspace::ClearClientViewMappings which will clear any client view mappings already added to the operation.
- If we detect that a FCreateWorkspace operation has both a stream set and client view mappings set then FPerforceCreateWorkspaceWorker will return an error. Perforce will allow us to creat a client spec with both entries, but will default to using the stream. Technically we could allow this too but it might cause unexpected behaviour to the caller. It is better to give a clear error and fail the workspace creation.
[CL 21316756 by paul chipchase in ue5-main branch]
#fyi paul.chipchase
Original CL Desc
-----------------------------------------------------------------
The virtualization manager will now read the newer config file location (Core.ContentVirtualization) first and only fall back to the older location (Core.ContentVirtualization) if it was not found.
#rb trivial
#jira UE-156189
#rnx
#preflight 62f38fed086f90bbc4d915f0
[CL 21316410 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f38fed086f90bbc4d915f0
- This will help if someone has updated their set up to the new location but not removed the old entries.
- We still warn that the old enteries exist, even if not used.
[CL 21316332 by paul chipchase in ue5-main branch]
