#rb Per.Larsson
#jira UE-161296
#rnx
#preflight 62fe3ce73d3fb466b229bcc0
- There are some usecases that require the VA system to initialize the first time it is accessed (usually the first time we attempt to pull a virtualized payload) rather than be initialized in the program start up. This change provides three different methods to achieve this:
-- Setting the define 'UE_VIRTUALIZATION_SYSTEM_LAZY_INIT' to 1 in a programs .target.cs
-- Setting [Core.ContentVirtualization]LazyInit=true in the Engine ini file
-- Running with the commandline option -VA-LazyInit
- If we detect that the source control backend is being initialized on a background thread we do not try to run the FConnect operation. The backend will still work but this does reduce the potential error checking on initialization. This is done because the FConnect operation currently only works on the main thread and to change this would be a bigger work item than we can schedule at the moment.
- UE::Virtualization::Initialize functions now take a EInitializationFlags enum as a parameter. This enum allows the call to ignore all lazy init settings and force the initialization immediately. This is useful for programs like the Virtualization standalone tool which just needs to start the system when needed.
-- The call to ::Initialize in LaunchEngineLoop passes in None and does not ignore lazy initialization.
-- Calls to ::Initialize in the UnrealVirtualizationTool however all use EInitializationFlags::ForceInitialize and ignore lazy initialization settings.
- Fixed an odd bug in UE::Virtualization::Initialize where the error path (if the config file cannot be found) was using a different start up code path.
- Add asserts when assigning to GVirtualizationSystem to make sure that it is null. This is not 100% safe but should catch some potential threading issues, if any.
- Add an assert after lazy initialization (IVirtualizationSystem::Get) to make sure that GVirtualizationSystem was assigned a valid object.
- Improve how we check for legacy values in [Core.ContentVirtualization]. We now support multiple allowed values.
- Added a way to poll if a VA system has been initialize yet or not, this allows us to avoid initializing a VA system if one has not yet been created and we try to:
-- Dump VA profiling stats after cooking
-- Send VA stats to studio analytics
- Note that currently using lazy init loading will probably cause the VA statistics panel not to work, this will be fixed in future work where we will allow the panel to register for a callback when the system is initialized.
[CL 21467510 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160954
#rnx
#preflight 62fb9de9086f90bbc4394829
- The problem with the old logic was that we checked all of the payloads initially to see which needed to be submitted but if enough payloads were requested we would split the submission into a number of batches. If a payload was in more than once batch then it would be submitted in the first batch and then fail the MarkForAdd command in the subsequent batches. When we detected this error we would fail the virtualization process.
- In the long term we should probably eliminate duplicate requests at the highest level (FVirtualizationManager) so no backend has to worry about it, but we need this working now so it was easiest to add duplicate checks to the source control backend
-- Note that if another process submits a payload between the status check and the mark for add calls then we will also get errors, but that is true for a lot of our source control code.
- Now when pushing payloads we first iterate over the requests and create a unique list of requests to submit. This eliminates the possibility of duplicate payloads being submitted and failing the MarkForAdd command.
- We now record the push request state for each payload in a map and then only apply it to the original requests if the push succeeds.
-- We still only really have failed/success statuses and nothing recording "already in the backend" which is scheduled to be fixed elsewhere in the backlog.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 21404920 via CL 21408168 via CL 21408189 via CL 21408222
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v975-21357124)
[CL 21411940 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160943, UE-151671
#rnx
#preflight 62fa31bbae3edb54c979e53e
- By checking the state of the payloads first we can avoid loading and writing a payload to disk, just to find that it is already in the depot. If we are very lucky we might find that all of the payloads are already uploaded, in which case we can avoid creating the submission workspace as well.
- The lock to prevent too many concurrent p4 connections does cover a wider area of code than it did before, in theory we could relinquish the lock while creating the payload files for submission, but generally the time it takes to do that is much smaller than the source control operations themselves so there is probably not much point.
- Any request that we find is already uploaded we can just mark as a success. We then create an array of pointers to the requests that were not found and operate on that for the rest of submission.
- Note that we also split up the payload existence checks into batches with the same max size limit of submission, although checking if a payload exists should be much less taxing on the system I'd rather not risk overloading the perforce server where possible.
#robomerge FNMain
[CL 21386511 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-160083
#rnx
#preflight 62f64714185b21882a875427
- When this backend was written, it shared the source control provider interface with the rest of the editor, so if we were unable to restore the workspace it would likely break the editor in interesting and hard to diagnose ways.
- Given the scope for problems and the low change of the operatation I opted to make the error fatal.
- Now that the backend owned it's own source control provider interface, the worst thing that can happen is that future virtualization attempts will also fail, but the user will not be left with a non functioning editor. With this in mind there is no longer a reason to consider this a fatal failure.
[CL 21353746 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f5f1a01e61d1ba0e6c893b
- Note that the older name 'SourceControl' will continue to work but will log a warning if used to notify the user to update their config files.
[CL 21352223 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-160619
#preflight 62f37c86b66d5d93133d67e3
- Based on work from Jess.Kube
### Virtualization
- The source control backend now takes an optional config option "ClientStream" which takes the name of the client stream to use.
- If a client stream is set then workspaces created for payload submission will use that and not provide a client-view mapping.
#ushell-cherrypick of 21195584 by Jess.Kube
### PerforceSourceControl
- Allow FCreateWorkspace to create workspaces with streams as well as classic workspaces cia FCreateWorkspace::SetStream.
- Add a method FCreateWorkspace::ClearClientViewMappings which will clear any client view mappings already added to the operation.
- If we detect that a FCreateWorkspace operation has both a stream set and client view mappings set then FPerforceCreateWorkspaceWorker will return an error. Perforce will allow us to creat a client spec with both entries, but will default to using the stream. Technically we could allow this too but it might cause unexpected behaviour to the caller. It is better to give a clear error and fail the workspace creation.
[CL 21316756 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#jira UE-159992
#preflight 62d7c9eb110da0f944670ecb
- When the depot path is parsed from the config file we now add the trailing '/' if it is not already there, this allows us to make the input more forgiving without changing the existing code.
- Removed FString::Printf when creating the path for payload_metainfo in favour of using WriteToString, this doesn't add much but gets rid of the %s.
- Reformatted some messages that had odd tab alignment.
[CL 21176124 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-158771
#rnx
#preflight 62c6c8f2f5590c326d77b0f3
- This is a temp fix to an internal problem, where we have a group of people unable to access the source control payload storage system due to permissions but do not actually need to access it at the moment.
- By supressing the connection failure pop up we can allow them to keep working undistrubed.
- Improving how we handle error notifications like this is already on the VA roadmap, at which point this change can be removed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20982887 via CL 20982891 via CL 20982896
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 20984627 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 628c9337a24685c36f50ec2f
- When the code was converted from single batch to support multiple batches I missed the check to return true if all payloads were already uploaded. So once we were operating on multiple batches if a batch was 100% already present in the backend we would return true and not process any further batches.
- Change the return true, to a continue so that future batches are processed.
[CL 20345282 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-151000
#rnx
#preflight 627291732f6d177be3c292e3
- When submitting large batches it was found that once a cl description reached a certain length, that p4v would lock up attempting to display it in the history tab, so we want to be able to make smaller submits to keep the description to a managable size.
- In addition it is better for our server set up to make multiple smaller batch submits than insanely large ones.
- The max size of the batch can be set when creating the virtualization graph in the config file.
[CL 20055325 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151000
#preflight 627271c83b2ed6f85363f53a
- In the old code when submitting packages for virtualization we would load all local payloads into memory then pass them to the virtualization system to be pushed to persistent storage. This works fine for an average users submits but when resaving large projects we can have submits with thousands of packages containing gigabytes of payloads.
- FPushRequest has been changed to accept either a payload directly (in the form of FCompressedBuffer) or accept a reference to a IPayloadProvider which will return the payload when asked.
- Most of the existing backends make no use of the batched pushing feature and only push one payload at a time. The default implementation of this will be to ask each request one at a time for the payload and then pass it to the backend for a single push. If the IPayloadProvider is being used we will therefor only ever have one payload loaded at a time avoiding memory spikes.
- The source control backend currently does implement the batched pushing feature. This backend writes each payload to disk before submitting them to perforce, so all we have to do is make sure that request one payload, write it to disk, then discard the payload to avoid memory spikes.
- This change required that FPushRequest make it's members private and provide accessors to them, as the caller shouldn't need to care if the payload is loaded or comes via the provider.
- NOTE that this implementation is less efficient if we have packages containing many different payloads as we will end up opening the package, parsing the trailer then load a payload many times over, where as the original code would load all payloads from the file in one go. In practice, given the overhead of the source control backend this didn't make a huge difference and is probably not worth the effort to optimize at this point.
[CL 20040609 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-147948
#preflight 6271209be16e280be60b5a62
- In the worst case scenario (where a machine can only access the source control backend) we will want to limit the potential number of concurrent connections that they can make.
- In testing, not limiting the connections could cause the occasional connection failure and I imagine having a full team running in the worse case scenario would quickly overwhelm the server.
- Currently we default to 8 connections (the limit allowed in UGS)
- The semaphore is a quick and dirty implementation similar to std::counting_semaphore. I am not spending much time on it as the worst case scenario is not expected to be common and if it does occur, the system is going to be painfully slow no matter how good the thread throttling.
### Future work
- This work could be taken a lot further by either
-- a) reusing connections rather than creating one for each pull, which will require changes to our source control api
-- b) gathering requests from many requesting threads once the limit has been reached and then sending the requests in batches
--c) reserve one connection for the game thread to prevent the user waiting?
[CL 20024594 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 6260f6e4360b45c32a84fae8
- The commandline -VA-MissBackends has been extended to a console command 'VA.MissBackends' that can be used at runtime to disable specific backends or reset the list and enable all of them again.
-- Entering VA.MissBackends without any args will print help documentation to the console/output log and show the user the names of the backends that they can disable.
- Technically we don't need the command line parsing anymore and users could use -dpcvars=VA.MissBackends=X instead but since we already have the code we might as well leave it untouched.
- Disabling backends from pulling payloads for debug purposes has been split off to it's own system stored as a
- IVirtualizationBackend::EOperations has been turned into bit flags, so we no longer need the 'both' entry
- IVirtualizationBackendnow has two EOperations members, one to record the operations that the backend supports and another that allows the operations to be disabled for debugging purposes.
- IVirtualizationBackend now has one method to poll if an operation is supported ::IsOperationSupported, which should not change over the lift time of the backend. In addition there is also a method for disabling/enabling a backend for debugging purposes as well as polling the debug state.
[CL 19845175 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Sebastian.Nordgren
#jira UE-147948
#preflight 6256ba2f647ad886b380d007
- In some cases people are getting minor connection failures to the perforce server which causes the p4 print command to fail. In these cases it would be better to retry the command rather than failing to get the payload entirely
- Currently the default will be 2 attempts to pull a payload with a 100ms wait between them, this can be changed when setting up the backend graph as optional config values "RetryCount=" and "RetryWaitTime="
-- Note that RetryWaitTime is in ms
- Moved the parsing of the command line to it's own method in the backend.
[CL 19752733 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#preflight 622f645a19287beb2c359176
- Use the SetRequireDirPathEndWithSeperator option with the FUpdateStatus operation as we are checking depot file paths. If we do not use this option they can be seen as network paths and will be checked against the file system which can take some time to time out.
- Wrapped this up in a utility function so that it can be more easily updated in the future.
[CL 19373558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-136480
#preflight 62289b59f4469cadac10eb91
- The backend can now connect to source control without forcing the rest of the editor to connect to source control.
- If the user disconnects the editor from source control, the backend will still be able to function using it's own connection.
-When the backend needs to change the current workspace in order to submit payloads we no longer invalidate the file cache used by the editor and so the user will no longer have to manually refresh files in the content browsers to restore the source control status icon.
[CL 19318970 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-142926
#preflight 621e1f177f2803279b575764
- Now when payloads are submitted, we will submit them from within the project's saved directory (by default) this now works because we write a .p4ignore file to the submission directory which will override any additional ignore file higher up in the directory structure.
-- A new utility function ::CreateSubmissionSessionDirectory was added to make sure that the directory we are submitting from has a p4ignore file. It is added in it's own scope as it is assumed that additional functionality will be added here later.
- If a user wishes to choose a different location to submit from, they can override the environment variable "UE-VirtualizationWorkingDir" to provide a different location. This might be useful if the user does not have a huge amount of spare space on the disk that the project is running from.
- There is also an optional entry to the ini file set up 'SubmitFromTempDir' (false by default) which when set to true will cause the system to attempt to submit the payload files from the machines temp directory instead.
-- There isn't an easy way for users to override this locally, which reduces the usefulness of the option. This might be worth improving in the future although that time would be better spent removing the need for the directory in the first place.
- We now try to find/create the root directory to submit payloads from when the backend is created. Failure at this point is not expected and will prevent the backend from working and so is considered an error.
- Added more logging to track the session guid when submitting and to be clear about where the files are being submitted from.
[CL 19197367 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-143680
#preflight 621776a4335015645d967662
#lockdown mark.lintott
- Strings inside of a LOCTEXT macro must be on a single line.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 19114310 in //UE5/Release-5.0/... via CL 19114561
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19152877 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#jira UE-143686
#rnx
#preflight 62189ed68849a7424f7d2e03
- Workspaces created to submit virtualized assets are now prefixed as "VASubmission-" rather than "MirageSubmission-"
-- Use of the term mirage is deprecated and should be replaced by VA or VirtualizedAsset(s)
- Set the description when we create the workspace to info the user that it was auto generated during the submission of virtualized payloads.
[CL 19143761 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-134434
#rnx
#preflight 620d1a10742ffef420223b15
- Changing a number of log items from 'Log' to 'Display' so that the info shows up more easily.
- Removed a number of log items that added no useful infomation
- Edited some of the existing log items to be clearer.
[CL 19031890 by paul chipchase in ue5-main branch]
