#rb trivial
#jira UE-160083
#rnx
#preflight 62f64714185b21882a875427
- When this backend was written, it shared the source control provider interface with the rest of the editor, so if we were unable to restore the workspace it would likely break the editor in interesting and hard to diagnose ways.
- Given the scope for problems and the low change of the operatation I opted to make the error fatal.
- Now that the backend owned it's own source control provider interface, the worst thing that can happen is that future virtualization attempts will also fail, but the user will not be left with a non functioning editor. With this in mind there is no longer a reason to consider this a fatal failure.
[CL 21353746 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f5f1a01e61d1ba0e6c893b
- Note that the older name 'SourceControl' will continue to work but will log a warning if used to notify the user to update their config files.
[CL 21352223 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-160619
#preflight 62f37c86b66d5d93133d67e3
- Based on work from Jess.Kube
### Virtualization
- The source control backend now takes an optional config option "ClientStream" which takes the name of the client stream to use.
- If a client stream is set then workspaces created for payload submission will use that and not provide a client-view mapping.
#ushell-cherrypick of 21195584 by Jess.Kube
### PerforceSourceControl
- Allow FCreateWorkspace to create workspaces with streams as well as classic workspaces cia FCreateWorkspace::SetStream.
- Add a method FCreateWorkspace::ClearClientViewMappings which will clear any client view mappings already added to the operation.
- If we detect that a FCreateWorkspace operation has both a stream set and client view mappings set then FPerforceCreateWorkspaceWorker will return an error. Perforce will allow us to creat a client spec with both entries, but will default to using the stream. Technically we could allow this too but it might cause unexpected behaviour to the caller. It is better to give a clear error and fail the workspace creation.
[CL 21316756 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#jira UE-159992
#preflight 62d7c9eb110da0f944670ecb
- When the depot path is parsed from the config file we now add the trailing '/' if it is not already there, this allows us to make the input more forgiving without changing the existing code.
- Removed FString::Printf when creating the path for payload_metainfo in favour of using WriteToString, this doesn't add much but gets rid of the %s.
- Reformatted some messages that had odd tab alignment.
[CL 21176124 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-158771
#rnx
#preflight 62c6c8f2f5590c326d77b0f3
- This is a temp fix to an internal problem, where we have a group of people unable to access the source control payload storage system due to permissions but do not actually need to access it at the moment.
- By supressing the connection failure pop up we can allow them to keep working undistrubed.
- Improving how we handle error notifications like this is already on the VA roadmap, at which point this change can be removed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20982887 via CL 20982891 via CL 20982896
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 20984627 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 628c9337a24685c36f50ec2f
- When the code was converted from single batch to support multiple batches I missed the check to return true if all payloads were already uploaded. So once we were operating on multiple batches if a batch was 100% already present in the backend we would return true and not process any further batches.
- Change the return true, to a continue so that future batches are processed.
[CL 20345282 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-151000
#rnx
#preflight 627291732f6d177be3c292e3
- When submitting large batches it was found that once a cl description reached a certain length, that p4v would lock up attempting to display it in the history tab, so we want to be able to make smaller submits to keep the description to a managable size.
- In addition it is better for our server set up to make multiple smaller batch submits than insanely large ones.
- The max size of the batch can be set when creating the virtualization graph in the config file.
[CL 20055325 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151000
#preflight 627271c83b2ed6f85363f53a
- In the old code when submitting packages for virtualization we would load all local payloads into memory then pass them to the virtualization system to be pushed to persistent storage. This works fine for an average users submits but when resaving large projects we can have submits with thousands of packages containing gigabytes of payloads.
- FPushRequest has been changed to accept either a payload directly (in the form of FCompressedBuffer) or accept a reference to a IPayloadProvider which will return the payload when asked.
- Most of the existing backends make no use of the batched pushing feature and only push one payload at a time. The default implementation of this will be to ask each request one at a time for the payload and then pass it to the backend for a single push. If the IPayloadProvider is being used we will therefor only ever have one payload loaded at a time avoiding memory spikes.
- The source control backend currently does implement the batched pushing feature. This backend writes each payload to disk before submitting them to perforce, so all we have to do is make sure that request one payload, write it to disk, then discard the payload to avoid memory spikes.
- This change required that FPushRequest make it's members private and provide accessors to them, as the caller shouldn't need to care if the payload is loaded or comes via the provider.
- NOTE that this implementation is less efficient if we have packages containing many different payloads as we will end up opening the package, parsing the trailer then load a payload many times over, where as the original code would load all payloads from the file in one go. In practice, given the overhead of the source control backend this didn't make a huge difference and is probably not worth the effort to optimize at this point.
[CL 20040609 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-147948
#preflight 6271209be16e280be60b5a62
- In the worst case scenario (where a machine can only access the source control backend) we will want to limit the potential number of concurrent connections that they can make.
- In testing, not limiting the connections could cause the occasional connection failure and I imagine having a full team running in the worse case scenario would quickly overwhelm the server.
- Currently we default to 8 connections (the limit allowed in UGS)
- The semaphore is a quick and dirty implementation similar to std::counting_semaphore. I am not spending much time on it as the worst case scenario is not expected to be common and if it does occur, the system is going to be painfully slow no matter how good the thread throttling.
### Future work
- This work could be taken a lot further by either
-- a) reusing connections rather than creating one for each pull, which will require changes to our source control api
-- b) gathering requests from many requesting threads once the limit has been reached and then sending the requests in batches
--c) reserve one connection for the game thread to prevent the user waiting?
[CL 20024594 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 6260f6e4360b45c32a84fae8
- The commandline -VA-MissBackends has been extended to a console command 'VA.MissBackends' that can be used at runtime to disable specific backends or reset the list and enable all of them again.
-- Entering VA.MissBackends without any args will print help documentation to the console/output log and show the user the names of the backends that they can disable.
- Technically we don't need the command line parsing anymore and users could use -dpcvars=VA.MissBackends=X instead but since we already have the code we might as well leave it untouched.
- Disabling backends from pulling payloads for debug purposes has been split off to it's own system stored as a
- IVirtualizationBackend::EOperations has been turned into bit flags, so we no longer need the 'both' entry
- IVirtualizationBackendnow has two EOperations members, one to record the operations that the backend supports and another that allows the operations to be disabled for debugging purposes.
- IVirtualizationBackend now has one method to poll if an operation is supported ::IsOperationSupported, which should not change over the lift time of the backend. In addition there is also a method for disabling/enabling a backend for debugging purposes as well as polling the debug state.
[CL 19845175 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Sebastian.Nordgren
#jira UE-147948
#preflight 6256ba2f647ad886b380d007
- In some cases people are getting minor connection failures to the perforce server which causes the p4 print command to fail. In these cases it would be better to retry the command rather than failing to get the payload entirely
- Currently the default will be 2 attempts to pull a payload with a 100ms wait between them, this can be changed when setting up the backend graph as optional config values "RetryCount=" and "RetryWaitTime="
-- Note that RetryWaitTime is in ms
- Moved the parsing of the command line to it's own method in the backend.
[CL 19752733 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#preflight 622f645a19287beb2c359176
- Use the SetRequireDirPathEndWithSeperator option with the FUpdateStatus operation as we are checking depot file paths. If we do not use this option they can be seen as network paths and will be checked against the file system which can take some time to time out.
- Wrapped this up in a utility function so that it can be more easily updated in the future.
[CL 19373558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-136480
#preflight 62289b59f4469cadac10eb91
- The backend can now connect to source control without forcing the rest of the editor to connect to source control.
- If the user disconnects the editor from source control, the backend will still be able to function using it's own connection.
-When the backend needs to change the current workspace in order to submit payloads we no longer invalidate the file cache used by the editor and so the user will no longer have to manually refresh files in the content browsers to restore the source control status icon.
[CL 19318970 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-142926
#preflight 621e1f177f2803279b575764
- Now when payloads are submitted, we will submit them from within the project's saved directory (by default) this now works because we write a .p4ignore file to the submission directory which will override any additional ignore file higher up in the directory structure.
-- A new utility function ::CreateSubmissionSessionDirectory was added to make sure that the directory we are submitting from has a p4ignore file. It is added in it's own scope as it is assumed that additional functionality will be added here later.
- If a user wishes to choose a different location to submit from, they can override the environment variable "UE-VirtualizationWorkingDir" to provide a different location. This might be useful if the user does not have a huge amount of spare space on the disk that the project is running from.
- There is also an optional entry to the ini file set up 'SubmitFromTempDir' (false by default) which when set to true will cause the system to attempt to submit the payload files from the machines temp directory instead.
-- There isn't an easy way for users to override this locally, which reduces the usefulness of the option. This might be worth improving in the future although that time would be better spent removing the need for the directory in the first place.
- We now try to find/create the root directory to submit payloads from when the backend is created. Failure at this point is not expected and will prevent the backend from working and so is considered an error.
- Added more logging to track the session guid when submitting and to be clear about where the files are being submitted from.
[CL 19197367 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-143680
#preflight 621776a4335015645d967662
#lockdown mark.lintott
- Strings inside of a LOCTEXT macro must be on a single line.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 19114310 in //UE5/Release-5.0/... via CL 19114561
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19152877 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#jira UE-143686
#rnx
#preflight 62189ed68849a7424f7d2e03
- Workspaces created to submit virtualized assets are now prefixed as "VASubmission-" rather than "MirageSubmission-"
-- Use of the term mirage is deprecated and should be replaced by VA or VirtualizedAsset(s)
- Set the description when we create the workspace to info the user that it was auto generated during the submission of virtualized payloads.
[CL 19143761 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-134434
#rnx
#preflight 620d1a10742ffef420223b15
- Changing a number of log items from 'Log' to 'Display' so that the info shows up more easily.
- Removed a number of log items that added no useful infomation
- Edited some of the existing log items to be clearer.
[CL 19031890 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Devin.Doucette
#jira UE-133497
#rnx
#preflight 61f835491c5ac5523462810a
- A lot of files have been touched but most of this is changing FPayload::IsValid to !FIoHash::IsZero, the main changes are in EditorBulkData/EditorBulkDataTests
- The only difference between FPayloadId and FIoHash is that the former considered the hash of an invalid or empty buffer to be 'invalid' too where as FIoHash would return a valid hash
-- To keep behaviour the same, we only hash payloads in EditorBulkData using a utility method ::HashBuffer which will not hash empty or invalid payloads and return a default FIoHash instead.
-- Unit tests have been extended to prove that the behaviour has not changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18806362 in //UE5/Release-5.0/... via CL 18808527 via CL 18821790
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822154 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-136612
#rnx
#preflight 61e589b7904123989a1817cc
- If a perforce connection issue is encountered when starting up the editor we will now print an error to the message log and then on the first tick of the editor display a notification to get the users attention. It will no longer result in a fatal error.
- Add a new category "Asset Virtualization" to the message log so that we can start to improve the feedback we give to end users rather than only reporting problems via UE_LOG. This also introduces a new localization text name space.
-- I will need to go over a lot of the virtualization modules error handling and move it to use the message log instead.
- When the source control backend first starts up we now specifically check if the connection works and if not try to give the user feedback on how to fix it.
-- Ideally we'd provide a login dialog for the user to fix the problem in place but due to how early in the start up process we need to be able to access the virtualization backends this is quite tricky.
- In the future we will build the concept of a backends connection into a formalized concept in the system along with better error handling. At this point FSourceControlBackend::OnConnectionError should be moved to general use for all backends. this will allow us to provide much better status info to the user.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637036 in //UE5/Release-5.0/... via CL 18637043 via CL 18637054
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637614 by paul chipchase in ue5-main branch]
