#rb Per.Larsson
#jira UE-203597
- When we first parse the value from the config file we should check to see if it contains and path separators and if so reject the backend entirely. Providing a full path rather than just the filename is an easy enough mistake to make and it is better for the user to find out sooner than later.
- The utility function ::TryCreateSubmissionSessionDirectory was extended after it was first created to also create a dummy ignore file, updated the name to reflect that it does more than just create a directory.
- We were relying ont he creation of the dummy file to also create the directory but that means it is harder for us to inform the user if an encountered problem is due to the directory creation or the file creation. Now we ensure that the directory is created first (with error handling) and only then try to create the dummy file.
- Added logging of the system error if we encounter issues to give the user a better idea what the issue was.
[CL 30532386 by paul chipchase in ue5-main branch]
#rb none
#jira UE-191791
#rnx
- When a connection attempt fails we now return the credentials used and the errors given by the system.
-- The credentials will be used to auto fill out the dialog, letting the user see what credentials were attempted in the last attempt.
-- The errors are shown in a box with red error text, letting them know what went wrong.
- We no longer let the revision control provider log errors when establishing a connection, so if the users correct info and successfully connects on a subsequent attempt they will not see an error in the logs.
-- If the connection completely fails we still log as before.
[CL 27302825 by paul chipchase in ue5-main branch]
#rb none
### Dialog
- When the source control backend tries to connect and fails we will now show a dialog allowing the user to try new settings rather than let the editor continue and crash.
- The dialog will allow the user to enter the server address (P4PORT) and their username (P4USER).
- Once the dialog is displayed the user will have three options to continue.
-- "Reset To Defaults" which will remove all settings saved to the users local ini files and attempt to use the environment defaults to connect.
-- "Retry Connection" which will take the new address and username that the user provided and attempt to connect with those settings.
-- "Skip" no connection attempt will be made. This may cause instability later if the user needs data that they cannot access.
- If the connection succeeds after the user has reset to defaults or provided new settings, then they will be saved in the local ini file for future sessions.
- The dialog will not show if:
-- Slate is disabled for the current processor has not yet initialized
-- The error is found not on the game thread. This means that lazy init will not currently work well with this dialog. Attempts to mashal the error to the game thread have been spotty at best and there is a high chance of thread locking so attempts were abandoned.
-- If the -unattended flag is set on the process.
-- If the dialog is disabled via the config file when setting up the backend (UseRetryConnectionDialog=False)
-- If 'engine.ini:[Core.VirtualizationModule]:UseLocalIniFileSettings' is false as future sessions would not be able to use the new settings and would become tiresome. In these cases the environment settings will need to be fixed.
- If the dialog is now shown then the backend is marked as unconnected and will not work.
### Known Issues
- We currently cannot get the settings that the failed p4 connection used, nor the error message dueto how the API works. So for now we just display an empty server address and username edit box. This will be fixed when the API is changed.
- There is currently no good way to inform the user where their settings come from.
- There is no easy way to reset the settings once saved. To allow the user to change the server address to a better proxy for example.
### Initialization
- In order to allow a slate dialog to be used, the initialization of the VA system has been moved later in pre init, to the first avaliable point after slate has initialized and its shaders have been built. This in theory should not cause a a problem as engine content cannot be virtualized.
-- Added a new option 'engine.ini:[Core.ContentVirtualization]:InitPreSlate' with a default of false. When set to true the VA system will initialize in the original, earlier portion of engine pre init. This is only included in case an existing virtualized project encounters a problem with the new ordering.
[CL 26115392 by paul chipchase in ue5-main branch]
#rb none
#jira UE-181418
#rnx
#preflight 6447c5482804595a04d1e375
- Each pull operation now takes a FText that can be filled in with additional info about any failures that will cause the new error dialog to be displayed.
-- At the moment we only have info to provide from the source control backend if the connection itself failed.
-- In the future we might want to completely overhaul our error logging and display all messages to the user but our logging system is not really built for that level of redirection at the moment.
- Only pass on the error if it came from a persistent backend, we don't need to worry the user about cached backends which are allowed to fail.
[CL 25178752 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-182205
#preflight 643932b2211b661dc4f594c6
- The enums don't contain any entries at the moment, the goal of this change is to reduce the friction that future API changes might have.
- The older versions of PullData/PushData have been deprecated but this won't flag compiler warnings if people have derived their own implementations of IVirtualizationBackend, so in addition to deprecating the methods have been set to 'final' and have checkNoEntry implementations which is the best we can do at the moment.
-- It is unlikely that people have actually done this as the documentation on adding new backend types will only go out in 5.3.
[CL 25039959 by paul chipchase in ue5-main branch]
#rb none
#jira UE-177956
#preflight
- Deprecate the config entry 'DepotRoot' and replace it with 'DepotPath'. The use of the term root was not clear and was causing people to think it was the name of the depot rather than the path to the root location to store payloads in.
- If 'DepotPath' is empty or missing, the value for 'ClientStream' will be used for it.
-- The 'DepotPath' can still be set when using streams, if the name of the stream is not a valid path (virtual streams for example.
[CL 25024791 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-180257
#preflight 6411cdbb5819afacaf06fbc9
- By default the value is true and the local ini file is used.
- Allows teams to opt out of loading settings from the locally saved ini file if they wish to.
[CL 24654053 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-175158
#preflight 63f8cd34ae54ee4ce9ea207b
- We really should just read this value out of the users p4 environment settings, but that would require a source control api change and would need to be done in a source control agnostic way. Letting a project just define the name is a quick way to expose the functionality.
- At some point if we do extend the source control api then this option can be deprecated.
[CL 24400577 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-175157
#rnx
#preflight 63f8b493ae54ee4ce9de6b78
- This feature was requested so that a team could opt into submitting from their appdata or temp directories instead of the project saved directory (our default)
- Deprecated the option '-SubmitFromTempDir' as it could be replacated with "-WorkingDir=$(Temp)/UnrealEngine/VASubmission"
[CL 24399707 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-175189
#rnx
#preflight 63d27eae450d5cdd0b4df619
- We were raising a FMessageLog to print a warning, then using ::OnConnectionError to defer the FMessageLog::Notify call until the next editor tick so an easy fix is to pass the warning to ::OnConnectionError and piggyback onto the defferal if needed.
- The messages used to be errors but were demoted during development, now that FSourceControlBackend has SuppressNotifications option and can supress the notifications on a per project basis we can restore the messages to errors.
- When calling ::OnConnectionError we will send the message to FMessageLog if we are currently on the game thread, then defer the notification until next tick as we were doing so previously. The deferral is needed in case the notify comes during editor start up (which for a connection problem is likely) so that the notification appears at the correct time. If we are not on the game thread we should still print the message to the log file so that debugging is easier, only the FMessageLog part needs to be deferred to the GameThread. In this case we need to tell FMessageLog to not print to the log file to avoid duplicates.
[CL 23919185 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63526abcae33b04ec1ee4a65
- The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker.
-- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use.
-- The DDC backend should be taking full advantage of the new batching API
-- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit.
- This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc.
-- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away.
[CL 22707955 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163834
#rnx
#preflight 632d98e4b4515b7e221654ec
### Virtualization
- At the moment there is no way in the editor UX to change the source control settings for the source control backend, so what ever settings are found in the global environment would be applied when the editor is first run and then saved to a config file under the <saved> directory. From this point onwards those settings would always be used when setting up the source control backend. If the settings were wrong, the first time that the editor was started, then we'd continue to use the incorrect settings even if the global enviroment was fixed. Making the backend work at that point would require that the user know about the local ini file and change the settings there.
- We cannot fix this by just ignoring the ini file as it has been requested that users can add their settings there in case they want to customize things, so we just need to avoid saving to it.
- Added documentation to the source control backends header file on how to set up the ini file if needed.
### Perforce
- FSourceControlInitSettings now accepts EConfigBehavior which describes how we want the source control provider to deal with its settings with regards to the ini file.
- Note that EConfigBehavior is not a bitfield as we do not wish to support only writing to the ini file, there is no point if it cannot be read from.
- When the source control provider is creatred we pass on the into from EConfigBehavior to the FPerforceSourceControlSettings to enable/disable saving and loading.
[CL 22163769 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-164284
#rnx
#preflight 632b182ffc7f1efbdf9638d9
- This is only useful if the entire team uses the same address for the perforce server
- When set, this will override the environment settings for P4PORT as well as override what ever is saved to SourceControlSettings.ini
[CL 22122737 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163103
#rnx
#preflight 6318989c2b7fe03eb664e9f0
### VirtualizationSystem/VirtualizationManager
- Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request
- Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety.
### IVirtualizationBackend
- The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs.
- Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters.
-- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942
- Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing.
- File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests.
-- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not.
[CL 21907558 by paul chipchase in ue5-main branch]
#rb Devin.Doucette
#jira UE-161599
#rnx
#preflight 6303c8d65a5d4e4624e7bf52
- There are some use cases that require the VA system to be initialized and configured correctly but would prefer that the backend connections only run if absolutely needed (usually when a payload is pulled or pushed for the first time), this change provides four different ways of doing this:
-- Setting [Core.VirtualizationModule]LazyInitConnections=true in the Engine ini file
-- Setting the define 'UE_VIRTUALIZATION_CONNECTION_LAZY_INIT' to 1 in a programs .target.cs
-- Running with the commandline option -VA-LazyInitConnections
-- Setting the cvar 'VA.LazyInitConnections' to 1 (only works if it is set before the VA system is initialized, changing it mid editor via the console does nothing)
--- Note that after the config file, each setting there only opts into lazy initializing the connections, setting the cvar to 0 for example will not prevent the cmdline from opting in etc.
- In the future we will allow the connection code to run async, so the latency can be hidden behind the editor loading, but for the current use case we are taking the minimal approach.
-- This means we only support the backend being in 3 states. No connection has been made yet, the connection is broken and the connection is working.
-- To keep things simple we only record if we have attempted to connect the backends or not. We don't check individual backends nor do we try to reconnect failed ones etc. This is all scheduled for a future work item.
- If the connections are not initialized when the VA system is, we wait until the first time someone calls one of the virtualization methods that will actually use a connection: Push/Pull/Query
-- We try connecting all of the backends at once, even if they won't be used in the call to keep things simple.
- Only the source control backend makes use of the connection system. The horde storage (http) backend could take advantage too, but it is currently unused and most likely going to just be deleted so there seemed little point updating it.
- If we try to run an operation on an unconnected backend we only log to verbose. This is to maintain existing behaviour where a failed backend would not be mounted at all. This logging will likely be revisited in a future work item.
[CL 21511855 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-156189
#rnx
#preflight 62f5f1a01e61d1ba0e6c893b
- Note that the older name 'SourceControl' will continue to work but will log a warning if used to notify the user to update their config files.
[CL 21352223 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-160619
#preflight 62f37c86b66d5d93133d67e3
- Based on work from Jess.Kube
### Virtualization
- The source control backend now takes an optional config option "ClientStream" which takes the name of the client stream to use.
- If a client stream is set then workspaces created for payload submission will use that and not provide a client-view mapping.
#ushell-cherrypick of 21195584 by Jess.Kube
### PerforceSourceControl
- Allow FCreateWorkspace to create workspaces with streams as well as classic workspaces cia FCreateWorkspace::SetStream.
- Add a method FCreateWorkspace::ClearClientViewMappings which will clear any client view mappings already added to the operation.
- If we detect that a FCreateWorkspace operation has both a stream set and client view mappings set then FPerforceCreateWorkspaceWorker will return an error. Perforce will allow us to creat a client spec with both entries, but will default to using the stream. Technically we could allow this too but it might cause unexpected behaviour to the caller. It is better to give a clear error and fail the workspace creation.
[CL 21316756 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-158771
#rnx
#preflight 62c6c8f2f5590c326d77b0f3
- This is a temp fix to an internal problem, where we have a group of people unable to access the source control payload storage system due to permissions but do not actually need to access it at the moment.
- By supressing the connection failure pop up we can allow them to keep working undistrubed.
- Improving how we handle error notifications like this is already on the VA roadmap, at which point this change can be removed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20982887 via CL 20982891 via CL 20982896
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 20984627 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-151000
#rnx
#preflight 627291732f6d177be3c292e3
- When submitting large batches it was found that once a cl description reached a certain length, that p4v would lock up attempting to display it in the history tab, so we want to be able to make smaller submits to keep the description to a managable size.
- In addition it is better for our server set up to make multiple smaller batch submits than insanely large ones.
- The max size of the batch can be set when creating the virtualization graph in the config file.
[CL 20055325 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-147948
#preflight 6271209be16e280be60b5a62
- In the worst case scenario (where a machine can only access the source control backend) we will want to limit the potential number of concurrent connections that they can make.
- In testing, not limiting the connections could cause the occasional connection failure and I imagine having a full team running in the worse case scenario would quickly overwhelm the server.
- Currently we default to 8 connections (the limit allowed in UGS)
- The semaphore is a quick and dirty implementation similar to std::counting_semaphore. I am not spending much time on it as the worst case scenario is not expected to be common and if it does occur, the system is going to be painfully slow no matter how good the thread throttling.
### Future work
- This work could be taken a lot further by either
-- a) reusing connections rather than creating one for each pull, which will require changes to our source control api
-- b) gathering requests from many requesting threads once the limit has been reached and then sending the requests in batches
--c) reserve one connection for the game thread to prevent the user waiting?
[CL 20024594 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Sebastian.Nordgren
#jira UE-147948
#preflight 6256ba2f647ad886b380d007
- In some cases people are getting minor connection failures to the perforce server which causes the p4 print command to fail. In these cases it would be better to retry the command rather than failing to get the payload entirely
- Currently the default will be 2 attempts to pull a payload with a 100ms wait between them, this can be changed when setting up the backend graph as optional config values "RetryCount=" and "RetryWaitTime="
-- Note that RetryWaitTime is in ms
- Moved the parsing of the command line to it's own method in the backend.
[CL 19752733 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-136480
#preflight 62289b59f4469cadac10eb91
- The backend can now connect to source control without forcing the rest of the editor to connect to source control.
- If the user disconnects the editor from source control, the backend will still be able to function using it's own connection.
-When the backend needs to change the current workspace in order to submit payloads we no longer invalidate the file cache used by the editor and so the user will no longer have to manually refresh files in the content browsers to restore the source control status icon.
[CL 19318970 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
