#rb none
### Dialog
- When the source control backend tries to connect and fails we will now show a dialog allowing the user to try new settings rather than let the editor continue and crash.
- The dialog will allow the user to enter the server address (P4PORT) and their username (P4USER).
- Once the dialog is displayed the user will have three options to continue.
-- "Reset To Defaults" which will remove all settings saved to the users local ini files and attempt to use the environment defaults to connect.
-- "Retry Connection" which will take the new address and username that the user provided and attempt to connect with those settings.
-- "Skip" no connection attempt will be made. This may cause instability later if the user needs data that they cannot access.
- If the connection succeeds after the user has reset to defaults or provided new settings, then they will be saved in the local ini file for future sessions.
- The dialog will not show if:
-- Slate is disabled for the current processor has not yet initialized
-- The error is found not on the game thread. This means that lazy init will not currently work well with this dialog. Attempts to mashal the error to the game thread have been spotty at best and there is a high chance of thread locking so attempts were abandoned.
-- If the -unattended flag is set on the process.
-- If the dialog is disabled via the config file when setting up the backend (UseRetryConnectionDialog=False)
-- If 'engine.ini:[Core.VirtualizationModule]:UseLocalIniFileSettings' is false as future sessions would not be able to use the new settings and would become tiresome. In these cases the environment settings will need to be fixed.
- If the dialog is now shown then the backend is marked as unconnected and will not work.
### Known Issues
- We currently cannot get the settings that the failed p4 connection used, nor the error message dueto how the API works. So for now we just display an empty server address and username edit box. This will be fixed when the API is changed.
- There is currently no good way to inform the user where their settings come from.
- There is no easy way to reset the settings once saved. To allow the user to change the server address to a better proxy for example.
### Initialization
- In order to allow a slate dialog to be used, the initialization of the VA system has been moved later in pre init, to the first avaliable point after slate has initialized and its shaders have been built. This in theory should not cause a a problem as engine content cannot be virtualized.
-- Added a new option 'engine.ini:[Core.ContentVirtualization]:InitPreSlate' with a default of false. When set to true the VA system will initialize in the original, earlier portion of engine pre init. This is only included in case an existing virtualized project encounters a problem with the new ordering.
[CL 26115735 by paul chipchase in 5.3 branch]
#rb none
#rnx
#preflight 645c95abaa3c584c0b1c91f0
- A url can now be provided via 'ini:Engine:[Core.VirtualizationModule]:ConnectionHelpUrl=""' that will be added to various connection errors. If the error message is via FMessageLog then the user will be able to click the url to go straight to the link.
- This sort of help will be very company specific, this url will allow teams to build up their own FAQ for their users.
- Currently used by the source control backend if the initial connection fails but will be expanded to other areas.
- The url is currently parsed and owned by the virtualization manager but at the moment the connection error is issued by each backend. So right now there is a static accessor allowing backends to get the value to use. This would not work properly for 3rd party backends and will be changed in the near term.
[CL 25424000 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 6458e2596c35ad81e60f2fa9
- I thought I had already done this but I see nothing in the p4 history, so it is likely that I forgot to actually submit the change.
- Fix some additional warnings found under clang (C5219)
[CL 25370075 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 6447d1954052d26a5a8c36cc
- Ends up pretty much the same,as FCompressedBuffer::FromCompressed will return a Null FCompressedBuffer if the provided data is null, but adding the check makes it clear when reading FSourceControlBackend::PullData that payloads might not be found and we are not setting them in that case.
[CL 25179230 by paul chipchase in ue5-main branch]
#rb none
#jira UE-181418
#rnx
#preflight 6447c5482804595a04d1e375
- Each pull operation now takes a FText that can be filled in with additional info about any failures that will cause the new error dialog to be displayed.
-- At the moment we only have info to provide from the source control backend if the connection itself failed.
-- In the future we might want to completely overhaul our error logging and display all messages to the user but our logging system is not really built for that level of redirection at the moment.
- Only pass on the error if it came from a persistent backend, we don't need to worry the user about cached backends which are allowed to fail.
[CL 25178752 by paul chipchase in ue5-main branch]
#rb none
#jira UE-182205
#preflight 643d2901c947f6523a2f5845
- Most backends support some form of existence check which is run before a push to prevent uploading data that is already there. Sometimes we want to ignore that check (bugs in the backend etc) and need a way to force that to happen.
- The new config option [Core.VirtualizationModule]:ForceCachingOnPull (default false) will do this when set to true.
- The file and DDC backends will respect the flag but for now the source control backend will continue to check to avoid accidental pointless revisions being added.
[CL 25066347 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-182205
#preflight 643932b2211b661dc4f594c6
- The enums don't contain any entries at the moment, the goal of this change is to reduce the friction that future API changes might have.
- The older versions of PullData/PushData have been deprecated but this won't flag compiler warnings if people have derived their own implementations of IVirtualizationBackend, so in addition to deprecating the methods have been set to 'final' and have checkNoEntry implementations which is the best we can do at the moment.
-- It is unlikely that people have actually done this as the documentation on adding new backend types will only go out in 5.3.
[CL 25039959 by paul chipchase in ue5-main branch]
#rb none
#jira UE-177956
#preflight
- Deprecate the config entry 'DepotRoot' and replace it with 'DepotPath'. The use of the term root was not clear and was causing people to think it was the name of the depot rather than the path to the root location to store payloads in.
- If 'DepotPath' is empty or missing, the value for 'ClientStream' will be used for it.
-- The 'DepotPath' can still be set when using streams, if the name of the stream is not a valid path (virtual streams for example.
[CL 25024791 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-180257
#preflight 6411cdbb5819afacaf06fbc9
- By default the value is true and the local ini file is used.
- Allows teams to opt out of loading settings from the locally saved ini file if they wish to.
[CL 24654053 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-170132
#rnx
#preflight 640f33c5363e9b40ab8cd7af
- Output the entire depot path when failing to download 'payload_metainfo' along with the address of the server.
- We ask the source control provider for the status and parse the p4 port from that as there is no way (currently) to get this info directly via the source control api, which currently works but is a bit flaky long term.
-- I am not sure if we want to keep the error reporting like this and so I am loath to make changes to the source control API unless I am sure we actually want the functionality.
- Removed duplicate code paths, there is now only one branch that will log an error.
- Removed call to FConnect as that does not do anything unless a client is set and we do not use one with this connection. (FConnect is mostly about client set up validation rather than actually connecting to the server since we always make a new perforce connection for each command anyway.
-- Instead we now check if the source control provider is available after we call Init, if this returns false then no connection was made.
- Documented the purpose behind calling ISourceControlProvider::Init
[CL 24613062 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-175158
#preflight 63f8cd34ae54ee4ce9ea207b
- We really should just read this value out of the users p4 environment settings, but that would require a source control api change and would need to be done in a source control agnostic way. Letting a project just define the name is a quick way to expose the functionality.
- At some point if we do extend the source control api then this option can be deprecated.
[CL 24400577 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-175157
#rnx
#preflight 63f8b493ae54ee4ce9de6b78
- This feature was requested so that a team could opt into submitting from their appdata or temp directories instead of the project saved directory (our default)
- Deprecated the option '-SubmitFromTempDir' as it could be replacated with "-WorkingDir=$(Temp)/UnrealEngine/VASubmission"
[CL 24399707 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-175189
#rnx
#preflight 63d27eae450d5cdd0b4df619
- We were raising a FMessageLog to print a warning, then using ::OnConnectionError to defer the FMessageLog::Notify call until the next editor tick so an easy fix is to pass the warning to ::OnConnectionError and piggyback onto the defferal if needed.
- The messages used to be errors but were demoted during development, now that FSourceControlBackend has SuppressNotifications option and can supress the notifications on a per project basis we can restore the messages to errors.
- When calling ::OnConnectionError we will send the message to FMessageLog if we are currently on the game thread, then defer the notification until next tick as we were doing so previously. The deferral is needed in case the notify comes during editor start up (which for a connection problem is likely) so that the notification appears at the correct time. If we are not on the game thread we should still print the message to the log file so that debugging is easier, only the FMessageLog part needs to be deferred to the GameThread. In this case we need to tell FMessageLog to not print to the log file to avoid duplicates.
[CL 23919185 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-174290
#rnx
#preflight 63c8fa22d15a37ae31487446
- Previously once the number of perforce connections allowed was reached we'd block all additional requests until an outstanding connection as been released at which point a random waiting thread would get the go ahead to make its own connection and so on. If the GameThread was making a request under these conditions the user could be left with a blocked editor while they wait for the GameThread to get a turn.
- This change allows FSourceControlBackend to create a semaphore that gives priority to the GameThread during contention so that the user gets the fastest possible response.
- FSemaphore now takes a bitfield of flags as a parameter in the constructor to allow us to specialize its behavior with the only support option being to prioritize the game thread.
- If the GameThread tries to acquire the semaphore and the prioritization option is enabled we use a slightly different path code path ::AcquireFromGameThread
-- Unlike the normal acquire code path, this path does not increment the count when waiting. This effectively reserves the next open slot for the calling thread.
-- Given that there is only one request on the GameThread at a time we don't have to check Counter once the event has been triggered. The even triggering means we know that one of the slots has been freed and we know we have already reserved that slot so we can just go ahead and return success at that point.
- Added a profile scope to acquire as that would've made the original problem easier to catch.
- Still need to look into replacing this with std::counting_semaphore once we can rely on c++ 20 support.
[CL 23771977 by paul chipchase in ue5-main branch]
Tested compiling fortnite, unrealeditor, lyra, qagame with non-unity/pch
#preflight 63635997876630122adeab9f
#rb none
[CL 22958990 by henrik karlsson in ue5-main branch]
#rb PJ.Kack
#jira UE-167578
#rnx
#preflight 635bdddd5b54febd38b7efd0
- Clean up the source control backend by removing the older code path that was ifdefed out and clean up the new path.
- FPerforceConnection no longer takes a TOptional<FSharedBuffer> but takes an optional pointer to a TArray of FSharedBuffer. When gathering files this should return one FSharedBuffer per file requested.
-- It might make more sense to attach the FSharedBuffer to the FP4Record but that would require some bigger changes and would be better suited to being some follow on work. All the code is private in the module and not exposed so it is fairly safe and easy to do these refactors later.
- Replaced the TArray buffer for gathering files with a purpose built class FP4DepotFile.
- When issuing a p4 print command with multiple files we will get one call to ClientUser::OutputStat followed by calls to either ClientUser::OutputText or ClientUser::OutputBinary until all of the file has been delivered, then we will get another call to ClientUser::OutputStat for the next file and so on.
-- Note with binary files we seem to get one final call to ClientUser::OutputBinary with zero size, which I assume is supposed to signal the end of the transfer, text files do not get this, they seem to just stop.
[CL 22888671 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63526abcae33b04ec1ee4a65
- The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker.
-- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use.
-- The DDC backend should be taking full advantage of the new batching API
-- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit.
- This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc.
-- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away.
[CL 22707955 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160942
#rnx
#preflight 6336f2ab5c2225fe5f6c08a5
- Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead.
-- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc.
- Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown.
- Changed the backends to use the new results system.
-- Not all paths were correctly setting the state in the file system backend.
-- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do.
- Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload.
[CL 22279127 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163834
#rnx
#preflight 632d98e4b4515b7e221654ec
### Virtualization
- At the moment there is no way in the editor UX to change the source control settings for the source control backend, so what ever settings are found in the global environment would be applied when the editor is first run and then saved to a config file under the <saved> directory. From this point onwards those settings would always be used when setting up the source control backend. If the settings were wrong, the first time that the editor was started, then we'd continue to use the incorrect settings even if the global enviroment was fixed. Making the backend work at that point would require that the user know about the local ini file and change the settings there.
- We cannot fix this by just ignoring the ini file as it has been requested that users can add their settings there in case they want to customize things, so we just need to avoid saving to it.
- Added documentation to the source control backends header file on how to set up the ini file if needed.
### Perforce
- FSourceControlInitSettings now accepts EConfigBehavior which describes how we want the source control provider to deal with its settings with regards to the ini file.
- Note that EConfigBehavior is not a bitfield as we do not wish to support only writing to the ini file, there is no point if it cannot be read from.
- When the source control provider is creatred we pass on the into from EConfigBehavior to the FPerforceSourceControlSettings to enable/disable saving and loading.
[CL 22163769 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-164284
#rnx
#preflight 632b182ffc7f1efbdf9638d9
- This is only useful if the entire team uses the same address for the perforce server
- When set, this will override the environment settings for P4PORT as well as override what ever is saved to SourceControlSettings.ini
[CL 22122737 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163103
#rnx
#preflight 6318989c2b7fe03eb664e9f0
### VirtualizationSystem/VirtualizationManager
- Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request
- Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety.
### IVirtualizationBackend
- The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs.
- Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters.
-- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942
- Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing.
- File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests.
-- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not.
[CL 21907558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161973
#rnx
#preflight 63074424ae13a5a09865a281
- We could try force loading and enabling the plugin at this point but I am not a fan of the idea of overriding the set up of the target in a way that is not immediately obvious to the end user.
- The virtualization module needs to include the 'Projects' module to be able to access the plugin system directly and while editing the file I took the opportunity to clean up how the module is declaring its dependencies.
[CL 21684932 by paul chipchase in ue5-main branch]
#rb Devin.Doucette
#jira UE-161599
#rnx
#preflight 6303c8d65a5d4e4624e7bf52
- There are some use cases that require the VA system to be initialized and configured correctly but would prefer that the backend connections only run if absolutely needed (usually when a payload is pulled or pushed for the first time), this change provides four different ways of doing this:
-- Setting [Core.VirtualizationModule]LazyInitConnections=true in the Engine ini file
-- Setting the define 'UE_VIRTUALIZATION_CONNECTION_LAZY_INIT' to 1 in a programs .target.cs
-- Running with the commandline option -VA-LazyInitConnections
-- Setting the cvar 'VA.LazyInitConnections' to 1 (only works if it is set before the VA system is initialized, changing it mid editor via the console does nothing)
--- Note that after the config file, each setting there only opts into lazy initializing the connections, setting the cvar to 0 for example will not prevent the cmdline from opting in etc.
- In the future we will allow the connection code to run async, so the latency can be hidden behind the editor loading, but for the current use case we are taking the minimal approach.
-- This means we only support the backend being in 3 states. No connection has been made yet, the connection is broken and the connection is working.
-- To keep things simple we only record if we have attempted to connect the backends or not. We don't check individual backends nor do we try to reconnect failed ones etc. This is all scheduled for a future work item.
- If the connections are not initialized when the VA system is, we wait until the first time someone calls one of the virtualization methods that will actually use a connection: Push/Pull/Query
-- We try connecting all of the backends at once, even if they won't be used in the call to keep things simple.
- Only the source control backend makes use of the connection system. The horde storage (http) backend could take advantage too, but it is currently unused and most likely going to just be deleted so there seemed little point updating it.
- If we try to run an operation on an unconnected backend we only log to verbose. This is to maintain existing behaviour where a failed backend would not be mounted at all. This logging will likely be revisited in a future work item.
[CL 21511855 by paul chipchase in ue5-main branch]
