#rb Per.Larsson
#rnx
- In addition to fixing up the filter dispaly I have also added a FSlowTask as this command can take longer on machines with slower HD when run on many packages at once.
[CL 35339449 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
- Currently we apply filtering to payloads at two points, package save and during virtualization. The reasons applied during package save are stored int he payload trailer and are easy to access when making reports but we needed a way to find the second set of filters.
- This set up is fairly hacky as we are accessing the default FVirtualizationManager directly, this is because I wanted to avoid making a public interface change as I am not 100% sure that we should be trying to fix filtering this way and am leaning towards either moving all filtering to a single point (both have their advantages and disadvantages) or making the difference between these two stages more explicit and for either solution I don't want to be tied to any interface changes we'd need to make here.
- The new utilities header I added is internal so we can change it as we need, it is also marked as experimental so licensees should be avoiding it anyway.
[CL 35335884 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-214357
#rnx
- TSAN picked up that FVirtualizationManager::bPendingBackendConnections could be accessed on multiple threads while being written too. Technically this is expected which is why the bool was checked a second time after acquiring the lock but to play things safe the bool has been changed to a std::atomic<bool>
- Changed InitCS from a FCriticalSystem (which can use an OS level primitive) to the less expensive FMutex since it's almost never going to be used.
[CL 33761948 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-147384
#rnx
- The previous functor based system was difficult to update as changes to the data being passed to the functor could break existing code, so ::GetAccumualtedPayloadActivityInfo has been replaced with ::GetBackendStatistics, a version that returns a struct for each backend in the system which contains the relevant statistics for that backend. Adding new data to a struct is much easier.
-- The new data added per backend is the storage type so that the caller can tell if the backend was for cached storage or persistent storage.
- To fit into this naming scheme ::GetAccumualtedPayloadActivityInfo has been deprecated in favor of ::GetSystemStatistics which is simply a rename.
- The deprecated methods should continue to work as before.
- Existing engine code had been updated to use the new methods.
- Removed a typo from the copyright boilerplate at the begining of VirtualizationSystem.h
[CL 33626815 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-147384
#rnx
- Note that these stats are only logged after selected commandlets have run (cooking, precaching etc)
- Replaces the data transfer rate statistic with an average request time to reflect recent changes to the statistics panel in the editor.
- Compacted the table to remove some unnecessary padding.
[CL 33602676 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-147384
#rnx
- When this code was moved from elsewhere to FVirtualizationManager we neglected to remove the calls to IVirtualizationSystem. The code was working just fine but on first glance it is more complicated that it needed to be as it looks like a static method (hence needing to find the active IVirtualizationSystem pointer) but it is not.
- Reserve the space we need in the attributes array.
[CL 33509346 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-168484
#rnx
- Add a verbose log message each time a payload is actually cached to a backend detailing the backend and the payload identifier.
-- Should only log if the payload actually uploads to the given backend, if it is rejected because the payload already exists in the backend we will not log anything.
- In order for this to work we need to reset the status of each FPushRequest being cached as we might have more than one caching backend and we don't want to includes pushes from one backend to another.
[CL 33452263 by paul chipchase in ue5-main branch]
#rb mark.lintott
#jira UE-165045
#rnx
- Add a new event (GetAnalyticsRecordEvent) to the UE::Virtualization namespace. Different telemetry systems can hook into this and respond when the virtualization system tries to send a new analytics event.
-- This even accepts an enum class of flags to provide better context for the event. At the moment the only valid flag is to request that the telemetry system flushes the event to make sure that it is sent before returning control to the event. In this way we can know that events are safely recorded before terminating the current process.
- To begin with we are only adding failure events and not recording any successes
-- Send an analytics even when the initial connection to perforce fails, the event should note if the user opted to retry their connection (after entering new details) or if they wanted to skip the check and load the editor anyway.
-- Send an analytics event when a payload fails to pull, the event should note if the user opted to retry to pull the payload or quit the process.
[CL 31087065 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-203381
-Some users have reported seeing their long cooks fail due to a machine having an unreliable connection or networkcard. In these cases the network outage can be swiftly fixed but if VA failed to pull a payload before then, the cook will terminate and need to be restarted which can cost a lot of time.
- It was requested that we add an optional way to have the system retry payload pulling when running in unattended mode but also to wait for X time (usually many minutes) before trying again. If it is likely that the connection will be restored within those few minutes then waiting will be much less costly than restarting a cook.
- That payloads can be pulled on many threads at the same time makes the logic a little tricky, so rather than counting how many payloads have failed vs the retry counter we count how many times we've logged a message to the user as this logging is protected by a critical section and acts as a way to "group" together failed pulls that occur around about the same time. We then reset this counter to 0 if we detect a successful pull.
- It is possible that a pull fails because the payload is missing, in which case this logic will probably cause the counter to reset frequently and the error to not become fatal for quite some time (possibly until the cook has almost finished) but it is quite unlikly to occur and due to this I have favored erring towards simple code rather than trying to track individual payload failures vs grouped failures vs successful pulls.
- Note: That when backends fail to pull payloads they generally log errors, which will eventually cause most of our processes to return non zero to indicate failure. VA should not log errors while we are inside of a retry loop and only print out errors when we detect a problem that we cannot solve to avoid this. This is being addressed as it's own work item.
[CL 30925886 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-203597
- When we first parse the value from the config file we should check to see if it contains and path separators and if so reject the backend entirely. Providing a full path rather than just the filename is an easy enough mistake to make and it is better for the user to find out sooner than later.
- The utility function ::TryCreateSubmissionSessionDirectory was extended after it was first created to also create a dummy ignore file, updated the name to reflect that it does more than just create a directory.
- We were relying ont he creation of the dummy file to also create the directory but that means it is harder for us to inform the user if an encountered problem is due to the directory creation or the file creation. Now we ensure that the directory is created first (with error handling) and only then try to create the dummy file.
- Added logging of the system error if we encounter issues to give the user a better idea what the issue was.
[CL 30532386 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
- When generating content browser thumnails GIsRunningUnattendedScript will be set to true which prevents us from showing the VA payload pull failed dialog, so if a payload does fail during thumbnail generation the process will just terminate rather than asking the user if they would like to retry the operation.
- We no longer consider 'GIsRunningUnattendedScript' when checking if the process is interactive.
- If the dialog is being displayed from the GameThread we can force GIsRunningUnattendedScript to false safely allowing the dialog to be displayed. If we fail a payload pull on a background thread while the GameThread is generating a thumbnail we will not be able to safely change GIsRunningUnattendedScript (which does not have proper thread guarding) so we will just have to accept that the dialog will fail to show and the process will terminate.
[CL 29747059 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-198807
#rnx
- We only needed the MessageLog module to setup a VA tab in the message log slate window which only matters if we are actually building for a target that uses slate.
[CL 29043098 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-180383
#rnx
- With the new error flow we block the process on a failed pull and then quit the process if the connection cannot be recovered, this means that we never reach the code that reports which payloads failed.
- Added similar reporting to FVirtualizationManager::OnPayloadPullError so we can get a log of which payloads failed to pull.
-- A pull failure should not really be dependant on the payload itself so this info isn't that important but it can be useful to eliminate potential issues by allowing us to confirm that the payload does in fact exist in persistent storage and the user was just unable to access it.
[CL 28833473 by paul chipchase in ue5-main branch]
#rb Martin Ridgers
#jira UE-191791
#rnx
- Add a new ::IsProcessInteractive function to UE::Virtualization::Utils namespace that will return true if we think the current process is being run in a scenario where a user will be present and that it is okay to show an interactive error dialog.
- Use this new check to make sure that we do not try to display the "connection error" dialog on start up (even though as a slate dialog this will usually fail to display in non-interactive scenarios anyway)
- Use the new check to avoid displaying the "payload pull failure" dialog and default to the process shutting down instead.
-- Added some code comments to explain why we force the process to exit rather than logging a fatal error.
[CL 28183133 by paul chipchase in ue5-main branch]
[Backout] - CL27745134
[FYI] stan.hormell
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others
[CL 27923017 by josh adams in ue5-main branch]
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others
[CL 27745141 by stan hormell in ue5-main branch]
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others
[CL 27731364 by josh adams in ue5-main branch]
#rb trivial
#jira UE-193786
#rnx
- Also removed some commented out code and a pointless local variable as code cleanup while I was on.
[CL 27341309 by paul chipchase in ue5-main branch]
#rb none
#jira UE-191791
#rnx
- When a connection attempt fails we now return the credentials used and the errors given by the system.
-- The credentials will be used to auto fill out the dialog, letting the user see what credentials were attempted in the last attempt.
-- The errors are shown in a box with red error text, letting them know what went wrong.
- We no longer let the revision control provider log errors when establishing a connection, so if the users correct info and successfully connects on a subsequent attempt they will not see an error in the logs.
-- If the connection completely fails we still log as before.
[CL 27302825 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-191791
#rnx
- Removed direct references to perforce from the dialog code, those strings are now passed in by the caller. This should make the dialog more usable by other revision control providers at some point.
- Make it more obvious that the dialog is related to virtualized assets.
- Add more references to the revision control provider name so that people realize which settings they are supposed to be entering.
- Removed some debug code for launching the dialog from a console command.
- Fixed a bug where clicking the button to reset the ini file settings and use the default connection would reset the settings but then not try to reconnect.
- Make better use of the dialog space by increasing the size of the input boxes at the expense of white space, so that longer server addresses can fit on screen.
[CL 26769098 by paul chipchase in ue5-main branch]
