Commit Graph

207 Commits

Author SHA1 Message Date
paul chipchase
2fd6372116 Add a new config option 'ForceCachingOnPull' to VA that when true will force backends to re-upload payloads when caching.
#rb none
#jira UE-182205
#preflight 643d2901c947f6523a2f5845

- Most backends support some form of existence check which is run before a push to prevent uploading data that is already there. Sometimes we want to ignore that check (bugs in the backend etc) and need a way to force that to happen.
- The new config option [Core.VirtualizationModule]:ForceCachingOnPull (default false) will do this when set to true.
- The file and DDC backends will respect the flag but for now the source control backend will continue to check to avoid accidental pointless revisions being added.

[CL 25066347 by paul chipchase in ue5-main branch]
2023-04-17 07:45:19 -04:00
paul chipchase
d3e14c816b Added code documentation for 'PullErrorAdditionalMsg' that was missing from CL 24714178
#rb trivial
#jira UE-180383
#rnx
#preflight 643cfc2c211b661dc4d3bc0c

[CL 25065493 by paul chipchase in ue5-main branch]
2023-04-17 04:18:35 -04:00
jamie dale
2fae9fc218 Use the FMessageDialog overloads that pass the optional title by-value
#jira
#rb none
#rnx

[CL 25062745 by jamie dale in ue5-main branch]
2023-04-15 22:21:07 -04:00
paul chipchase
c65eb35e8c Add bitflag enums to IVirtualizationBackend::PullData and IVirtualizationBackend::PushData so that it is easier to specialize behavior in the future.
#rb Per.Larsson
#jira UE-182205
#preflight 643932b2211b661dc4f594c6

- The enums don't contain any entries at the moment, the goal of this change is to reduce the friction that future API changes might have.
- The older versions of PullData/PushData have been deprecated but this won't flag compiler warnings if people have derived their own implementations of IVirtualizationBackend, so in addition to deprecating the methods have been set to 'final' and have checkNoEntry implementations which is the best we can do at the moment.
-- It is unlikely that people have actually done this as the documentation on adding new backend types will only go out in 5.3.

[CL 25039959 by paul chipchase in ue5-main branch]
2023-04-14 10:13:46 -04:00
paul chipchase
b96c7d6df6 Clean up the config setup options for the P4SourceControl VA backend.
#rb none
#jira UE-177956
#preflight

- Deprecate the config entry 'DepotRoot' and replace it with 'DepotPath'. The use of the term root was not clear and was causing people to think it was the name of the depot rather than the path to the root location to store payloads in.
- If 'DepotPath' is empty or missing, the value for 'ClientStream' will be used for it.
-- The 'DepotPath' can still be set when using streams, if the name of the stream is not a valid path (virtual streams for example.

[CL 25024791 by paul chipchase in ue5-main branch]
2023-04-13 10:11:47 -04:00
paul chipchase
3de71b52ed Log the data reduction when virtualizing packages
#rb trivial
#jira UE-182542
#preflight 64369ea4d03b1c87dd91f15b

- We now log the total size saving by running the virtualization process as well as the original size of the packages being virtualized and their resulting size.
- If LogVirtualization is set to verbose logging we will also log each individual package and its size reduction.

[CL 25005336 by paul chipchase in ue5-main branch]
2023-04-12 08:24:53 -04:00
paul chipchase
70e8a4d7b3 Reduce the disk space overhead when virtualizing packages.
#rb none
#jira UE-182542
#preflight 643699cdeb6cd0e63a6bd729

- Previously we were duplicating all packages being virtualized, without the recently virtualized payloads, then try to move them one at a time to replace the original packages.
- Now we do all of the prep work such as checking the packages out of revision control and checking that they can be written to first, then duplicate & move each package one at a time.
-- This way we only ever have a single additional duplicated package at a time.
- Moved the final summary logging to an exit scope so we can return when ever we want in the function.
- Added early out for error case.
- Added early out if there are no packages to remove virtualized payloads from.
- Improved the code commands in IVirtualizationSystem::TryVirtualizePackages so that the caller knows the potential failure case scenarios.

[CL 25005153 by paul chipchase in ue5-main branch]
2023-04-12 08:02:36 -04:00
paul chipchase
31fd4ad8e2 CIS - Code formatting fix, remove newline that was added at the top of the cpp
#rb trivial
#jira none
#rnx
#preflight

[CL 25003922 by paul chipchase in ue5-main branch]
2023-04-12 04:51:15 -04:00
paul chipchase
b0a5ab7783 Change the VA filesystem backend to log the absolute file path of its directory so that it is easier for users to find.
#rb trivial
#jira none
#preflight 64365842d03b1c87dd7ca47d

- When the path being logged is relative, there is sometimes confusion as to which directory it is relative to.

[CL 25003785 by paul chipchase in ue5-main branch]
2023-04-12 04:13:35 -04:00
paul chipchase
43751b70d9 Use a specific subdirectory under 'Saved' when duplicating packages for virtualization/hydration so that it is easier to identify disk bloat caused by the duplication and to clean up failed processes.
#rb trivial
#jira UE-182542
#rnx
#preflight 643657e30c4277fc0b72f928

[CL 25003781 by paul chipchase in ue5-main branch]
2023-04-12 04:12:33 -04:00
paul chipchase
89483c750a Add some more profiling scopes to the package virtualization project
#rb trivial
#jira UE-182542
#rnx
#preflight 64365867eb6cd0e63a533159

- Attempting to hunt down a performance issue that is currently only occuring for a single user and cannot be reproduced elsewhere.

[CL 25003776 by paul chipchase in ue5-main branch]
2023-04-12 04:11:47 -04:00
paul chipchase
57da929556 Add code comment explaining why we need to check the file states in revision control to avoid the check being removed in the name of optimization.
#rb trivial
#jira none
#rnx
#preflight 64365189b67b6017ea310d45

[CL 25003548 by paul chipchase in ue5-main branch]
2023-04-12 03:03:43 -04:00
paul chipchase
fb6ed82fb8 Improve the logging during the virtualization process.
#rb trivial
#jira UE-182542
#rnx
#preflight 642ea49d8f078cc70375b7ab

[CL 24942618 by paul chipchase in ue5-main branch]
2023-04-06 07:03:22 -04:00
paul chipchase
47114cc9fd Move the virtualization utilities to be public so that plugins can access them.
#rb trivial
#jira UE-161982
#rnx
#preflight

[CL 24832278 by paul chipchase in ue5-main branch]
2023-03-29 03:25:28 -04:00
mark lintott
ad70d0d087 Fix up of Event Attribute names to support Grafana in StudioAnalytics, DerivedData, Zen, Virtualization and CookProfiling. Grafana does not like the . separator so these have been replaced by _. I have checked this with the DPE team ( and verified ) this is still compatible with Snowflake/Tableau telemetry and such will not requires any changes to Snowflake or exisitng Tableau dashboards.
[FYI] wes.hunt, eric.knapik, paul.chipchase, devin.doucette
#rb devin.doucette
#preflight 641c6e31c44ce895fc452a84

[CL 24764989 by mark lintott in ue5-main branch]
2023-03-23 13:43:00 -04:00
paul chipchase
8cf12f5179 Add code documentation describing the perforce set up required for storing VA payloads.
#rb trivial
#jira UE-180708
#rnx
#preflight 64186329691c5ebc156cdb24

[CL 24715179 by paul chipchase in ue5-main branch]
2023-03-20 09:49:52 -04:00
paul chipchase
ae6c359ca8 Allow the error message displayed when VA payloads fail to pull to be customized.
#rb none
#jira UE-180383
#preflight 64183a83691c5ebc155d21a2

- The payload pull error dialog now suports an additional custom line in the dialog that can be set via Engine:[Core.VirtualizationModule]:AdditionalPullErrorMsg. The intent is to allow teams to provide additional help and info to their users.
- The default error text is now localized.

[CL 24714178 by paul chipchase in ue5-main branch]
2023-03-20 07:40:06 -04:00
paul chipchase
633a13ec9c Add a new error handling flow (off by default) to VA when payloads fail to pull.
#rb none
#jira UE-180383
#preflight 64147be21c44ff98b969b1aa

- The new flow will display an error dialog to the user informing them of the problem and ask if they want to retry or quit the process. This is quite a shift if how we have been handling failed payload pulls up until and this is the first rough implementation so for now it is off by default.
- We only show one dialog at a time and if a new error occurs while we are waiting for user input, that thread is set to wait on the existing dialogs outcome.

[CL 24689856 by paul chipchase in ue5-main branch]
2023-03-17 11:15:24 -04:00
paul chipchase
9be422f486 Clean up VA config file parsing code
#rb trivial
#jira none
#rnx
#preflight 6413305ef81cce1dd8daae8b

- We no longer error if the values cannot be parsed from the config file, our defaults in BaseEngine.ini are the same defaults that we have in code anyway.
- We now trust that FConfigFile will not write to the variable passed into it on parse failure so there is no need to parse to a local variable first, then apply that to the member if the parse was a success.

[CL 24671648 by paul chipchase in ue5-main branch]
2023-03-16 11:23:43 -04:00
paul chipchase
40df3d2c61 Add an option ''-UseLocalIniFileSettings=True/False" to the VA source control backend, which controls if we allow the user of the locally saved 'SourceControlSettings.ini' or not
#rb trivial
#jira UE-180257
#preflight 6411cdbb5819afacaf06fbc9

- By default the value is true and the local ini file is used.
- Allows teams to opt out of loading settings from the locally saved ini file if they wish to.

[CL 24654053 by paul chipchase in ue5-main branch]
2023-03-15 10:17:32 -04:00
paul chipchase
6a78587b59 Improve error message shown to the user if they fail to connect to the VA perforce backend
#rb Per.Larsson
#jira UE-170132
#rnx
#preflight 640f33c5363e9b40ab8cd7af

- Output the entire depot path when failing to download 'payload_metainfo' along with the address of the server.
- We ask the source control provider for the status and parse the p4 port from that as there is no way (currently) to get this info directly via the source control api, which currently works but is a bit flaky long term.
-- I am not sure if we want to keep the error reporting like this and so I am loath to make changes to the source control API unless I am sure we actually want the functionality.
- Removed duplicate code paths, there is now only one branch that will log an error.
- Removed call to FConnect as that does not do anything unless a client is set and we do not use one with this connection. (FConnect is mostly about client set up validation rather than actually connecting to the server since we always make a new perforce connection for each command anyway.
-- Instead we now check if the source control provider is available after we call Init, if this returns false then no connection was made.
- Documented the purpose behind calling ISourceControlProvider::Init

[CL 24613062 by paul chipchase in ue5-main branch]
2023-03-13 11:10:53 -04:00
paul chipchase
4166d55804 Fix typo in VA source control backend logging.
#rb trivial
#jira none
#rnx
#preflight 6409819ca92b27de3c82f080

[CL 24572518 by paul chipchase in ue5-main branch]
2023-03-09 02:40:45 -05:00
paul chipchase
ae52fd7713 Clean up some of the code documentation regarding virtualization filtering.
#rb trivial
#jira none
#rnx
#preflight 6408843a827cf9251b123cb1

[CL 24558187 by paul chipchase in ue5-main branch]
2023-03-08 08:06:42 -05:00
paul chipchase
8a3c4c85ec Add a config file option '-IgnoreFile=' that allows a project to define the name of the p4ignore file used in the perforce environment.
#rb trivial
#jira UE-175158
#preflight 63f8cd34ae54ee4ce9ea207b

- We really should just read this value out of the users p4 environment settings, but that would require a source control api change and would need to be done in a source control agnostic way. Letting a project just define the name is a quick way to expose the functionality.
- At some point if we do extend the source control api then this option can be deprecated.

[CL 24400577 by paul chipchase in ue5-main branch]
2023-02-24 09:58:54 -05:00
paul chipchase
9992c0d589 Add a new config file option (-WorkingDir=ABC) when setting up a VA source control backend that allows finer grain control over where payloads are submitted from.
#rb trivial
#jira UE-175157
#rnx
#preflight 63f8b493ae54ee4ce9de6b78

- This feature was requested so that a team could opt into submitting from their appdata or temp directories instead of the project saved directory (our default)
- Deprecated the option '-SubmitFromTempDir' as it could be replacated with "-WorkingDir=$(Temp)/UnrealEngine/VASubmission"

[CL 24399707 by paul chipchase in ue5-main branch]
2023-02-24 08:50:29 -05:00