Commit Graph

223 Commits

Author SHA1 Message Date
bryan sefcik
da92084a12 Optimized out more private modules includes and dependencies.
#preflight 64627c382965f6ea8ea83bd6

[CL 25479683 by bryan sefcik in ue5-main branch]
2023-05-15 16:26:12 -04:00
paul chipchase
0a9f2e615e Change TryCopyPackageWithoutTrailer to take the length of the trailer to remove rather than the trailer structure itself.
#rb trivial
#jira none
#'rnx
#preflight 646218cb2965f6ea8e87db6e

[CL 25471700 by paul chipchase in ue5-main branch]
2023-05-15 07:55:37 -04:00
paul chipchase
7cd5e648dd Allow an optional url to be shown along side VA connection errors to make it easier for users to find the help that they need.
#rb none
#rnx
#preflight 645c95abaa3c584c0b1c91f0

- A url can now be provided via 'ini:Engine:[Core.VirtualizationModule]:ConnectionHelpUrl=""' that will be added to various connection errors. If the error message is via FMessageLog then the user will be able to click the url to go straight to the link.
- This sort of help will be very company specific, this url will allow teams to build up their own FAQ for their users.
- Currently used by the source control backend if the initial connection fails but will be expanded to other areas.
- The url is currently parsed and owned by the virtualization manager but at the moment the connection error is issued by each backend. So right now there is a static accessor allowing backends to get the value to use. This would not work properly for 3rd party backends and will  be changed in the near term.

[CL 25424000 by paul chipchase in ue5-main branch]
2023-05-11 06:35:58 -04:00
paul chipchase
6a1a0551da Enable UnsafeTypeCastWarningLevel errors for the virtualization module
#rb trivial
#jira none
#rnx
#preflight 6458e2596c35ad81e60f2fa9

- I thought I had already done this but I see nothing in the p4 history, so it is likely that I forgot to actually submit the change.
- Fix some additional warnings found under clang (C5219)

[CL 25370075 by paul chipchase in ue5-main branch]
2023-05-08 09:40:17 -04:00
paul chipchase
fef4f75edc Do not assign payloads downloaded from source control if they did not actually download.
#rb trivial
#jira none
#rnx
#preflight 6447d1954052d26a5a8c36cc

- Ends up pretty much the same,as FCompressedBuffer::FromCompressed will return a Null FCompressedBuffer if the provided data is null, but adding the check makes it clear when reading FSourceControlBackend::PullData that payloads might not be found and we are not setting them in that case.

[CL 25179230 by paul chipchase in ue5-main branch]
2023-04-25 09:32:46 -04:00
paul chipchase
107e71c495 Add a way for VA backends to supply additional errors for failed pull operations when calling FVirtualizationManager::OnPayloadPullError
#rb none
#jira UE-181418
#rnx
#preflight 6447c5482804595a04d1e375

- Each pull operation now takes a FText that can be filled in with additional info about any failures that will cause the new error dialog to be displayed.
-- At the moment we only have info to provide from the source control backend if the connection itself failed.
-- In the future we might want to completely overhaul our error logging and display all messages to the user but our logging system is not really built for that level of redirection at the moment.
- Only pass on the error if it came from a persistent backend, we don't need to worry the user about cached backends which are allowed to fail.

[CL 25178752 by paul chipchase in ue5-main branch]
2023-04-25 08:47:06 -04:00
paul chipchase
1400b2ddc4 Remove duplicate package entries from the list of packages to virtualize/rehydrate.
#rb none
#jira UE-183776
#rnx
#preflight 6441439073470c177c0ce09c

- One crash was reported where a user has somehow managed to try to submit a list of packages containing duplicate entries. This means we try to strip the trailer from that package multiple times which would cause the file system to assert during a call to truncate.
- This is technically the fault of higher level systems outside the control of VA but we should still handle it properly.
- Now duplicate entries are detected and removed so that each package is only processed once.
- If duplicates are detected we log a warning.

[CL 25127877 by paul chipchase in ue5-main branch]
2023-04-20 11:05:45 -04:00
paul chipchase
447771b1a7 Remove debug code accidently submitted in CL 25066347
#rb trivial
#jira none
#rnx
#preflight 643d4f59930b3b84922485c2

[CL 25067672 by paul chipchase in ue5-main branch]
2023-04-17 10:15:35 -04:00
paul chipchase
a8717da576 Do not record VA caching statistics if no payload was actually cached because they were already present in the backend.
#rb trivial
#jira UE-182205
#rnx
#preflight 643d3249a35280ed4fca3097

- If all payloads were already in the backend we would still record the time taken to query but do not increment the number of hits nor the size of the payloads. Without a stat for "payloads skipped" which we do not currently have any interest in adding, recording the time taken to query where no transfer takes place just ends up being misleading and looks like a bug.

[CL 25066475 by paul chipchase in ue5-main branch]
2023-04-17 08:06:12 -04:00
paul chipchase
2fd6372116 Add a new config option 'ForceCachingOnPull' to VA that when true will force backends to re-upload payloads when caching.
#rb none
#jira UE-182205
#preflight 643d2901c947f6523a2f5845

- Most backends support some form of existence check which is run before a push to prevent uploading data that is already there. Sometimes we want to ignore that check (bugs in the backend etc) and need a way to force that to happen.
- The new config option [Core.VirtualizationModule]:ForceCachingOnPull (default false) will do this when set to true.
- The file and DDC backends will respect the flag but for now the source control backend will continue to check to avoid accidental pointless revisions being added.

[CL 25066347 by paul chipchase in ue5-main branch]
2023-04-17 07:45:19 -04:00
paul chipchase
d3e14c816b Added code documentation for 'PullErrorAdditionalMsg' that was missing from CL 24714178
#rb trivial
#jira UE-180383
#rnx
#preflight 643cfc2c211b661dc4d3bc0c

[CL 25065493 by paul chipchase in ue5-main branch]
2023-04-17 04:18:35 -04:00
jamie dale
2fae9fc218 Use the FMessageDialog overloads that pass the optional title by-value
#jira
#rb none
#rnx

[CL 25062745 by jamie dale in ue5-main branch]
2023-04-15 22:21:07 -04:00
paul chipchase
c65eb35e8c Add bitflag enums to IVirtualizationBackend::PullData and IVirtualizationBackend::PushData so that it is easier to specialize behavior in the future.
#rb Per.Larsson
#jira UE-182205
#preflight 643932b2211b661dc4f594c6

- The enums don't contain any entries at the moment, the goal of this change is to reduce the friction that future API changes might have.
- The older versions of PullData/PushData have been deprecated but this won't flag compiler warnings if people have derived their own implementations of IVirtualizationBackend, so in addition to deprecating the methods have been set to 'final' and have checkNoEntry implementations which is the best we can do at the moment.
-- It is unlikely that people have actually done this as the documentation on adding new backend types will only go out in 5.3.

[CL 25039959 by paul chipchase in ue5-main branch]
2023-04-14 10:13:46 -04:00
paul chipchase
b96c7d6df6 Clean up the config setup options for the P4SourceControl VA backend.
#rb none
#jira UE-177956
#preflight

- Deprecate the config entry 'DepotRoot' and replace it with 'DepotPath'. The use of the term root was not clear and was causing people to think it was the name of the depot rather than the path to the root location to store payloads in.
- If 'DepotPath' is empty or missing, the value for 'ClientStream' will be used for it.
-- The 'DepotPath' can still be set when using streams, if the name of the stream is not a valid path (virtual streams for example.

[CL 25024791 by paul chipchase in ue5-main branch]
2023-04-13 10:11:47 -04:00
paul chipchase
3de71b52ed Log the data reduction when virtualizing packages
#rb trivial
#jira UE-182542
#preflight 64369ea4d03b1c87dd91f15b

- We now log the total size saving by running the virtualization process as well as the original size of the packages being virtualized and their resulting size.
- If LogVirtualization is set to verbose logging we will also log each individual package and its size reduction.

[CL 25005336 by paul chipchase in ue5-main branch]
2023-04-12 08:24:53 -04:00
paul chipchase
70e8a4d7b3 Reduce the disk space overhead when virtualizing packages.
#rb none
#jira UE-182542
#preflight 643699cdeb6cd0e63a6bd729

- Previously we were duplicating all packages being virtualized, without the recently virtualized payloads, then try to move them one at a time to replace the original packages.
- Now we do all of the prep work such as checking the packages out of revision control and checking that they can be written to first, then duplicate & move each package one at a time.
-- This way we only ever have a single additional duplicated package at a time.
- Moved the final summary logging to an exit scope so we can return when ever we want in the function.
- Added early out for error case.
- Added early out if there are no packages to remove virtualized payloads from.
- Improved the code commands in IVirtualizationSystem::TryVirtualizePackages so that the caller knows the potential failure case scenarios.

[CL 25005153 by paul chipchase in ue5-main branch]
2023-04-12 08:02:36 -04:00
paul chipchase
31fd4ad8e2 CIS - Code formatting fix, remove newline that was added at the top of the cpp
#rb trivial
#jira none
#rnx
#preflight

[CL 25003922 by paul chipchase in ue5-main branch]
2023-04-12 04:51:15 -04:00
paul chipchase
b0a5ab7783 Change the VA filesystem backend to log the absolute file path of its directory so that it is easier for users to find.
#rb trivial
#jira none
#preflight 64365842d03b1c87dd7ca47d

- When the path being logged is relative, there is sometimes confusion as to which directory it is relative to.

[CL 25003785 by paul chipchase in ue5-main branch]
2023-04-12 04:13:35 -04:00
paul chipchase
43751b70d9 Use a specific subdirectory under 'Saved' when duplicating packages for virtualization/hydration so that it is easier to identify disk bloat caused by the duplication and to clean up failed processes.
#rb trivial
#jira UE-182542
#rnx
#preflight 643657e30c4277fc0b72f928

[CL 25003781 by paul chipchase in ue5-main branch]
2023-04-12 04:12:33 -04:00
paul chipchase
89483c750a Add some more profiling scopes to the package virtualization project
#rb trivial
#jira UE-182542
#rnx
#preflight 64365867eb6cd0e63a533159

- Attempting to hunt down a performance issue that is currently only occuring for a single user and cannot be reproduced elsewhere.

[CL 25003776 by paul chipchase in ue5-main branch]
2023-04-12 04:11:47 -04:00
paul chipchase
57da929556 Add code comment explaining why we need to check the file states in revision control to avoid the check being removed in the name of optimization.
#rb trivial
#jira none
#rnx
#preflight 64365189b67b6017ea310d45

[CL 25003548 by paul chipchase in ue5-main branch]
2023-04-12 03:03:43 -04:00
paul chipchase
fb6ed82fb8 Improve the logging during the virtualization process.
#rb trivial
#jira UE-182542
#rnx
#preflight 642ea49d8f078cc70375b7ab

[CL 24942618 by paul chipchase in ue5-main branch]
2023-04-06 07:03:22 -04:00
paul chipchase
667d0814f3 Do not assume that all VA backends will be inside of a UE::Virtualization namespace
#rb trivial
#jira #jira UE-161982
#rnx
#preflight 64242859b72410fc1731c762

- The UE_REGISTER_VIRTUALIZATION_BACKEND_FACTORY macro would only compile when inside of the UE::Virtualization namespace, which is true for all of the default implementations that we provide but will probably not be true for any extension backends that third parties might write.

[CL 24834268 by paul chipchase in ue5-main branch]
2023-03-29 08:15:53 -04:00
paul chipchase
47114cc9fd Move the virtualization utilities to be public so that plugins can access them.
#rb trivial
#jira UE-161982
#rnx
#preflight

[CL 24832278 by paul chipchase in ue5-main branch]
2023-03-29 03:25:28 -04:00
mark lintott
ad70d0d087 Fix up of Event Attribute names to support Grafana in StudioAnalytics, DerivedData, Zen, Virtualization and CookProfiling. Grafana does not like the . separator so these have been replaced by _. I have checked this with the DPE team ( and verified ) this is still compatible with Snowflake/Tableau telemetry and such will not requires any changes to Snowflake or exisitng Tableau dashboards.
[FYI] wes.hunt, eric.knapik, paul.chipchase, devin.doucette
#rb devin.doucette
#preflight 641c6e31c44ce895fc452a84

[CL 24764989 by mark lintott in ue5-main branch]
2023-03-23 13:43:00 -04:00