Commit Graph

45 Commits

Author SHA1 Message Date
paul chipchase
318c56e475 The VA statistics panel once again informs the user if the project does not have the VA system enabled.
#rb PJ.Kack
#rnx

- If the virtualization system is not enabled for the current project then we should display a message informing the user of this rather than just displaying the normal statistics grid without any backends.
- This code was accidentally removed from the last refactor of this panel in CL 33601457, note that we also removed the lazyinit code path, this is fine as that code path is effectively deprecated anyway.

[CL 34090342 by paul chipchase in ue5-main branch]
2024-06-04 07:15:51 -04:00
paul chipchase
ecea53ce65 Added new methods to IVirtualizationSystem for accessing statistics and deprecated the old methods.
#rb PJ.Kack
#jira UE-147384
#rnx

- The previous functor based system was difficult to update as changes to the data being passed to the functor could break existing code, so ::GetAccumualtedPayloadActivityInfo has been replaced with ::GetBackendStatistics, a version that returns a struct for each backend in the system which contains the relevant statistics for that backend. Adding new data to a struct is much easier.
-- The new data added per backend is the storage type so that the caller can tell if the backend was for cached storage or persistent storage.
- To fit into this naming scheme ::GetAccumualtedPayloadActivityInfo has been deprecated in favor of ::GetSystemStatistics which is simply a rename.
- The deprecated methods should continue to work as before.
- Existing engine code had been updated to use the new methods.
- Removed a typo from the copyright boilerplate at the begining of VirtualizationSystem.h

[CL 33626815 by paul chipchase in ue5-main branch]
2024-05-14 07:59:24 -04:00
paul chipchase
842abbf97a Compile fix for clang cast warnings introduced in CL 33601457
#rb PJ.Kack
#jira UE-147384
#rnx

[CL 33603011 by paul chipchase in ue5-main branch]
2024-05-13 09:20:26 -04:00
paul chipchase
51d8962cfd Enable casting warnings for the VirtualizationEditor module
#rb Per.Larsson
#rnx

[CL 33602460 by paul chipchase in ue5-main branch]
2024-05-13 08:47:27 -04:00
paul chipchase
4d93776204 Improve the virtualization statistics panel.
#rb PJ.Kack
#jira UE-147384
#rnx

- Added boxes around each set of categories (pull, push, cache) to make it easier to see the differences between them.
- Moved the "Push" statistics to be last in order as it is the least interest of the statistics, infact when an external submit tool is being used with the editor it won't display any useful info at all (we should look into removing that set of stats if we can detect such tool use)
- Changed the total time spent in request to an average time per request
- Refactored the code to reduce code duplication.

[CL 33601468 by paul chipchase in ue5-main branch]
2024-05-13 07:26:28 -04:00
paul chipchase
3891c50bc1 Improve the parsing of '-PackageDir=' and '-PackageFolder=; in virtualization commandlets.
#rb Martin.Ridgers
#rnx

- Allow more than one path to be specified at a time. Additional directories can be chained with either ',' or '+'
- Log each root directory searched or that the project itself is being searched.

[CL 31044961 by paul chipchase in ue5-main branch]
2024-01-31 08:53:45 -05:00
paul chipchase
bf1405e2c0 The commandlet 'VirtualizationEditor.PrecachePayloads' can now be configured to only precache specific directories by using the cmdline arg '-PackageDir=' or '-PackageFolder='
#rb Martin.Ridgers
#rnx

[CL 31044886 by paul chipchase in ue5-main branch]
2024-01-31 08:51:43 -05:00
steve robb
6d5b974842 Fixed up a lot of bool-taking container resize functions to take EAllowShrinking instead.
[CL 30735396 by steve robb in ue5-main branch]
2024-01-19 19:41:56 -05:00
marc audy
40118bf525 Silence warnings
[CL 30444877 by marc audy in ue5-main branch]
2023-12-22 17:21:37 -05:00
kirill zorin
b6ee3a6c64 Fix UE_LOG callsites that have format string-related UB
#rb trivial
#preflight 647107780a6634dbb236a0a6

[CL 25651638 by kirill zorin in ue5-main branch]
2023-05-26 15:52:39 -04:00
paul chipchase
fa35ec4043 Add a new console command 'ValidatePackagePayloads' as a quick way to check the status of payloads in a package.
#rb none
#jira none
#rnx
#preflight 646492502d446eac96a6fff9

- Each payload in the package is loaded or pulled and checked to make sure that it:
-- Could be accessed.
-- The hash stored in the FCompressedBuffer is the same as the original requested hash.
-- That the payload can be decompressed.
-- That the decompressed payload when hashed has the same hash as the original request.
- This should catch cases where the payload cannot be found or is corrupted in some way.

[CL 25503830 by paul chipchase in ue5-main branch]
2023-05-17 04:53:43 -04:00
paul chipchase
54b51c99e3 Fix some typos in code documentation
#rb trivial
#jira none
#rnx
#preflight 6461dc5ab87b09aadddc17c8

[CL 25470354 by paul chipchase in ue5-main branch]
2023-05-15 04:03:12 -04:00
paul chipchase
fdf6f97cfc Allow the output of the GeneratePayloadManifest commandlet to be spit up over multiple csv files if it gets too large.
#rb trivial
#jira none
#preflight 645cc3cc2965f6ea8e81cfab

- If a project contained too many packages the final .csv output file might be too large to import into some spread sheet systems, in these cases we need to split the output into multiple smaller .csv files.
- The output is now sent to its own subdirectory under saved, ''/payloadmanifest/"
- When done the commandlet will report the file path of each .csv written to disk
- Add the number of payloads discovered to the final output summary.

[CL 25424197 by paul chipchase in ue5-main branch]
2023-05-11 06:50:09 -04:00
paul chipchase
b96e6b41da Add a new cmdline parameter '-LocalOnly' to the 'GeneratePayloadManifest commandlet. When used only payloads that are still stored locally will be added to the manifest.
#rb trivial
#jira none
#preflight 645ca98ccf788a2558b003ef

[CL 25423192 by paul chipchase in ue5-main branch]
2023-05-11 04:51:23 -04:00
paul chipchase
73d6b003e8 Allow the virtualize and rehydrate project commandlets to work on specific directories if requested.
#rb none
#jira UE-178523
#preflight 6437a727d03b1c87ddf5ca15

- The commandlets now accept '-PackageDir=' and '-PackageFolder=' commnad line args.

[CL 25021806 by paul chipchase in ue5-main branch]
2023-04-13 03:44:12 -04:00
paul chipchase
c6b707f171 Add a new cmdline option '-ValidateContent' to the 'ValidateVirtualizedContent commandlet
#rb none
#jira UE-182196
#rnx
#preflight 642d2b1389339023eb66b663

- The new cmdline option will download each payload and make sure that the data is correct in addition to checking that the payload exists.
- Note: There is no way to control which backend returns the payload so the VA graph, or cmdline parameters would need to be changed be the caller to make sure they are checking the backend that they are interested in.
-- This new option is fairly slow and is intended to be used as part of an automated checking system rather than by users themselves, so needing to change the graph and running the commandlet once per backend that needs testing shouldn't be too hard to do.

[CL 24924696 by paul chipchase in ue5-main branch]
2023-04-05 04:22:43 -04:00
paul chipchase
1619183e14 Factor out the multithreaded pull system from PrecachePayloadsCommandlet to the general utilities so that other VA commandlets can make use of it.
#rb trivial
#jira UE-182196
#rnx
#preflight 642c20f5d6069e71b1f39f80

[CL 24909424 by paul chipchase in ue5-main branch]
2023-04-04 09:16:58 -04:00
paul chipchase
9b96e61b1b Move some comon functionality from UPrecachePayloadsCommandlet to the general utilities so that other commandlets can use it.
#rb trivial
#jira UE-182196
#rnx
#preflight 642bff7689339023ebf971fa

[CL 24908348 by paul chipchase in ue5-main branch]
2023-04-04 07:05:12 -04:00
paul chipchase
b4d5ca8fbb Improve the VA PrecachePayloads commandlet
#rb none
#rnx
#preflight 641c01e2a6092dfb8dfa1478

- The previous default iteration mode could saturdate the worker threads, which would prevent the Zen DDC backend from running clean up code resulting in very large memory spikes.
- We now launch a limited number of tasks that process a limited number of payloads before completing. Once a task is finsihed we issue another until there is no more work to do.
- This seems to strike a good balance of keeping the network transfer at peak speed but allowing other systems time on the cpu.
- Removed all the experimental iteration modes and going all in on this solution, which makes the code easier to parse.

[CL 24759860 by paul chipchase in ue5-main branch]
2023-03-23 07:02:27 -04:00
paul chipchase
6a7f58fedf Fix a typo in the VA commandlet documentation.
#rb trivial
#jira none
#rnx
#preflight 640b3ef2d778f88975290bdb

[CL 24591048 by paul chipchase in ue5-main branch]
2023-03-10 09:43:59 -05:00
paul chipchase
425e201624 Improve VA commandlet documentation to avoid problems when people invoke them via powershell.
#rb trivial
#jira none
#rnx
#preflight 640adf1eaf3fc352929322a9

- When a tester tried '-run=VirtualizationEditor.???' via powershell a spece was added so the editor got '-run=VirtualizationEditor .???' which doesn't parse correctly.
- Change the documentation to suggest using '-run="VirtualizationEditor.???"' so that the original text is preserved.

[CL 24588709 by paul chipchase in ue5-main branch]
2023-03-10 02:56:22 -05:00
paul chipchase
9e48b8d00b Fix the 'CheckForVirtualizedContent ' commandlet to be able to check engine content once again.
#rb none
#jira UE-174785
#rnx
#preflight 63ff0ceeae54ee4ce912c17c

- Recently how we discover packages for the VA commandlets was changed to be much faster but it was also changed to automatically filter out all engine content as we removed the optional (opt in) feature of virtualizing engine content.
- This meant that 'CheckForVirtualizedContent' would not be able to check engine packages if '-CheckEngine' was set as we were not providing the commandlet with them.
- ::FindAllPackages has been renamed ::FindPackages and both it and ::DiscoverPackages now accept a flags parameter that allows the caller to request that engine content be excluded.
- Although most commandlets do not want engine content, having the engine content filter as opt in closer matches existing apis and will probably lead to less mistakes.

[CL 24459148 by paul chipchase in ue5-main branch]
2023-03-01 03:47:24 -05:00
paul chipchase
4a9de4ab1b The VA commandlet 'ValidateVirtualizedContent' now iterates over all packages in a project and makes sure that any virtualized payloads encountered can be found in persistent storage. The original version of this commandlet has been renamed to 'CheckForVirtualizedContent' and continues to be used to make sure that virtualized packages do not get submitted to places that they should not.
#rb trivial
#jira UE-176611
#preflight 63fe18fcf43e53f68189d869

- Renamed UValidateVirtualizedContentCommandlet to UCheckForVirtualizedContentCommandlet. P4 rename was not done because I wanted to repurpose UValidateVirtualizedContentCommandlet and adding new versions of the code files after renaming them is likely to cause weird p4 merge issues in the future.

[CL 24442899 by paul chipchase in ue5-main branch]
2023-02-28 10:48:04 -05:00
paul chipchase
ea9afe6a78 Add '-AutoCheckIn' cmdline option to the Virtualize/Rehydrate project commandlets.
#rb trivial
#jira UE-178518
#preflight 63fcb6f560163ed3c22e3d35

- The option will only checkin packages that were checked out by the commandlet. Local files or packages that were already open for edit will not be submitted.

[CL 24422913 by paul chipchase in ue5-main branch]
2023-02-27 09:34:47 -05:00
paul chipchase
5b0faa30ca Speed up package discovery for VA commandlets.
#rb Per.Larsson
#jira UE-167454
#rnx
#preflight 63f5e5b9ac06ce789f047431

- Replace use of ::NormalizePackageNames with the asset registry for finding packages.
- We can use the async version of SearchAllAssets so that we don't have to wait on the asset registry cache being saved to disk.
- ::FindAllPackages now returns all paths in our standard form (relative to Engine\Binaries\<Platform>) where as NormalizePackageNames would return a mix of formats depending on the mount point location.

[CL 24359197 by paul chipchase in ue5-main branch]
2023-02-22 07:39:51 -05:00