#rb none
#jira UE-183776
#rnx
#preflight 6441439073470c177c0ce09c
- One crash was reported where a user has somehow managed to try to submit a list of packages containing duplicate entries. This means we try to strip the trailer from that package multiple times which would cause the file system to assert during a call to truncate.
- This is technically the fault of higher level systems outside the control of VA but we should still handle it properly.
- Now duplicate entries are detected and removed so that each package is only processed once.
- If duplicates are detected we log a warning.
[CL 25127877 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-182542
#preflight 64369ea4d03b1c87dd91f15b
- We now log the total size saving by running the virtualization process as well as the original size of the packages being virtualized and their resulting size.
- If LogVirtualization is set to verbose logging we will also log each individual package and its size reduction.
[CL 25005336 by paul chipchase in ue5-main branch]
#rb none
#jira UE-182542
#preflight 643699cdeb6cd0e63a6bd729
- Previously we were duplicating all packages being virtualized, without the recently virtualized payloads, then try to move them one at a time to replace the original packages.
- Now we do all of the prep work such as checking the packages out of revision control and checking that they can be written to first, then duplicate & move each package one at a time.
-- This way we only ever have a single additional duplicated package at a time.
- Moved the final summary logging to an exit scope so we can return when ever we want in the function.
- Added early out for error case.
- Added early out if there are no packages to remove virtualized payloads from.
- Improved the code commands in IVirtualizationSystem::TryVirtualizePackages so that the caller knows the potential failure case scenarios.
[CL 25005153 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-182542
#rnx
#preflight 64365867eb6cd0e63a533159
- Attempting to hunt down a performance issue that is currently only occuring for a single user and cannot be reproduced elsewhere.
[CL 25003776 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-176605
#preflight 63ea68d9b91ae11c1cbab311
- A common problem that has been reported is that the user has tried to virtualize packages with the virtualization tool but not seen any virtualization occur. When debugged the issue turns out to be that the packages are too old and need to be re-saved.
- If the user submits via the editor then the package will be auto reverted if it has not yet been re-saved so this problem is not seen in that flow.
- Although this is a problem that will solve itself over time, it would be helpful to the person testing the system if we explicitly log that X number of packages were too old.
- Given how annoying this has proven to people the log message is currently set to warning.
- We don't actually return why a package trailer fails to load from FPackageTrailer::LoadTrailer, as working that out requires additional file reading and most the time we don't really care.
- Instead I added a new utility to the virtualization module so that we can opt into checking the reason.
- At the moment we only report if the package is too old and not for every possible reason.
[CL 24206813 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 639c4112012902cb8db43e13
- This allows us to provide the user with more ways to customize the virtualization and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- Previously if we virtualized a package that was not checked out in revision control we would warn the user and then skip updating the package file on disk. This means the payloads would be uploaded but the user would be left with no local changes. Since sometimes we know we don't need to check out any package (virtualizing the packages in a change list for example) we don't want to always incur the cost of polling reivision control to see which packages do need checking out. This is why we now allow the caller to request package files be checked out via the new options enum EVirtualizationOptions.
-- If the EVirtualizationOptions::Checkout flag is provided we will poll the revision control status of all package files and then check out those which need it.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- Added a new utility function to SourceControlUtilties to make it easier to check out packages. There is similar functionality elsewhere in the code base but the virtualization module is too low level to make use of it.
- Updated existing code that calls ::TryVirtualizePackages and add cases of ''using namespace UE::Virtualization' where required to improve readability.
- The UnrealVirtualizationTool now supports a new cmdline option "-checkout" that can be used when virtualizing packages. This will checkout any package that was actually virtualized so the result can be saved back out to the workspace domain. This means we no longer require the caller to have checked out the packages before running the tool.
[CL 23536832 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-171492
#rnx
#preflight 6391f02d67018b14b581c0b0
- It is possible that a project might want to have persistent storage enabled but no caching storage, it is probably a bad idea but it should work. However before this fix doing so would cause an error to be logged during the virtualization of a package and fail the process.
- During the virtualization process we now check if pushing is enabled for cache storage before trying to push there, if not then we log that the phase is being skipped.
- Even though the above fixes the problem we would still get odd results when pushing payloads to cached storage without any backends enabled.
-- We now check if the pushing process is disabled before processing requests, so we can fail faster. There is now also a proper error value for this that can be stored in each request (FPushRequest) where as previously they'd all still display "pending"
-- We also check to see if the requested push type has any associated backend before processing requests and have a specific error value for this. Note that we consider
- We also now force the visibility of the slow task progress bar, as otherwise we might not end up displaying the virtualization progress to the progress dialog (this is a work around to a long standing problem with the slow task system due to the time limit associated with updating the dialog via EnterProgressFrame, this needs to be fixed elsewhere)
#ushell-cherrypick of 23445439 by paul.chipchase
[CL 23461405 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 634805f7f93be0f634b8ed5b
- Added a new FArchive 'FFixedBufferWriterArchive' that will let me serialize to an already allocated, fixed sized memory buffer.
- Moved the code for opening and reading packages to shared utility functions so that it is easier to keep the error messages consistent.
- Moved common code to a new function ::TryRehydrateBuilder so that both rehydration methods can share it.
- Note that technically we are changing the API since now we accept a TConstArrayView for the package file paths for both the virtualization and rehydration paths, but existing code will be compatible with this change.
[CL 22511406 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160942
#rnx
#preflight 6336f2ab5c2225fe5f6c08a5
- Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead.
-- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc.
- Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown.
- Changed the backends to use the new results system.
-- Not all paths were correctly setting the state in the file system backend.
-- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do.
- Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload.
[CL 22279127 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160943, UE-151671
#rnx
#preflight 62fa3334153b17e7462954b3
### Problem
- The virtualization process first checks to see if any of the local payloads are already stored in persistent storage and only try to submit those which aren't, which would mean if we are submitting a package with a local payload that is already in the persistent storage system but not in cached storage we have no way to cache it.
- This logic also introduced problems where filtered out payloads were being virtualized which was fixed with the addition of ENABLE_FILTERING_HACK which was not a robust long term solution.
- Checking if the payloads need to be pushed or not complicated the logic of the virtualization process and makes it harder to understand and make fixes too.
### Fix
- Removed code for ENABLE_FILTERING_HACK and UE_PRECHECK_PAYLOAD_STATUS from the virtualization process
- Removed some code loops from the virtualization process but we still end up with one loop finding all of the package trailers and a second loop setting up the FWorkspaceDomainPayloadProvider
-- This second loop will be removed in a future work item to avoid trying to push duplicate payloads (unless I end up doing that work at the virtualization manager level instead)
- We make one push for cached storage and another for persistent
-- Cached storage push failure will only result in a warning, failing the persistent push will error out as before
-- We need to reset the request states after the cache push. In a future work item we will likely allow a single push to specific both storage solutions, so I am leaving this code using the same set of requests for both pushes.
#robomerge FNMain
[CL 21386503 by paul chipchase in ue5-main branch]
#rb none
#rnx
#jira UE-159597
#preflight
- The option is [Core.VirtualizationModule]VirtualizationProcessTag=
- If left blank then no tag will be added
- The internal virtualization process code no longer needs to return a tag since it will now be applied by the virtualization manager instead.
-- Note that we still follow the same logic, we only stamp if no errors were encountered.
[CL 21217286 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62d5509047779a730a1e4602
- The code in the files isn't really a check anymore but the actual process for virtualizing packages (taking the payloads stored in the package trailer, pushing to the backends then modifying the trailer) updating the name reflects this.
- It also helps match the PackageRehydrationProcess.
[CL 21147691 by paul chipchase in ue5-main branch]
