#rb trivial
#jira UE-175158
#preflight 63f8cd34ae54ee4ce9ea207b
- We really should just read this value out of the users p4 environment settings, but that would require a source control api change and would need to be done in a source control agnostic way. Letting a project just define the name is a quick way to expose the functionality.
- At some point if we do extend the source control api then this option can be deprecated.
[CL 24400577 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-175157
#rnx
#preflight 63f8b493ae54ee4ce9de6b78
- This feature was requested so that a team could opt into submitting from their appdata or temp directories instead of the project saved directory (our default)
- Deprecated the option '-SubmitFromTempDir' as it could be replacated with "-WorkingDir=$(Temp)/UnrealEngine/VASubmission"
[CL 24399707 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 63ee42a2d669f2ed779879c7
- The commandlet runs quite slowly at the moment, logging the progress every 10 seconds or so helps the user see that work is being done and the process hasn't locked up.
[CL 24257562 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-177013
#preflight 63ece283956709374a03eb8a
- The existing config file option 'EnableCacheAfterPull' is now deprecated. If we detect that it is still in the config file we will warn the user about it but continue to apply the value to maintain compatibility.
- Add a new config file option 'EnableCacheOnPull' which controls if a payload is pushed to cached storage after it has been pulled from persistent storage. (Default true)
- Add a new config file option 'EnableCacheOnPush' which controls if a payload is pushed to cached storage before it is pushed to persistent storage. (Default true)
- We now store the info about caching in a single enum bitfield ECachingPolicy.
- Added an up to date comment section about ini file set up for FVirtualizationManager which was previously missing.
[CL 24230636 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-177015
#preflight 63eb71ea956709374ab61d5e
- So far nobody has proposed any worthwhile use case for virtualizing engine content given the following downsides:
-- We cannot ever do so internally as we would be unable to distribute the virtualized packages.
-- Projects and the engine packages cannot always be accessed via the same client spec, leading to problems if we try.
-- Engine content is typically small and not suited to virtualization in the first place.
- If we remove the option then we can exclude the engine packages from our VA commandlets which will give less confusing output (at the moment if you try to virtualize a project containing a single package we will report that several thousand packages were found as we need to parse engine content too)
- Deprecated the ini file options, if we detect them we will warn the user that the option is still present and that engine content now will never virtualize.
[CL 24208899 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-176605
#preflight 63ea68d9b91ae11c1cbab311
- A common problem that has been reported is that the user has tried to virtualize packages with the virtualization tool but not seen any virtualization occur. When debugged the issue turns out to be that the packages are too old and need to be re-saved.
- If the user submits via the editor then the package will be auto reverted if it has not yet been re-saved so this problem is not seen in that flow.
- Although this is a problem that will solve itself over time, it would be helpful to the person testing the system if we explicitly log that X number of packages were too old.
- Given how annoying this has proven to people the log message is currently set to warning.
- We don't actually return why a package trailer fails to load from FPackageTrailer::LoadTrailer, as working that out requires additional file reading and most the time we don't really care.
- Instead I added a new utility to the virtualization module so that we can opt into checking the reason.
- At the moment we only report if the package is too old and not for every possible reason.
[CL 24206813 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-176605
#preflight 63e4bb8a046845470872df95
- Instead of just giving an error message stating that the trailer could not be found, we now try to return why the trailer cannot be found, if the package is corrupted, or was last saved before the package trailer concept was introduced etc.
[CL 24094160 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-175189
#rnx
#preflight 63d27eae450d5cdd0b4df619
- We were raising a FMessageLog to print a warning, then using ::OnConnectionError to defer the FMessageLog::Notify call until the next editor tick so an easy fix is to pass the warning to ::OnConnectionError and piggyback onto the defferal if needed.
- The messages used to be errors but were demoted during development, now that FSourceControlBackend has SuppressNotifications option and can supress the notifications on a per project basis we can restore the messages to errors.
- When calling ::OnConnectionError we will send the message to FMessageLog if we are currently on the game thread, then defer the notification until next tick as we were doing so previously. The deferral is needed in case the notify comes during editor start up (which for a connection problem is likely) so that the notification appears at the correct time. If we are not on the game thread we should still print the message to the log file so that debugging is easier, only the FMessageLog part needs to be deferred to the GameThread. In this case we need to tell FMessageLog to not print to the log file to avoid duplicates.
[CL 23919185 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 63d7b0135428dc67b160b3aa
- CL 23859876 introduced optimizations that consider a filepath to be invalid if the path contains consecutive path separators (DirA//DirB for example instead of DirA/DirB) and although no issues were seen when testing we fairly quickly ran into a user facing errors when trying to submit work from the editor.
- The users client spec root had a '\' at the end of it where as most internal devs do not. This in turn caused problems when parsing file info from source control as the code was assuming that the client spec root had no trailing separator and was adding one.
- Although the system providing the bad filepaths will be fixed it makes sense to sanitize the filepath input for these two processes as any system, including 3rd party ones could be providing the paths.
[CL 23919107 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-174290
#rnx
#preflight 63c8fa22d15a37ae31487446
- Previously once the number of perforce connections allowed was reached we'd block all additional requests until an outstanding connection as been released at which point a random waiting thread would get the go ahead to make its own connection and so on. If the GameThread was making a request under these conditions the user could be left with a blocked editor while they wait for the GameThread to get a turn.
- This change allows FSourceControlBackend to create a semaphore that gives priority to the GameThread during contention so that the user gets the fastest possible response.
- FSemaphore now takes a bitfield of flags as a parameter in the constructor to allow us to specialize its behavior with the only support option being to prioritize the game thread.
- If the GameThread tries to acquire the semaphore and the prioritization option is enabled we use a slightly different path code path ::AcquireFromGameThread
-- Unlike the normal acquire code path, this path does not increment the count when waiting. This effectively reserves the next open slot for the calling thread.
-- Given that there is only one request on the GameThread at a time we don't have to check Counter once the event has been triggered. The even triggering means we know that one of the slots has been freed and we know we have already reserved that slot so we can just go ahead and return success at that point.
- Added a profile scope to acquire as that would've made the original problem easier to catch.
- Still need to look into replacing this with std::counting_semaphore once we can rely on c++ 20 support.
[CL 23771977 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 63c4fa7fb0652247508bcb57
- Fleshed out the comment for the function to better describe the return value.
[CL 23727404 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 63bec803c543a64b7d38d89f
- If no 'Path' option is found the path will default to the projects saved directory, under a 'VAPayloads' subdirectory
- This works as an okay default if the backend is being used for cached storage.
[CL 23645795 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd670a71079a8d1c0e837b
- Since the API was forcing the caller to pass in a results structure to be filled in, we might as well make it the return value.
- Added a ::WasSuccessful method to the results structures that can be used instead of checking if the result had errors or not.
- Remove the reset method from FVirtualizationResult/FRehydrationResult as they no longer need it.
- The older deprecated methods still use the results enum, so we cannot easily deprecate those enums yet.
[CL 23626072 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 639c4112012902cb8db43e13
- This allows us to provide the user with more ways to customize the virtualization and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- Previously if we virtualized a package that was not checked out in revision control we would warn the user and then skip updating the package file on disk. This means the payloads would be uploaded but the user would be left with no local changes. Since sometimes we know we don't need to check out any package (virtualizing the packages in a change list for example) we don't want to always incur the cost of polling reivision control to see which packages do need checking out. This is why we now allow the caller to request package files be checked out via the new options enum EVirtualizationOptions.
-- If the EVirtualizationOptions::Checkout flag is provided we will poll the revision control status of all package files and then check out those which need it.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- Added a new utility function to SourceControlUtilties to make it easier to check out packages. There is similar functionality elsewhere in the code base but the virtualization module is too low level to make use of it.
- Updated existing code that calls ::TryVirtualizePackages and add cases of ''using namespace UE::Virtualization' where required to improve readability.
- The UnrealVirtualizationTool now supports a new cmdline option "-checkout" that can be used when virtualizing packages. This will checkout any package that was actually virtualized so the result can be saved back out to the workspace domain. This means we no longer require the caller to have checked out the packages before running the tool.
[CL 23536832 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-171492
#rnx
#preflight 6391f02d67018b14b581c0b0
- It is possible that a project might want to have persistent storage enabled but no caching storage, it is probably a bad idea but it should work. However before this fix doing so would cause an error to be logged during the virtualization of a package and fail the process.
- During the virtualization process we now check if pushing is enabled for cache storage before trying to push there, if not then we log that the phase is being skipped.
- Even though the above fixes the problem we would still get odd results when pushing payloads to cached storage without any backends enabled.
-- We now check if the pushing process is disabled before processing requests, so we can fail faster. There is now also a proper error value for this that can be stored in each request (FPushRequest) where as previously they'd all still display "pending"
-- We also check to see if the requested push type has any associated backend before processing requests and have a specific error value for this. Note that we consider
- We also now force the visibility of the slow task progress bar, as otherwise we might not end up displaying the virtualization progress to the progress dialog (this is a work around to a long standing problem with the slow task system due to the time limit associated with updating the dialog via EnterProgressFrame, this needs to be fixed elsewhere)
#ushell-cherrypick of 23445439 by paul.chipchase
[CL 23461405 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-170012
#rnx
#preflight 63724fa79e3bea8079f2ed76
- The original for loop was added at a time where FCompressedBuffer did not support serialization directly, at that time the code worked.
- Now it looks like the FCompressedBuffer returned by the call to ::GetPayload is going out of scope early, which means that the FSharedBuffer ends up zero length (and quite possibly unsafe to use)
- However nowadays FCompressedBuffer has both a operator<< and direct ::Load/::Save methods so we can remove the for loop and simplify the code.
[CL 23133173 by paul chipchase in ue5-main branch]
