Tested compiling fortnite, unrealeditor, lyra, qagame with non-unity/pch
#preflight 63635997876630122adeab9f
#rb none
[CL 22958990 by henrik karlsson in ue5-main branch]
#rb PJ.Kack
#jira UE-167578
#rnx
#preflight 635bdddd5b54febd38b7efd0
- Clean up the source control backend by removing the older code path that was ifdefed out and clean up the new path.
- FPerforceConnection no longer takes a TOptional<FSharedBuffer> but takes an optional pointer to a TArray of FSharedBuffer. When gathering files this should return one FSharedBuffer per file requested.
-- It might make more sense to attach the FSharedBuffer to the FP4Record but that would require some bigger changes and would be better suited to being some follow on work. All the code is private in the module and not exposed so it is fairly safe and easy to do these refactors later.
- Replaced the TArray buffer for gathering files with a purpose built class FP4DepotFile.
- When issuing a p4 print command with multiple files we will get one call to ClientUser::OutputStat followed by calls to either ClientUser::OutputText or ClientUser::OutputBinary until all of the file has been delivered, then we will get another call to ClientUser::OutputStat for the next file and so on.
-- Note with binary files we seem to get one final call to ClientUser::OutputBinary with zero size, which I assume is supposed to signal the end of the transfer, text files do not get this, they seem to just stop.
[CL 22888671 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-168452
#rnx
#preflight 635a4773f1493f97864fb660
- IVirtualizationBackend can now have its operations disabled after it is created.
-- This means that a backend that only supported pushing and not pulling would end up with no valid operations if marked as read only. This will not assert because we do this after the backend is created and we only check on creation.
[CL 22810918 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-168057
#rnx
#preflight 635656e6e6096564af1bc117
- Work on UE-165308 broke the following commandlines as they were accidently converted to cmdline switches rather than values
-- VABackendGraph
-- VAMissBackends
-- VAMissChance
- ::IsCmdLineSet has now been split into two utility wrappers, one for params (bools) and one for values.
[CL 22792790 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63526abcae33b04ec1ee4a65
- The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker.
-- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use.
-- The DDC backend should be taking full advantage of the new batching API
-- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit.
- This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc.
-- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away.
[CL 22707955 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-165308
#rnx
#preflight 634ff17e8449f85a47390fbe
- It was pointed out that a command line option containing '-' characters could confuse parsers and it is not worth the risk to keep them.
- Now we accept versions of each command line without the '-' character which will be the version that we document for users in 5.1
[CL 22650940 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 634805f7f93be0f634b8ed5b
- Added a new FArchive 'FFixedBufferWriterArchive' that will let me serialize to an already allocated, fixed sized memory buffer.
- Moved the code for opening and reading packages to shared utility functions so that it is easier to keep the error messages consistent.
- Moved common code to a new function ::TryRehydrateBuilder so that both rehydration methods can share it.
- Note that technically we are changing the API since now we accept a TConstArrayView for the package file paths for both the virtualization and rehydration paths, but existing code will be compatible with this change.
[CL 22511406 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-160942
#rnx
#preflight 634557f5c272487ed4ab2014
- In addition to adding the hit counter (which can be very useful when debugging) the stats logging code has been factored out to a single lambda tht is called for each set of stats to make it easier to update in future.
[CL 22457126 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 633ad936c250f60502b2d8cb
- Everything was working but when a cvar is set via a device profile we expect the cvar to be registered sooner, hence the warning.
- Added a new define UE_USE_GLOBAL_CVAR to FVirtualizationManager.cpp, when set we use a static global cvar for 'VA.AllowPkgVirtualization' rather then creating and registering one when the system is mounted.
- Since the cvar can now outlive the manager we also needed to add a way to remove the bound lambda from the cvar when the manager is destroyed.
[CL 22322164 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160942
#rnx
#preflight 6336f2ab5c2225fe5f6c08a5
- Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead.
-- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc.
- Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown.
- Changed the backends to use the new results system.
-- Not all paths were correctly setting the state in the file system backend.
-- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do.
- Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload.
[CL 22279127 by paul chipchase in ue5-main branch]
Before:
Total CPU Time: 53783.640625 s
Total time in Parallel executor: 558.66 seconds
After:
Total CPU Time: 47886.140625 s
Total time in Parallel executor: 498.81 seconds
#jira
[CL 22173145 by bryan sefcik in ue5-main branch]
#rb Per.Larsson
#jira UE-163834
#rnx
#preflight 632d98e4b4515b7e221654ec
### Virtualization
- At the moment there is no way in the editor UX to change the source control settings for the source control backend, so what ever settings are found in the global environment would be applied when the editor is first run and then saved to a config file under the <saved> directory. From this point onwards those settings would always be used when setting up the source control backend. If the settings were wrong, the first time that the editor was started, then we'd continue to use the incorrect settings even if the global enviroment was fixed. Making the backend work at that point would require that the user know about the local ini file and change the settings there.
- We cannot fix this by just ignoring the ini file as it has been requested that users can add their settings there in case they want to customize things, so we just need to avoid saving to it.
- Added documentation to the source control backends header file on how to set up the ini file if needed.
### Perforce
- FSourceControlInitSettings now accepts EConfigBehavior which describes how we want the source control provider to deal with its settings with regards to the ini file.
- Note that EConfigBehavior is not a bitfield as we do not wish to support only writing to the ini file, there is no point if it cannot be read from.
- When the source control provider is creatred we pass on the into from EConfigBehavior to the FPerforceSourceControlSettings to enable/disable saving and loading.
[CL 22163769 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-164278
#rnx
#preflight 63284a248c3def91aa62bd83
- This replaces the old cvar 'VA.DisableVirtualization' which is being removed.
- We will log each time the value is switched to aid in future debugging.
- The name needed to be changed as we needed to invert how it worked, also "DiableVirtualization" was a bit misleading and made people think it turned the entire system off, not just the process of virtualizing packages on submit.
-- The member bEnablePayloadVirtualization was also renamed to bAllowPackageVirtualization to make its purpose clearer.
- The cmdline version was changed from 'VA-DisableVirtualization' to 'VA-SkipPkgVirtualization' for similar reasons.
[CL 22146910 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-164284
#rnx
#preflight 632b182ffc7f1efbdf9638d9
- This is only useful if the entire team uses the same address for the perforce server
- When set, this will override the environment settings for P4PORT as well as override what ever is saved to SourceControlSettings.ini
[CL 22122737 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163103
#rnx
#preflight 6318989c2b7fe03eb664e9f0
### VirtualizationSystem/VirtualizationManager
- Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request
- Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety.
### IVirtualizationBackend
- The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs.
- Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters.
-- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942
- Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing.
- File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests.
-- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not.
[CL 21907558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163016
#rnx
#preflight
- The backend is no longer used as we access horde storage via the ddc backend.
[CL 21862840 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161973
#rnx
#preflight 63074424ae13a5a09865a281
- We could try force loading and enabling the plugin at this point but I am not a fan of the idea of overriding the set up of the target in a way that is not immediately obvious to the end user.
- The virtualization module needs to include the 'Projects' module to be able to access the plugin system directly and while editing the file I took the opportunity to clean up how the module is declaring its dependencies.
[CL 21684932 by paul chipchase in ue5-main branch]
- Set VA.DisableVirtualization CVar for CookedEditor targets
- Use UE_VIRTUALIZATION_CONNECTION_LAZY_INIT for [redacted] CookedEditor targets
- setup FLinuxDeviceProfileSelectorModule so Linux CookedEditor use the right device profile
- Fix bLazyInitConnections not being set by UE_VIRTUALIZATION_CONNECTION_LAZY_INIT and fix logging
- Add AllowVirtualAssets to automation scripts for [redacted]
[REVIEW] [at]Josh.Adams [at]Eric.Knapik [at]Paul.Chipchase
#preflight https://horde.devtools.epicgames.com/job/630664c1516bef57ffef3c24
[CL 21553248 by justin marcus in ue5-main branch]
#rb Per.Larsson
#jira UE-161685
#rnx
#preflight 6304e0dfae13a5a098e2b9c9
- Added a cvar 'VA.DisableVirtualization' and a commandline '-VA-DisableVirtualization-' that when set are equivalent to setting '[Core.VirtualizationModule].EnablePayloadVirtualization=false' and will. prevent the VA system from virtualizing any packages on submit.
[CL 21524913 by paul chipchase in ue5-main branch]
#rb Devin.Doucette
#jira UE-161599
#rnx
#preflight 6303c8d65a5d4e4624e7bf52
- There are some use cases that require the VA system to be initialized and configured correctly but would prefer that the backend connections only run if absolutely needed (usually when a payload is pulled or pushed for the first time), this change provides four different ways of doing this:
-- Setting [Core.VirtualizationModule]LazyInitConnections=true in the Engine ini file
-- Setting the define 'UE_VIRTUALIZATION_CONNECTION_LAZY_INIT' to 1 in a programs .target.cs
-- Running with the commandline option -VA-LazyInitConnections
-- Setting the cvar 'VA.LazyInitConnections' to 1 (only works if it is set before the VA system is initialized, changing it mid editor via the console does nothing)
--- Note that after the config file, each setting there only opts into lazy initializing the connections, setting the cvar to 0 for example will not prevent the cmdline from opting in etc.
- In the future we will allow the connection code to run async, so the latency can be hidden behind the editor loading, but for the current use case we are taking the minimal approach.
-- This means we only support the backend being in 3 states. No connection has been made yet, the connection is broken and the connection is working.
-- To keep things simple we only record if we have attempted to connect the backends or not. We don't check individual backends nor do we try to reconnect failed ones etc. This is all scheduled for a future work item.
- If the connections are not initialized when the VA system is, we wait until the first time someone calls one of the virtualization methods that will actually use a connection: Push/Pull/Query
-- We try connecting all of the backends at once, even if they won't be used in the call to keep things simple.
- Only the source control backend makes use of the connection system. The horde storage (http) backend could take advantage too, but it is currently unused and most likely going to just be deleted so there seemed little point updating it.
- If we try to run an operation on an unconnected backend we only log to verbose. This is to maintain existing behaviour where a failed backend would not be mounted at all. This logging will likely be revisited in a future work item.
[CL 21511855 by paul chipchase in ue5-main branch]
