Commit Graph

22 Commits

Author SHA1 Message Date
henrik karlsson
9f12e202b4 [UBA]
* Added ProcessStartInfo::writeOutputFilesOnFail that can be set to true if we want the output files to be written/sent back even though process exited with errors
* Changed so session client use ProcessStartInfoHolder and moved serialization code to that class

[CL 30576299 by henrik karlsson in ue5-main branch]
2024-01-11 15:42:50 -05:00
henrik karlsson
8827665b22 [UBA]
* Beeeautiful hack to try to get around ETXTBSY error when spawning ispc processes on linux. Our theory is that with perfectly bad timing the symlink creation of ispc could happen at the exact same time as a posix_spawn happening and inside symlink there is a temporarily open file descriptor that gets cloned into the child and kept open. and this causes ETXTBSY when trying to open that binary.

Solution is simply to have a writelock around the symlink creation and a readlock around posix_spawn

[CL 30513020 by henrik karlsson in ue5-main branch]
2024-01-09 14:26:32 -05:00
henrik karlsson
e0e929073b [Uba]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.

[CL 30507663 by henrik karlsson in ue5-main branch]
2024-01-09 12:04:16 -05:00
henrik karlsson
9b1e22319d [UBA]
* Fixed so paths returned from SearchPathForFile are cleaned up (not having .. etc)
* Fixed so UbaAgent handle WSAPoll POLLERR as timeout so it retries when failing (this made the process exit on wine)
* Fixed so returned processes that had been reused returned the reused process to the scheduler
* Fixed so EnsureBinaryFile is using correct fileNameKeys
* Removed assert in detoured GetFullPathNameA since it is always calling GetFullPathNameW

[CL 30465533 by henrik karlsson in ue5-main branch]
2024-01-03 16:23:58 -05:00
henrik karlsson
ea673427b2 [UBA]
* Implemented process reuse logic in UbaScheduler. It is now possible for running processes to fetch more work
* Fixed bugs in custom message path
* Fixed bugs in FlushWrittenFiles message
* Changed So UpdateEnvironment is resetting stats
* Fixed potential race condition related to process reuse
* Added special rule for ShaderCompileWorker.exe which detours ImageGetDigestStream in Imagehlp.dll (because wine implementation does not match windows implementation)
* Renamed "exitedUserData" to just userData since it is used for more than when exiting process
* Added a couple more dlls to "known system dlls"
* Fixed so visualizer can visualize process reuse properly

[CL 30462059 by henrik karlsson in ue5-main branch]
2024-01-02 17:42:42 -05:00
henrik karlsson
9b8409f622 [UBA]
* Fixed so when downloading application modules, thread itself helps out downloading

[CL 30438265 by henrik karlsson in ue5-main branch]
2023-12-22 03:35:12 -05:00
henrik karlsson
85e56e8590 [UBA]
* Fixed macos compile error

[CL 30438003 by henrik karlsson in ue5-main branch]
2023-12-22 02:57:12 -05:00
henrik karlsson
32f666ea42 [UBA]
* Added Scheduler class that is a very simple scheduler that handles processes that has no intra dependencies. It handles scheduling remotely and also reschedule processes returned from remote machines.
* Changed so application dependencies are retrieved in parallel by clients
* Added so custom assert handler can be set from the outside
* Added traceEnabled to SessionCreateInfo so trace shared mem can be created without needing to launch visualizer or write to file
* Added so userData can be provided in RemoteProcessAvailable and RemoteProcessReturned callbacks
* Added more files to known system files (based on what exists in wine)
* Improved FindImports code and made it available in export. (it is now automatically filtering out known system files)
* Moved ProcessHandle to its own file
* Added ProcessStartInfoHolder which is a class that can wrap a ProcessStartInfo and make sure all strings are allocated

[CL 30437951 by henrik karlsson in ue5-main branch]
2023-12-22 02:41:43 -05:00
henrik karlsson
0c68ced47b [UBA]
* Fixed a bug where uba failed to download dlls to remote if it was in the system32 folder but did not exist on the remote machine's system32 and was not part of known system modules. (this solves the problem with ShaderCompilerWorker not downloading opengl32.dll to barebone machines)
* Fixed bug where relative path of dll was sent into RetriveCasFile which in turn failed because it couldn't find the file
* Fixed so code could handle loading libraries with circular dependencies
* Removed assert for now related to GetThreadPreferredUILanguages and asking for language name instead of id for remote machines)

[CL 30407004 by henrik karlsson in ue5-main branch]
2023-12-19 19:53:32 -05:00
henrik karlsson
12afed80b7 [UBA]
* Implemented knownInputs support. It is now fully supported to provide known inputs when doing RunProcessRemote. This is optional and can be a nice speedup if latency to helpers is high since it reduce number of messages going back and forth.
* Changed so GetMemoryMapAlignment is checking only file name and no flags. Flags is now handled on the outside. This made it possible to send over map alignment for files without having to send the file names
* Renamed WorkManagerImpl::Wait to WorkManagerImpl::FlushWork
* Renamed willBeUsedUnCompressed to storeUncompressed
* Added ScopedCriticalSection::Enter

[CL 30388720 by henrik karlsson in ue5-main branch]
2023-12-19 01:53:01 -05:00
henrik karlsson
ed5170f3c4 [UBA]
* Stats fixes for network transfers

[CL 30361605 by henrik karlsson in ue5-main branch]
2023-12-15 16:57:54 -05:00
henrik karlsson
9e7c0fe803 [UBA]
* Reduced stack usage in client session by reusing 256kb stack binary reader
* Changed so update dir table and hash table messages are not sent if client is already enough up to date
* Fixed race condition writing dir table memory size. This has been in here forever and maybe can't happen in practice.. but wrong lock was taken when updating table memory size (different than what GetDirectoryTableSize was using)

[CL 30358736 by henrik karlsson in ue5-main branch]
2023-12-15 15:38:35 -05:00
henrik karlsson
9a536983eb [Uba]
* Removed usage of "select" in tcp socket connect code because it can't handle file descriptors over 1024 so segfaults (phew, hard one to find)
* Added option to have clients send back log to server (SessionServerCreateInfo.remoteLogEnabled) .. this forced a network protocol bump
* Improved asserts for when network messages are corrupt
* Some small fixes of asserts in debug in win32 detoured code

[CL 30343891 by henrik karlsson in ue5-main branch]
2023-12-15 02:18:13 -05:00
henrik karlsson
37ad7c3e59 [UBA]
* Cleanup, no logical changes

[CL 30284862 by henrik karlsson in ue5-main branch]
2023-12-12 21:28:16 -05:00
henrik karlsson
6ca7d5a11c [Uba]
* Removed special case code that should not be needed anymore

[CL 30256948 by henrik karlsson in ue5-main branch]
2023-12-11 22:57:05 -05:00
henrik karlsson
37b5c7298e [Uba]
* Implemented RunProcess which can be used to spawn and wait on raw processes (not detoured). (Note, calling program needs to report file system modifications since it is not known to UBA)
* Changed so UBAExecutor is using UBA for non-detoured processes as well. This will give correct profiling tracking and reuse existing code.
* Fixed handle leak in windows
* Fixed race condition on linux since wordexp() is n ot thread safe
* Added tests
* New binaries

[CL 30216046 by henrik karlsson in ue5-main branch]
2023-12-08 13:44:47 -05:00
henrik karlsson
bd65b88a24 [UBA]
* Fixed so libDetours.so can be in a path containing spaces (by adding quotes around path)

[CL 30166337 by henrik karlsson in ue5-main branch]
2023-12-06 15:35:09 -05:00
henrik karlsson
9d7ed4c71a [Uba]
* Fixed so TMPDIR is local-only
* Fixed so HOME envvar is used when resolving paths with ~

[CL 30159897 by henrik karlsson in ue5-main branch]
2023-12-06 12:42:50 -05:00
henrik karlsson
e17a938d11 [Uba]
* Fixed so memory block takes alignment for allocations

[CL 30143636 by henrik karlsson in ue5-main branch]
2023-12-05 19:50:46 -05:00
henrik karlsson
7f47a136b8 [Uba]
* Added disallowed path support. It is not possible to register paths that are disallowed for serving to clients

[CL 30140725 by henrik karlsson in ue5-main branch]
2023-12-05 19:02:30 -05:00
henrik karlsson
ec124f0a2e [Uba]
* Fixed so temp files stored in file mappings are cleared directly when parent process dies

[CL 30109027 by henrik karlsson in ue5-main branch]
2023-12-05 01:40:20 -05:00
joe kirchoff
9affe69d97 UnrealBuildAccelerator: Experimental for 5.4
#jira UE-197967

[CL 30002804 by joe kirchoff in ue5-main branch]
2023-11-29 18:47:11 -05:00