Commit Graph

37 Commits

Author SHA1 Message Date
zack neyland
c91ba1805b UBA: Fixes various small bugs.
[CL 30662921 by zack neyland in ue5-main branch]
2024-01-17 13:07:52 -05:00
henrik karlsson
ade0725e14 [UBA]
* Fixed so imagehlp.dll and dbghelp.dll are detoured when loaded. Detour ImageGetDigestStream and SymLoadModuleExW because both of them cause trouble on wine
* Fixed so reuse of processes also honor log files so a new log file is created when reuse happen
* Improved log file naming so host can set name and log files are sent back properly with the right name

[CL 30638193 by henrik karlsson in ue5-main branch]
2024-01-16 13:24:28 -05:00
henrik karlsson
22cf61f84b [UBA]
* Added GetNextProcess as a real message type (not using Custom) to be able to stop fetching work if helper is being terminated by aws or if we want to scale down number of workers
* Did some cleanup in the Tcp backend code and made sure not to call close socket multiple times (since it can cause the code to close a different socket than what is owned)

[CL 30584707 by henrik karlsson in ue5-main branch]
2024-01-11 21:01:56 -05:00
henrik karlsson
e0e929073b [Uba]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.

[CL 30507663 by henrik karlsson in ue5-main branch]
2024-01-09 12:04:16 -05:00
henrik karlsson
ea673427b2 [UBA]
* Implemented process reuse logic in UbaScheduler. It is now possible for running processes to fetch more work
* Fixed bugs in custom message path
* Fixed bugs in FlushWrittenFiles message
* Changed So UpdateEnvironment is resetting stats
* Fixed potential race condition related to process reuse
* Added special rule for ShaderCompileWorker.exe which detours ImageGetDigestStream in Imagehlp.dll (because wine implementation does not match windows implementation)
* Renamed "exitedUserData" to just userData since it is used for more than when exiting process
* Added a couple more dlls to "known system dlls"
* Fixed so visualizer can visualize process reuse properly

[CL 30462059 by henrik karlsson in ue5-main branch]
2024-01-02 17:42:42 -05:00
henrik karlsson
32f666ea42 [UBA]
* Added Scheduler class that is a very simple scheduler that handles processes that has no intra dependencies. It handles scheduling remotely and also reschedule processes returned from remote machines.
* Changed so application dependencies are retrieved in parallel by clients
* Added so custom assert handler can be set from the outside
* Added traceEnabled to SessionCreateInfo so trace shared mem can be created without needing to launch visualizer or write to file
* Added so userData can be provided in RemoteProcessAvailable and RemoteProcessReturned callbacks
* Added more files to known system files (based on what exists in wine)
* Improved FindImports code and made it available in export. (it is now automatically filtering out known system files)
* Moved ProcessHandle to its own file
* Added ProcessStartInfoHolder which is a class that can wrap a ProcessStartInfo and make sure all strings are allocated

[CL 30437951 by henrik karlsson in ue5-main branch]
2023-12-22 02:41:43 -05:00
joe kirchoff
ca00303128 UnrealBuildAccelerator: Fix Linux build
#rnx

[CL 30406871 by joe kirchoff in ue5-main branch]
2023-12-19 19:45:26 -05:00
joe kirchoff
cf26cfb90c UnrealBuildAccelerator: Workarounds to get latest PVS-Studio working
#rnx
#rb henrik.karlsson

[CL 30406199 by joe kirchoff in ue5-main branch]
2023-12-19 18:56:20 -05:00
henrik karlsson
12afed80b7 [UBA]
* Implemented knownInputs support. It is now fully supported to provide known inputs when doing RunProcessRemote. This is optional and can be a nice speedup if latency to helpers is high since it reduce number of messages going back and forth.
* Changed so GetMemoryMapAlignment is checking only file name and no flags. Flags is now handled on the outside. This made it possible to send over map alignment for files without having to send the file names
* Renamed WorkManagerImpl::Wait to WorkManagerImpl::FlushWork
* Renamed willBeUsedUnCompressed to storeUncompressed
* Added ScopedCriticalSection::Enter

[CL 30388720 by henrik karlsson in ue5-main branch]
2023-12-19 01:53:01 -05:00
joe kirchoff
522e63e9a4 UnrealBuildAccelerator: Fix frequency of wall time on mac and linux
#rnx

[CL 30364108 by joe kirchoff in ue5-main branch]
2023-12-15 18:00:48 -05:00
henrik karlsson
5ebc3e8754 [UBA]
* Fixed one more stats visualization number that was wrong (not taking frequency in to account)

[CL 30364013 by henrik karlsson in ue5-main branch]
2023-12-15 17:57:18 -05:00
henrik karlsson
9e7c0fe803 [UBA]
* Reduced stack usage in client session by reusing 256kb stack binary reader
* Changed so update dir table and hash table messages are not sent if client is already enough up to date
* Fixed race condition writing dir table memory size. This has been in here forever and maybe can't happen in practice.. but wrong lock was taken when updating table memory size (different than what GetDirectoryTableSize was using)

[CL 30358736 by henrik karlsson in ue5-main branch]
2023-12-15 15:38:35 -05:00
henrik karlsson
058cb6c708 [UBA]
* Fixed path validation code
* Added null guard in NetworkBackendTcp::Connect and collapsed to early outs into one

[CL 30353516 by henrik karlsson in ue5-main branch]
2023-12-15 13:02:50 -05:00
henrik karlsson
054d26f056 [UBA]
* Fixed so all visualization of times in visualizer is taking timer frequency into account

[CL 30344197 by henrik karlsson in ue5-main branch]
2023-12-15 03:06:12 -05:00
henrik karlsson
9a536983eb [Uba]
* Removed usage of "select" in tcp socket connect code because it can't handle file descriptors over 1024 so segfaults (phew, hard one to find)
* Added option to have clients send back log to server (SessionServerCreateInfo.remoteLogEnabled) .. this forced a network protocol bump
* Improved asserts for when network messages are corrupt
* Some small fixes of asserts in debug in win32 detoured code

[CL 30343891 by henrik karlsson in ue5-main branch]
2023-12-15 02:18:13 -05:00
henrik karlsson
59cfe7c7d4 [UBA]
* Added delete to copy-ctor and assign operator for Thread and event and fixed related code
* Some minor cleanups in thread class
* Fixed so UbaAgent is not writing timeout waiting for connection if server was connected but quickly disconnected

[CL 30325575 by henrik karlsson in ue5-main branch]
2023-12-14 13:31:47 -05:00
henrik karlsson
251533b3e9 [UBA]
* Disabled asserts in shipping again for linux platform since we haven't seen an assert since enabling it

[CL 30315296 by henrik karlsson in ue5-main branch]
2023-12-14 03:16:31 -05:00
henrik karlsson
37ad7c3e59 [UBA]
* Cleanup, no logical changes

[CL 30284862 by henrik karlsson in ue5-main branch]
2023-12-12 21:28:16 -05:00
henrik karlsson
4e02297201 [Uba]
* Added sigsegv handler to Uba that is initialized on CreateCallbackLogWriter. This is a temporary hack just to try to get info from a seg fault happening on the farm now and then

[CL 30257270 by henrik karlsson in ue5-main branch]
2023-12-11 23:50:40 -05:00
henrik karlsson
ca04570337 [Uba]
* Fixed handling of paths that has dirs that has double dots in it.. still a valid path

[CL 30247269 by henrik karlsson in ue5-main branch]
2023-12-11 15:21:10 -05:00
henrik karlsson
e6b2b50ab7 [UBA]
* Removed "EARLY_INIT" macro since this doesn't seem to work properly on macos.
* Changed so only one __attribute(constructor..) exists
* Replaced g_applicationRules with GetApplicationRules() to try to resolve initializationordering issues

[CL 30232292 by henrik karlsson in ue5-main branch]
2023-12-11 02:07:47 -05:00
henrik karlsson
d3d5176d3b [UBA]
* Implement simple commandline parser and added unit tests
* Removed usage of wordexp on non-windows platforms because it seems to have a _huge_ penalty on macos..
* Added trace scope around WriteFilesToDisk
* Fixed assert in stdout/err pipes on macos and added handling for poll hangup
*

[CL 30225204 by henrik karlsson in ue5-main branch]
2023-12-09 03:07:04 -05:00
zack neyland
a6873d78e7 UBA: Additional changes for Mac support.
* Fixes an issue with socket closing
* Adds support for getting number of procs.

[CL 30186045 by zack neyland in ue5-main branch]
2023-12-07 13:26:32 -05:00
henrik karlsson
9d7ed4c71a [Uba]
* Fixed so TMPDIR is local-only
* Fixed so HOME envvar is used when resolving paths with ~

[CL 30159897 by henrik karlsson in ue5-main branch]
2023-12-06 12:42:50 -05:00
henrik karlsson
e17a938d11 [Uba]
* Fixed so memory block takes alignment for allocations

[CL 30143636 by henrik karlsson in ue5-main branch]
2023-12-05 19:50:46 -05:00