Commit Graph

28 Commits

Author SHA1 Message Date
henrik karlsson
9f12e202b4 [UBA]
* Added ProcessStartInfo::writeOutputFilesOnFail that can be set to true if we want the output files to be written/sent back even though process exited with errors
* Changed so session client use ProcessStartInfoHolder and moved serialization code to that class

[CL 30576299 by henrik karlsson in ue5-main branch]
2024-01-11 15:42:50 -05:00
henrik karlsson
8827665b22 [UBA]
* Beeeautiful hack to try to get around ETXTBSY error when spawning ispc processes on linux. Our theory is that with perfectly bad timing the symlink creation of ispc could happen at the exact same time as a posix_spawn happening and inside symlink there is a temporarily open file descriptor that gets cloned into the child and kept open. and this causes ETXTBSY when trying to open that binary.

Solution is simply to have a writelock around the symlink creation and a readlock around posix_spawn

[CL 30513020 by henrik karlsson in ue5-main branch]
2024-01-09 14:26:32 -05:00
henrik karlsson
e0e929073b [Uba]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.

[CL 30507663 by henrik karlsson in ue5-main branch]
2024-01-09 12:04:16 -05:00
henrik karlsson
ea673427b2 [UBA]
* Implemented process reuse logic in UbaScheduler. It is now possible for running processes to fetch more work
* Fixed bugs in custom message path
* Fixed bugs in FlushWrittenFiles message
* Changed So UpdateEnvironment is resetting stats
* Fixed potential race condition related to process reuse
* Added special rule for ShaderCompileWorker.exe which detours ImageGetDigestStream in Imagehlp.dll (because wine implementation does not match windows implementation)
* Renamed "exitedUserData" to just userData since it is used for more than when exiting process
* Added a couple more dlls to "known system dlls"
* Fixed so visualizer can visualize process reuse properly

[CL 30462059 by henrik karlsson in ue5-main branch]
2024-01-02 17:42:42 -05:00
henrik karlsson
79fad227cd [UBA]
* Fixed linux compile error

[CL 30407078 by henrik karlsson in ue5-main branch]
2023-12-19 20:02:10 -05:00
josh adams
91a7b42e3e - Removed unneeded PLATFORM_MAC
#rb henrik.karlsson

[CL 30403458 by josh adams in ue5-main branch]
2023-12-19 16:32:11 -05:00
josh adams
5c95a66a67 - Lowered the priority of the processes Uba spawns, so that too many clangs don't starve the session process. Doesn't always repro, but it was causing some networking errors when talking to horde
#rb henrik.karlsson

[CL 30403426 by josh adams in ue5-main branch]
2023-12-19 16:31:15 -05:00
henrik karlsson
bb93072bd6 [Uba]
* #if 0 on code that does not exist on mac

[CL 30389279 by henrik karlsson in ue5-main branch]
2023-12-19 02:50:28 -05:00
josh adams
53e1c01e3f Fixed UBA on Mac to hve children die when session process goes away even if not notified
#rb henrik.karlsson

[CL 30385745 by josh adams in ue5-main branch]
2023-12-18 18:48:15 -05:00
henrik karlsson
5ebc3e8754 [UBA]
* Fixed one more stats visualization number that was wrong (not taking frequency in to account)

[CL 30364013 by henrik karlsson in ue5-main branch]
2023-12-15 17:57:18 -05:00
henrik karlsson
054d26f056 [UBA]
* Fixed so all visualization of times in visualizer is taking timer frequency into account

[CL 30344197 by henrik karlsson in ue5-main branch]
2023-12-15 03:06:12 -05:00
henrik karlsson
f0e6540060 [UBA]
* Added retry for ETXTBSY failures when doing posix_spawn. Have no idea how these can happen but they do.

[CL 30311542 by henrik karlsson in ue5-main branch]
2023-12-13 19:52:40 -05:00
henrik karlsson
37ad7c3e59 [UBA]
* Cleanup, no logical changes

[CL 30284862 by henrik karlsson in ue5-main branch]
2023-12-12 21:28:16 -05:00
henrik karlsson
7814c8dfd9 [UBA]
* Fixes to make macos work properly with remote helpers

[CL 30280233 by henrik karlsson in ue5-main branch]
2023-12-12 17:49:54 -05:00
henrik karlsson
e6b2b50ab7 [UBA]
* Removed "EARLY_INIT" macro since this doesn't seem to work properly on macos.
* Changed so only one __attribute(constructor..) exists
* Replaced g_applicationRules with GetApplicationRules() to try to resolve initializationordering issues

[CL 30232292 by henrik karlsson in ue5-main branch]
2023-12-11 02:07:47 -05:00
henrik karlsson
d3d5176d3b [UBA]
* Implement simple commandline parser and added unit tests
* Removed usage of wordexp on non-windows platforms because it seems to have a _huge_ penalty on macos..
* Added trace scope around WriteFilesToDisk
* Fixed assert in stdout/err pipes on macos and added handling for poll hangup
*

[CL 30225204 by henrik karlsson in ue5-main branch]
2023-12-09 03:07:04 -05:00
henrik karlsson
37b5c7298e [Uba]
* Implemented RunProcess which can be used to spawn and wait on raw processes (not detoured). (Note, calling program needs to report file system modifications since it is not known to UBA)
* Changed so UBAExecutor is using UBA for non-detoured processes as well. This will give correct profiling tracking and reuse existing code.
* Fixed handle leak in windows
* Fixed race condition on linux since wordexp() is n ot thread safe
* Added tests
* New binaries

[CL 30216046 by henrik karlsson in ue5-main branch]
2023-12-08 13:44:47 -05:00
henrik karlsson
b863d56435 [Uba]
* Fixed so puts is properly detoured so output logging going through there is captured by uba
* Fixed so linux application paths can contain spaces

[CL 30196216 by henrik karlsson in ue5-main branch]
2023-12-07 18:37:44 -05:00
henrik karlsson
bd65b88a24 [UBA]
* Fixed so libDetours.so can be in a path containing spaces (by adding quotes around path)

[CL 30166337 by henrik karlsson in ue5-main branch]
2023-12-06 15:35:09 -05:00
henrik karlsson
0e05059a5a [Uba]
* Brought back code where parent waits for children's m_hasExited because it can actually happen!! (Need to investigate)

[CL 30149195 by henrik karlsson in ue5-main branch]
2023-12-06 01:36:40 -05:00
henrik karlsson
247e634dcc [Uba]
* Don't write output file if process is cancelled

[CL 30141292 by henrik karlsson in ue5-main branch]
2023-12-05 19:10:55 -05:00
zack neyland
03cebfacc3 UBA: Various fixes for POSIX APIs.
* Checking returns a bit better
* Handling "/" in a friendly way since realpath calls stat with "/"
* Hooks up lockFd so we don't accidently close fd1 and fd2 (aka stdout/stderr)
* MacOS doesn't have subreaper so implement another way using sysctl

[CL 30129626 by zack neyland in ue5-main branch]
2023-12-05 14:40:41 -05:00
henrik karlsson
ddf2bb314d [UBA]
* Fixed bug where mapped memory were released while still used

[CL 30109545 by henrik karlsson in ue5-main branch]
2023-12-05 03:16:52 -05:00
henrik karlsson
ec124f0a2e [Uba]
* Fixed so temp files stored in file mappings are cleared directly when parent process dies

[CL 30109027 by henrik karlsson in ue5-main branch]
2023-12-05 01:40:20 -05:00
henrik karlsson
909f4ef9e0 [UBA]
* Fixed bug with time stamps on linux.. was using seconds accuracy
* Fixed a few bugs on linux paths related to not updating directory caches properly when files changed
* Fixed so asserts are logging properly on linux
* Added debug logger for session side to be able to track all tracking of file caches (disabled by default)

[CL 30104061 by henrik karlsson in ue5-main branch]
2023-12-04 19:43:52 -05:00