Commit Graph

47 Commits

Author SHA1 Message Date
henrik karlsson
eda007f51f [UBA]
* Added support for most features in UbaScheduler. Dependencies, weights etc etc.
* Added support for loading yaml file with processes for UbaScheduler.
* Added so UbaCli interprets yaml file as a file with processes use it to populate a scheduler which is then executed
* Fixed so all threads inside same process spawned by uba ends up in the same thread group.
* Fixed linux crash where process comunication memory was deleted when cancel event was called (added lock around code)
* Fixed deadlock that could happen if flush dead processes were called at the after lock but before processhandle dtor in Session::ProcessExited
* Changed new[]/delete[] to aligned_alloc/free because for some reason new/delete trigger asan on linux and don't know why.

[CL 31372220 by henrik karlsson in ue5-main branch]
2024-02-11 04:00:51 -05:00
henrik karlsson
c5b06cbba0 [UBA]
* Added more information for when process dies without sending exit message

[CL 31367591 by henrik karlsson in ue5-main branch]
2024-02-10 12:38:52 -05:00
henrik karlsson
cca66a4ea3 [UBA]
* Removed logging entries that was only added to make horde not timeout.. since this was implemented horde has added ping logic so this is not needed anymore
* Fixed bug in callstack logging on posix
* Changed a bunch of %hs to %s
* Added allocator debug code to try to figure out asan issues

[CL 31281576 by henrik karlsson in ue5-main branch]
2024-02-07 20:36:18 -05:00
sergio gardeazabal
9728fc156b [UBA] Disable power throttling in created processes to ensure P-Cores are preferred over E-Cores on Intel Hybrid architectures platforms
#jira UE-205966

#rb henrik.karlsson

[CL 31193922 by sergio gardeazabal in ue5-main branch]
2024-02-05 17:56:36 -05:00
henrik karlsson
eee33ba668 [UBA]
* Fixed so stdout/err is redirected to pipe on detoured processes on linux/mac.. this should fix so errors are visualized in the right place
* Added -nostdout to UbaCli to be able to check that redirected stdout/err works
* Added version for process messages to catch issues where ppl have old ubaagent.exe but new ubadetours.dll
* Added capacity parameter to FixPath and added asserts to check that we never write outside capacity
* Fixed code creating g_virtualApplication buffer.
* Disabled asserts for mac non debug builds
* Removed detoured write/fwrite for posix now when we instead redirect stdout/err
* Enabled some unit tests for linux now when they work as intended regarding redirected stdout/err

[CL 31029918 by henrik karlsson in ue5-main branch]
2024-01-30 19:05:51 -05:00
henrik karlsson
0eaf4b8111 [UBA]
* Fixed some error handling for posix
* Improved some logging
* Disabled some unit tests on linux because they always fail

[CL 30991796 by henrik karlsson in ue5-main branch]
2024-01-30 02:17:34 -05:00
henrik karlsson
3ea6ef91db [UBA]
* Removed error on linux for failing poll call

[CL 30987771 by henrik karlsson in ue5-main branch]
2024-01-29 23:59:19 -05:00
zack neyland
505cac5c83 UBA: Add if some defs around POLLUP for platform differences
[CL 30983584 by zack neyland in ue5-main branch]
2024-01-29 21:01:39 -05:00
zack neyland
59077f46f4 UBA: Close fds when done with polling to stop leaking on MacOS.
[CL 30980705 by zack neyland in ue5-main branch]
2024-01-29 18:45:46 -05:00
henrik karlsson
0d51a68c53 [Uba]
* Fixed file descriptor leak when running native processes on posix platforms
* Removed hangup test on poll for pipe file descriptors since it exited to early and missed reads

[CL 30963156 by henrik karlsson in ue5-main branch]
2024-01-29 02:13:15 -05:00
josh adams
2c6d16634e Fixing working dir on Mac
#rb henrik.karlsson

[CL 30891145 by josh adams in ue5-main branch]
2024-01-25 12:31:40 -05:00
zack neyland
c10257601e UBA: Bump cmdline limit to 64k
[CL 30888128 by zack neyland in ue5-main branch]
2024-01-25 11:10:13 -05:00
henrik karlsson
def11cc5d0 [UBA]
* Improved error handling for setpriority even more. There is a risk that a non-detoured process have exited already when we get to this function

[CL 30878827 by henrik karlsson in ue5-main branch]
2024-01-25 01:12:09 -05:00
henrik karlsson
29914bf077 [UBA]
* Fixed so if setpriority fails on linux because of not having permissions we don't fail the build

[CL 30878645 by henrik karlsson in ue5-main branch]
2024-01-25 01:03:44 -05:00
henrik karlsson
abae032917 [UBA]
* Improved error description when failing to create pipe on posix

[CL 30875937 by henrik karlsson in ue5-main branch]
2024-01-24 22:22:28 -05:00
henrik karlsson
659c4c9101 [UBA]
* Changed so com memory is not split up in read and write since they never overlap.. so both read and write can use all the memory

[CL 30871523 by henrik karlsson in ue5-main branch]
2024-01-24 20:33:10 -05:00
henrik karlsson
ade0725e14 [UBA]
* Fixed so imagehlp.dll and dbghelp.dll are detoured when loaded. Detour ImageGetDigestStream and SymLoadModuleExW because both of them cause trouble on wine
* Fixed so reuse of processes also honor log files so a new log file is created when reuse happen
* Improved log file naming so host can set name and log files are sent back properly with the right name

[CL 30638193 by henrik karlsson in ue5-main branch]
2024-01-16 13:24:28 -05:00
henrik karlsson
4b1c00af7d [UBA]
* Fixed stats reporting bug for process reuse

[CL 30588189 by henrik karlsson in ue5-main branch]
2024-01-12 01:51:57 -05:00
henrik karlsson
22cf61f84b [UBA]
* Added GetNextProcess as a real message type (not using Custom) to be able to stop fetching work if helper is being terminated by aws or if we want to scale down number of workers
* Did some cleanup in the Tcp backend code and made sure not to call close socket multiple times (since it can cause the code to close a different socket than what is owned)

[CL 30584707 by henrik karlsson in ue5-main branch]
2024-01-11 21:01:56 -05:00
henrik karlsson
9f12e202b4 [UBA]
* Added ProcessStartInfo::writeOutputFilesOnFail that can be set to true if we want the output files to be written/sent back even though process exited with errors
* Changed so session client use ProcessStartInfoHolder and moved serialization code to that class

[CL 30576299 by henrik karlsson in ue5-main branch]
2024-01-11 15:42:50 -05:00
henrik karlsson
8827665b22 [UBA]
* Beeeautiful hack to try to get around ETXTBSY error when spawning ispc processes on linux. Our theory is that with perfectly bad timing the symlink creation of ispc could happen at the exact same time as a posix_spawn happening and inside symlink there is a temporarily open file descriptor that gets cloned into the child and kept open. and this causes ETXTBSY when trying to open that binary.

Solution is simply to have a writelock around the symlink creation and a readlock around posix_spawn

[CL 30513020 by henrik karlsson in ue5-main branch]
2024-01-09 14:26:32 -05:00
henrik karlsson
e0e929073b [Uba]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.

[CL 30507663 by henrik karlsson in ue5-main branch]
2024-01-09 12:04:16 -05:00
henrik karlsson
ea673427b2 [UBA]
* Implemented process reuse logic in UbaScheduler. It is now possible for running processes to fetch more work
* Fixed bugs in custom message path
* Fixed bugs in FlushWrittenFiles message
* Changed So UpdateEnvironment is resetting stats
* Fixed potential race condition related to process reuse
* Added special rule for ShaderCompileWorker.exe which detours ImageGetDigestStream in Imagehlp.dll (because wine implementation does not match windows implementation)
* Renamed "exitedUserData" to just userData since it is used for more than when exiting process
* Added a couple more dlls to "known system dlls"
* Fixed so visualizer can visualize process reuse properly

[CL 30462059 by henrik karlsson in ue5-main branch]
2024-01-02 17:42:42 -05:00
henrik karlsson
79fad227cd [UBA]
* Fixed linux compile error

[CL 30407078 by henrik karlsson in ue5-main branch]
2023-12-19 20:02:10 -05:00
josh adams
91a7b42e3e - Removed unneeded PLATFORM_MAC
#rb henrik.karlsson

[CL 30403458 by josh adams in ue5-main branch]
2023-12-19 16:32:11 -05:00