Commit Graph

100 Commits

Author SHA1 Message Date
zack neyland
e35aaea9ec UBA: Fix up tests on Mac.
* Need to grab the actual clang binary location and the SDK path.

[CL 30670808 by zack neyland in ue5-main branch]
2024-01-17 17:20:04 -05:00
henrik karlsson
11ed67aeed [UBA]
* Added log entries to trace file

[CL 30669670 by henrik karlsson in ue5-main branch]
2024-01-17 16:49:50 -05:00
henrik karlsson
fa2772e27d [UBA]
* Added so in-process proxy can use storage of client
* Fixed reset of trace reader
* Fixed annoying warnings when closing invalid sockets

[CL 30667316 by henrik karlsson in ue5-main branch]
2024-01-17 15:28:30 -05:00
henrik karlsson
ade0725e14 [UBA]
* Fixed so imagehlp.dll and dbghelp.dll are detoured when loaded. Detour ImageGetDigestStream and SymLoadModuleExW because both of them cause trouble on wine
* Fixed so reuse of processes also honor log files so a new log file is created when reuse happen
* Improved log file naming so host can set name and log files are sent back properly with the right name

[CL 30638193 by henrik karlsson in ue5-main branch]
2024-01-16 13:24:28 -05:00
henrik karlsson
460bdb7637 [UBA/Shader]
* Changed Scheduler_GetStats to separate active into activeLocal and activeRemote
* Added Storage_DeleteFile that can be used to delete cas entries if we know a file will never be used again (no point taking up space)
* Fixed so ubaagents spawned through horde has a idle timeout of 15 seconds (also made them quiet)
* Changed so UbaJobProcessor shutdown uba after 100 seconds instead of 10 seconds of idling without tasks
* Added so .uba.in and .uba.out is deleted from cas storage after used.. no point storing them there since they will never be used again
* Fixed so number of local SCW instances are decreased when more remote helpers are running
* Fixed race condition where tasks could be added when uba was shutting down causing tasks to be cancelled

[CL 30607957 by henrik karlsson in ue5-main branch]
2024-01-12 18:46:59 -05:00
henrik karlsson
c635451c61 [UBA]
* Fixed so sendfile stats is correct for reuse process entries

[CL 30588393 by henrik karlsson in ue5-main branch]
2024-01-12 02:35:20 -05:00
henrik karlsson
4b1c00af7d [UBA]
* Fixed stats reporting bug for process reuse

[CL 30588189 by henrik karlsson in ue5-main branch]
2024-01-12 01:51:57 -05:00
henrik karlsson
22cf61f84b [UBA]
* Added GetNextProcess as a real message type (not using Custom) to be able to stop fetching work if helper is being terminated by aws or if we want to scale down number of workers
* Did some cleanup in the Tcp backend code and made sure not to call close socket multiple times (since it can cause the code to close a different socket than what is owned)

[CL 30584707 by henrik karlsson in ue5-main branch]
2024-01-11 21:01:56 -05:00
henrik karlsson
350ca0b6b4 [UBA]
* Fixed so two traces can't be writing to the same shared memory at the same time
* Added commented out code to track number of active file mappings

[CL 30576568 by henrik karlsson in ue5-main branch]
2024-01-11 15:49:37 -05:00
henrik karlsson
9f12e202b4 [UBA]
* Added ProcessStartInfo::writeOutputFilesOnFail that can be set to true if we want the output files to be written/sent back even though process exited with errors
* Changed so session client use ProcessStartInfoHolder and moved serialization code to that class

[CL 30576299 by henrik karlsson in ue5-main branch]
2024-01-11 15:42:50 -05:00
henrik karlsson
ae997cd583 [UBA]
* Enabled overlap write and bottleneck max to 16 when writing files to disk... this seems to give best results on both machines with ntfs compression and non-compression

[CL 30575881 by henrik karlsson in ue5-main branch]
2024-01-11 15:29:03 -05:00
henrik karlsson
24864b248d [UBA]
* Downgraded tcp bind error to info since it could happen if ppl are running compile and cook at the same time.

[CL 30574550 by henrik karlsson in ue5-main branch]
2024-01-11 14:35:13 -05:00
henrik karlsson
70c75ab3cd [UBA]
* Added Bottleneck type that can be used in combination with BottleneckScope to prevent more than x threads to pass through at the same time

[CL 30574267 by henrik karlsson in ue5-main branch]
2024-01-11 14:30:02 -05:00
henrik karlsson
8400320d41 [UBA]
* Removed assert in detouring code for GetFileAttributesA since we have verfied it is forwarded to GetFileAttributesW on both windows and wine
* Added code to fixup path sent in when creating storage server through export functions
* Added path separator fix in session setup code

[CL 30547691 by henrik karlsson in ue5-main branch]
2024-01-10 15:43:04 -05:00
henrik karlsson
9b81535da2 [UBA]
* Added unit test for scheduler

[CL 30523388 by henrik karlsson in ue5-main branch]
2024-01-09 20:29:17 -05:00
henrik karlsson
2242d42396 [UBA]
* Fixed so scheduler can take in 0 as max local processors and use 0. Default is now ~0u which becomes max logical processors

[CL 30519168 by henrik karlsson in ue5-main branch]
2024-01-09 18:03:55 -05:00
henrik karlsson
8827665b22 [UBA]
* Beeeautiful hack to try to get around ETXTBSY error when spawning ispc processes on linux. Our theory is that with perfectly bad timing the symlink creation of ispc could happen at the exact same time as a posix_spawn happening and inside symlink there is a temporarily open file descriptor that gets cloned into the child and kept open. and this causes ETXTBSY when trying to open that binary.

Solution is simply to have a writelock around the symlink creation and a readlock around posix_spawn

[CL 30513020 by henrik karlsson in ue5-main branch]
2024-01-09 14:26:32 -05:00
henrik karlsson
e0e929073b [Uba]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.

[CL 30507663 by henrik karlsson in ue5-main branch]
2024-01-09 12:04:16 -05:00
henrik karlsson
9b1e22319d [UBA]
* Fixed so paths returned from SearchPathForFile are cleaned up (not having .. etc)
* Fixed so UbaAgent handle WSAPoll POLLERR as timeout so it retries when failing (this made the process exit on wine)
* Fixed so returned processes that had been reused returned the reused process to the scheduler
* Fixed so EnsureBinaryFile is using correct fileNameKeys
* Removed assert in detoured GetFullPathNameA since it is always calling GetFullPathNameW

[CL 30465533 by henrik karlsson in ue5-main branch]
2024-01-03 16:23:58 -05:00
henrik karlsson
a42437b0b6 [UBA]
* Fixed bug with detoured process requesting more processes if custom handler is not registered

[CL 30462271 by henrik karlsson in ue5-main branch]
2024-01-02 18:22:46 -05:00
henrik karlsson
ea673427b2 [UBA]
* Implemented process reuse logic in UbaScheduler. It is now possible for running processes to fetch more work
* Fixed bugs in custom message path
* Fixed bugs in FlushWrittenFiles message
* Changed So UpdateEnvironment is resetting stats
* Fixed potential race condition related to process reuse
* Added special rule for ShaderCompileWorker.exe which detours ImageGetDigestStream in Imagehlp.dll (because wine implementation does not match windows implementation)
* Renamed "exitedUserData" to just userData since it is used for more than when exiting process
* Added a couple more dlls to "known system dlls"
* Fixed so visualizer can visualize process reuse properly

[CL 30462059 by henrik karlsson in ue5-main branch]
2024-01-02 17:42:42 -05:00
henrik karlsson
9b8409f622 [UBA]
* Fixed so when downloading application modules, thread itself helps out downloading

[CL 30438265 by henrik karlsson in ue5-main branch]
2023-12-22 03:35:12 -05:00
henrik karlsson
85e56e8590 [UBA]
* Fixed macos compile error

[CL 30438003 by henrik karlsson in ue5-main branch]
2023-12-22 02:57:12 -05:00
henrik karlsson
32f666ea42 [UBA]
* Added Scheduler class that is a very simple scheduler that handles processes that has no intra dependencies. It handles scheduling remotely and also reschedule processes returned from remote machines.
* Changed so application dependencies are retrieved in parallel by clients
* Added so custom assert handler can be set from the outside
* Added traceEnabled to SessionCreateInfo so trace shared mem can be created without needing to launch visualizer or write to file
* Added so userData can be provided in RemoteProcessAvailable and RemoteProcessReturned callbacks
* Added more files to known system files (based on what exists in wine)
* Improved FindImports code and made it available in export. (it is now automatically filtering out known system files)
* Moved ProcessHandle to its own file
* Added ProcessStartInfoHolder which is a class that can wrap a ProcessStartInfo and make sure all strings are allocated

[CL 30437951 by henrik karlsson in ue5-main branch]
2023-12-22 02:41:43 -05:00
henrik karlsson
79fad227cd [UBA]
* Fixed linux compile error

[CL 30407078 by henrik karlsson in ue5-main branch]
2023-12-19 20:02:10 -05:00