Commit Graph

256 Commits

Author SHA1 Message Date
henrik karlsson
06f1fc5332 [UBA]
* Lowered stack usage for some unit tests to see if that solves macos failures

[CL 32146672 by henrik karlsson in ue5-main branch]
2024-03-10 00:06:20 -05:00
henrik karlsson
b42a0e2d39 [UBA]
* Removed hard coded epic strings that should not be in code

[CL 32137608 by henrik karlsson in ue5-main branch]
2024-03-08 18:50:54 -05:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
a4d5c72c7c [UBA]
* Added custom application rule for ShaderCompileWorker to prevent it from using storage proxy for .uba.in files since they are always unique anyway

[CL 32131073 by henrik karlsson in ue5-main branch]
2024-03-08 16:07:08 -05:00
henrik karlsson
5511083ddb [UBA]
* Fixed bug in binarywriter/reader for wide characters that has a higher numeric value (like korean characters)

[CL 32077280 by henrik karlsson in ue5-main branch]
2024-03-06 21:21:30 -05:00
henrik karlsson
d394b4b515 [UBA]
* Added session notification message to be able to see why client is about to go away.. hopefully this will help us identify if there are crashes happening on remotes that we don't track
* Added ApplicationRules::AllowStorageProxy and changed so .obj files are never going through the storage proxy on fetch (they are only read by one process so no point feeding them through the proxy
* Fixed the file write-through code and gave the #if 0 a name (#if UBA_USE_WRITE_THROUGH).. still disabled since it didn't give any benefits enabling it

[CL 32025069 by henrik karlsson in ue5-main branch]
2024-03-05 12:41:52 -05:00
henrik karlsson
56e1dbbf78 [UBA]
* Fixed missing assignment of process weight in assignment operator.. this caused remote machines to think all processes had the processor weight of 1 which is wrong for processes like cl.exe which cost 1.5 processors.

[CL 32024695 by henrik karlsson in ue5-main branch]
2024-03-05 12:34:44 -05:00
henrik karlsson
692b672f93 [UBA]
* Fixed bug where additional work could be added without worker picking it up causing a deadlock

[CL 32022334 by henrik karlsson in ue5-main branch]
2024-03-05 11:32:25 -05:00
henrik karlsson
632b9434be [UBA]
* Added null guard for WriteAssertInfo text.. should have been part of earlier changelist

[CL 31982586 by henrik karlsson in ue5-main branch]
2024-03-03 23:48:30 -05:00
henrik karlsson
3ea3ccd9c1 [UBA]
* Changed so in-process storage proxy is using memory network backend for communication with client

[CL 31982524 by henrik karlsson in ue5-main branch]
2024-03-03 23:44:00 -05:00
henrik karlsson
3522a7bcd6 [UBA]
* Added __try/__catch to UbaAgent main and disabled the other seh exception handling to see if we get more luck with this on wine/linux
* Moved code from SessionClient ctor to start function to get around ordering issue where session can get connection callback in ctor
* Compile fix for test code caused by previous changelist

[CL 31982469 by henrik karlsson in ue5-main branch]
2024-03-03 23:33:30 -05:00
henrik karlsson
bbf8e06ccb [UBA]
* Added support for handling server side messages and send response later
* Changed so storage proxy errors are muted when disconnected
* Improved error messages

[CL 31981923 by henrik karlsson in ue5-main branch]
2024-03-03 21:30:10 -05:00
henrik karlsson
ff59263a86 [UBA]
* Added support for sending .uba trace files from helpers to server

[CL 31981882 by henrik karlsson in ue5-main branch]
2024-03-03 21:18:10 -05:00
henrik karlsson
ffbd2809a5 [UBA]
* Fixed race condition in UbaNetworkClient related to message ids and returning message ids after use
* Implemented network backend that is using memory for communication (will be used between client and proxy server when running inproc)
* Added connection uid provided by network backend to be able to improve error messages
* Added NetworkBackend GetTotalSendAndRecv that can be used to fetch all traffic on backend

[CL 31981782 by henrik karlsson in ue5-main branch]
2024-03-03 21:07:10 -05:00
henrik karlsson
0b3f5e6f95 [UBA]
* Extracted work tracker out from work manager so it can be used for multiple work managers when running with proxies

[CL 31981685 by henrik karlsson in ue5-main branch]
2024-03-03 20:51:45 -05:00
henrik karlsson
5e6dc44ec3 [UBA]
* Added PrefetchVirtualMemory function
* Fixed bug where code writing callstacks checks end of module to figure out if address belongs to module
* Changed so FileAccessor::CreateMemoryWrite set always allow read because you can't create file mappings that are write-only
* Added option to scheduler to force all runs to run native
* Fixed so bottleneck class has enter/leave function
* Added ScopedReadLock::Enter
* Moved LogWriter to its own file
* Visualizer - Fixed so when hovering over timeline time under pointer is showing
* Visualizer - Fixed so hovering over work records shows name, start and duration
* Visualizer - Added context menu item that can show/hide text of process bars
* Improved assert message
* Changed so lock is not on stack but is leaked to prevent shutdown warnings in debug
* Added missing includes

[CL 31981637 by henrik karlsson in ue5-main branch]
2024-03-03 20:42:41 -05:00
henrik karlsson
d9ce0963c2 [Uba]
* Fixed so visualizer does not crash with version 21 traces that contains log messages that have over 100 lines

[CL 31900281 by henrik karlsson in ue5-main branch]
2024-02-29 02:18:27 -05:00
henrik karlsson
d7599db4bd [UBA]
* Changed NetworkMessage to use a callback for when message is done and moved event out of message. This is a change in preparation for UbaStorageProxy optimization

[CL 31865186 by henrik karlsson in ue5-main branch]
2024-02-28 00:30:17 -05:00
henrik karlsson
2ba1550eba [Uba]
* Fixed bug in trace code where errors are more than 100 lines long

[CL 31855407 by henrik karlsson in ue5-main branch]
2024-02-27 17:34:12 -05:00
henrik karlsson
6bcbb38bd8 [UBA]
* Added potential race condition fix that could happen during certain disconnect situations in storage proxy
* Added Deploy-Linux.bat that can be used to build all linux binaries. It also enables tsan when running "Deploy-Linux.bat debug"

[CL 31832747 by henrik karlsson in ue5-main branch]
2024-02-27 00:49:18 -05:00
henrik karlsson
361532ab08 [UBA]
* Reduced contention in SessionClient when testing if processes are done and spawning new processes
* Changed so FileMappingBuffer is lazily closing mapped memory. For some reason these calls end up being very expensive and we don't have to close them straight away

[CL 31796195 by henrik karlsson in ue5-main branch]
2024-02-26 01:50:10 -05:00
henrik karlsson
1bd3b05737 [UBA]
* Changed UbaAgent max connection count from 4 to 8 to improve throughput when being proxy
* Changed UbaAgent worker count for storage proxy from 16 to 192.. these workers are just receiving/sending data and need to be able to go wide
* Fixed so UbaCli use storage proxy when testing in process client

[CL 31796094 by henrik karlsson in ue5-main branch]
2024-02-26 01:37:10 -05:00
henrik karlsson
79f24ee323 [UBA]
* Changed so sending in a 0 workercount will use logical core count
* Cleaned up code around work tracking

[CL 31796079 by henrik karlsson in ue5-main branch]
2024-02-26 01:35:40 -05:00
henrik karlsson
488b4334ab [UBA]
* Elevate thread priority for the threads doing all the networking to make sure proxy and host are prioritizing receiving/sending network data

[CL 31796023 by henrik karlsson in ue5-main branch]
2024-02-26 01:29:10 -05:00
henrik karlsson
79ea88294a [UBA]
* Changed so StorageProxy is not reading entire files first before sending them out to clients. Instead it now lazily loads each segment based on requests and send it

[CL 31795976 by henrik karlsson in ue5-main branch]
2024-02-26 01:26:10 -05:00