29 Commits

Author SHA1 Message Date
henrik karlsson
b3e6772c6f [UBA]
* Added so it is possible to configure number of workers when spawning UbaAgent (mostly for debugging)

[CL 35873914 by henrik karlsson in ue5-main branch]
2024-08-28 13:33:28 -04:00
henrik karlsson
82f4e890bb [UBA]
* Changed so network client/server listen to 0.0.0.0 by default instead of traversing network adapters.

Thanks ZeroErrors[at]github for reporting the issue with the network adapter traversal.

[CL 35535974 by henrik karlsson in ue5-main branch]
2024-08-14 11:43:45 -04:00
henrik karlsson
70329d1cf1 [UBA]
* Changed event to use clock CLOCK_MONOTONIC for linux and changed to pthread_cond_timedwait_relative_np for mac.
* Changed GetTime to use clock_gettime_nsec_np(CLOCK_UPTIME_RAW) for mac.

These changes are to attempt to fix the 10minute event timeouts that are not 10 minutes but rather a second.

[CL 35063177 by henrik karlsson in ue5-main branch]
2024-07-24 15:26:01 -04:00
henrik karlsson
057592973a [UBA]
* Moved event initialization code to be outside lock in order to reduce contention

[CL 35006365 by henrik karlsson in ue5-main branch]
2024-07-22 19:11:51 -04:00
henrik karlsson
74257b2783 [UBA]
* Added validation code for creating event used to wait on message response

[CL 35006273 by henrik karlsson in ue5-main branch]
2024-07-22 19:08:56 -04:00
henrik karlsson
9f0be19f61 [UBA]
* Changed so actual wait time when timing out waiting for message is logged

[CL 35005070 by henrik karlsson in ue5-main branch]
2024-07-22 18:35:42 -04:00
henrik karlsson
b6d23cd6e0 [UBA]
* Fixed non windows compile errors
* Fixed bug with m_connectionsIt not being set

[CL 34859444 by henrik karlsson in ue5-main branch]
2024-07-16 21:58:09 -04:00
henrik karlsson
829e2b41da [UBA]
* Fixed potential race condition related to additional connections and them being added while send call picks up and use the connection before callbacks are set and there is an disconnect

[CL 34858451 by henrik karlsson in ue5-main branch]
2024-07-16 20:07:03 -04:00
henrik karlsson
f6ced08c48 [UBA]
* Added more details to logging when uba cache client fails to connect

[CL 34774526 by henrik karlsson in ue5-main branch]
2024-07-04 12:56:04 -04:00
henrik karlsson
14ff4f319e [UBA]
* Moved m_isConnected event Set call to inside write lock for connected functions to make sure connected functions are only called once
* Added delete of copy ctor and assignment

[CL 34773910 by henrik karlsson in ue5-main branch]
2024-07-04 10:58:32 -04:00
henrik karlsson
8e5bfe4d9b [UBA]
* Added error code to NetworkMessage to be able to diagnose "failed to connect" errors in cache client

[CL 34385738 by henrik karlsson in ue5-main branch]
2024-06-14 17:41:02 -04:00
henrik karlsson
9ba008c8b4 [UBA]
* Optimized cache entry matching
* Added StoreCasKey and changed some StoreCasFile to StoreCasKey in CacheClient which does not need to store cas files
* Changed some parts of the code to use string keys instead of strings since the keys are designed to have correct casing

[CL 34027529 by henrik karlsson in ue5-main branch]
2024-05-31 02:18:27 -04:00
henrik karlsson
19f7cccff9 [UBA]
* Fixed serious bug related to child processes on helper machines getting process ids that collides with ids of root processes coming from server
* Fixed bug in trace reader causing reusable processes to get no stats
* Added good-to-have asserts

[CL 33064711 by henrik karlsson in ue5-main branch]
2024-04-18 01:10:16 -04:00
henrik karlsson
44a70ba88f [UBA]
* Fixed so tracked inputs can be bigger than 512kb (ouch)
* Added to BinaryWriter so you can always write utf8 even though platform is not utf8
* Fixed bug in IsKnownSystemFile (there was a missing comma on one line). Added unit test
* Added Swap function to MemoryBlock which can be used to swap content of memoryblocks
* Added Parse function to StringBuffer that can parse a wchar stringbuffer into a char array
* Fixed TimeToText for times over 24 hours
* Added some accessors to Session
* Fixed so some temp files are not tracked as inputs for detoured process
* Added assert when message is created without body (it is a special case scenario)
* Removed final keyword on NetworkClient to be able to subclass in export logic (it is just easier)

[CL 32870061 by henrik karlsson in ue5-main branch]
2024-04-10 20:21:25 -04:00
henrik karlsson
428298308f [UBA]
* Fixed so NetworkMessage can be created without initialized
* Removed unused code
* Fixed StorageClient::SendBatchMessages so it deletes correct number of messages if it disconnects in the middle of transfer

[CL 32262740 by henrik karlsson in ue5-main branch]
2024-03-14 18:21:41 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
3ea3ccd9c1 [UBA]
* Changed so in-process storage proxy is using memory network backend for communication with client

[CL 31982524 by henrik karlsson in ue5-main branch]
2024-03-03 23:44:00 -05:00
henrik karlsson
ffbd2809a5 [UBA]
* Fixed race condition in UbaNetworkClient related to message ids and returning message ids after use
* Implemented network backend that is using memory for communication (will be used between client and proxy server when running inproc)
* Added connection uid provided by network backend to be able to improve error messages
* Added NetworkBackend GetTotalSendAndRecv that can be used to fetch all traffic on backend

[CL 31981782 by henrik karlsson in ue5-main branch]
2024-03-03 21:07:10 -05:00
henrik karlsson
d7599db4bd [UBA]
* Changed NetworkMessage to use a callback for when message is done and moved event out of message. This is a change in preparation for UbaStorageProxy optimization

[CL 31865186 by henrik karlsson in ue5-main branch]
2024-02-28 00:30:17 -05:00
henrik karlsson
4e4877105e [UBA]
* Added Uba NetworkClient::StopAll
* Fixed shutdown race condition in NetworkClient
* Changed client work count from 16 to no of logical cores

[CL 31795950 by henrik karlsson in ue5-main branch]
2024-02-26 01:23:42 -05:00
henrik karlsson
0b12758059 [UBA]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1

[CL 31795889 by henrik karlsson in ue5-main branch]
2024-02-26 01:14:42 -05:00
henrik karlsson
4481da98d7 [Uba]
* Fixed deadlock in client that could happen if client connecting timed out at the same time as it managed to connect. There were lots of race conditions in this code around these scenarios. Current implementation should be properly fixed

[CL 31756699 by henrik karlsson in ue5-main branch]
2024-02-23 00:59:47 -05:00
henrik karlsson
eb7f4122df [Uba]
* Fixed shutdown issues where client and server did not wait for connection to be closed before closing down
* Added error handling for trying to decompress bad cas files

[CL 31715626 by henrik karlsson in ue5-main branch]
2024-02-22 01:56:35 -05:00
henrik karlsson
8255fb00b5 [Uba]
* Added more description when failing to drop cas db entry
* Added cas entry lock around code that copy or link file out
* Added more fixes for things reported by tsan/asan

[CL 31708309 by henrik karlsson in ue5-main branch]
2024-02-21 20:09:27 -05:00
henrik karlsson
4373e37640 [Uba]
* Fixed a bug in UbaWorkManager shutdown where intrusive linked list contains bad instances while deleting workers
* Fixed more tsan errors when doing ctrl-c on host

[CL 31676297 by henrik karlsson in ue5-main branch]
2024-02-21 02:52:38 -05:00