Commit Graph

28 Commits

Author SHA1 Message Date
henrik karlsson
7647b785f2 [UBA]
* Added support for strings and comments in config files
* Added tests
* Fixed for uba to work integrated in other programs (proper dll exports)
* Added process id to process info in visualizer

[CL 34364596 by henrik karlsson in ue5-main branch]
2024-06-14 02:35:27 -04:00
henrik karlsson
bbe78dfb31 [UBA]
* Added code to make cache server return result without waiting for unmapview of file being done

[CL 34132357 by henrik karlsson in ue5-main branch]
2024-06-05 14:00:34 -04:00
henrik karlsson
5a263c548a [UBA]
* Added extra validation when writing cache entries
* Disabled directory preparsing when fetching cache entries for now

[CL 34100764 by henrik karlsson in ue5-main branch]
2024-06-04 14:07:46 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
9ba008c8b4 [UBA]
* Optimized cache entry matching
* Added StoreCasKey and changed some StoreCasFile to StoreCasKey in CacheClient which does not need to store cas files
* Changed some parts of the code to use string keys instead of strings since the keys are designed to have correct casing

[CL 34027529 by henrik karlsson in ue5-main branch]
2024-05-31 02:18:27 -04:00
henrik karlsson
639a380cda [UBA]
* Fixed static analysis errors

[CL 33948663 by henrik karlsson in ue5-main branch]
2024-05-28 15:07:55 -04:00
henrik karlsson
d8bc6303d1 [UBA]
* Fixed fallback code for not finding caskey for file to compare caskeys with matching compression flags

[CL 33907932 by henrik karlsson in ue5-main branch]
2024-05-24 17:45:03 -04:00
henrik karlsson
d279e02711 [UBA]
* Added rsaenh.dll to known files list

[CL 33730633 by henrik karlsson in ue5-main branch]
2024-05-17 14:00:34 -04:00
henrik karlsson
48cfe3206c [UBA]
* Fixed code related to process reuse. It was not resetting variables properly
* Added support for detoured SHGetKnownFolderPath
* Fixed so logs for remote child processes are networked over to server when remoteLogs are enabled
* Improved some assert messages
* Fixed some binaryreaderwriter mismatches (they where harmless, just caused asserts)
* Fixed so local system temp folder can always be found when recursing from root to system temp. This is a horrible hack because it could cause bad conflicts between host file system and helper file system.. but solves current problems
* Moved removal of \\?\ into FixPath

[CL 33517723 by henrik karlsson in ue5-main branch]
2024-05-08 12:53:11 -04:00
henrik karlsson
d6cab679ff [UBA]
* Added header to compressed obj files containing the cas key of the compressed file
* Fixed so compressed obj files work with cache system

[CL 33236459 by henrik karlsson in ue5-main branch]
2024-04-25 13:10:20 -04:00
henrik karlsson
04c89f60f5 [UBA/UBT]
* Fixed so linux UbaCacheService can host windows clients

[CL 33219738 by henrik karlsson in ue5-main branch]
2024-04-24 20:07:13 -04:00
henrik karlsson
c7a5dc7f43 [Uba]
* Fixed option to manually handle storage overflow
* Enabled this option in cache server
* Fixed bug where cas files received from client was not added to written files linked list used to decide which files to evict first

[CL 32950775 by henrik karlsson in ue5-main branch]
2024-04-12 19:42:04 -04:00
henrik karlsson
55918e1ada [UBA]
* Added artifact cache support. It is now possible to skip running process and instead just download outputs if inputs matches cache entry.

[CL 32870344 by henrik karlsson in ue5-main branch]
2024-04-10 20:29:18 -04:00
henrik karlsson
643c4ffe51 [UBA]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum

[CL 32777954 by henrik karlsson in ue5-main branch]
2024-04-05 19:10:53 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
3ea3ccd9c1 [UBA]
* Changed so in-process storage proxy is using memory network backend for communication with client

[CL 31982524 by henrik karlsson in ue5-main branch]
2024-03-03 23:44:00 -05:00
henrik karlsson
bbf8e06ccb [UBA]
* Added support for handling server side messages and send response later
* Changed so storage proxy errors are muted when disconnected
* Improved error messages

[CL 31981923 by henrik karlsson in ue5-main branch]
2024-03-03 21:30:10 -05:00
henrik karlsson
0b12758059 [UBA]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1

[CL 31795889 by henrik karlsson in ue5-main branch]
2024-02-26 01:14:42 -05:00
henrik karlsson
18005e4601 [UBA]
* Disabled custom signal handler in UbaHost in case this is the reason we get crashes in ubt
* Changed some logging from Info to Detail

[CL 31742437 by henrik karlsson in ue5-main branch]
2024-02-22 18:46:44 -05:00
henrik karlsson
5adb64e10e [UBA]
* Fixed read-after-free bug found by TSAN in StorageServer.cpp
* Fixed write-after-free bug found in UbaProcess.cpp (m_cancelEvent.Set() could be called after memory was freed)
* Lots of minor fixes found using TSAN. Most are harmless but still nice to cleanup
* Disabled mimalloc for linux again.. seems like tsan does not like it so maybe there are bugs in it

[CL 31676014 by henrik karlsson in ue5-main branch]
2024-02-21 01:43:26 -05:00
henrik karlsson
dd7143b230 [UBA]
* Fixed write-after-free bug in code writing cas file to disk
* Fixed write-after-free bug in code cleaning up finished processes in session client code
* Added some asserts
* UbaCli - disabled remote logging in non-debug configs

[CL 31632262 by henrik karlsson in ue5-main branch]
2024-02-19 18:41:16 -05:00
henrik karlsson
b4b421de64 [UBA]
* Fixed so server side code sending cas to client does not have to send messages in order. If file is larger than 16mb we use file mapping instead with random access.

[CL 31607373 by henrik karlsson in ue5-main branch]
2024-02-19 01:59:59 -05:00
henrik karlsson
5673117e65 [UBA]
* Added code to handling uba::ReadFile trying to read file with less bytes than requested. It will try for 3 seconds and then fail with a message saying it read 0 bytes for 3 seconds
* Added more information to error messages

[CL 31461069 by henrik karlsson in ue5-main branch]
2024-02-14 01:37:37 -05:00
henrik karlsson
34b52deafb [UBA]
* Added hint to unmapfileview to be able to add useful information to asserts on fails
* Added error handling for vsnprintf returning error
* Fixed memory stomp in unit test
* Removed usage of %hs on non-windows since asan complains about it
* Changed UBA_USE_MIMALLOC to always be defined and be 0 or 1 instead of using ifdef
* Enabled mimalloc on linux

[CL 31355077 by henrik karlsson in ue5-main branch]
2024-02-09 16:12:12 -05:00
henrik karlsson
5314f45f1e [UBA]
* Fixed bug that could cause an infinite build hang if agent with multiple tcp connections disconnected while retreiving files

[CL 30284803 by henrik karlsson in ue5-main branch]
2023-12-12 21:25:58 -05:00