Commit Graph

33 Commits

Author SHA1 Message Date
henrik karlsson
e90be1e1c7 [UBA]
* Moved around variables in UbaStorage
* Added log warning to show if something happens that should never happen

[CL 35689934 by henrik karlsson in ue5-main branch]
2024-08-21 02:51:44 -04:00
henrik karlsson
e666d30940 [UBA]
* Improved some logging to help find reason for error "... that is already being written"

[CL 35190817 by henrik karlsson in ue5-main branch]
2024-07-30 13:47:15 -04:00
henrik karlsson
4f86424225 [UBA]
* Added more logging info around active fetches

[CL 35071303 by henrik karlsson in ue5-main branch]
2024-07-24 18:51:02 -04:00
henrik karlsson
da448ec91c [Uba]
* Fixed variable name

[CL 35048255 by henrik karlsson in ue5-main branch]
2024-07-24 02:26:22 -04:00
henrik karlsson
2d29aa233e [UBA]
* Added debug logging stuff to session server FetchBegin to try to figure out why clients timeout now and then

[CL 35004964 by henrik karlsson in ue5-main branch]
2024-07-22 18:34:54 -04:00
henrik karlsson
7647b785f2 [UBA]
* Added support for strings and comments in config files
* Added tests
* Fixed for uba to work integrated in other programs (proper dll exports)
* Added process id to process info in visualizer

[CL 34364596 by henrik karlsson in ue5-main branch]
2024-06-14 02:35:27 -04:00
henrik karlsson
bbe78dfb31 [UBA]
* Added code to make cache server return result without waiting for unmapview of file being done

[CL 34132357 by henrik karlsson in ue5-main branch]
2024-06-05 14:00:34 -04:00
henrik karlsson
5a263c548a [UBA]
* Added extra validation when writing cache entries
* Disabled directory preparsing when fetching cache entries for now

[CL 34100764 by henrik karlsson in ue5-main branch]
2024-06-04 14:07:46 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
9ba008c8b4 [UBA]
* Optimized cache entry matching
* Added StoreCasKey and changed some StoreCasFile to StoreCasKey in CacheClient which does not need to store cas files
* Changed some parts of the code to use string keys instead of strings since the keys are designed to have correct casing

[CL 34027529 by henrik karlsson in ue5-main branch]
2024-05-31 02:18:27 -04:00
henrik karlsson
639a380cda [UBA]
* Fixed static analysis errors

[CL 33948663 by henrik karlsson in ue5-main branch]
2024-05-28 15:07:55 -04:00
henrik karlsson
d8bc6303d1 [UBA]
* Fixed fallback code for not finding caskey for file to compare caskeys with matching compression flags

[CL 33907932 by henrik karlsson in ue5-main branch]
2024-05-24 17:45:03 -04:00
henrik karlsson
d279e02711 [UBA]
* Added rsaenh.dll to known files list

[CL 33730633 by henrik karlsson in ue5-main branch]
2024-05-17 14:00:34 -04:00
henrik karlsson
48cfe3206c [UBA]
* Fixed code related to process reuse. It was not resetting variables properly
* Added support for detoured SHGetKnownFolderPath
* Fixed so logs for remote child processes are networked over to server when remoteLogs are enabled
* Improved some assert messages
* Fixed some binaryreaderwriter mismatches (they where harmless, just caused asserts)
* Fixed so local system temp folder can always be found when recursing from root to system temp. This is a horrible hack because it could cause bad conflicts between host file system and helper file system.. but solves current problems
* Moved removal of \\?\ into FixPath

[CL 33517723 by henrik karlsson in ue5-main branch]
2024-05-08 12:53:11 -04:00
henrik karlsson
d6cab679ff [UBA]
* Added header to compressed obj files containing the cas key of the compressed file
* Fixed so compressed obj files work with cache system

[CL 33236459 by henrik karlsson in ue5-main branch]
2024-04-25 13:10:20 -04:00
henrik karlsson
04c89f60f5 [UBA/UBT]
* Fixed so linux UbaCacheService can host windows clients

[CL 33219738 by henrik karlsson in ue5-main branch]
2024-04-24 20:07:13 -04:00
henrik karlsson
c7a5dc7f43 [Uba]
* Fixed option to manually handle storage overflow
* Enabled this option in cache server
* Fixed bug where cas files received from client was not added to written files linked list used to decide which files to evict first

[CL 32950775 by henrik karlsson in ue5-main branch]
2024-04-12 19:42:04 -04:00
henrik karlsson
55918e1ada [UBA]
* Added artifact cache support. It is now possible to skip running process and instead just download outputs if inputs matches cache entry.

[CL 32870344 by henrik karlsson in ue5-main branch]
2024-04-10 20:29:18 -04:00
henrik karlsson
643c4ffe51 [UBA]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum

[CL 32777954 by henrik karlsson in ue5-main branch]
2024-04-05 19:10:53 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
3ea3ccd9c1 [UBA]
* Changed so in-process storage proxy is using memory network backend for communication with client

[CL 31982524 by henrik karlsson in ue5-main branch]
2024-03-03 23:44:00 -05:00
henrik karlsson
bbf8e06ccb [UBA]
* Added support for handling server side messages and send response later
* Changed so storage proxy errors are muted when disconnected
* Improved error messages

[CL 31981923 by henrik karlsson in ue5-main branch]
2024-03-03 21:30:10 -05:00
henrik karlsson
0b12758059 [UBA]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1

[CL 31795889 by henrik karlsson in ue5-main branch]
2024-02-26 01:14:42 -05:00
henrik karlsson
18005e4601 [UBA]
* Disabled custom signal handler in UbaHost in case this is the reason we get crashes in ubt
* Changed some logging from Info to Detail

[CL 31742437 by henrik karlsson in ue5-main branch]
2024-02-22 18:46:44 -05:00
henrik karlsson
5adb64e10e [UBA]
* Fixed read-after-free bug found by TSAN in StorageServer.cpp
* Fixed write-after-free bug found in UbaProcess.cpp (m_cancelEvent.Set() could be called after memory was freed)
* Lots of minor fixes found using TSAN. Most are harmless but still nice to cleanup
* Disabled mimalloc for linux again.. seems like tsan does not like it so maybe there are bugs in it

[CL 31676014 by henrik karlsson in ue5-main branch]
2024-02-21 01:43:26 -05:00