42 Commits

Author SHA1 Message Date
henrik karlsson
7b39171a86 [UBA]
* Fixed so casentries collected through populating from disk have their verified and exists set to true
* Changed so copyorlink the cas data to temporary folders for execution are never linked on macos.. for some reason this does not always work which is strange

[CL 35908391 by henrik karlsson in ue5-main branch]
2024-08-29 18:05:36 -04:00
henrik karlsson
6a8b0c6c8a [UBA]
* Fix for mistake where file lookup db was populated with compressed caskeys

[CL 35731393 by henrik karlsson in ue5-main branch]
2024-08-22 00:22:17 -04:00
henrik karlsson
21e61a2b7f [UBA]
* Fixed so caskey is stored as not compressed when populating cas from dir

[CL 35731117 by henrik karlsson in ue5-main branch]
2024-08-21 23:58:47 -04:00
henrik karlsson
6e5f130451 [UBA]
* More optimizations in PopulateCasFromDirs

[CL 35731080 by henrik karlsson in ue5-main branch]
2024-08-21 23:49:47 -04:00
henrik karlsson
e9a4eefa88 [UBA]
* More optimizations around PopulateCasFromDirs

[CL 35730709 by henrik karlsson in ue5-main branch]
2024-08-21 23:07:47 -04:00
henrik karlsson
907121dc6c [UBA]
* Fixed so PopulateCasFromDirs now use file lookup table to prevent cas calculation if file has been calculated before

[CL 35730628 by henrik karlsson in ue5-main branch]
2024-08-21 23:00:47 -04:00
henrik karlsson
7818bba17a [UBA]
* Expanded lock a little bit when populating local cas files because it is now populated at the same time as being used
* Fixed uninitialized variable that caused cas population from all xcode installations

[CL 35728595 by henrik karlsson in ue5-main branch]
2024-08-21 20:13:16 -04:00
henrik karlsson
e29818bdf4 [UBA]
* Fixed so agent populates cas after it has started the session.. this in order to prevent timeouts after connecting if populating cas from local path takes a long time

[CL 35728234 by henrik karlsson in ue5-main branch]
2024-08-21 19:56:16 -04:00
henrik karlsson
ef9f400171 [UBA]
* Changed so all remote processes that fails that have a failed message will send back for retry

[CL 35349174 by henrik karlsson in ue5-main branch]
2024-08-06 15:16:58 -04:00
henrik karlsson
b6b1d6dead [UBA]
* Found another bug in cas file transfer code where if segments where aligned on a very specific size the receiving side would throw an error.

[CL 35058262 by henrik karlsson in ue5-main branch]
2024-07-24 13:03:44 -04:00
henrik karlsson
1bd7940eef [UBA]
* Fixed so bad file path entries are not saved to casdb file
* Added so allow proxy can be set from config files
* Added assert for if bad files have entered the cas entry directory on disk

[CL 34476627 by henrik karlsson in ue5-main branch]
2024-06-18 16:23:41 -04:00
henrik karlsson
0d5d90f935 [UBA]
* Added MapMemoryCopy function that wraps memcpy and add timer

[CL 34058937 by henrik karlsson in ue5-main branch]
2024-06-02 18:19:46 -04:00
henrik karlsson
b7e3cef7bf [UBA]
* Added retry code for when failing to decompress cas file to memory on client.. it could be that cas file is corrupt so we delete the file and re-download it.
* Made CheckCasFiles parallel
* Fixed so CheckCasFiles check all files that was created after .isRunning was created to see that their content look valid
* Added time to log when correcting/validation cas store

[CL 33390412 by henrik karlsson in ue5-main branch]
2024-05-02 00:46:58 -04:00
henrik karlsson
d6cab679ff [UBA]
* Added header to compressed obj files containing the cas key of the compressed file
* Fixed so compressed obj files work with cache system

[CL 33236459 by henrik karlsson in ue5-main branch]
2024-04-25 13:10:20 -04:00
henrik karlsson
55918e1ada [UBA]
* Added artifact cache support. It is now possible to skip running process and instead just download outputs if inputs matches cache entry.

[CL 32870344 by henrik karlsson in ue5-main branch]
2024-04-10 20:29:18 -04:00
henrik karlsson
44a70ba88f [UBA]
* Fixed so tracked inputs can be bigger than 512kb (ouch)
* Added to BinaryWriter so you can always write utf8 even though platform is not utf8
* Fixed bug in IsKnownSystemFile (there was a missing comma on one line). Added unit test
* Added Swap function to MemoryBlock which can be used to swap content of memoryblocks
* Added Parse function to StringBuffer that can parse a wchar stringbuffer into a char array
* Fixed TimeToText for times over 24 hours
* Added some accessors to Session
* Fixed so some temp files are not tracked as inputs for detoured process
* Added assert when message is created without body (it is a special case scenario)
* Removed final keyword on NetworkClient to be able to subclass in export logic (it is just easier)

[CL 32870061 by henrik karlsson in ue5-main branch]
2024-04-10 20:21:25 -04:00
henrik karlsson
643c4ffe51 [UBA]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum

[CL 32777954 by henrik karlsson in ue5-main branch]
2024-04-05 19:10:53 -04:00
henrik karlsson
428298308f [UBA]
* Fixed so NetworkMessage can be created without initialized
* Removed unused code
* Fixed StorageClient::SendBatchMessages so it deletes correct number of messages if it disconnects in the middle of transfer

[CL 32262740 by henrik karlsson in ue5-main branch]
2024-03-14 18:21:41 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
3ea3ccd9c1 [UBA]
* Changed so in-process storage proxy is using memory network backend for communication with client

[CL 31982524 by henrik karlsson in ue5-main branch]
2024-03-03 23:44:00 -05:00
henrik karlsson
d7599db4bd [UBA]
* Changed NetworkMessage to use a callback for when message is done and moved event out of message. This is a change in preparation for UbaStorageProxy optimization

[CL 31865186 by henrik karlsson in ue5-main branch]
2024-02-28 00:30:17 -05:00
henrik karlsson
0b12758059 [UBA]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1

[CL 31795889 by henrik karlsson in ue5-main branch]
2024-02-26 01:14:42 -05:00
henrik karlsson
18005e4601 [UBA]
* Disabled custom signal handler in UbaHost in case this is the reason we get crashes in ubt
* Changed some logging from Info to Detail

[CL 31742437 by henrik karlsson in ue5-main branch]
2024-02-22 18:46:44 -05:00
henrik karlsson
8255fb00b5 [Uba]
* Added more description when failing to drop cas db entry
* Added cas entry lock around code that copy or link file out
* Added more fixes for things reported by tsan/asan

[CL 31708309 by henrik karlsson in ue5-main branch]
2024-02-21 20:09:27 -05:00
henrik karlsson
529009dcd9 [Uba]
* Changed so ClientSession struct is allocated using aligned_alloc instead of new.. we can't figure out why this is failing with new on the farm and noone can repro so this is a very nasty workaround until we do understand how it can go wrong.
* Fixed potential race condition in scheduler
* Reduced lock scope in StorageClient to remove chance of deadlock
* Enabled mimalloc on linux

[CL 31660233 by henrik karlsson in ue5-main branch]
2024-02-20 18:03:50 -05:00