Commit Graph

32 Commits

Author SHA1 Message Date
henrik karlsson
1bd7940eef [UBA]
* Fixed so bad file path entries are not saved to casdb file
* Added so allow proxy can be set from config files
* Added assert for if bad files have entered the cas entry directory on disk

[CL 34476627 by henrik karlsson in ue5-main branch]
2024-06-18 16:23:41 -04:00
henrik karlsson
0d5d90f935 [UBA]
* Added MapMemoryCopy function that wraps memcpy and add timer

[CL 34058937 by henrik karlsson in ue5-main branch]
2024-06-02 18:19:46 -04:00
henrik karlsson
b7e3cef7bf [UBA]
* Added retry code for when failing to decompress cas file to memory on client.. it could be that cas file is corrupt so we delete the file and re-download it.
* Made CheckCasFiles parallel
* Fixed so CheckCasFiles check all files that was created after .isRunning was created to see that their content look valid
* Added time to log when correcting/validation cas store

[CL 33390412 by henrik karlsson in ue5-main branch]
2024-05-02 00:46:58 -04:00
henrik karlsson
d6cab679ff [UBA]
* Added header to compressed obj files containing the cas key of the compressed file
* Fixed so compressed obj files work with cache system

[CL 33236459 by henrik karlsson in ue5-main branch]
2024-04-25 13:10:20 -04:00
henrik karlsson
55918e1ada [UBA]
* Added artifact cache support. It is now possible to skip running process and instead just download outputs if inputs matches cache entry.

[CL 32870344 by henrik karlsson in ue5-main branch]
2024-04-10 20:29:18 -04:00
henrik karlsson
44a70ba88f [UBA]
* Fixed so tracked inputs can be bigger than 512kb (ouch)
* Added to BinaryWriter so you can always write utf8 even though platform is not utf8
* Fixed bug in IsKnownSystemFile (there was a missing comma on one line). Added unit test
* Added Swap function to MemoryBlock which can be used to swap content of memoryblocks
* Added Parse function to StringBuffer that can parse a wchar stringbuffer into a char array
* Fixed TimeToText for times over 24 hours
* Added some accessors to Session
* Fixed so some temp files are not tracked as inputs for detoured process
* Added assert when message is created without body (it is a special case scenario)
* Removed final keyword on NetworkClient to be able to subclass in export logic (it is just easier)

[CL 32870061 by henrik karlsson in ue5-main branch]
2024-04-10 20:21:25 -04:00
henrik karlsson
643c4ffe51 [UBA]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum

[CL 32777954 by henrik karlsson in ue5-main branch]
2024-04-05 19:10:53 -04:00
henrik karlsson
428298308f [UBA]
* Fixed so NetworkMessage can be created without initialized
* Removed unused code
* Fixed StorageClient::SendBatchMessages so it deletes correct number of messages if it disconnects in the middle of transfer

[CL 32262740 by henrik karlsson in ue5-main branch]
2024-03-14 18:21:41 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
3ea3ccd9c1 [UBA]
* Changed so in-process storage proxy is using memory network backend for communication with client

[CL 31982524 by henrik karlsson in ue5-main branch]
2024-03-03 23:44:00 -05:00
henrik karlsson
d7599db4bd [UBA]
* Changed NetworkMessage to use a callback for when message is done and moved event out of message. This is a change in preparation for UbaStorageProxy optimization

[CL 31865186 by henrik karlsson in ue5-main branch]
2024-02-28 00:30:17 -05:00
henrik karlsson
0b12758059 [UBA]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1

[CL 31795889 by henrik karlsson in ue5-main branch]
2024-02-26 01:14:42 -05:00
henrik karlsson
18005e4601 [UBA]
* Disabled custom signal handler in UbaHost in case this is the reason we get crashes in ubt
* Changed some logging from Info to Detail

[CL 31742437 by henrik karlsson in ue5-main branch]
2024-02-22 18:46:44 -05:00
henrik karlsson
8255fb00b5 [Uba]
* Added more description when failing to drop cas db entry
* Added cas entry lock around code that copy or link file out
* Added more fixes for things reported by tsan/asan

[CL 31708309 by henrik karlsson in ue5-main branch]
2024-02-21 20:09:27 -05:00
henrik karlsson
529009dcd9 [Uba]
* Changed so ClientSession struct is allocated using aligned_alloc instead of new.. we can't figure out why this is failing with new on the farm and noone can repro so this is a very nasty workaround until we do understand how it can go wrong.
* Fixed potential race condition in scheduler
* Reduced lock scope in StorageClient to remove chance of deadlock
* Enabled mimalloc on linux

[CL 31660233 by henrik karlsson in ue5-main branch]
2024-02-20 18:03:50 -05:00
henrik karlsson
34b52deafb [UBA]
* Added hint to unmapfileview to be able to add useful information to asserts on fails
* Added error handling for vsnprintf returning error
* Fixed memory stomp in unit test
* Removed usage of %hs on non-windows since asan complains about it
* Changed UBA_USE_MIMALLOC to always be defined and be 0 or 1 instead of using ifdef
* Enabled mimalloc on linux

[CL 31355077 by henrik karlsson in ue5-main branch]
2024-02-09 16:12:12 -05:00
henrik karlsson
b35413a335 [UBA]
* Proper fix of bug where code could write 8 bytes outside memory allocation

[CL 31127482 by henrik karlsson in ue5-main branch]
2024-02-02 11:27:44 -05:00
henrik karlsson
85b847a049 [Uba]
* Fixed bug in storage client code that directly decompress network data to disk/mem. It could write outside buffer by 8 bytes
* Added UbaCli -checkcas2  .. this codepath creates a storage server and client and networks all cas files over.. just to check so code paths properly work
* Fixed UbaCli -checkaws bug

[CL 31118503 by henrik karlsson in ue5-main branch]
2024-02-02 02:00:29 -05:00
henrik karlsson
fa2772e27d [UBA]
* Added so in-process proxy can use storage of client
* Fixed reset of trace reader
* Fixed annoying warnings when closing invalid sockets

[CL 30667316 by henrik karlsson in ue5-main branch]
2024-01-17 15:28:30 -05:00
henrik karlsson
e0e929073b [Uba]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.

[CL 30507663 by henrik karlsson in ue5-main branch]
2024-01-09 12:04:16 -05:00
henrik karlsson
12afed80b7 [UBA]
* Implemented knownInputs support. It is now fully supported to provide known inputs when doing RunProcessRemote. This is optional and can be a nice speedup if latency to helpers is high since it reduce number of messages going back and forth.
* Changed so GetMemoryMapAlignment is checking only file name and no flags. Flags is now handled on the outside. This made it possible to send over map alignment for files without having to send the file names
* Renamed WorkManagerImpl::Wait to WorkManagerImpl::FlushWork
* Renamed willBeUsedUnCompressed to storeUncompressed
* Added ScopedCriticalSection::Enter

[CL 30388720 by henrik karlsson in ue5-main branch]
2023-12-19 01:53:01 -05:00
henrik karlsson
4ac7fefdef [UBA]
* Fixed a bug where compressed cas files ended up with the uncompressed file size on disk

[CL 30388511 by henrik karlsson in ue5-main branch]
2023-12-19 01:06:09 -05:00
henrik karlsson
7804809758 [UBA]
* Fixed so storage client keeps track of last tested proxy to prevent testing of same proxy over and over again
* Removed Sleep to emulate timeout if timeout variable is not set

[CL 30315232 by henrik karlsson in ue5-main branch]
2023-12-14 03:04:13 -05:00
henrik karlsson
cce03720c8 [UBA]
* Added guards for null function pointers in tcp backend
* Changed signal handler for SIGSEGV to use sigaction
* Added time to log entry when populating cas client side

[CL 30303417 by henrik karlsson in ue5-main branch]
2023-12-13 15:59:16 -05:00
henrik karlsson
7814c8dfd9 [UBA]
* Fixes to make macos work properly with remote helpers

[CL 30280233 by henrik karlsson in ue5-main branch]
2023-12-12 17:49:54 -05:00