Commit Graph

58 Commits

Author SHA1 Message Date
henrik karlsson
81920d7856 [UBA]
* Fixed validation code in scheduler for process reuse
* Moved all code inside ServerSession::HandleMessage into one function per message to more easily be able to read callstacks
* Added more validation around missing file errors.. now server logs out lots of information too

[CL 34414815 by henrik karlsson in ue5-main branch]
2024-06-17 02:02:04 -04:00
henrik karlsson
07e85ed2d9 [UBA]
* Added so path environment variable is networked over to helper. Needed when programs do their own search to find executable from name

[CL 34393859 by henrik karlsson in ue5-main branch]
2024-06-15 01:59:17 -04:00
henrik karlsson
339e9c9d52 [UBA]
* Added more code to be able to diagnose rare issue on the farm

[CL 34393839 by henrik karlsson in ue5-main branch]
2024-06-15 01:57:17 -04:00
henrik karlsson
978134e2bf [UBA]
* Changed so ProcessStartupInfoHolder inherits from ProcessStartupInfo

[CL 34393813 by henrik karlsson in ue5-main branch]
2024-06-15 01:56:18 -04:00
henrik karlsson
d97e1a534e [UBA]
* Added tests for Create/Remove directory
* Misc small cleanups

[CL 34393738 by henrik karlsson in ue5-main branch]
2024-06-15 01:29:18 -04:00
henrik karlsson
4ae98f4792 [UBA]
* Added support for GetLongPathName and RemoveDirectory for remote helpers

[CL 34393494 by henrik karlsson in ue5-main branch]
2024-06-15 00:37:17 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
502a9e44e2 [UBA]
* Added StringView type and replaced a ton of string+stringLen to StringView
* Fixed bug in obj file preloader.
* Fixed so clang targets can used compressed obj files
* Fixed bug in cache client when entry count is zero
* Added magic to compressed obj file header to be able to see if an obj file is compressed or not
* Moved ParseArguments to its own file and added so it can handle both char and wchar

[CL 33933853 by henrik karlsson in ue5-main branch]
2024-05-27 23:22:26 -04:00
henrik karlsson
22767fb360 [UBA]
* Improved error message on session client when failing to upload file to server

[CL 33642344 by henrik karlsson in ue5-main branch]
2024-05-14 20:26:45 -04:00
henrik karlsson
9ed810f4c8 [UBA]
* Fixed so additional error logging for missing files is not including files that does not match name

[CL 33579258 by henrik karlsson in ue5-main branch]
2024-05-10 12:56:21 -04:00
henrik karlsson
48cfe3206c [UBA]
* Fixed code related to process reuse. It was not resetting variables properly
* Added support for detoured SHGetKnownFolderPath
* Fixed so logs for remote child processes are networked over to server when remoteLogs are enabled
* Improved some assert messages
* Fixed some binaryreaderwriter mismatches (they where harmless, just caused asserts)
* Fixed so local system temp folder can always be found when recursing from root to system temp. This is a horrible hack because it could cause bad conflicts between host file system and helper file system.. but solves current problems
* Moved removal of \\?\ into FixPath

[CL 33517723 by henrik karlsson in ue5-main branch]
2024-05-08 12:53:11 -04:00
henrik karlsson
b7e3cef7bf [UBA]
* Added retry code for when failing to decompress cas file to memory on client.. it could be that cas file is corrupt so we delete the file and re-download it.
* Made CheckCasFiles parallel
* Fixed so CheckCasFiles check all files that was created after .isRunning was created to see that their content look valid
* Added time to log when correcting/validation cas store

[CL 33390412 by henrik karlsson in ue5-main branch]
2024-05-02 00:46:58 -04:00
henrik karlsson
3756c4828b [UBA]
* Added more info to logging trying to catch bug where header files are not "found" remotely

[CL 33377278 by henrik karlsson in ue5-main branch]
2024-05-01 13:58:08 -04:00
henrik karlsson
23db012ceb [UBA]
* Added more logging information when farm gets a include not found error. This is temporary and will be deleted once we have figured out why this can happen now and then.

[CL 33368684 by henrik karlsson in ue5-main branch]
2024-05-01 02:26:42 -04:00
henrik karlsson
19f7cccff9 [UBA]
* Fixed serious bug related to child processes on helper machines getting process ids that collides with ids of root processes coming from server
* Fixed bug in trace reader causing reusable processes to get no stats
* Added good-to-have asserts

[CL 33064711 by henrik karlsson in ue5-main branch]
2024-04-18 01:10:16 -04:00
henrik karlsson
44a70ba88f [UBA]
* Fixed so tracked inputs can be bigger than 512kb (ouch)
* Added to BinaryWriter so you can always write utf8 even though platform is not utf8
* Fixed bug in IsKnownSystemFile (there was a missing comma on one line). Added unit test
* Added Swap function to MemoryBlock which can be used to swap content of memoryblocks
* Added Parse function to StringBuffer that can parse a wchar stringbuffer into a char array
* Fixed TimeToText for times over 24 hours
* Added some accessors to Session
* Fixed so some temp files are not tracked as inputs for detoured process
* Added assert when message is created without body (it is a special case scenario)
* Removed final keyword on NetworkClient to be able to subclass in export logic (it is just easier)

[CL 32870061 by henrik karlsson in ue5-main branch]
2024-04-10 20:21:25 -04:00
henrik karlsson
643c4ffe51 [UBA]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum

[CL 32777954 by henrik karlsson in ue5-main branch]
2024-04-05 19:10:53 -04:00
henrik karlsson
9b17fef999 [UBA]
* Added support for storing .obj files compressed by cl.exe and then decompressed in-memory by link.exe. Note this is not properly implemented and only work for cl.exe+link.exe combo

[CL 32625054 by henrik karlsson in ue5-main branch]
2024-03-29 17:49:27 -04:00
henrik karlsson
1c70c24b63 [Uba]
* Fixed stack overflow issue on mac silicon machines

[CL 32195720 by henrik karlsson in ue5-main branch]
2024-03-12 18:16:36 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
d394b4b515 [UBA]
* Added session notification message to be able to see why client is about to go away.. hopefully this will help us identify if there are crashes happening on remotes that we don't track
* Added ApplicationRules::AllowStorageProxy and changed so .obj files are never going through the storage proxy on fetch (they are only read by one process so no point feeding them through the proxy
* Fixed the file write-through code and gave the #if 0 a name (#if UBA_USE_WRITE_THROUGH).. still disabled since it didn't give any benefits enabling it

[CL 32025069 by henrik karlsson in ue5-main branch]
2024-03-05 12:41:52 -05:00
henrik karlsson
3522a7bcd6 [UBA]
* Added __try/__catch to UbaAgent main and disabled the other seh exception handling to see if we get more luck with this on wine/linux
* Moved code from SessionClient ctor to start function to get around ordering issue where session can get connection callback in ctor
* Compile fix for test code caused by previous changelist

[CL 31982469 by henrik karlsson in ue5-main branch]
2024-03-03 23:33:30 -05:00
henrik karlsson
ff59263a86 [UBA]
* Added support for sending .uba trace files from helpers to server

[CL 31981882 by henrik karlsson in ue5-main branch]
2024-03-03 21:18:10 -05:00
henrik karlsson
361532ab08 [UBA]
* Reduced contention in SessionClient when testing if processes are done and spawning new processes
* Changed so FileMappingBuffer is lazily closing mapped memory. For some reason these calls end up being very expensive and we don't have to close them straight away

[CL 31796195 by henrik karlsson in ue5-main branch]
2024-02-26 01:50:10 -05:00
henrik karlsson
0b12758059 [UBA]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1

[CL 31795889 by henrik karlsson in ue5-main branch]
2024-02-26 01:14:42 -05:00