Commit Graph

60 Commits

Author SHA1 Message Date
henrik karlsson
81920d7856 [UBA]
* Fixed validation code in scheduler for process reuse
* Moved all code inside ServerSession::HandleMessage into one function per message to more easily be able to read callstacks
* Added more validation around missing file errors.. now server logs out lots of information too

[CL 34414815 by henrik karlsson in ue5-main branch]
2024-06-17 02:02:04 -04:00
henrik karlsson
07e85ed2d9 [UBA]
* Added so path environment variable is networked over to helper. Needed when programs do their own search to find executable from name

[CL 34393859 by henrik karlsson in ue5-main branch]
2024-06-15 01:59:17 -04:00
henrik karlsson
339e9c9d52 [UBA]
* Added more code to be able to diagnose rare issue on the farm

[CL 34393839 by henrik karlsson in ue5-main branch]
2024-06-15 01:57:17 -04:00
henrik karlsson
978134e2bf [UBA]
* Changed so ProcessStartupInfoHolder inherits from ProcessStartupInfo

[CL 34393813 by henrik karlsson in ue5-main branch]
2024-06-15 01:56:18 -04:00
henrik karlsson
dae1ab42db [Uba]
* Changed Config GetTable to return a pointer that can be null if table does not exist

[CL 34393645 by henrik karlsson in ue5-main branch]
2024-06-15 01:11:17 -04:00
henrik karlsson
4ae98f4792 [UBA]
* Added support for GetLongPathName and RemoveDirectory for remote helpers

[CL 34393494 by henrik karlsson in ue5-main branch]
2024-06-15 00:37:17 -04:00
henrik karlsson
cccb57fb9f [UBA]
* Changed so casdb entry is not invalidated when file is stored from helper since we already have caskey and everything ready
* Added options to disable usage of cache hits in cacheclient.. this is a debugging feature
* Added option to not use path roots in cache client
* Added more debug information to try to figure out bug where cache client sees wrong cas key of file

[CL 34170695 by henrik karlsson in ue5-main branch]
2024-06-06 15:45:03 -04:00
henrik karlsson
b629a98f81 [UBA]
* Changed individual cancel message from Info to Detail

[CL 34080563 by henrik karlsson in ue5-main branch]
2024-06-03 18:52:18 -04:00
henrik karlsson
367be10315 [UBA]
* Fixed annoying assert when pressing ctrl-c when running in debug
* Fixed so trace stream writes out processed stopped through cancel

[CL 34078000 by henrik karlsson in ue5-main branch]
2024-06-03 17:26:03 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
502a9e44e2 [UBA]
* Added StringView type and replaced a ton of string+stringLen to StringView
* Fixed bug in obj file preloader.
* Fixed so clang targets can used compressed obj files
* Fixed bug in cache client when entry count is zero
* Added magic to compressed obj file header to be able to see if an obj file is compressed or not
* Moved ParseArguments to its own file and added so it can handle both char and wchar

[CL 33933853 by henrik karlsson in ue5-main branch]
2024-05-27 23:22:26 -04:00
henrik karlsson
b7c47c1473 [UBA]
* Improved error messages related to files sent to server. Messages will now show process they are related to

[CL 33635098 by henrik karlsson in ue5-main branch]
2024-05-14 15:36:20 -04:00
henrik karlsson
48cfe3206c [UBA]
* Fixed code related to process reuse. It was not resetting variables properly
* Added support for detoured SHGetKnownFolderPath
* Fixed so logs for remote child processes are networked over to server when remoteLogs are enabled
* Improved some assert messages
* Fixed some binaryreaderwriter mismatches (they where harmless, just caused asserts)
* Fixed so local system temp folder can always be found when recursing from root to system temp. This is a horrible hack because it could cause bad conflicts between host file system and helper file system.. but solves current problems
* Moved removal of \\?\ into FixPath

[CL 33517723 by henrik karlsson in ue5-main branch]
2024-05-08 12:53:11 -04:00
henrik karlsson
ac3ef463db [UBA]
* Removed code in SessionServer::WaitOnAllTasks that terminates processes on windows. that code was never supposed to be there
* Removed printing of error when trying to traverse directory on volume that has bitlock enabled and not allowed to access

[CL 33490527 by henrik karlsson in ue5-main branch]
2024-05-07 14:41:51 -04:00
henrik karlsson
55d472590f [UBA]
* Added support for providing custom environment variables when creating UbaSession
* Reverted code using real environment variables when spawning processes...

[CL 33469655 by henrik karlsson in ue5-main branch]
2024-05-06 16:54:39 -04:00
henrik karlsson
d6cab679ff [UBA]
* Added header to compressed obj files containing the cas key of the compressed file
* Fixed so compressed obj files work with cache system

[CL 33236459 by henrik karlsson in ue5-main branch]
2024-04-25 13:10:20 -04:00
henrik karlsson
adfccca28f [UBA]
* Fixed bug in text normalization code.. was missing one character
* Fixed so cache processes are colored differently in visualizer
* Improved assert information when pressing ctrl-c during build and an assert triggers

[CL 32957554 by henrik karlsson in ue5-main branch]
2024-04-13 19:21:42 -04:00
henrik karlsson
c88d02914d [UBA]
* Implemented V1 of CompactPathTable which instead stores string segments separately.. Reduces the size table by ~30%.. slightly slower perf
* Added "RootPaths" which is a class that holds a list of root paths needed to normalize paths into non-environment dependent paths
* Added stats for cache that visualizes properly in visualizer
* Changed hash functions to have better quality
* Cleaned up code visualizing stats in visualizer.. now boxes popping up should have a nice matching size to strings
* Cleaned up UbaExports and formatted all functions the same way.. old are deprecated and will be removed once new binaries are in

[CL 32922089 by henrik karlsson in ue5-main branch]
2024-04-12 01:05:04 -04:00
henrik karlsson
44a70ba88f [UBA]
* Fixed so tracked inputs can be bigger than 512kb (ouch)
* Added to BinaryWriter so you can always write utf8 even though platform is not utf8
* Fixed bug in IsKnownSystemFile (there was a missing comma on one line). Added unit test
* Added Swap function to MemoryBlock which can be used to swap content of memoryblocks
* Added Parse function to StringBuffer that can parse a wchar stringbuffer into a char array
* Fixed TimeToText for times over 24 hours
* Added some accessors to Session
* Fixed so some temp files are not tracked as inputs for detoured process
* Added assert when message is created without body (it is a special case scenario)
* Removed final keyword on NetworkClient to be able to subclass in export logic (it is just easier)

[CL 32870061 by henrik karlsson in ue5-main branch]
2024-04-10 20:21:25 -04:00
henrik karlsson
643c4ffe51 [UBA]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum

[CL 32777954 by henrik karlsson in ue5-main branch]
2024-04-05 19:10:53 -04:00
henrik karlsson
9b17fef999 [UBA]
* Added support for storing .obj files compressed by cl.exe and then decompressed in-memory by link.exe. Note this is not properly implemented and only work for cl.exe+link.exe combo

[CL 32625054 by henrik karlsson in ue5-main branch]
2024-03-29 17:49:27 -04:00
henrik karlsson
fabf93ee7c [Uba]
* Added lots of more logging around processes disappearing without exit message
* Added descriptions to lots of asserts since macos refuse to give us good line numbers
* Added more error handling

[CL 32255038 by henrik karlsson in ue5-main branch]
2024-03-14 15:26:23 -04:00
henrik karlsson
6a027cfd06 [UBA]
* Fixed so ref count of remote process is kept up in order to prevent it from being deleted at wrong point in time

[CL 32158427 by henrik karlsson in ue5-main branch]
2024-03-11 13:12:54 -04:00
henrik karlsson
5dc54a8104 [UBA]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer

[CL 32137083 by henrik karlsson in ue5-main branch]
2024-03-08 18:31:48 -05:00
henrik karlsson
d394b4b515 [UBA]
* Added session notification message to be able to see why client is about to go away.. hopefully this will help us identify if there are crashes happening on remotes that we don't track
* Added ApplicationRules::AllowStorageProxy and changed so .obj files are never going through the storage proxy on fetch (they are only read by one process so no point feeding them through the proxy
* Fixed the file write-through code and gave the #if 0 a name (#if UBA_USE_WRITE_THROUGH).. still disabled since it didn't give any benefits enabling it

[CL 32025069 by henrik karlsson in ue5-main branch]
2024-03-05 12:41:52 -05:00