* Fixed validation code in scheduler for process reuse
* Moved all code inside ServerSession::HandleMessage into one function per message to more easily be able to read callstacks
* Added more validation around missing file errors.. now server logs out lots of information too
[CL 34414815 by henrik karlsson in ue5-main branch]
* Added directory preparsing in UbaCacheClient where it parse the entire folder when needing the hash of a file
[CL 34060027 by henrik karlsson in ue5-main branch]
* Lots of fixes related to cache and prepared code for integration with UBT (ubt will provide input and output files for cache entries)
[CL 33209814 by henrik karlsson in ue5-main branch]
* Bumped table sizes to prevent oom when building our largest non unity builds
* Added bucket support in cache system
* Added ParallelFor function to WorkManager
* Fixed so annoying logging when pressing ctrl-c is muted
[CL 32947130 by henrik karlsson in ue5-main branch]
* Implemented V1 of CompactPathTable which instead stores string segments separately.. Reduces the size table by ~30%.. slightly slower perf
* Added "RootPaths" which is a class that holds a list of root paths needed to normalize paths into non-environment dependent paths
* Added stats for cache that visualizes properly in visualizer
* Changed hash functions to have better quality
* Cleaned up code visualizing stats in visualizer.. now boxes popping up should have a nice matching size to strings
* Cleaned up UbaExports and formatted all functions the same way.. old are deprecated and will be removed once new binaries are in
[CL 32922089 by henrik karlsson in ue5-main branch]
* Added artifact cache support. It is now possible to skip running process and instead just download outputs if inputs matches cache entry.
[CL 32870344 by henrik karlsson in ue5-main branch]
* Added so process input dependencies can be tracked on remote compiles and sent back to host.
* Changed so ApplicationRules is a pointer on the ProcessStartInfo instead.. this in preparation of being able to have custom application rules but also be able to query rules before process instance is created
* Added support for suppressing error log if FileAccessor open file fails
* Fixed so BytesToText show bytes when under 1kb
* Fixed potential shutdown race related to process exiting and session instance being destroyed
* Some minor fixes here and there related to logging
* Changed process IsDetoured to GetExecutionType enum
[CL 32777954 by henrik karlsson in ue5-main branch]
* Huge stability push... all (known) paths have been tested heavily on linux with tsan and every single found race condition report has been fixed. Lots of locks have been added/moved/changed and some instances of things have been leaked on purpose to prevent tsan reports during shutdown
* More efficient storage proxy implementation which immediately forward segments to clients once they are available in proxy
* Added UbaAgent -command=x which can be used to send commands to host. Supported commands are "status"which prints out status of all remote sessions. "abort/abortproxy/abortnonproxy" that can be used to abort remote sessions and "disableremote" to have host stop accepting more helpers
* Fixed scheduler::stop bug if remotes were still requesting processes
* Added support for process reuse on linux/macos
* Added support for Coordinator interface and dynamically load coordinator dll in UbaCli
* Restructured code a little bit to be able to queue up all writes in parallel
* Added Show create/write colors to visualizer (defaults to on)
* Fixed so write file times are visualized in visualizer
* Improved socket path for visualizer
* Improved a lot of error messages
* Fixed double close of memory handle in StorageServer
* Changed some ScopedWriteLock to SCOPED_WRITE_LOCK (same for read locks)
* Fixed some missing cleanup of trace view when starting a new trace view in visualizer
[CL 32137083 by henrik karlsson in ue5-main branch]
* Added PrefetchVirtualMemory function
* Fixed bug where code writing callstacks checks end of module to figure out if address belongs to module
* Changed so FileAccessor::CreateMemoryWrite set always allow read because you can't create file mappings that are write-only
* Added option to scheduler to force all runs to run native
* Fixed so bottleneck class has enter/leave function
* Added ScopedReadLock::Enter
* Moved LogWriter to its own file
* Visualizer - Fixed so when hovering over timeline time under pointer is showing
* Visualizer - Fixed so hovering over work records shows name, start and duration
* Visualizer - Added context menu item that can show/hide text of process bars
* Improved assert message
* Changed so lock is not on stack but is leaked to prevent shutdown warnings in debug
* Added missing includes
[CL 31981637 by henrik karlsson in ue5-main branch]
* Changed so all ScopedReadLock and ScopedWriteLock are macros (SCOPED_READ_LOCK and SCOPED_WRITE_LOCK).. this to be able to add code around the lock to measure contention. Contention testing code can be enabled by setting UBA_TRACK_CONTENTION to 1
[CL 31795889 by henrik karlsson in ue5-main branch]
* Changed so ClientSession struct is allocated using aligned_alloc instead of new.. we can't figure out why this is failing with new on the farm and noone can repro so this is a very nasty workaround until we do understand how it can go wrong.
* Fixed potential race condition in scheduler
* Reduced lock scope in StorageClient to remove chance of deadlock
* Enabled mimalloc on linux
[CL 31660233 by henrik karlsson in ue5-main branch]
* Added support for most features in UbaScheduler. Dependencies, weights etc etc.
* Added support for loading yaml file with processes for UbaScheduler.
* Added so UbaCli interprets yaml file as a file with processes use it to populate a scheduler which is then executed
* Fixed so all threads inside same process spawned by uba ends up in the same thread group.
* Fixed linux crash where process comunication memory was deleted when cancel event was called (added lock around code)
* Fixed deadlock that could happen if flush dead processes were called at the after lock but before processhandle dtor in Session::ProcessExited
* Changed new[]/delete[] to aligned_alloc/free because for some reason new/delete trigger asan on linux and don't know why.
[CL 31372220 by henrik karlsson in ue5-main branch]
* Fixed so imagehlp.dll and dbghelp.dll are detoured when loaded. Detour ImageGetDigestStream and SymLoadModuleExW because both of them cause trouble on wine
* Fixed so reuse of processes also honor log files so a new log file is created when reuse happen
* Improved log file naming so host can set name and log files are sent back properly with the right name
[CL 30638193 by henrik karlsson in ue5-main branch]
* Changed Scheduler_GetStats to separate active into activeLocal and activeRemote
* Added Storage_DeleteFile that can be used to delete cas entries if we know a file will never be used again (no point taking up space)
* Fixed so ubaagents spawned through horde has a idle timeout of 15 seconds (also made them quiet)
* Changed so UbaJobProcessor shutdown uba after 100 seconds instead of 10 seconds of idling without tasks
* Added so .uba.in and .uba.out is deleted from cas storage after used.. no point storing them there since they will never be used again
* Fixed so number of local SCW instances are decreased when more remote helpers are running
* Fixed race condition where tasks could be added when uba was shutting down causing tasks to be cancelled
[CL 30607957 by henrik karlsson in ue5-main branch]
* Added GetNextProcess as a real message type (not using Custom) to be able to stop fetching work if helper is being terminated by aws or if we want to scale down number of workers
* Did some cleanup in the Tcp backend code and made sure not to call close socket multiple times (since it can cause the code to close a different socket than what is owned)
[CL 30584707 by henrik karlsson in ue5-main branch]
* Fixed so scheduler can take in 0 as max local processors and use 0. Default is now ~0u which becomes max logical processors
[CL 30519168 by henrik karlsson in ue5-main branch]
* Added RegisterDeleteFile to external api that should be used when file is deleted outside of uba but uba needs to know about it
* Reduced lock scope around FlushDeadProcesses
* Fixed disconnect issue crash in ubaagent that could cause access violation in UbaStorage. Fix was to cleanup when leaving StorageClient::SendAllSegments
* Changed so UbaRequestNextProcess is doing a full environment update (not resetting stats) if no next process is found
* Fixed so Scheduler enableProcessReuse=false works properly
* Improved some assert descriptions
* Fixed so files in m_outputFiles in session client is erased when flushed to server
* Fixed so Rpc_GetFullName cleans up .. in paths
* Fixed so files that existed but has been deleted by external process is seen as not existing by remote detoured process.
[CL 30507663 by henrik karlsson in ue5-main branch]
* Fixed so paths returned from SearchPathForFile are cleaned up (not having .. etc)
* Fixed so UbaAgent handle WSAPoll POLLERR as timeout so it retries when failing (this made the process exit on wine)
* Fixed so returned processes that had been reused returned the reused process to the scheduler
* Fixed so EnsureBinaryFile is using correct fileNameKeys
* Removed assert in detoured GetFullPathNameA since it is always calling GetFullPathNameW
[CL 30465533 by henrik karlsson in ue5-main branch]
* Implemented process reuse logic in UbaScheduler. It is now possible for running processes to fetch more work
* Fixed bugs in custom message path
* Fixed bugs in FlushWrittenFiles message
* Changed So UpdateEnvironment is resetting stats
* Fixed potential race condition related to process reuse
* Added special rule for ShaderCompileWorker.exe which detours ImageGetDigestStream in Imagehlp.dll (because wine implementation does not match windows implementation)
* Renamed "exitedUserData" to just userData since it is used for more than when exiting process
* Added a couple more dlls to "known system dlls"
* Fixed so visualizer can visualize process reuse properly
[CL 30462059 by henrik karlsson in ue5-main branch]
