Commit Graph

64 Commits

Author SHA1 Message Date
henrik karlsson
7b39171a86 [UBA]
* Fixed so casentries collected through populating from disk have their verified and exists set to true
* Changed so copyorlink the cas data to temporary folders for execution are never linked on macos.. for some reason this does not always work which is strange

[CL 35908391 by henrik karlsson in ue5-main branch]
2024-08-29 18:05:36 -04:00
henrik karlsson
907121dc6c [UBA]
* Fixed so PopulateCasFromDirs now use file lookup table to prevent cas calculation if file has been calculated before

[CL 35730628 by henrik karlsson in ue5-main branch]
2024-08-21 23:00:47 -04:00
henrik karlsson
5f4f27dd8e [UBA]
* Fixed printf bug in code path handling corrupt cas files...

[CL 35443632 by henrik karlsson in ue5-main branch]
2024-08-10 01:41:38 -04:00
henrik karlsson
a146ebe918 [UBA]
* Added support for including log lines in cache entries. Design is based on that the log lines are the same most of the time and we store a shared log line per cache key that is part of the response when requesting cache entries for a key. If entries have different log lines inside the same key we will receive those as part of the ReportUsedEntry message.
* Fixed linux arm compile error

[CL 35145540 by henrik karlsson in ue5-main branch]
2024-07-29 13:48:36 -04:00
henrik karlsson
fd7243a7c8 [UBA]
* More info added to some error message related to decompressing memory to memory

[CL 35058731 by henrik karlsson in ue5-main branch]
2024-07-24 13:17:08 -04:00
henrik karlsson
72647ce0ce [UBA]
* Fixed crash in code handling corrupt cas files

[CL 35000523 by henrik karlsson in ue5-main branch]
2024-07-22 17:11:49 -04:00
henrik karlsson
1bd7940eef [UBA]
* Fixed so bad file path entries are not saved to casdb file
* Added so allow proxy can be set from config files
* Added assert for if bad files have entered the cas entry directory on disk

[CL 34476627 by henrik karlsson in ue5-main branch]
2024-06-18 16:23:41 -04:00
henrik karlsson
dae1ab42db [Uba]
* Changed Config GetTable to return a pointer that can be null if table does not exist

[CL 34393645 by henrik karlsson in ue5-main branch]
2024-06-15 01:11:17 -04:00
henrik karlsson
7647b785f2 [UBA]
* Added support for strings and comments in config files
* Added tests
* Fixed for uba to work integrated in other programs (proper dll exports)
* Added process id to process info in visualizer

[CL 34364596 by henrik karlsson in ue5-main branch]
2024-06-14 02:35:27 -04:00
henrik karlsson
87e57a366d [UBA]
* Removed lots of debug code now when bug has been found

[CL 34189212 by henrik karlsson in ue5-main branch]
2024-06-07 00:03:54 -04:00
henrik karlsson
c50a3b17e1 [UBA]
* Fixed bug where renamed file reported wrong stringkey for name to session from detoured process
* Fixed so compressed obj files wrote the right size on disk to file lookuptable

[CL 34184061 by henrik karlsson in ue5-main branch]
2024-06-06 20:30:32 -04:00
henrik karlsson
cccb57fb9f [UBA]
* Changed so casdb entry is not invalidated when file is stored from helper since we already have caskey and everything ready
* Added options to disable usage of cache hits in cacheclient.. this is a debugging feature
* Added option to not use path roots in cache client
* Added more debug information to try to figure out bug where cache client sees wrong cas key of file

[CL 34170695 by henrik karlsson in ue5-main branch]
2024-06-06 15:45:03 -04:00
henrik karlsson
bc5f780f83 [UBA]
* Reverted back to use OpenFile/GetFileInformation instead of GetFileAttributes because weirdly enough it seems to be much faster on cloud setups like AWS.. and I don't know why. You would think 1 syscall is faster than three (open,getinfo.close). GetFileInformation even contains more info than GetFileAttributes :-)

[CL 34144760 by henrik karlsson in ue5-main branch]
2024-06-05 19:57:55 -04:00
henrik karlsson
023755cc3c [UBA]
* Added more debug logging to writetocache mismatch
* Renamed all "entry" to either casEntry or fileEntry in order to be able to more easily do find-in-files
* Fixed so store entry done message is sent on fail as well (so server can clean up)

[CL 34142236 by henrik karlsson in ue5-main branch]
2024-06-05 17:57:46 -04:00
henrik karlsson
791502389d [UBA]
* Attempt at fixing problem with cache client seeing the wrong file when writing to cache. Problem is related to a race condition between directory crawler threads and writing files. We can end up in this scenario
1. Crawler query directory, file information gets stored in temporary state. and crawler starts traversing state
2. File is written and new date is set to file on disk, and file db entry is set to verified false.
3. Crawler hits the same file and now checks temporary state with file db entry and notice the lastwritten is the same, and set it back to verified. (but it is not the same)

[CL 34132081 by henrik karlsson in ue5-main branch]
2024-06-05 13:52:28 -04:00
henrik karlsson
5a263c548a [UBA]
* Added extra validation when writing cache entries
* Disabled directory preparsing when fetching cache entries for now

[CL 34100764 by henrik karlsson in ue5-main branch]
2024-06-04 14:07:46 -04:00
henrik karlsson
152812809b [UBA]
* Removed duplicated code
* Added asserts to make sure no compressed obj files are used to calculate cas key
* Added missing locks around some code that should have locks

[CL 34072568 by henrik karlsson in ue5-main branch]
2024-06-03 14:37:42 -04:00
henrik karlsson
e01c4e0b1c [UBA]
* Added directory preparsing in UbaCacheClient where it parse the entire folder when needing the hash of a file

[CL 34060027 by henrik karlsson in ue5-main branch]
2024-06-03 01:31:27 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
9ba008c8b4 [UBA]
* Optimized cache entry matching
* Added StoreCasKey and changed some StoreCasFile to StoreCasKey in CacheClient which does not need to store cas files
* Changed some parts of the code to use string keys instead of strings since the keys are designed to have correct casing

[CL 34027529 by henrik karlsson in ue5-main branch]
2024-05-31 02:18:27 -04:00
henrik karlsson
a567e860a1 [UBA]
* Fixed bug in previous submit.. FileExists would return true for directories.. while OpenForRead would return false for directories

[CL 33986027 by henrik karlsson in ue5-main branch]
2024-05-29 16:35:45 -04:00
henrik karlsson
f90752c220 [UBA]
* Added more stats to cache fetch to be able to figure out where time goes
* Changed code to use FileExists instead of GetFileInformation.. since we only need information that can be fetched with FileExists
* Removed usage of sprintf when creating cas filename

[CL 33979482 by henrik karlsson in ue5-main branch]
2024-05-29 13:31:50 -04:00
henrik karlsson
1273801622 [UBA]
* Added -expiration=<seconds> to UbaCacheService.. This controls when cache entries should be deleted. If an entry has been unused for more than <expiration> time it gets deleted. Defaults to two days.
* Added reason why cache entries are deleted
* Changed so cas database is always checked in startup.

[CL 33963630 by henrik karlsson in ue5-main branch]
2024-05-29 01:07:06 -04:00
henrik karlsson
299e76b7f4 [UBA]
* Fixed write path when store compressed obj files are enabled

[CL 33934060 by henrik karlsson in ue5-main branch]
2024-05-28 00:03:55 -04:00
henrik karlsson
502a9e44e2 [UBA]
* Added StringView type and replaced a ton of string+stringLen to StringView
* Fixed bug in obj file preloader.
* Fixed so clang targets can used compressed obj files
* Fixed bug in cache client when entry count is zero
* Added magic to compressed obj file header to be able to see if an obj file is compressed or not
* Moved ParseArguments to its own file and added so it can handle both char and wchar

[CL 33933853 by henrik karlsson in ue5-main branch]
2024-05-27 23:22:26 -04:00