Commit Graph

58 Commits

Author SHA1 Message Date
henrik karlsson
1bd7940eef [UBA]
* Fixed so bad file path entries are not saved to casdb file
* Added so allow proxy can be set from config files
* Added assert for if bad files have entered the cas entry directory on disk

[CL 34476627 by henrik karlsson in ue5-main branch]
2024-06-18 16:23:41 -04:00
henrik karlsson
dae1ab42db [Uba]
* Changed Config GetTable to return a pointer that can be null if table does not exist

[CL 34393645 by henrik karlsson in ue5-main branch]
2024-06-15 01:11:17 -04:00
henrik karlsson
7647b785f2 [UBA]
* Added support for strings and comments in config files
* Added tests
* Fixed for uba to work integrated in other programs (proper dll exports)
* Added process id to process info in visualizer

[CL 34364596 by henrik karlsson in ue5-main branch]
2024-06-14 02:35:27 -04:00
henrik karlsson
87e57a366d [UBA]
* Removed lots of debug code now when bug has been found

[CL 34189212 by henrik karlsson in ue5-main branch]
2024-06-07 00:03:54 -04:00
henrik karlsson
c50a3b17e1 [UBA]
* Fixed bug where renamed file reported wrong stringkey for name to session from detoured process
* Fixed so compressed obj files wrote the right size on disk to file lookuptable

[CL 34184061 by henrik karlsson in ue5-main branch]
2024-06-06 20:30:32 -04:00
henrik karlsson
cccb57fb9f [UBA]
* Changed so casdb entry is not invalidated when file is stored from helper since we already have caskey and everything ready
* Added options to disable usage of cache hits in cacheclient.. this is a debugging feature
* Added option to not use path roots in cache client
* Added more debug information to try to figure out bug where cache client sees wrong cas key of file

[CL 34170695 by henrik karlsson in ue5-main branch]
2024-06-06 15:45:03 -04:00
henrik karlsson
bc5f780f83 [UBA]
* Reverted back to use OpenFile/GetFileInformation instead of GetFileAttributes because weirdly enough it seems to be much faster on cloud setups like AWS.. and I don't know why. You would think 1 syscall is faster than three (open,getinfo.close). GetFileInformation even contains more info than GetFileAttributes :-)

[CL 34144760 by henrik karlsson in ue5-main branch]
2024-06-05 19:57:55 -04:00
henrik karlsson
023755cc3c [UBA]
* Added more debug logging to writetocache mismatch
* Renamed all "entry" to either casEntry or fileEntry in order to be able to more easily do find-in-files
* Fixed so store entry done message is sent on fail as well (so server can clean up)

[CL 34142236 by henrik karlsson in ue5-main branch]
2024-06-05 17:57:46 -04:00
henrik karlsson
791502389d [UBA]
* Attempt at fixing problem with cache client seeing the wrong file when writing to cache. Problem is related to a race condition between directory crawler threads and writing files. We can end up in this scenario
1. Crawler query directory, file information gets stored in temporary state. and crawler starts traversing state
2. File is written and new date is set to file on disk, and file db entry is set to verified false.
3. Crawler hits the same file and now checks temporary state with file db entry and notice the lastwritten is the same, and set it back to verified. (but it is not the same)

[CL 34132081 by henrik karlsson in ue5-main branch]
2024-06-05 13:52:28 -04:00
henrik karlsson
5a263c548a [UBA]
* Added extra validation when writing cache entries
* Disabled directory preparsing when fetching cache entries for now

[CL 34100764 by henrik karlsson in ue5-main branch]
2024-06-04 14:07:46 -04:00
henrik karlsson
152812809b [UBA]
* Removed duplicated code
* Added asserts to make sure no compressed obj files are used to calculate cas key
* Added missing locks around some code that should have locks

[CL 34072568 by henrik karlsson in ue5-main branch]
2024-06-03 14:37:42 -04:00
henrik karlsson
e01c4e0b1c [UBA]
* Added directory preparsing in UbaCacheClient where it parse the entire folder when needing the hash of a file

[CL 34060027 by henrik karlsson in ue5-main branch]
2024-06-03 01:31:27 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
9ba008c8b4 [UBA]
* Optimized cache entry matching
* Added StoreCasKey and changed some StoreCasFile to StoreCasKey in CacheClient which does not need to store cas files
* Changed some parts of the code to use string keys instead of strings since the keys are designed to have correct casing

[CL 34027529 by henrik karlsson in ue5-main branch]
2024-05-31 02:18:27 -04:00
henrik karlsson
a567e860a1 [UBA]
* Fixed bug in previous submit.. FileExists would return true for directories.. while OpenForRead would return false for directories

[CL 33986027 by henrik karlsson in ue5-main branch]
2024-05-29 16:35:45 -04:00
henrik karlsson
f90752c220 [UBA]
* Added more stats to cache fetch to be able to figure out where time goes
* Changed code to use FileExists instead of GetFileInformation.. since we only need information that can be fetched with FileExists
* Removed usage of sprintf when creating cas filename

[CL 33979482 by henrik karlsson in ue5-main branch]
2024-05-29 13:31:50 -04:00
henrik karlsson
1273801622 [UBA]
* Added -expiration=<seconds> to UbaCacheService.. This controls when cache entries should be deleted. If an entry has been unused for more than <expiration> time it gets deleted. Defaults to two days.
* Added reason why cache entries are deleted
* Changed so cas database is always checked in startup.

[CL 33963630 by henrik karlsson in ue5-main branch]
2024-05-29 01:07:06 -04:00
henrik karlsson
299e76b7f4 [UBA]
* Fixed write path when store compressed obj files are enabled

[CL 33934060 by henrik karlsson in ue5-main branch]
2024-05-28 00:03:55 -04:00
henrik karlsson
502a9e44e2 [UBA]
* Added StringView type and replaced a ton of string+stringLen to StringView
* Fixed bug in obj file preloader.
* Fixed so clang targets can used compressed obj files
* Fixed bug in cache client when entry count is zero
* Added magic to compressed obj file header to be able to see if an obj file is compressed or not
* Moved ParseArguments to its own file and added so it can handle both char and wchar

[CL 33933853 by henrik karlsson in ue5-main branch]
2024-05-27 23:22:26 -04:00
henrik karlsson
89fe909dcb [UBA]
* Added more times to steps to be able to track what is taking time

[CL 33919937 by henrik karlsson in ue5-main branch]
2024-05-26 23:28:21 -04:00
henrik karlsson
943b1d84d6 [Uba]
* WorkManager - Fixed bug in ParallelFor. deleted event could be set
* CacheService - More optimizations around maintenance

[CL 33918909 by henrik karlsson in ue5-main branch]
2024-05-26 18:07:50 -04:00
henrik karlsson
898ef9c01e [UBA]
* Fixed bug in storage proxy where files at very specific sizes could end up missing data so cas on the client side could have zeroed out bytes at the end of the cas file

[CL 33671306 by henrik karlsson in ue5-main branch]
2024-05-15 17:20:14 -04:00
henrik karlsson
1cd492036b [UBA]
* Added UbaCli.exe -checkfiletable that can be used to check so files on disk match what uba thinks they are

[CL 33440260 by henrik karlsson in ue5-main branch]
2024-05-03 18:25:21 -04:00
henrik karlsson
b58d8af568 [UBA]
* Reverted CasTableVersion bump to prevent helpers shared between branches from jumping back and forth

[CL 33391440 by henrik karlsson in ue5-main branch]
2024-05-02 02:11:28 -04:00
henrik karlsson
b7e3cef7bf [UBA]
* Added retry code for when failing to decompress cas file to memory on client.. it could be that cas file is corrupt so we delete the file and re-download it.
* Made CheckCasFiles parallel
* Fixed so CheckCasFiles check all files that was created after .isRunning was created to see that their content look valid
* Added time to log when correcting/validation cas store

[CL 33390412 by henrik karlsson in ue5-main branch]
2024-05-02 00:46:58 -04:00