42 Commits

Author SHA1 Message Date
henrik karlsson
a146ebe918 [UBA]
* Added support for including log lines in cache entries. Design is based on that the log lines are the same most of the time and we store a shared log line per cache key that is part of the response when requesting cache entries for a key. If entries have different log lines inside the same key we will receive those as part of the ReportUsedEntry message.
* Fixed linux arm compile error

[CL 35145540 by henrik karlsson in ue5-main branch]
2024-07-29 13:48:36 -04:00
henrik karlsson
dae1ab42db [Uba]
* Changed Config GetTable to return a pointer that can be null if table does not exist

[CL 34393645 by henrik karlsson in ue5-main branch]
2024-06-15 01:11:17 -04:00
henrik karlsson
ac7fa08ab7 [UBA]
* CacheClient - Added error information when failing to connect to server
* Changed description -> GetDescription()

[CL 34390338 by henrik karlsson in ue5-main branch]
2024-06-14 20:06:57 -04:00
henrik karlsson
87e57a366d [UBA]
* Removed lots of debug code now when bug has been found

[CL 34189212 by henrik karlsson in ue5-main branch]
2024-06-07 00:03:54 -04:00
henrik karlsson
f235d46593 [UBA]
* Fixed so cacheclient useRoots set to true does not affect bucket id

[CL 34172207 by henrik karlsson in ue5-main branch]
2024-06-06 16:15:14 -04:00
henrik karlsson
cccb57fb9f [UBA]
* Changed so casdb entry is not invalidated when file is stored from helper since we already have caskey and everything ready
* Added options to disable usage of cache hits in cacheclient.. this is a debugging feature
* Added option to not use path roots in cache client
* Added more debug information to try to figure out bug where cache client sees wrong cas key of file

[CL 34170695 by henrik karlsson in ue5-main branch]
2024-06-06 15:45:03 -04:00
henrik karlsson
000bde8513 [UBA]
* Added very basic solution for loading a config file. There are lots of internal things that we want to be able to config without having to release new binaries all the time
* Added CacheClientCreateInfo support for config file

[CL 34163080 by henrik karlsson in ue5-main branch]
2024-06-06 13:00:09 -04:00
henrik karlsson
023755cc3c [UBA]
* Added more debug logging to writetocache mismatch
* Renamed all "entry" to either casEntry or fileEntry in order to be able to more easily do find-in-files
* Fixed so store entry done message is sent on fail as well (so server can clean up)

[CL 34142236 by henrik karlsson in ue5-main branch]
2024-06-05 17:57:46 -04:00
henrik karlsson
5a263c548a [UBA]
* Added extra validation when writing cache entries
* Disabled directory preparsing when fetching cache entries for now

[CL 34100764 by henrik karlsson in ue5-main branch]
2024-06-04 14:07:46 -04:00
henrik karlsson
b787ab9d90 [UBA]
* Fixed bug in code that prefetches dir content

[CL 34084776 by henrik karlsson in ue5-main branch]
2024-06-03 23:14:56 -04:00
henrik karlsson
152812809b [UBA]
* Removed duplicated code
* Added asserts to make sure no compressed obj files are used to calculate cas key
* Added missing locks around some code that should have locks

[CL 34072568 by henrik karlsson in ue5-main branch]
2024-06-03 14:37:42 -04:00
henrik karlsson
e01c4e0b1c [UBA]
* Added directory preparsing in UbaCacheClient where it parse the entire folder when needing the hash of a file

[CL 34060027 by henrik karlsson in ue5-main branch]
2024-06-03 01:31:27 -04:00
henrik karlsson
fdd2fd52c4 [UBA]
* Renamed system stats to kernel stats
* Changed all stats storing to use bitfield first to say which fields that are non-zero

[CL 34058881 by henrik karlsson in ue5-main branch]
2024-06-02 18:14:50 -04:00
henrik karlsson
9ba008c8b4 [UBA]
* Optimized cache entry matching
* Added StoreCasKey and changed some StoreCasFile to StoreCasKey in CacheClient which does not need to store cas files
* Changed some parts of the code to use string keys instead of strings since the keys are designed to have correct casing

[CL 34027529 by henrik karlsson in ue5-main branch]
2024-05-31 02:18:27 -04:00
henrik karlsson
233c2cddb2 [UBA]
* Added stats tracking for get file attributes to get more information on what is stalling on the farm
* Enabled storage and system stats for cache misses

[CL 34020773 by henrik karlsson in ue5-main branch]
2024-05-30 19:43:04 -04:00
henrik karlsson
9c0a3c3812 [UBA]
* Added rootpaths version to bucket id in order to make sure we can change system root paths and still be backward compatible

[CL 34019465 by henrik karlsson in ue5-main branch]
2024-05-30 19:25:13 -04:00
henrik karlsson
68695416c9 [UBA]
* Added support for explicit id for root paths. This will make it easier to make things backward compatible

[CL 34018903 by henrik karlsson in ue5-main branch]
2024-05-30 19:17:36 -04:00
henrik karlsson
f90752c220 [UBA]
* Added more stats to cache fetch to be able to figure out where time goes
* Changed code to use FileExists instead of GetFileInformation.. since we only need information that can be fetched with FileExists
* Removed usage of sprintf when creating cas filename

[CL 33979482 by henrik karlsson in ue5-main branch]
2024-05-29 13:31:50 -04:00
henrik karlsson
0799091ec5 [UBA]
* Changed so when fetching cache entries we only calculate cas for localized files once

[CL 33955441 by henrik karlsson in ue5-main branch]
2024-05-28 18:03:47 -04:00
henrik karlsson
639a380cda [UBA]
* Fixed static analysis errors

[CL 33948663 by henrik karlsson in ue5-main branch]
2024-05-28 15:07:55 -04:00
henrik karlsson
502a9e44e2 [UBA]
* Added StringView type and replaced a ton of string+stringLen to StringView
* Fixed bug in obj file preloader.
* Fixed so clang targets can used compressed obj files
* Fixed bug in cache client when entry count is zero
* Added magic to compressed obj file header to be able to see if an obj file is compressed or not
* Moved ParseArguments to its own file and added so it can handle both char and wchar

[CL 33933853 by henrik karlsson in ue5-main branch]
2024-05-27 23:22:26 -04:00
henrik karlsson
1a9837227d [UBA]
* Added so fails to write on UbaCacheClient just produce infos and not errors/warnings.. since we don't want a failed cache download to cause failed builds

[CL 33916306 by henrik karlsson in ue5-main branch]
2024-05-25 17:52:40 -04:00
henrik karlsson
5d8c42793d [UBA]
* Changed so UbaCacheServer stores ranges of input offsets for entries in the same key. It is likely we have many matching inputs between entries in the same key
* Added database creation time and store all times in relation to creation time
* Added lastUsedTime to entries to be able to add support for cleaning up entries that have not been used for x time (not implemented yet)

[CL 33916166 by henrik karlsson in ue5-main branch]
2024-05-25 16:58:10 -04:00
henrik karlsson
fc56ce5b68 [UBA]
* Fixed deadlock that could happen when lots of clients are connecting while maintenance is running
* Added UbaCli -cachecommand=<cmd> that can be used to query status of cache service
* Fixed code setting compressed byte on caskey to only use a bit
* Changed so a bit on the caskey can tell if a file is normalized or not
* Added stats for writing to cache to trace file
*

[CL 33880362 by henrik karlsson in ue5-main branch]
2024-05-23 19:16:15 -04:00
henrik karlsson
f7ba9d6726 [UBA]
* Changed normalized file magic size from 8 bytes to 2..
* Fixed bug with wrong offset calculations

[CL 33855669 by henrik karlsson in ue5-main branch]
2024-05-23 03:02:41 -04:00