Commit Graph

1142 Commits

Author SHA1 Message Date
Ben Marsh
7c433940c6 Horde: Prevent content replication when disabled for a stream.
#preflight none

[CL 19387210 by Ben Marsh in ue5-main branch]
2022-03-15 12:02:51 -04:00
Joakim Lindqvist
9c306f22c8 Move per namespace settings into the namespace policy
Introduced a setting for controlling if we do last access tracking for namespaces

This gives us better overview of what settings are enabled per namespace (and easier toggling for legacy vs new behavior)

#preflight none

[CL 19386732 by Joakim Lindqvist in ue5-main branch]
2022-03-15 11:28:44 -04:00
Ben Marsh
de4eaf1457 Horde: Fix infinite loop in scheduling of builds initiated by gates, which whould eventually time out and result in no jobs being started.
- PerforceService was not returning files as part of change details, due to specifying -S argument when querying changelist. This would cause any code/content change limitations to fail.
- Maximum change to query for gated steps was not being updated while scanning for triggers.
- Added additional context to error message when timeouts occur so we can more easily track down similar errors in future.
- Added test to check correct behavior.

#preflight none

[CL 19386580 by Ben Marsh in ue5-main branch]
2022-03-15 11:20:25 -04:00
Josh Engebretson
ee77d50ae5 Horde: Bump config service version so it picks up the new issue promotion template flags
#jira none
#skipci
#preflight none
#rnx
#fyi ben.marsh

[CL 19384715 by Josh Engebretson in ue5-main branch]
2022-03-15 09:52:51 -04:00
Joakim Lindqvist
91c6bea31f Horde Storage - Make polling services start at a different offsets so that they do not all run at exactly the same time (which generates needless clustering of load)
#preflight none

[CL 19384160 by Joakim Lindqvist in ue5-main branch]
2022-03-15 08:07:12 -04:00
Ben Marsh
dca1affebb Horde: Fix tests.
#preflight none

[CL 19381096 by Ben Marsh in ue5-main branch]
2022-03-14 21:07:12 -04:00
Ben Marsh
d9ffceda32 Horde: Missing file.
#preflight none

[CL 19381022 by Ben Marsh in ue5-main branch]
2022-03-14 20:57:27 -04:00
Ben Marsh
a8b95e2727 Horde: Remove upgrade logic for old job documents. This is causing invalid user documents to be created.
#preflight none

[CL 19379672 by Ben Marsh in ue5-main branch]
2022-03-14 18:19:04 -04:00
Josh Engebretson
fff5ab600b Horde: Adding PromoteIssuesByDefault to job templates with test case
#preflight none
#skipci
#jira none
#rnx
#fyi ben.marsh

[CL 19378188 by Josh Engebretson in ue5-main branch]
2022-03-14 16:48:18 -04:00
Ben Marsh
61b5663941 Horde: Add per-stream settings for replicating commit data.
- Remove ref ids from commit objects, so we can keep multiple refs for each depending on configured settings.
- Replication is still disabled by default for streams.

#preflight none

[CL 19377931 by Ben Marsh in ue5-main branch]
2022-03-14 16:22:49 -04:00
Ben Marsh
fd9d9932c8 Horde: Encode settings used to replicate data to Perforce in the ref ids. Settings are now copied into a CommitKey object, which is encoded as compact binary and hashed, and also embedded in its original form in the bundle ref.
#preflight none

[CL 19376293 by Ben Marsh in ue5-main branch]
2022-03-14 14:51:49 -04:00
Ben Marsh
f035ee86ae Horde: Rename ReadObject/WriteObject methods on IStorageClient to ReadBlob/WriteBlob for consistency. Objects and blobs are generally treated as identical now.
#preflight none

[CL 19374912 by Ben Marsh in ue5-main branch]
2022-03-14 13:37:17 -04:00
Ben Marsh
ab05edcdd0 Horde: Add stats for writing out new bundles to storage.
#preflight none

[CL 19373739 by Ben Marsh in ue5-main branch]
2022-03-14 12:14:47 -04:00
Joakim Lindqvist
b104ab36df Horde Storage - Make sure we do not throw exceptions into the timer update code and instead log them so our structured logging sees it.
#preflight none

[CL 19373519 by Joakim Lindqvist in ue5-main branch]
2022-03-14 11:58:06 -04:00
Joakim Lindqvist
a2793ee592 Horde Storage - Made sure we do not commit content to the blob index until we are sure it was succesfully written to the blob store. Might help fix some inconsistency we have seen between the state of S3 and the state of the blob index.
Also made sure that when the blob store consistency check runs, it does in fact update the blob index when it deletes a bad blob and not just removes it from the store (this check hasnt really found any issues for a while so this is likely not related but still good to fix).
Lastly added some error output when a blob that we expect to find (as we have done a exists check for it) is in fact not present when using the inline type.

#preflight https://horde.devtools.epicgames.com/job/622f5fba0a342dea04ee8686

[CL 19373481 by Joakim Lindqvist in ue5-main branch]
2022-03-14 11:55:00 -04:00
Ben Marsh
c7aebbce2b Horde: Prevent issues automatically being assigned an owner (and triggerring notifications) if there is only one possible suspect. We ALWAYS want to respect the promote-by-default settings for jobs in order to reduce false positives for intermittent failures.
#preflight none

[CL 19373390 by Ben Marsh in ue5-main branch]
2022-03-14 11:44:23 -04:00
Joakim Lindqvist
fddc7c8437 Horde Storage - Added endpoint for uploading content without knowings its hash (its returned back in the response). This hash should still be compared against the content to make sure the correct content was uploaded but allows for verification after upload rather then before.
#preflight none

[CL 19372140 by Joakim Lindqvist in ue5-main branch]
2022-03-14 09:40:50 -04:00
Joakim Lindqvist
382bf970fc Horde Storage - Fixed failing test
#preflight none

[CL 19371940 by Joakim Lindqvist in ue5-main branch]
2022-03-14 09:11:02 -04:00
Joakim Lindqvist
524c7f490b Horde Storage - Added resource names to a bunch of datadog scopes to make it easier to follow what queries are being made.
#preflight none

[CL 19371895 by Joakim Lindqvist in ue5-main branch]
2022-03-14 09:02:13 -04:00
Joakim Lindqvist
2bfb9fec52 Horde Storage - Added ability to ondemand replicate blobs from other regions. This is opt in by specifying both a feature toggle as well as enabling it for the given namespace.
#preflight none

[CL 19371763 by Joakim Lindqvist in ue5-main branch]
2022-03-14 08:42:45 -04:00
Joakim Lindqvist
019564671f Horde Storage - Cleanup GC settings which was a mess, they were inverted both in how they were configured and what they did (so end result was correct luckily).
Renamed CleanNamespacesV1 setting to CleanNamespacesLegacy as is used to cleanup the legacy (dynamo based) namespaces.

#preflight none

[CL 19371425 by Joakim Lindqvist in ue5-main branch]
2022-03-14 07:03:17 -04:00
Joakim Lindqvist
5c9441ecb2 Horde Storage - If a replicator gets multiple use snapshot responses it can just stop running as it will have reached the latest state.
#preflight none

[CL 19371413 by Joakim Lindqvist in ue5-main branch]
2022-03-14 07:00:33 -04:00
Ben Marsh
180c64b3cf Horde: More bundle fixes.
- Fixed tracking of parent ownership for child nodes when adding new nodes to the tree.
- Fixed incorrect offsets in exports when multiple items are in a single compression packet.
- Document and enforce invariants when switching nodes between different states. Nodes read from refs are now immediately turned into standalone nodes. Removes unnecessary logic for caching/decompressing nodes in the root ref on demand.

#preflight none

[CL 19368974 by Ben Marsh in ue5-main branch]
2022-03-12 20:43:58 -04:00
Joe Kirchoff
bb75809a0e Horde: Add IStorageClient.FindMissingBlobsAsync
#rnx
#rb none
#preflight none

[CL 19358297 by Joe Kirchoff in ue5-main branch]
2022-03-11 13:32:51 -05:00
Josh Engebretson
0b9316a86c Horde: Add generic tool exited event messages to systemic matcher
#skipci
#preflight none
#rnx
#jira none

[CL 19357716 by Josh Engebretson in ue5-main branch]
2022-03-11 13:07:59 -05:00