fix: name the SRE team by its current name - #224
Conversation
The five dead-letter queue alerts label their owner platform-sre, a name the team stopped using. The datum-cloud organization configuration records platform-sre as a previous name of the team now called sre, so the label matches nothing the alerting standard recognizes and routes to nobody while still reading as owned. Rename the label on all five rules, and the section banner above them so the file does not contradict itself.
|
hurfity durf |
scotwells
left a comment
scotwells
left a comment
There was a problem hiding this comment.
@ecv we should actually reconsider all of these alerts being attached to the SRE team. Some of these alerts (e.g. policy failures) are a result of service developers having misconfigurations in their Activity Policies that they are responsible for triaging and investigating. For example, alerts for the billing activity policy should actually be surfaced to the billing team directly. Maybe we need to consider namespace level "team" overrides or resource level team overrides?
|
Agreed, and the correct team name is still the wrong answer for several of these. A policy failure caused by a service's own misconfiguration belongs to that service's team, not to whoever operates the pipeline. Split it in two, since neither half works alone. #225 here: the alerts have to carry the failing policy and where it lives. A router cannot resolve an owner from a label the alert never had, and today these expressions sum the policy away. datum-cloud/infra#3984: resolving an owner from the alert's subject, with the namespace and resource level overrides you describe. Filed against the alerting standard, because the same defect shows up in the |
2 participants
Five dead-letter queue alerts name an owning team that no longer goes by that name.
They read as owned while matching nothing the alerting standard recognises, so anything they raise routes to nobody.
This renames the label to the team's current name, and the banner above the rules with it, from the ownership audit in datum-cloud/infra#3974
Test plan