diff --git a/CLAUDE.md b/CLAUDE.md index 65a0e981..af090430 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,6 +6,41 @@ Guidance for Claude Code when working in this repository (KloudMate documentatio [AGENTS.md](AGENTS.md) is the authoritative guide for this repo: project context, the writing style and voice rules (Section 2, read it before writing any content), the tech stack (Astro + Starlight), and the directory and routing structure. Follow it. The rules below are additions specific to the KloudMate agent docs (`src/content/docs/docs/kloudmate-agent/**`), not replacements. +## Writing style: kill these AI tells (all docs, self-check before you finish) + +AGENTS.md §2 is the full voice guide — read it before writing. This section is the focused checklist of tells that keep slipping into drafts and getting flagged. Self-review every page against it and fix the hits **before** handing it over. The user should not have to point these out again. + +Write the way Stripe, Datadog, and Google developer docs actually read: lead with the task or the outcome, keep each section to a few tight sentences, stay in second person with imperative steps, and use concrete verbs with real values. For a concrete model, [Datadog's Step Functions overview](https://docs.datadoghq.com/serverless/step_functions/) opens by stating what the service is and what the product adds, uses outcome-led headings ("Monitor the overall health…", "Reduce Step Function debugging time…"), keeps every section to 2–3 sentences, names concrete outcomes ("identify what states are problematic or have a high latency"), and never announces a count or calls a screen "a summary of activity." + +### Structural tells — state the fact, don't announce it + +The pattern the user flags most: **announcing a quantity + a vague nominalization + a balanced "…and they…" clause**, instead of just stating the fact. State the concrete thing; drop the count when the list right below already shows it; use an active verb instead of "is a … of." + +| Don't write | Why it reads as AI | Write instead | +|---|---|---| +| "Two levels of data are available, and they need different amounts of setup:" | count + vague nominal + balanced "and they" clause | "Some of this works the moment you connect an account. The rest needs execution logging:" | +| "The tab has three parts:" · "they need two things:" · "Check two things:" | announcing a count the list below already shows | "The tab shows:" · "Check both of these:" | +| "The Overview tab is a workspace-wide summary of execution activity." | copula + nominalization ("is a summary of activity") | "The Overview tab summarizes the last 24 hours of executions." | +| "Both workflow types are supported." | flat passive announcement | "KloudMate supports both Standard and Express workflows." | +| "…with three tabs: Executions, Metrics, Configuration." | needless count | "…with tabs for Executions, Metrics, and Configuration." | + +### Tone (AGENTS.md §2 has the full word lists; these are the repeat offenders) + +- **Em dashes for asides** → period, comma, colon, or parentheses. +- **No drama or voiceover:** rhetorical build-ups ("X is the answer"), trailing tags ("…, which it is"), figurative labels ("delivery vehicle"), cutesy asides ("the same treatment for your Lambda functions"). Write the plain fact. +- **No reflexive benefit-tails:** don't glue "…, so you can [vague upside]" onto every sentence. Keep it only when the cause and effect is real and specific. +- **No hype or filler:** powerful, seamless, robust, effortless; leverage/utilize → use, via → with, "in order to" → to, "simply/just/easily". + +### Do this + +- Open with what the reader accomplishes, not background or history. +- Verb-led headings for task sections ("Turn on execution logging"); noun headings that name a screen are fine for reference sections ("The State machines table"). +- Vary sentence length and openers. Don't start three sentences in a row the same way, especially with the product name. +- Be concrete: real labels and values (`level ALL`, `/aws/vendedlogs/states/`), a real error string. +- Close a section with the next step (a link to setup, Explore, or Alerts). + +When asked to review for these, rewrite the offending lines — don't just list them. + ## Agent-docs terminology: plain words, no product jargon - Do not use **"baseline."** It means nothing to a reader. Use **"eBPF monitoring"** or **"ETW monitoring"** for the eBPF or ETW layer, and **"automatic monitoring"** (or "what the agent collects automatically") for the general idea. Drop the redundant article too: write "eBPF monitoring needs kernel 4.14," not "the eBPF monitoring needs…". It reads as a mass noun, and the docs already start sentences with a bare "eBPF." diff --git a/astro.config.mjs b/astro.config.mjs index 7e655f26..46ce9f66 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -74,6 +74,9 @@ export default defineConfig({ customCss: ['./src/styles/brand.css'], components: { Header: './src/components/Header.astro', + // Wraps starlight-llm-actions' Page Actions dropdown in + // `data-pagefind-ignore` so its menu labels stay out of the search index. + PageTitle: './src/components/PageTitle.astro', Sidebar: './src/components/Sidebar.astro', }, sidebar: [ diff --git a/src/components/PageTitle.astro b/src/components/PageTitle.astro new file mode 100644 index 00000000..753f2a06 --- /dev/null +++ b/src/components/PageTitle.astro @@ -0,0 +1,58 @@ +--- +/** + * Replaces the `PageTitle` override that starlight-llm-actions injects, so the + * Page Actions dropdown can be wrapped in `data-pagefind-ignore`. + * + * Starlight puts `data-pagefind-body` on `
`, and the dropdown renders + * inside it, directly after the `

`. Without the wrapper, Pagefind indexes + * the menu's labels ("Copy page", "View as Markdown", "Open in Claude", …) as + * the first ~380 characters of every page, so they lead the search excerpts and + * every page matches a search for those words. + * + * The plugin detects a user `PageTitle` override, skips its own injection, and + * logs a build warning pointing at exactly this pattern. Note that the skipped + * override is also where the plugin renders its optional `printNotice` banner; + * that option isn't configured here, so nothing is lost, but adding it later + * means porting the banner markup into this file. + * + * The `

` stays outside the ignored subtree on purpose: Starlight sets no + * `data-pagefind-meta="title"`, so Pagefind takes each search result's title + * from the first indexed `

`. + */ +import Default from '@astrojs/starlight/components/PageTitle.astro'; +import PageActions from 'starlight-llm-actions/components/PageActions.astro'; +--- + +
+ +
+ +
+
+ + diff --git a/src/content/docs/docs/alerts/alert-groups.mdx b/src/content/docs/docs/alerts/alert-groups.mdx index 00903b97..c2e1efad 100644 --- a/src/content/docs/docs/alerts/alert-groups.mdx +++ b/src/content/docs/docs/alerts/alert-groups.mdx @@ -1,13 +1,22 @@ --- title: "Alert Groups" -description: "Bundle related signals into a single notifiable group with firing instances, an audit log, and dispatched notifications all in one place." +description: "Group related alerts into a single notifiable incident, with firing instances, an audit log, and dispatched notifications all in one place." sidebar: order: 6 --- -When matching alerts fire close together, KloudMate bundles them into a single **Alert Group**. Instead of paging on every individual signal, you page once on the group; new signals append to the group rather than fan out into separate notifications. +When related alerts fire close together, KloudMate groups them into a single **Alert Group**. Instead of notifying on every individual alert, you get one notification for the group, and new alerts append to it rather than fan out into separate notifications. -A group survives restarts and deduplicates against the same incident over time, so the same recurring issue keeps showing up on the same group rather than spawning a fresh one each time. +A group survives restarts and deduplicates against the same incident over time, so a recurring issue keeps showing up on the same group rather than spawning a fresh one each time. + +## How groups form + +An Alert Group forms by static grouping or by the AI correlation engine. Both produce the same kind of group: + +- **Static grouping** folds alerts together by the deterministic group-by keys on a routing rule. Alerts that share those label values land in one group. +- **Auto (AI) grouping** hands the decision to the correlation engine, which links related alerts on its own and learns which alerts tend to fire together over time. + +You choose the mode per routing rule; see [Routing Rules](../routing-rules/#alert-grouping-static-or-auto-ai). For how membership is decided in each mode, and why a single rule firing on many hosts stays one Auto group, see [How Alert Grouping Works](../how-alert-grouping-works/). However a group forms, it behaves the same everywhere else on this page. An Auto-correlated group also records [why its alerts were grouped](#why-these-were-grouped). ## Where to find groups @@ -15,31 +24,32 @@ Open **Alerts → Alert groups** in the left navigation. This is the workspace-w ![Alert Groups list](./images/alert-groups-list.png) -Columns: **Group** (title + label chips), **State** (Open / Resolved), **Signals**, **Opened**, **Last signal**, and an investigation indicator when [Auto-RCA](../auto-rca/) ran. +Columns: **Group** (title), **State** (Open / Resolved), **Instances** (the firing-instance count), **Opened**, **Last update**, and an RCA indicator when an investigation is attached (a manual **Run RCA** or [Auto-RCA](../auto-rca/)). -Expand any row to preview that group's signals inline — alert name, state, severity, received time, and labels — without leaving the list. The group title links straight to the full detail page. +Expand any row to preview that group's alerts inline: alert name, state, severity, received time, and labels, without leaving the list. The group title links straight to the full detail page. Filters across the top: -- **State** — All / Open / Resolved toggle. Defaults to **Open**. -- **Routing rule** — dropdown filtered to rules in the workspace. -- **Label matchers** — chip input where you type `key=value`. Press Enter to add a chip. +- **State:** All / Open / Resolved toggle. Defaults to **Open**. +- **Routing rule:** dropdown filtered to rules in the workspace. +- **Label matchers:** chip input where you type `key=value`. Press Enter to add a chip. -Severity isn't a filter on the list — it lives on the per-instance signals inside a group, not on the group itself. +Severity isn't a filter on the list. It lives on the per-instance alerts inside a group, not on the group itself. ## Anatomy of a group A group carries: -- **Title** — set by the first signal that opened the group. **Immutable** after that, even if a higher-severity signal joins later. A lock icon next to the title indicates this; hovering it reads *"Set by first signal — cannot be changed."* -- **State** — `Open` while at least one underlying instance is firing; `Resolved` once everything quiets. A firing group whose firing instances are **all silenced** stays `Open` but carries a **Muted** badge (*"Firing, but all instances are silenced — notifications muted"*) — muting gates notifications, it doesn't resolve the group. -- **Labels** — the keys you grouped by on the routing rule, derived from the signals that joined the group. -- **Signal count** — total signals attached to the group so far. -- **Routing rule** — the rule that opened the group. -- **Group ID** — the durable identifier you can share with teammates or paste into the assistant. -- **Attached investigation** — present when [Auto-RCA](../auto-rca/) ran on the group. +- **Title:** set by the first alert that opened the group, and **immutable** afterward, even if a higher-severity alert joins later. A lock icon next to the title marks it. +- **State:** `Open` while at least one underlying instance is firing; `Resolved` once everything quiets. A firing group whose instances are **all silenced** stays `Open` but carries a **Muted** badge. Muting gates notifications; it doesn't resolve the group. +- **Labels:** the labels that define the group, read from the alerts that joined it (the group-by keys, in Static mode). +- **Alerts and instances:** how many distinct alerts and firing instances the group holds so far, shown as *"N alerts · N firing instances"*. +- **Routing rule:** the rule that opened the group. +- **Group ID:** the durable identifier you can share with teammates or paste into the assistant. +- **Correlation reason:** for an Auto-correlated group, why these alerts were grouped. See [Why these were grouped](#why-these-were-grouped). +- **Investigation:** a root-cause investigation, when one is attached. Start it manually with **Run RCA**, or automatically with [Auto-RCA](../auto-rca/); see [Run RCA and View RCA](#run-rca-and-view-rca). -Severity isn't a single group-level field — each underlying instance carries its own severity, and you read them on the **Firing instances** panel and the **Signals** tab. +Severity isn't a single group-level field. Each underlying instance carries its own severity, which you read on the **Firing instances** panel and the **Alerts** tab. ## Group detail @@ -49,59 +59,75 @@ Click any row to open the group detail page. ### Header -- **Title** with a lock icon (hover: *"Set by first signal — cannot be changed"*). -- **Ask KloudMate Assistant** — opens the assistant chat panel with the group's labels, state, signal count, and routing rule pre-loaded into the prompt, so you can jump into investigation without retyping context. -- **Silence this group** — opens the silence creator pre-filled with the group's labels as matchers and bound to the group via `auto_expire_group_id`. +- **Title** with a lock icon: it's set by the first alert and can't be changed. +- **Run RCA / View RCA:** starts or opens a root-cause investigation for the group. See [Run RCA and View RCA](#run-rca-and-view-rca). +- **Ask KloudMate Assistant:** opens the assistant chat panel with the group's labels, state, alert and instance counts, and routing rule pre-loaded into the prompt, so you can start investigating without retyping context. +- **Silence this group:** opens the silence creator pre-filled with the group's labels as matchers and bound to the group via `auto_expire_group_id`. -### Meta strip +### Firing instances -Below the title: state chip, signal count (`N signals in this group`), Opened relative time, Resolved relative time (when applicable), and Last signal relative time. +The **Firing instances** panel is the largest section on the page. It's a table of the unique alert instances that joined this group, deduplicated by the per-instance labels the grouping engine uses. -### Labels, routing rule, and Group ID +- **Common labels bar** at the top: labels shared by every instance, so the per-row labels column only shows what varies. +- **Columns:** + - **State:** the instance's current state: **Firing**, **No Data**, **Error**, or **Resolved**. Silenced instances are flagged as muted, so you can see at a glance which ones are firing but suppressed. + - **Alert:** the alert rule name. Links to the rule when the alert is KloudMate-native (carries an `alarm_id` label). + - **Instance labels:** the labels that distinguish this instance from its siblings (common labels are stripped out). + - **Severity:** the per-instance severity, if the alert carries one. + - **Since:** how long this instance has been in its current state. -A bordered card directly under the meta strip shows: +Use this panel to see what's firing inside the group at a glance. For example, service A's `prod` and `staging` are firing while `dev` has already recovered. -- **Labels** — the group's label chips. Per-instance label keys (`alarm_id`, `instance_key`, etc.) are filtered out here so you see the labels that actually defined the group. -- **Routing rule** — a link to the rule that opened the group. -- **Group ID** — monospace, for sharing or pasting into tooling. +### Run RCA and View RCA -### Firing instances +The group header carries a single root-cause button, and its label reflects whether an investigation is attached yet: + +- **Run RCA:** shown when no investigation is attached. It starts a root-cause investigation for the group on demand. +- **View RCA:** shown once an investigation is attached. It opens a side drawer with the root-cause summary and a **View full investigation** link. -The **Firing instances** panel is the dominant body section. It's a table of the unique alert instances that joined this group, deduplicated by the per-instance labels the grouping engine uses. +This is the manual trigger. To investigate every group a rule opens automatically instead, enable [Auto-RCA](../auto-rca/) on the routing rule; an Auto-RCA result surfaces through the same **View RCA** drawer. -- **Common labels bar** at the top — labels shared by every instance, so the per-row labels column only shows what varies. -- **Columns**: - - **State** — the instance's current state: **Firing**, **No Data**, **Error**, or **Resolved**. Silenced instances are flagged as muted, so you can see at a glance which ones are firing but suppressed. - - **Alarm** — the alert rule name. Links to the rule when the signal is KloudMate-native (carries an `alarm_id` label). - - **Instance labels** — the labels that distinguish this instance from siblings (common labels are stripped out). - - **Severity** — the per-instance severity, if the signal carries one. - - **Since** — how long this instance has been in its current state. -- A summary above the table reports the alert and instance counts with a per-state breakdown, e.g. *"2 alerts · 7 instances · 4 firing, 3 resolved"*. +![View RCA drawer with the root-cause summary and a View full investigation link](./images/alert-group-rca-drawer.png) -Use this panel to see what's firing inside the group at a glance — for example, service A's `prod` and `staging` are firing while `dev` has already recovered. +### Tabs -### Attached investigation +- **Alerts:** every underlying alert as a row. Columns: Alert (linked to the rule when available), State, Severity, Received at, Labels. Use this when you need the raw event-by-event stream rather than the instance-level rollup. +- **Blast radius:** what the group's firing resources connect to. See [Blast radius](#blast-radius). +- **Audit:** a vertical timeline of everything that happened to the group: opened, alerts appended, silence applied, notification dispatched, resolved. Auto-correlated groups also log correlation events here: **"Group opened by AI"** and **"Alert correlated into group"**. +- **Notifications:** per-channel dispatch outcomes (ok / failed / suppressed-by-silence), with deep links to where the notification landed: Slack thread URL, Jira ticket, KloudMate Incidents incident, and so on. -When the routing rule had Auto-RCA enabled, an info-styled card sits between the Firing instances panel and the tabs: +### Blast radius -- Investigation title and status chip (`Completed` / `In progress`). -- The root-cause analysis text. -- A **View full investigation** link. +The **Blast radius** tab resolves the group's firing alerts to the resources they name, then graphs what each one depends on, so you can see how far the incident reaches. -See [Auto-RCA](../auto-rca/) for how this gets created. +Each affected resource sits at the center of its own graph, with neighbors branching off along labeled edges: **runs on**, **talks to**, **calls**. Neighbors of the same kind collapse into one node (*116 Pods · runs on*) that you click to list them. -### Tabs +The same graph appears on a single alert's **Blast radius** tab and on a resource's **Dependencies** tab in Cloud Inventory. An empty graph means KloudMate hasn't mapped a connection yet, not that the lookup failed. When nothing is firing or the alerts don't match a known resource, the tab says so instead of drawing one. + +![The Blast radius tab: a row of affected resource chips above a graph of each resource and what it runs on](./images/alert-group-blast-radius.png) + +### Why these were grouped + +A **Why these were grouped** section sits at the bottom of the detail page, with the reason in plain English, the shared labels, and supporting detail. + +The chip and section only appear when the engine actually correlated more than one alert. A lone alert the engine evaluated and left on its own shows no chip, which is expected (see [Cold start](#cold-start)). + +### Cold start + +A new workspace has little co-occurrence history for the engine to learn from, so it leaves most alerts as separate singletons with no correlation chip. That's expected, not a failure. Correlation strengthens over the following days as the engine sees which alerts tend to fire together. + +### Feedback + +Under the reason, a **Was this grouping correct?** control takes feedback with **Correct** and **Wrong**. -- **Signals** — every underlying firing as a row. Columns: Alert (linked to the rule when available), State, Severity, Received at, Labels. Use this when you need the raw event-by-event stream rather than the instance-level rollup. -- **Audit** — vertical timeline of everything that happened to the group: opened, appended, silence applied, notification dispatched, resolved. -- **Notifications** — per-channel dispatch outcomes (ok / failed / suppressed-by-silence), with deep links to where the notification landed: Slack thread URL, Jira ticket, KloudMate Incidents incident, etc. +Choosing **Wrong** lets you point at the specific alert that doesn't belong, or mark the whole group as wrong. That feedback trains the correlation engine: a **Wrong** verdict makes it less likely to pair those alerts again, and **Correct** reinforces the grouping. ## When a group resolves -KloudMate marks a group `Resolved` once every underlying signal has resolved and a short flap-grace period has elapsed (a few minutes — long enough to absorb signals that flap rapidly without closing the group prematurely). Resolved groups stay in the list under the **Resolved** filter and remain searchable indefinitely. +KloudMate marks a group `Resolved` once every underlying instance has resolved and a short grace period for flapping has elapsed (a few minutes, long enough to absorb instances that flap rapidly without closing the group prematurely). Resolved groups stay in the list under the **Resolved** filter and remain searchable indefinitely. ## Related -- [Routing Rules](../routing-rules/) — define which alerts a group accepts and how it's notified. -- [Silences](../silences/) — suppress notifications for matching labels. -- [Auto-RCA](../auto-rca/) — automatic investigations on group open. +- [Routing Rules](../routing-rules/): define which alerts a group accepts, and whether it groups by keys (Static) or by the correlation engine (Auto). +- [Silences](../silences/): suppress notifications for matching labels. +- [Auto-RCA](../auto-rca/): automatic investigations on group open. diff --git a/src/content/docs/docs/alerts/alert-lifecycle.mdx b/src/content/docs/docs/alerts/alert-lifecycle.mdx index 80b27c67..4ca5baea 100644 --- a/src/content/docs/docs/alerts/alert-lifecycle.mdx +++ b/src/content/docs/docs/alerts/alert-lifecycle.mdx @@ -8,14 +8,14 @@ sidebar: import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -Once you've [created an alert](../create-alerts/), KloudMate evaluates it on a schedule and moves it through a series of states. The control knobs in **Configure evaluation settings** — **Evaluate every**, **Pending duration**, **Recovery period**, **Alert state if No data**, and **Alert state if Error** — each shape *when* and *how* those transitions happen. +Once you've [created an alert](../create-alerts/), KloudMate evaluates it on a schedule and moves it through a series of states. The control knobs in **Configure evaluation settings** (**Evaluate every**, **Pending duration**, **Recovery period**, **Alert state if No data**, and **Alert state if Error**) each shape *when* and *how* those transitions happen. -This page explains the lifecycle so the behavior never surprises you. If you've ever wondered "why did my alert fire instantly?" or "why didn't No Data wait out my pending duration?", the answer is here. +This page explains the full lifecycle: why an alert can fire instantly, and why No Data and Error don't wait out the pending duration. ## How evaluation works -1. KloudMate runs the rule's queries and expressions on the cadence set by **Evaluate every** (e.g. every `60s`). -2. A single rule can produce many **instances** — one per series returned by the query (one per host, function, endpoint, …). Each instance is tracked independently and has its own state. +1. KloudMate runs the rule's queries and expressions on the cadence set by **Evaluate every** (for example, every `60s`). +2. A single rule can produce many **instances**, one per series returned by the query (one per host, function, endpoint, …). Each instance is tracked independently and has its own state. 3. Every evaluation produces one raw **outcome** per instance, *before* any of the timing knobs are applied: | Outcome | What it means | @@ -23,7 +23,7 @@ This page explains the lifecycle so the behavior never surprises you. If you've | **Breaching** | The alert condition evaluated to true. For a [Condition Expression](../expressions/), that's your threshold being crossed. | | **Normal** | The alert condition evaluated to false. | | **No Data** | The query returned no data points for the window (or the value was null / `NaN`). | -| **Error** | A query failed, or the rule couldn't be evaluated at all — bad query, source unreachable, or an evaluation timeout. | +| **Error** | A query failed, or the rule couldn't be evaluated at all: bad query, source unreachable, or an evaluation timeout. | :::tip A raw metric is treated as **breaching** whenever its value is non-zero. To compare against a threshold instead, wrap it in a **Condition Expression** so the outcome reflects `value > threshold` (true/false), not just "non-zero". See [Writing Expressions](../expressions/). @@ -37,23 +37,23 @@ These raw outcomes are then fed through the timing knobs to produce the instance |---|---| | **Normal** | The condition is within threshold. Nothing to do. | | **Pending** | The condition is currently breaching but hasn't held long enough to fire yet. | -| **Firing** | The condition has breached for at least the **Pending duration**. This is what notifies. +| **Firing** | The condition has breached for at least the **Pending duration**. This is what notifies. | | **Recovering** | The condition has cleared, but the instance is still firing while it waits out the **Recovery period**. It has *not* resolved yet. | -| **No Data** | The query returned no data — and the rule is configured to surface that as its own state. | -| **Error** | The evaluation errored — and the rule is configured to surface that as its own state. | +| **No Data** | The query returned no data, and the rule is configured to surface that as its own state. | +| **Error** | The evaluation errored, and the rule is configured to surface that as its own state. | -**Pending** and **Recovering** are *derived* states — KloudMate computes them from the raw outcome plus the timing knobs. The evaluator itself never reports them. +**Pending** and **Recovering** are *derived* states. KloudMate computes them from the raw outcome plus the timing knobs; the evaluator itself never reports them. ## The firing episode -The core of the lifecycle is the **firing episode** — the path an instance takes from quiet, to firing, and back to quiet: +The core of the lifecycle is the **firing episode**, the path an instance takes from quiet, to firing, and back to quiet: ``` Normal ──breach──▶ Pending ──held ≥ pending duration──▶ Firing ──clears──▶ Recovering ──clear held ≥ recovery period──▶ Normal ``` -- **Pending duration** gates the way *in*. The condition must breach **continuously** for this long before the instance fires. A single Normal evaluation in the middle resets it back to Normal — the breach has to be sustained, not intermittent. Set it to `0` (leave it empty) to fire on the first breaching evaluation. -- **Recovery period** gates the way *out*. After an instance clears, it stays **Firing** (shown as **Recovering**) until the condition has stayed clear for this long. This stops a flapping metric from resolving and immediately re-firing — which, because a resolved alert is terminal, would otherwise open a brand-new alert group on every flap. Set it to `0` (leave it empty) to resolve the instant the condition clears. +- **Pending duration** gates the way *in*. The condition must breach **continuously** for this long before the instance fires. A single Normal evaluation in the middle resets it back to Normal, so the breach has to be sustained, not intermittent. Set it to `0` (leave it empty) to fire on the first breaching evaluation. +- **Recovery period** gates the way *out*. After an instance clears, it stays **Firing** (shown as **Recovering**) until the condition has stayed clear for this long. This stops a flapping metric from resolving and immediately re-firing. Because a resolved alert is terminal, that would otherwise open a brand-new alert group on every flap. Set it to `0` (leave it empty) to resolve the instant the condition clears. Exact transitions: @@ -63,15 +63,15 @@ Exact transitions: | Normal | breaching | **Firing** | Pending duration = 0 (fire immediately) | | Pending | breaching | **Firing** | breach has been sustained ≥ Pending duration | | Pending | breaching | **Pending** | still inside the Pending duration | -| Pending | clears | **Normal** | resets — the breach was never confirmed | +| Pending | clears | **Normal** | resets; the breach was never confirmed | | Firing | clears | **Recovering** | Recovery period > 0 | | Firing | clears | **Normal** (resolved) | Recovery period = 0 (resolve immediately) | | Recovering | clears | **Normal** (resolved) | clear has been sustained ≥ Recovery period | | Recovering | clears | **Recovering** | still inside the Recovery period | -| Recovering | breaching | **Firing** | a re-breach during recovery — snaps straight back to Firing | +| Recovering | breaching | **Firing** | a re-breach during recovery snaps straight back to Firing | :::note -A **re-breach during Recovering** snaps the instance back to **Firing** without starting a fresh Pending wait and without re-notifying — from the episode's point of view it never stopped firing. Only a clear that survives the whole Recovery period actually resolves the alert. +A **re-breach during Recovering** snaps the instance back to **Firing** without starting a fresh Pending wait and without re-notifying. From the episode's point of view it never stopped firing. Only a clear that survives the whole Recovery period actually resolves the alert. ::: ## The control knobs at a glance @@ -79,18 +79,18 @@ A **re-breach during Recovering** snaps the instance back to **Firing** without | Knob (form label) | What it controls | Default | Inherits from [folder](../folders/)? | |---|---|---|---| | **Evaluate every** | How often the rule runs. | `60s` | Yes | -| **Pending duration** | How long a breach must hold before **Firing**. | Fire immediately | No — per rule | -| **Recovery period** | How long a clear must hold before resolving. | Resolve immediately | No — per rule | +| **Pending duration** | How long a breach must hold before **Firing**. | Fire immediately | No (per rule) | +| **Recovery period** | How long a clear must hold before resolving. | Resolve immediately | No (per rule) | | **Alert state if No data** | Which state a no-data result becomes. | No Data | Yes | | **Alert state if Error** | Which state an error result becomes. | Error | Yes | ## How No Data and Error fit in -This is the part that most often surprises people, and it's the answer to a common question: +No Data and Error behave differently from a threshold breach. A common question: -> *I configured the same query, pending duration, and recovery period. Only the **Firing** state goes through the pending and recovery windows — **No Data** and **Error** don't. Is that expected?* +> *I configured the same query, pending duration, and recovery period. Only the **Firing** state goes through the pending and recovery windows; **No Data** and **Error** don't. Is that expected?* -**Yes — that's expected.** The **Pending duration** and **Recovery period** are properties of the *firing episode*. They gate the ramp **into** and **out of** Firing. **No Data** and **Error** are separate evaluation outcomes — by default they are not part of a firing episode, so there is nothing for those windows to debounce. An instance switches into (and out of) No Data or Error **immediately**. +**Yes, that's expected.** The **Pending duration** and **Recovery period** are properties of the *firing episode*. They gate the ramp **into** and **out of** Firing. **No Data** and **Error** are separate evaluation outcomes. By default they aren't part of a firing episode, so there is nothing for those windows to debounce. An instance switches into (and out of) No Data or Error **immediately**. What each setting does on a no-data or error result: @@ -99,7 +99,7 @@ What each setting does on a no-data or error result: | Target | Behavior when the query returns no data | |---|---| | **No Data** (default) | Instance switches to **No Data** immediately. No pending wait, no recovery wait. | -| **Normal** | Instance switches to **Normal** immediately — treated as a clear. | +| **Normal** | Instance switches to **Normal** immediately, treated as a clear. | | **Error** | Instance switches to **Error** immediately. | | **Firing** | Treated exactly like a threshold breach: it ramps through the **Pending duration** on the way in, and once firing it honors the **Recovery period** when data returns. An already-firing instance rides through the gap without resetting. | @@ -108,23 +108,23 @@ What each setting does on a no-data or error result: | Target | Behavior when the evaluation errors | |---|---| | **Error** (default) | Instance switches to **Error** immediately. No pending wait, no recovery wait. | -| **Normal** | Instance switches to **Normal** immediately — treated as a clear (so a firing instance enters the Recovery period). | +| **Normal** | Instance switches to **Normal** immediately, treated as a clear (so a firing instance enters the Recovery period). | | **Firing** | Treated exactly like a threshold breach: it ramps through the **Pending duration**, and honors the **Recovery period** when the query recovers. | :::note[The takeaway] -If you want a no-data or error condition to be **debounced by the same Pending duration and Recovery period as a threshold breach**, set **Alert state if No data** / **Alert state if Error** to **Firing**. Left on their default (**No Data** / **Error**), they are immediate, deliberate signals — a gap or a broken query shows up right away rather than waiting out a timer. +If you want a no-data or error condition to be **debounced by the same Pending duration and Recovery period as a threshold breach**, set **Alert state if No data** / **Alert state if Error** to **Firing**. Left on their default (**No Data** / **Error**), they're immediate and deliberate: a gap or a broken query shows up right away rather than waiting out a timer. ::: :::tip -Infrastructure failures — a query timeout, an unreachable source, or any error that prevents the rule from running — are routed through **Alert state if Error**, exactly like an error *returned* by the query. So "couldn't evaluate" behaves consistently with how you configured the error state. +Infrastructure failures (a query timeout, an unreachable source, or any error that prevents the rule from running) are routed through **Alert state if Error**, exactly like an error *returned* by the query. So "couldn't evaluate" behaves consistently with how you configured the error state. ::: ## Other behaviors you might notice -- **Transient blips on a firing alert.** A firing instance only *rides through* a momentary No Data or Error gap when that state is mapped to **Firing**. If No Data / Error are left on their defaults, a firing instance switches to **No Data** / **Error** on the next bad evaluation — it does not stay "Firing" through the gap. -- **A series that disappears entirely.** When a whole series stops appearing in the query results (a host is decommissioned, a metric is dropped), KloudMate waits a short grace window — about **two evaluation intervals** — and then treats the vanished instance as resolved and closes it. This is distinct from **No Data**, which is the query *running* and returning nothing. +- **Transient blips on a firing alert.** A firing instance only *rides through* a momentary No Data or Error gap when that state is mapped to **Firing**. If No Data / Error are left on their defaults, a firing instance switches to **No Data** / **Error** on the next bad evaluation; it does not stay "Firing" through the gap. +- **A series that disappears entirely.** When a whole series stops appearing in the query results (a host is decommissioned, a metric is dropped), KloudMate waits a short grace window (about **two evaluation intervals**) and then treats the vanished instance as resolved and closes it. This is distinct from **No Data**, which is the query *running* and returning nothing. To alert on the disappearance instead of closing it quietly, turn on [Instance Absence Detection](../instance-absence-detection/). - **Multi-instance rules.** A rule's overall state reflects its instances: if any instance is firing (or recovering), the rule reads as **Firing**; otherwise it surfaces the most significant instance state. Each instance still runs its own lifecycle independently. -- **Silences and maintenance windows don't change state.** They gate the outbound *notification*, not the lifecycle. A silenced instance keeps evaluating and transitioning, and its state changes still land in history — only the notification is withheld. See [Silences](../silences/) and [Maintenance Windows](../maintenance-windows/). +- **Silences and maintenance windows don't change state.** They gate the outbound *notification*, not the lifecycle. A silenced instance keeps evaluating and transitioning, and its state changes still land in history; only the notification is withheld. See [Silences](../silences/) and [Maintenance Windows](../maintenance-windows/). ## Related diff --git a/src/content/docs/docs/alerts/annotations-and-severity.mdx b/src/content/docs/docs/alerts/annotations-and-severity.mdx index f0c92ad2..b72e5eab 100644 --- a/src/content/docs/docs/alerts/annotations-and-severity.mdx +++ b/src/content/docs/docs/alerts/annotations-and-severity.mdx @@ -8,34 +8,34 @@ sidebar: import { Steps } from '@astrojs/starlight/components'; -KloudMate attaches custom metadata to your alerts using **annotations** — key-value pairs that flow into notifications and incidents along with the alert itself. +**Annotations** are key-value pairs you attach to an alert. They flow into notifications and incidents along with the alert itself. Annotation values support **Liquid templating**, so you can pull live data from the firing alert into the notification text. The reserved key `severity` drives how incident-management and routing tools prioritize the alert. ## Where to find annotations -Annotations live in the **Responder context** section of the alert rule editor (inside the **Add alarm details** step). Five annotation keys are pre-defined and render as labelled form fields: +Annotations live in the **Responder context** section of the alert rule editor (inside the **Add alarm details** step). The pre-defined annotation keys render as labeled form fields: -- **Severity** — free-form severity string, e.g. `sev1`, `critical`, `p1`. Plain text or Liquid template. -- **Summary** — multiline text, shown as the alert headline in notifications. Supports templates. -- **Dashboard** — pick a workspace dashboard to link from the notification. -- **Panel** — when a dashboard is picked, narrows the link to a specific panel. -- **Playbook URL** — link to the runbook responders should follow. Supports templates. +- **Severity:** free-form severity string, for example `sev1`, `critical`, or `p1`. Plain text or a Liquid template. +- **Summary:** multiline text, shown as the alert headline in notifications. Supports templates. +- **Dashboard:** pick a workspace dashboard to link from the notification. +- **Panel:** when a dashboard is picked, narrows the link to a specific panel. +- **Playbook URL:** link to the runbook responders should follow. Supports templates. -Any other key is a **custom annotation**. Custom annotations live under a collapsible **Custom annotations** accordion below the fixed fields — click **Add Annotation** to create a new key-value row. +Any other key is a **custom annotation**. Custom annotations live under a collapsible **Custom annotations** accordion below the fixed fields. Click **Add Annotation** to create a new key-value row. ## Add a custom annotation -1. **Open the rule editor.** Scroll to **Responder context** → expand the **Custom annotations** accordion. +1. **Open the rule editor.** Scroll to **Responder context** and expand the **Custom annotations** accordion. 2. **Click Add Annotation.** A new key-value row appears. -3. **Provide a key.** Use any string — e.g. `service_owner`, `region`, `runbook_section`. +3. **Provide a key.** Use any string, for example `service_owner`, `region`, or `runbook_section`. 4. **Provide a value.** This can be static text (`payments-team`) or a Liquid template (`{{ labels.service }} is degraded`). -5. **Save the rule.** Annotation values render at notification time, not at save time — see [Failure mode](#failure-mode) below. +5. **Save the rule.** Annotation values render at notification time, not at save time. See [Failure mode](#failure-mode) below. ## Using Liquid templates @@ -66,12 +66,12 @@ You can interpolate live alert data into any annotation value with the `{{ }}` s ### Common references -- `{{ labels.service }}` — the value of the `service` label on the firing instance. -- `{{ state.value }}` — the evaluated number that crossed the threshold. -- `{{ state.values.A }}`, `{{ state.values.B }}` — raw outputs of each query / expression node (by node ID). -- `{{ state.current_state }}` — `Alerting`, `Normal`, etc. -- `{{ state.resolved }}` — boolean; useful in `{% if state.resolved %}…{% endif %}` branches. -- `{{ rule.name }}`, `{{ rule.id }}` — the alert rule's name and id. Prefer these over `labels.alarm_id` / `labels.alarm_name` — those internal keys are stripped before render. +- `{{ labels.service }}`: the value of the `service` label on the firing instance. +- `{{ state.value }}`: the evaluated number that crossed the threshold. +- `{{ state.values.A }}`, `{{ state.values.B }}`: raw outputs of each query or expression node (by node ID). +- `{{ state.current_state }}`: `Alerting`, `Normal`, and so on. +- `{{ state.resolved }}`: boolean, useful in `{% if state.resolved %}…{% endif %}` branches. +- `{{ rule.name }}`, `{{ rule.id }}`: the alert rule's name and id. Prefer these over `labels.alarm_id` / `labels.alarm_name`; those internal keys are stripped before render. ### Example: contextual summary @@ -90,7 +90,7 @@ Every templated value field has a **Test template** icon to its right. Click it - A **Sample context** panel lets you tweak the labels / state / rule JSON the template renders against. - Click **Test** to see the rendered output below. Parse errors surface inline so you can fix them before saving. -The drawer is purely opt-in — KloudMate doesn't validate templates on save. If a template can't render at notification time, KloudMate falls back to the raw template string rather than dropping the notification. +The drawer is opt-in; KloudMate doesn't validate templates on save. If a template can't render at notification time, KloudMate falls back to the raw template string rather than dropping the notification. ### Failure mode @@ -122,15 +122,15 @@ At notification time, KloudMate evaluates the template against the firing alert' ## Severity vs. tags -Earlier versions of KloudMate used **notification tags** on the rule as the routing key. Tags have been replaced by: +Earlier versions of KloudMate routed by **notification tags** on the rule. Tags are gone now, replaced by: -- **Labels** — auto-derived from query dimensions and folders. These are what [Routing Rules](../routing-rules/) match against. -- **Annotations** — human-readable context that flows into the notification. -- **Severity** — the reserved annotation key, free-form, optionally Liquid-templated. +- **Labels:** auto-derived from query dimensions and folders. These are what [Routing Rules](../routing-rules/) match against. +- **Annotations:** human-readable context that flows into the notification. +- **Severity:** the reserved annotation key, free-form, optionally Liquid-templated. -If you used to route by tag, switch to a routing rule that matches on the equivalent label (e.g. `service=checkout-api`). +If you used to route by tag, switch to a routing rule that matches on the equivalent label (for example, `service=checkout-api`). ## Related -- [Routing Rules](../routing-rules/) — how labels match alerts to channels. -- [Creating Alerts](../create-alerts/) — where annotations live in the rule editor. +- [Routing Rules](../routing-rules/): how labels match alerts to channels. +- [Creating Alerts](../create-alerts/): where annotations live in the rule editor. diff --git a/src/content/docs/docs/alerts/auto-rca.mdx b/src/content/docs/docs/alerts/auto-rca.mdx index 55a9e5a0..b45621d7 100644 --- a/src/content/docs/docs/alerts/auto-rca.mdx +++ b/src/content/docs/docs/alerts/auto-rca.mdx @@ -5,42 +5,41 @@ sidebar: order: 8 --- -Auto-RCA hands an [Alert Group](../alert-groups/) to KloudMate's AI investigator the moment it opens, so by the time you open the group there's already a root-cause summary waiting. You enable it per [Routing Rule](../routing-rules/) — every group the rule produces gets investigated. +Auto-RCA hands an [Alert Group](../alert-groups/) to KloudMate's AI investigator the moment it opens, so by the time you open the group there's already a root-cause summary waiting. You enable it per [Routing Rule](../routing-rules/); every group the rule produces gets investigated. + +Auto-RCA is the automatic trigger: it runs on every group a rule opens, without anyone asking. You can also start an investigation manually on any group with the **Run RCA** button. Once an investigation attaches, the button becomes **View RCA** and opens it in a side drawer. See [Run RCA and View RCA](../alert-groups/#run-rca-and-view-rca). ## How it works -When a routing rule with Auto-RCA enabled opens a group, KloudMate schedules an investigation after the configurable **Auto-RCA delay** (default: 5 minutes). The delay gives related signals time to fold into the group so the investigation sees the full picture, not only the first signal. +When a routing rule with Auto-RCA enabled opens a group, KloudMate schedules an investigation after the configurable **Auto-RCA delay** (default: 5 minutes). The delay gives related alerts time to fold into the group so the investigation sees the full picture, not only the first alert. When the investigation completes, its summary attaches to the group: -- The group detail page shows the **Attached investigation** block with the summary excerpt and a **View full investigation** link. -- The notifications dispatched from the group include the investigation summary (where the channel format supports it — Slack thread reply, KloudMate Incidents comment, etc.). +- On the group detail page, the header's RCA button becomes **View RCA**. Opening it shows the root-cause summary and a **View full investigation** link. See [Run RCA and View RCA](../alert-groups/#run-rca-and-view-rca). +- The notifications dispatched from the group include the investigation summary, where the channel format supports it: a Slack thread reply, a KloudMate Incidents comment, and so on. :::note[One investigation per group] -Auto-RCA runs only once per alert group — when the group is first opened. Alerts that later join an existing group (for example, a Memory alert folding into a group that a CPU alert already opened) do **not** trigger a new investigation. Those alerts still send the usual group-transition notifications, but no additional Auto-RCA report is generated for them. +Auto-RCA runs only once per alert group, when the group is first opened. Alerts that later join an existing group (for example, a Memory alert folding into a group that a CPU alert already opened) do **not** trigger a new investigation. Those alerts still send the usual group-transition notifications, but Auto-RCA doesn't generate an additional report for them. ::: -:::note[Placeholder image] -Screenshot pending — an alert group detail page showing an attached investigation block with summary excerpt and the View full investigation link. -::: -![Attached investigation on group detail](./images/alert-group-investigation.png) +![View RCA drawer with the root-cause summary and a View full investigation link](./images/auto-rca-single-investigation.png) ## Enable Auto-RCA on a routing rule 1. Open **Alerts → Routing Rules** and edit the rule you want to enrich. 2. Scroll to the **Auto-RCA** section. 3. Toggle **Auto-RCA** on. -4. Set the **Delay** if the default 5 minutes isn't right. A shorter delay surfaces answers faster; a longer one captures more signals before the investigation runs. +4. Set the **Delay** if the default 5 minutes isn't right. A shorter delay surfaces answers faster; a longer one captures more alerts before the investigation runs. 5. Save. -Every group this rule opens from now on will trigger an investigation after the configured delay. +Every group this rule opens from now on triggers an investigation after the configured delay. ## When Auto-RCA is most useful -Auto-RCA works best on rules that bundle related signals — a spike after a deploy, cascading errors across services, or downstream impact from an upstream incident. For very narrow per-host or per-instance rules, the investigation has less to work with and you may prefer to disable it to reduce compute usage. +Auto-RCA works best on rules that group related alerts: a spike after a deploy, cascading errors across services, or downstream impact from an upstream incident. For very narrow per-host or per-instance rules, the investigation has less to work with, and you may prefer to disable it to reduce compute usage. ## Related -- [Routing Rules](../routing-rules/) — where the Auto-RCA toggle lives. -- [Alert Groups](../alert-groups/) — where the investigation surfaces. -- [KloudMate Assistant](../../kloudmate-assistant/) — the broader investigator surface. +- [Routing Rules](../routing-rules/): where the Auto-RCA toggle lives. +- [Alert Groups](../alert-groups/): where the investigation surfaces. +- [KloudMate Assistant](../../kloudmate-assistant/): the broader investigator surface. diff --git a/src/content/docs/docs/alerts/create-alerts.mdx b/src/content/docs/docs/alerts/create-alerts.mdx index 4bc9f18a..bba931ac 100644 --- a/src/content/docs/docs/alerts/create-alerts.mdx +++ b/src/content/docs/docs/alerts/create-alerts.mdx @@ -4,15 +4,16 @@ description: "Step-by-step guide to setting up and configuring KloudMate alerts. sidebar: order: 2 --- -KloudMate lets you create and configure alerts for events that are critical to your application. Each alert watches a query, evaluates a condition, and — when the condition holds long enough — fires into the grouping and routing engine so the right people are notified. + +An alert watches a query, evaluates a condition, and when the condition holds long enough, fires into the grouping and routing engine so the right people get notified. This page walks through creating one, step by step. ## Getting started -Navigate to the **Alerts** section from the left navigation menu. +Open the **Alerts** section from the left navigation. ![Alerts list](./images/setting-up-kloudmate-alarms-1.png) -The **Alerts** screen displays a list of all existing alert rules along with their current state, name, and description. The summary at the top shows the total number of rules, including how many are currently **Firing** or **Pending**. Rules can be grouped into [Folders](../folders/), which appear as collapsible header rows. +The **Alerts** screen lists your alert rules with their current state, name, and description. A summary at the top shows how many rules you have, and how many are currently **Firing** or **Pending**. You can group rules into [Folders](../folders/), which appear as collapsible header rows. From the **more options (⋯)** icon on any rule, you can: @@ -25,59 +26,59 @@ From the **more options (⋯)** icon on any rule, you can: ![Alert row actions](./images/setting-up-kloudmate-alarms-1.jpeg) -**Pause Notifications** opens a dialog that creates a temporary [silence](../silences/) scoped to this alert: the alert keeps evaluating, but its notifications are suppressed until the silence expires. While it's paused the rule shows a **Silenced** badge on the alerts list and on its detail page; view or end the pause from **Alerts → Silences**. Add matchers to limit it to specific instances. +**Pause Notifications** creates a temporary [silence](../silences/) scoped to this alert: the rule keeps evaluating, but KloudMate suppresses its notifications until the silence expires. While it's paused, the rule shows a **Silenced** badge on the alerts list and its detail page; view or end the pause from **Alerts → Silences**. Add matchers to limit it to specific instances. -To learn about the key concepts of KloudMate Alerts, see the [Alerts Overview](../). +For the key concepts, see the [Alerts Overview](../). ## Creating a new alert -Click the **Create Alert** button at the top-right corner of the **Alerts** screen. A dialog appears with three ways to create an alert: +Click the **Create Alert** button at the top right of the **Alerts** screen. A dialog appears with three ways to create an alert: ![Create Alert dialog](./images/create-alarm-dialog.png) -- **From Template** — Start with a pre-configured alert for a common monitoring scenario. -- **From Scratch** — Create a custom alert from an empty configuration. -- **Using AI** — Open **KloudMate Builder** and describe the alert you want in plain English. +- **From Template:** start with a pre-configured alert for a common monitoring scenario. +- **From Scratch:** create a custom alert from an empty configuration. +- **Using AI:** open **KloudMate Builder** and describe the alert you want in plain English. ### From Template -Instead of building an alert from scratch, you can start from a pre-configured template that covers common monitoring scenarios and best practices. +Instead of building an alert from scratch, start from a pre-configured template that covers a common monitoring scenario. 1. In the **Create Alert** dialog, select **From Template**. 2. Click the **Select a template** dropdown and choose a template that matches your monitoring needs. ![Template selection dropdown](./images/from-template-dropdown.png) -3. Click **Create Alert**. The alert is created and appears in the **Alerts** list. +3. Click **Create Alert**. The alert appears in the **Alerts** list. 4. To open and configure it, click the menu next to the alert and select **Edit**. -5. The alert opens pre-configured with the query, aggregation, and threshold settings from the template. Review and adjust any filters to match your environment. -6. Click **Save** or **Save & Close** when done. +5. The alert opens pre-configured with the query, aggregation, and threshold from the template. Review it and adjust any filters to match your environment. +6. Click **Save** or **Save & Close** when you're done. ### Using AI -KloudMate's assistant can automatically generate queries and thresholds based on your natural language prompt. +The KloudMate assistant generates the queries and thresholds from a plain-English prompt. 1. In the **Create Alert** dialog, select **Using AI**. 2. A text box appears. Describe the alert you want to create. ![Using AI prompt](./images/using-ai-prompt.png) -3. Click **Create Alert**. KloudMate generates the alert configuration based on your description. +3. Click **Create Alert**. KloudMate generates the alert configuration from your description. 4. To review or adjust the settings, click the menu next to the alert and select **Edit**. ### From Scratch To build a fully custom alert, select **From Scratch** in the **Create Alert** dialog and click **Create Alert**. -This opens the alert creation form where you can choose a data source, configure the metric or query to monitor, and define the alert condition on a single page. +This opens the alert creation form, where you choose a data source, configure the metric or query to monitor, and define the alert condition on a single page. -You can create multiple queries and expressions using the **Add Query** and **Add Expression** buttons. Each query or expression is assigned a unique alphabetical notation such as **A**, **B**, or **C**. You can duplicate any query or expression using the copy icon at the top-right corner of each block. +Create more queries and expressions with the **Add Query** and **Add Expression** buttons. KloudMate assigns each one a letter, such as **A**, **B**, or **C**, and you can duplicate any block with the copy icon at its top right. -To access advanced query and expression options such as **Math expressions**, **Reduce**, and **Condition expressions**, click **Advanced mode** at the top of the form. +To reach advanced options such as **Math expressions**, **Reduce**, and **Condition expressions**, click **Advanced mode** at the top of the form. ![Advanced mode options](./images/advanced-mode-options.png) -The rule editor walks you through four steps: **Setup query conditions and expressions**, **Configure evaluation settings**, **Add alarm details**, and **Notifications**. +The rule editor walks you through **Setup query conditions and expressions**, **Configure evaluation settings**, **Add alarm details**, and **Notifications**, then a final save. ### 1. Setup query conditions and expressions @@ -85,10 +86,10 @@ The rule editor walks you through four steps: **Setup query conditions and expre ![OpenTelemetry / KloudMate query setup](./images/setting-up-kloudmate-alarms-2.png) -- **Data Set:** Select the dataset you want to retrieve from your data source. -- **Metric to Aggregate:** Select the metric associated with the selected dataset that you want to monitor. -- **Group By:** Enter the attributes used to group the data points. -- **Filters:** Add filters to narrow down the retrieved data points. +- **Data Set:** select the dataset you want to retrieve from your data source. +- **Metric to Aggregate:** select the metric you want to monitor from that dataset. +- **Group By:** enter the attributes used to group the data points. +- **Filters:** add filters to narrow the retrieved data points. OpenTelemetry users can also use **Prometheus query language** to retrieve data and configure alerts. @@ -96,12 +97,12 @@ OpenTelemetry users can also use **Prometheus query language** to retrieve data ![AWS CloudWatch query setup](./images/setting-up-kloudmate-alarms-3.png) -- **Time Range:** Set the duration for which data should be fetched using the dropdown, or enter a custom value in seconds. -- **Region:** Select the AWS region of the service you want to monitor. -- **Namespace:** Select the AWS service namespace you want to create an alert for. -- **Metric:** Select the metric associated with the selected namespace. -- **Statistic:** Select the statistical function to use when calculating data points. -- **Dimensions:** Optionally configure the alert for grouped resources within the selected namespace. For example, for EC2, you can filter by autoscaling group name, image ID, instance type, and more. +- **Time Range:** set how far back to fetch data with the dropdown, or enter a custom value in seconds. +- **Region:** select the AWS region of the service you want to monitor. +- **Namespace:** select the AWS service namespace you want to alert on. +- **Metric:** select the metric from that namespace. +- **Statistic:** select the statistical function to apply when calculating data points. +- **Dimensions:** optionally scope the alert to grouped resources within the namespace. For EC2, for example, you can filter by autoscaling group name, image ID, instance type, and more. Click **Run Query** to fetch data. @@ -110,92 +111,93 @@ Click **Run Query** to fetch data. Query time ranges support the following: - **Operators:** `-` for subtracting time -- **Supported values:** The same units and keywords used in dashboards +- **Supported values:** the same units and keywords used in dashboards - **Examples:** `now`, `now-5m` #### Setting up evaluation expressions -Expressions let you apply logic to query results. Reference any configured query or expression using its alphabetical notation, such as **A**, **B**, or **C**. An expression can be passed as a parameter only when multiple expressions are configured. +Expressions let you apply logic to query results. Reference any query or expression by its letter, such as **A**, **B**, or **C**. An expression can be passed as a parameter only when multiple expressions are configured. -Choose from the following expression types: +Choose from these expression types: -- **Math Expression:** Enter a mathematical expression to apply to the value of a query or expression. Examples: `$A+1`, `$A<$B`, `$A && $C`. For more information, see [Alert Expressions](../expressions/). -- **Reduce:** Select a function to aggregate the values of a query or expression into a single number, then select the target query or expression from the **Input** dropdown. Available functions include `mean()`, `max()`, `min()`, `sum()`, `last()`, and `count()`. -- **Condition Expression:** Select a function and a query or expression, then choose a condition and provide a threshold value to evaluate against. You can add multiple conditions and combine them using **AND** or **OR** logical operators. +- **Math Expression:** enter a mathematical expression to apply to a query or expression, such as `$A+1`, `$A<$B`, or `$A && $C`. See [Alert Expressions](../expressions/). +- **Reduce:** aggregate a query or expression into a single number with a function, then pick the target from the **Input** dropdown. Available functions include `mean()`, `max()`, `min()`, `sum()`, `last()`, and `count()`. +- **Condition Expression:** pick a function and a query or expression, then choose a condition and a threshold to evaluate against. You can add multiple conditions and combine them with **AND** or **OR**. -Click **Run Queries** to execute all configured queries and expressions. +Click **Run Queries** to execute everything you've configured. :::tip -To avoid the **NoData** issue when using multiple queries in a single alert, use the `ifNull` operator to assign a default value. Read more in [Alert Expressions](../expressions/). +To avoid the **NoData** issue when a single alert uses multiple queries, use the `ifNull` operator to assign a default value. See [Alert Expressions](../expressions/). ::: ### 2. Configure evaluation settings -This section opens with a **Folder** dropdown — select an existing folder (or type a new name to create one inline) to organize the rule and inherit shared defaults. The folder's `interval_seconds`, `no_data_state`, and `eval_error_state` flow into the rule as inheritable defaults you can override per-field. See [Folders](../folders/). +This step opens with a **Folder** dropdown. Select an existing folder, or type a new name to create one inline, to organize the rule and inherit shared defaults. The folder's `interval_seconds`, `no_data_state`, and `eval_error_state` flow into the rule as defaults you can override per field. See [Folders](../folders/). ![Evaluation settings](./images/setting-up-kloudmate-alarms-4.png) -- **Alert condition** — Select the query or expression that should trigger the alert, such as **A**, **B**, or **C**. -- **Evaluate every** — How frequently the alert condition should be evaluated (e.g. `60s`, `1m`). -- **Pending duration** — How long the condition must remain true before the alert fires (e.g. `5m`). Leave empty to fire immediately. -- **Recovery period** — How long the condition must stay within threshold before the rule resolves (e.g. `5m`), which stops a flapping metric from resolving and immediately re-firing. While the rule waits out this window, the instance shows as **Recovering** — still firing, not yet back to Normal. Leave empty to resolve as soon as the condition clears. -- **Alert state if No data** — Which state the alert enters when the query returns no data points. Options: **Firing**, **No Data**, **Normal**, or **Error**. -- **Alert state if Error** — Which state the alert enters when a query returns an error. Options: **Firing**, **Error**, or **Normal**. +- **Alert condition:** the query or expression that triggers the alert, such as **A**, **B**, or **C**. +- **Evaluate every:** how often the alert condition is evaluated (for example, `60s` or `1m`). +- **Pending duration:** how long the condition must stay true before the alert fires (for example, `5m`). Leave empty to fire immediately. +- **Recovery period:** how long the condition must stay within threshold before the rule resolves (for example, `5m`). This stops a flapping metric from resolving and immediately re-firing. While the rule waits out this window, the instance shows as **Recovering**: still firing, not yet back to Normal. Leave empty to resolve as soon as the condition clears. +- **Alert state if No data:** which state the alert enters when the query returns no data points. Options: **Firing**, **No Data**, **Normal**, or **Error**. +- **Alert state if Error:** which state the alert enters when a query returns an error. Options: **Firing**, **Error**, or **Normal**. +- **Alert when an instance stops reporting:** track each instance the query returns, and alert when one goes silent while the others keep reporting. Turning this on also sets **Alert state if No data** to **Firing**. An **Auto-close after** field appears alongside it, controlling how long a silent instance stays tracked. See [Instance Absence Detection](../instance-absence-detection/). :::note[How “Firing” interacts with the pending duration] -When **Alert state if No data** or **Alert state if Error** is set to **Firing**, the alert doesn't fire the instant data goes missing or a query errors. It fires only after the condition has held for the **Pending duration** above — so a brief gap or a transient error won't page anyone, and a series that's already firing rides through the blip without resetting. Set the **Pending duration** to `0` to fire immediately instead. +When **Alert state if No data** or **Alert state if Error** is set to **Firing**, the alert doesn't fire the instant data goes missing or a query errors. It fires only after the condition has held for the **Pending duration** above, so a brief gap or a transient error won't notify anyone, and a series that's already firing rides through the blip without resetting. Set the **Pending duration** to `0` to fire immediately instead. ::: :::tip -For the full lifecycle — how **Pending duration**, **Recovery period**, and the **No data** / **Error** states interact across an alert's states — see [Alert Lifecycle & States](../alert-lifecycle/). +For the full lifecycle, and how **Pending duration**, **Recovery period**, and the **No data** / **Error** states interact across an alert's states, see [Alert Lifecycle & States](../alert-lifecycle/). ::: Click **Preview alerts** to run the query immediately and check the result. :::tip -If the rule belongs to a folder with shared defaults, **Evaluate every**, **Alert state if No data**, and **Alert state if Error** appear with placeholder values inherited from the folder. Leave them blank to keep inheriting, or type a value to override. A small **Reset to folder default** link appears under any field you've overridden. +If the rule belongs to a folder with shared defaults, **Evaluate every**, **Alert state if No data**, and **Alert state if Error** show placeholder values inherited from the folder. Leave them blank to keep inheriting, or type a value to override. A small **Reset to folder default** link appears under any field you've overridden. ::: ### 3. Add alarm details ![Alert details](./images/setting-up-kloudmate-alarms-5.png) -Two top-level fields and one section: +Set the alert's name and description, then fill in the responder context: -- **Alert name** — Enter a name for the alert. -- **Description** — Add a description to help identify the alert's purpose. +- **Alert name:** a name for the alert. +- **Description:** a short description of the alert's purpose. -**Responder context** — a labelled section that holds the five fixed annotations responders see when the notification lands. The hint above the section reads e.g. *"Help on-call responders understand the alarm and act quickly."* +**Responder context** is a labeled section that holds the annotations responders see when the notification lands. The hint above it reads, for example, *"Help on-call responders understand the alarm and act quickly."* -- **Severity** — Free-form severity (e.g. `sev1`, `critical`, `p1`). Supports Liquid templates so severity can depend on the firing value. -- **Summary** — Multiline message included in notifications. Supports templates. -- **Dashboard** — Optional dashboard link surfaced with the notification. -- **Panel** — When a dashboard is picked, narrows the link to a specific panel. -- **Playbook URL** — Optional runbook URL. +- **Severity:** free-form severity (for example, `sev1`, `critical`, or `p1`). Supports Liquid templates, so severity can depend on the firing value. +- **Summary:** multiline message included in notifications. Supports templates. +- **Dashboard:** optional dashboard link surfaced with the notification. +- **Panel:** when you pick a dashboard, narrows the link to a specific panel. +- **Playbook URL:** optional runbook URL. -**Custom annotations** — collapsed accordion at the bottom. Open it to add custom key-value pairs (e.g. `service_owner`, `region`). Values support Liquid templates. See [Annotations & Severity](../annotations-and-severity/). +**Custom annotations** is a collapsed accordion at the bottom. Open it to add your own key-value pairs (for example, `service_owner` or `region`). Values support Liquid templates. See [Annotations & Severity](../annotations-and-severity/). ### 4. Notifications -The Notifications step covers how this alert flows into the grouping engine. KloudMate replaced free-form notification tags with **labels** that routing rules match on: +The Notifications step controls how this alert flows into the grouping engine. Routing rules match on **labels**, not free-form tags: -- **Labels** — Key/value pairs attached to every signal this alert emits. Add them with **Add label**; the list can also be left empty. Routing rules match on these labels, plus the reserved `alarm_id`, `alarm_rule_folder_id`, and per-instance query labels, to decide which channels notify and how signals are grouped. -- **Routing** is decided by [Routing Rules](../routing-rules/) in **Alerts → Routing rules**, which match alerts by label and send them to one or more notification channels. -- **Severity** flows through the reserved `severity` annotation from the previous step — downstream tools use it to prioritize. +- **Labels:** key/value pairs attached to each alert this rule fires. Add them with **Add label**, or leave the list empty. Routing rules match on these labels, plus the reserved `alarm_id`, `alarm_rule_folder_id`, and per-instance query labels, to decide which channels notify and how alerts group. +- **Routing:** [Routing Rules](../routing-rules/) in **Alerts → Routing rules** match alerts by label and send them to one or more notification channels. +- **Severity:** flows through the reserved `severity` annotation from the previous step; downstream tools use it to prioritize. -If you're migrating an existing rule that used notification tags, the rule keeps working; routing now matches on labels instead. +If you're migrating a rule that used notification tags, it keeps working; routing now matches on labels instead. ### 5. Save the alert -Click **Save** to save the alert, or **Save & Close** to save and return to the **Alerts** screen. A confirmation message appears when the alert is created successfully. +Click **Save** to save the alert, or **Save & Close** to save and return to the **Alerts** screen. A confirmation appears once the alert is created. ## Viewing an alert -To open an alert, click the menu next to it and select **View**. This opens the detail page with four tabs: +To open an alert, click the menu next to it and select **View**. The detail page has these tabs: ![Alert detail overview](./images/alarm-detail-overview.png) -- **Overview** — Shows instance states, breaching instances with labels, reason, and duration, along with recent state transitions. -- **Instances** — Shows the full list of alert instances and their current states. -- **History** — Shows the state change history over time. -- **Rule** — Shows the alert configuration and query definition. +- **Overview:** instance states, breaching instances with their labels, reason, and duration, plus recent state transitions. +- **Instances:** the full list of alert instances and their current states. +- **History:** the state-change history over time. +- **Rule:** the alert configuration and query definition. diff --git a/src/content/docs/docs/alerts/expressions.mdx b/src/content/docs/docs/alerts/expressions.mdx index ace30c9f..4e1c017c 100644 --- a/src/content/docs/docs/alerts/expressions.mdx +++ b/src/content/docs/docs/alerts/expressions.mdx @@ -4,24 +4,26 @@ description: "Write and use expressions to evaluate metrics in KloudMate alerts. sidebar: order: 10 --- -Expressions let you manipulate query results using math and other operations. Alerts can only target numeric data, so if a query returns a time series, you must reduce it with an expression before it can drive an alert. -Each query or expression is represented by a unique alphabetical notation (A, B, C, and so on). While writing an expression, one or more queries or other expressions can be passed as parameters using their notation. +Expressions let you manipulate query results with math and other operations. Alerts can only target numeric data, so if a query returns a time series, you must reduce it with an expression before it can drive an alert. + +Each query or expression has a unique letter (A, B, C, and so on). In an expression, reference other queries or expressions by their letter. ## Expression operations -KloudMate supports three types of operations in alert expressions: +Alert expressions support three operations: + - Math expression - Reduce - Condition expression ### Math Expression -Math expression takes time series or number data returned by a query or an expression and turns them into different time series and numbers using mathematical operations or functions. +A math expression takes the time series or numbers returned by a query or expression and transforms them with mathematical operations or functions. ![image](./images/writing-expressions-for-kloudmate-alarms-1.png) -**Input:** Any mathematical operation to apply to the data returned from other queries or expressions. Queries and expressions are passed as parameters using their alphabetical notation prefixed with a dollar sign, for example, `$A`. +**Input:** any mathematical operation to apply to the data returned from other queries or expressions. Reference queries and expressions by their letter, prefixed with a dollar sign, for example `$A`. #### Mathematical Operations @@ -55,91 +57,91 @@ Examples: `$A==0`, `!$A!=0`, `$A<$B`, `$A>10`, `$A<=$D`, `$A>=$D` Examples: `$A && $D`, `$A and $D`, `$A or $D`, `$A || $D`, `ifNull($A, 0)` :::info -For the above operations: -- When both `$A` and `$B` are numbers, the operation is performed between the two numbers. -- If one parameter is a number and the other is a time series, the operation is performed between the number and each value in the time series individually. -- If both `$A` and `$B` are time series data, the operation is performed between each value in the two series having the same timestamps. +For the operations above: +- When both `$A` and `$B` are numbers, the operation runs between the two numbers. +- If one parameter is a number and the other is a time series, the operation runs between the number and each value in the time series individually. +- If both `$A` and `$B` are time series, the operation runs between the values that share the same timestamp. - Boolean and logical operations return `0` for false and `1` for true. ::: #### Using Expressions for Multiple Query Conditions -When working with multiple queries in a single alert: -- Use the Reduce function for all time series outputs (e.g., mean, max, last). -- Use the `ifNull` logic with all reduced values so that if any query returns NoData, it does not affect the evaluation of other queries. +When a single alert uses multiple queries: +- Use the Reduce function on every time series output (for example, mean, max, or last). +- Use `ifNull` on every reduced value, so a query returning NoData doesn't affect the evaluation of the others. Example: `ifNull($A, 0) > 50 || ifNull($B, 0) > 90` -- `ifNull` — checks for NoData and assigns the given value (0) to that node if true. -- `>` — arithmetic operation that checks the condition. -- `||` — logical OR operation that evaluates both query nodes. +- `ifNull`: checks for NoData and assigns the given value (0) to that node if true. +- `>`: the arithmetic operation that checks the condition. +- `||`: the logical OR that evaluates both query nodes. #### Mathematical Functions -- `abs` — Returns the absolute value. E.g., `abs(-1)` or `abs($A)` -- `log` — Returns the natural logarithm. Returns NaN if the value is less than 0. E.g., `log(-1)` or `log($A)` -- `round` — Returns a rounded integer value. E.g., `round(3.123)` or `round($A)` -- `ceil` — Rounds up to the nearest integer. E.g., `ceil(3.123)` returns `4` -- `floor` — Rounds down to the nearest integer. E.g., `floor(3.123)` returns `3` +- `abs`: returns the absolute value. For example, `abs(-1)` or `abs($A)`. +- `log`: returns the natural logarithm. Returns NaN if the value is less than 0. For example, `log(-1)` or `log($A)`. +- `round`: returns a rounded integer. For example, `round(3.123)` or `round($A)`. +- `ceil`: rounds up to the nearest integer. `ceil(3.123)` returns `4`. +- `floor`: rounds down to the nearest integer. `floor(3.123)` returns `3`. #### Built-in Time-Range Variables -To get queried data in a data-per-time-range format (e.g., data/min), divide the data by the appropriate time-range variable. The following four built-in variables are available: -- `$range:m` — Time range in minutes -- `$range:s` — Time range in seconds -- `$range:h` — Time range in hours -- `$range:d` — Time range in days +To get data in a per-time-range format (for example, data per minute), divide the data by the appropriate time-range variable. These built-in variables are available: +- `$range:m`: time range in minutes +- `$range:s`: time range in seconds +- `$range:h`: time range in hours +- `$range:d`: time range in days ### Reduce -Reduce takes one or more time series or numbers of data returned from a query or an expression and turns them into a single number. +Reduce takes the time series or numbers returned by a query or expression and turns them into a single number. -- **Reduction function:** Select the function to apply — `mean()`, `max()`, `min()`, `sum()`, `last()`, or `count()`. +- **Reduction function:** select the function to apply: `mean()`, `max()`, `min()`, `sum()`, `last()`, or `count()`. ![image](./images/writing-expressions-for-kloudmate-alarms-2.png) -- **Input:** Select the query or expression to reduce using its alphabetical notation. +- **Input:** select the query or expression to reduce by its letter. ![image](./images/writing-expressions-for-kloudmate-alarms-3.png) -The selected reduction function aggregates the values of the query or expression into a single value. +The reduction function aggregates the values of the query or expression into a single value. ### Condition Expression -Condition expression takes one or more time series or numbers of data returned from queries or expressions and gives a boolean value as a result — `0` (False) if the condition is not met, or `1` (True) if it is met. +A condition expression takes the time series or numbers returned by queries or expressions and gives a boolean result: `0` (False) if the condition isn't met, or `1` (True) if it is. -- **WHEN:** Select the reduction function — `last()`, `mean()`, `max()`, `min()`, `sum()`, or `count()`. +- **WHEN:** select the reduction function: `last()`, `mean()`, `max()`, `min()`, `sum()`, or `count()`. ![image](./images/writing-expressions-for-kloudmate-alarms-4.png) -- **OF:** Select the query or expression using its alphabetical notation (e.g., A, C). +- **OF:** select the query or expression by its letter (for example, A or C). ![image](./images/writing-expressions-for-kloudmate-alarms-5.png) -- **Condition string:** Select the comparison condition — `IS ABOVE`, `IS SAME OR ABOVE`, `IS BELOW`, or `IS SAME OR BELOW`. +- **Condition string:** select the comparison: `IS ABOVE`, `IS SAME OR ABOVE`, `IS BELOW`, or `IS SAME OR BELOW`. ![image](./images/writing-expressions-for-kloudmate-alarms-6.png) -- **Threshold:** Enter the numeric value to evaluate the condition against. +- **Threshold:** enter the numeric value to evaluate the condition against. ![image](./images/writing-expressions-for-kloudmate-alarms-7.png) -The selected reduction function is applied to the selected query or expression to generate a single value. This value is then compared against the provided threshold according to the selected condition. When the input is a collection of time series or number data, the reduction function is applied to each element individually. Each reduced output is evaluated against the condition individually, and the outputs are then combined using the AND operation. +The reduction function is applied to the query or expression to produce a single value, which is then compared against the threshold using the condition you chose. When the input is a collection of time series or numbers, the reduction function is applied to each element individually, each reduced output is evaluated against the condition, and the outputs are combined with AND. -You can add multiple conditions in a condition expression using the **Add condition** button and choose how they work together using **AND** or **OR** logical operators. +Add more conditions with the **Add condition** button, and choose how they combine with the **AND** or **OR** operators. ## Nodes -Nodes represent individual queries or expressions used in the creation of alerts and dashboards. Each node is assigned an alphabetical identifier by default (e.g., A, B, C), based on the order in which it was created. +Nodes represent the individual queries or expressions used to build alerts and dashboards. KloudMate assigns each node a letter by default (for example, A, B, or C), based on the order it was created. ![image](./images/writing-expressions-for-kloudmate-alarms-1.jpeg) -Nodes serve as reference points, allowing users to identify and evaluate specific queries or expressions during analysis. +Nodes serve as reference points, so you can identify and evaluate specific queries or expressions during analysis. ![image](./images/writing-expressions-for-kloudmate-alarms-2.jpeg) -While default alphabetical naming is applied automatically, nodes can be renamed for better clarity. +KloudMate names nodes alphabetically by default, but you can rename them for clarity. ![image](./images/writing-expressions-for-kloudmate-alarms-3.jpeg) -If the custom node name includes spaces, it must be enclosed in **${}**. Example: `${CPU usage}` +If a custom node name includes spaces, enclose it in **${}**. For example: `${CPU usage}` ![image](./images/writing-expressions-for-kloudmate-alarms-4.jpeg) diff --git a/src/content/docs/docs/alerts/folders.mdx b/src/content/docs/docs/alerts/folders.mdx index ef042408..b9d2c3cb 100644 --- a/src/content/docs/docs/alerts/folders.mdx +++ b/src/content/docs/docs/alerts/folders.mdx @@ -5,21 +5,21 @@ sidebar: order: 4 --- -Folders organize related alert rules and let member rules inherit shared evaluation defaults. They follow the dashboards-style grouping pattern applied to alerts: opt into a folder view, see your rules collapsed under their owning folder, and let one set of defaults propagate to whichever rules need them. +Folders organize related alert rules and let member rules inherit shared evaluation defaults. They bring the dashboards-style grouping pattern to alerts: opt into a folder view, see your rules collapsed under their owning folder, and let one set of defaults propagate to whichever rules need them. On the **Alerts** list, the layout toggle in the toolbar switches between a flat **List view** and a grouped **Folder view**. In folder view, each folder renders as a collapsible header row with its member rules underneath; rules without a folder land in a pinned **Ungrouped** section. ![Alerts list in folder view](./images/alerts-folder-view.png) -There's no separate Folders page or sidebar entry — everything happens from the Alerts list. +There's no separate Folders page or sidebar entry; everything happens from the Alerts list. ## Create a folder -Folders are created inline from the alert rule editor's **Folder** picker — there's no separate "New folder" button. +You create folders inline from the alert rule editor's **Folder** picker; there's no separate "New folder" button. 1. Open an alert rule (create a new one or edit an existing one). 2. In the **Configure evaluation settings** section, click the **Folder** dropdown. -3. Type a new folder name. A `+ Create ""` option appears at the bottom of the dropdown — click it. +3. Type a new folder name. A `+ Create ""` option appears at the bottom of the dropdown. Click it. 4. KloudMate creates the folder and assigns the current rule to it. A toast confirms the creation. 5. To set folder-level defaults (interval, no-data state, eval-error state), switch to **Folder view** on the Alerts list, open the folder's kebab menu, and select **Edit folder**. @@ -33,7 +33,7 @@ In Folder view, each folder header row has a kebab menu with **Edit folder** and |---|---| | **Name** | Required, unique within the workspace. | | **Description** | Optional. | -| **Evaluate every** | Default evaluation interval (e.g. `60s`, `1m`, `2h`). Members inherit when blank. | +| **Evaluate every** | Default evaluation interval (for example `60s`, `1m`, or `2h`). Members inherit when blank. | | **No data state** | Default state when queries return no data. Options: `Normal`, `Alerting`, `NoData`. | | **Eval error state** | Default state when evaluation errors out. Options: `Normal`, `Alerting`, `Error`. | @@ -41,10 +41,10 @@ A note above these reads: *"Member alerts inherit these defaults unless they set ## Move rules into a folder -Two ways: +Move rules from the rule editor, or in bulk from the list: -- **From the rule editor** — open the rule and pick the folder from the **Folder** dropdown in the **Configure evaluation settings** section. -- **Bulk-move from the list** — select rules using the checkboxes (a toolbar appears with the selection count). Click **Move to folder…**, select a destination from the radio list (existing folders or **Ungrouped**), and confirm. +- **From the rule editor:** open the rule and pick the folder from the **Folder** dropdown in the **Configure evaluation settings** section. +- **Bulk-move from the list:** select rules with the checkboxes (a toolbar appears with the selection count). Click **Move to folder…**, select a destination from the radio list (existing folders or **Ungrouped**), and confirm. Moving a rule does **not** rewrite its `interval_seconds`, `no_data_state`, or `eval_error_state` columns. Whatever the rule had before the move stays. The folder defaults only apply where the rule's field is blank. @@ -58,7 +58,7 @@ A rule's eval-config fields follow a clear three-state model: | Filled (overriding) | The rule's stored value + `Overrides folder default` helper line + **Reset to folder default** link | The rule's value | | No folder selected | Plain input, no helper text | The rule's value (or system default if blank) | -Clicking **Reset to folder default** clears the field and flips the helper text back to inheriting. Saving with the field blank persists `NULL` for that column — so when the folder default later changes, the rule picks up the new value automatically. +Clicking **Reset to folder default** clears the field and flips the helper text back to inheriting. Saving with the field blank persists `NULL` for that column, so when the folder default later changes, the rule picks up the new value automatically. :::tip To detach a rule from its folder entirely, change the **Folder** dropdown to the ungrouped option. The eval-config columns keep their stored values; the rule stops inheriting. @@ -66,16 +66,16 @@ To detach a rule from its folder entirely, change the **Folder** dropdown to the ## Delete a folder -From the folder kebab menu, pick **Delete folder**. The confirmation explains: e.g. *"Member alerts will become ungrouped — their stored eval-config values stay intact."* KloudMate detaches the members first and then deletes the folder row. Rules survive the deletion. +From the folder kebab menu, pick **Delete folder**. The confirmation explains that member alerts will become ungrouped, with their stored eval-config values intact. KloudMate detaches the members first and then deletes the folder row. Rules survive the deletion. ## Folder labels in alert groups A rule's folder identity flows into the alert's labels as `alarm_rule_folder_id` and `alarm_rule_folder_name`. That means: -- [Routing Rules](../routing-rules/) can match on folder identity (e.g. `alarm_rule_folder_name in [Kubernetes, Networking]`). +- [Routing Rules](../routing-rules/) can match on folder identity (for example, `alarm_rule_folder_name in [Kubernetes, Networking]`). - The folder name shows as a label chip on the [Alert Group](../alert-groups/) detail header automatically. ## Related -- [Creating Alerts](../create-alerts/) — picking a folder during rule creation. -- [Routing Rules](../routing-rules/) — matching on folder labels. +- [Creating Alerts](../create-alerts/): picking a folder during rule creation. +- [Routing Rules](../routing-rules/): matching on folder labels. diff --git a/src/content/docs/docs/alerts/how-alert-grouping-works.mdx b/src/content/docs/docs/alerts/how-alert-grouping-works.mdx new file mode 100644 index 00000000..12414665 --- /dev/null +++ b/src/content/docs/docs/alerts/how-alert-grouping-works.mdx @@ -0,0 +1,74 @@ +--- +title: "How Alert Grouping Works" +description: "How KloudMate's AI groups related firing alerts into one incident, and how to get per-host groups when you need them." +sidebar: + order: 6.5 +--- + +A single problem rarely fires a single alert. One deploy can spike CPU across forty hosts at once; one failing database can set off the app, the queue, and the health checks together. That's how a real issue turns into an alert storm. + +Alert grouping correlates those related alerts and folds them into a single [Alert Group](../alert-groups/), so one incident reaches you instead of forty notifications. This page shows how the engine decides what belongs together, and how to steer it. + +## What gets grouped + +KloudMate groups individual firings, not whole rules. + +One alert rule can fire on many targets at once. A `High CPU` rule fires once per host that crosses the line; a pod-restart rule fires once per pod. Each firing is its own **instance**, carrying its own labels (`host_name`, `pod`, and so on). Grouping works at the instance level, so a single rule's firings can spread across a group, each placed by its own labels. + +## Static or Auto grouping + +Every alert runs through a [routing rule](../routing-rules/), and each rule groups its alerts one of two ways. + +**Static** grouping is deterministic: you name the labels, and alerts that share those values land together. **Auto (AI)** grouping hands the decision to the correlation engine, which links related alerts on its own and learns which of your alerts tend to fire together over time. + +| | **Static** | **Auto (AI)** | +|---|---|---| +| Who decides | You, with **Group-by keys** | KloudMate's correlation engine | +| Based on | The **values** of your group-by labels: one group per distinct value | The strongest connection the engine can find | +| One rule on `host=minikube` and `host=fedora` | **Two groups**, one per host | **One group** | + +You set the mode per rule with the **Alert grouping** control. See [Routing Rules](../routing-rules/#alert-grouping-static-or-auto-ai). + +## One rule stays one group in Auto mode + +Auto grouping treats every firing of one rule as the same issue on different targets, so they always stay in a single group, even when the hosts, pods, or containers differ. + +Take that deploy again. Your `High CPU` rule fires on `minikube`, `fedora`, and dozens more. Auto grouping keeps them as **one** incident instead of scattering them into dozens, because they're the same rule reporting on different machines. A rule firing on 500 hosts reaches you as one incident with 500 instances, not 500 separate notifications. + +:::note[Want a group per host instead?] +Some rules you *do* want split per host, like a disk-filling-up check. Switch that rule to **Static** and add `host_name` as a Group-by key, and each host gets its own group. +::: + +## When to use which + +- Choose **Static** when you already know the dimension you care about and want deterministic groups: one per host, per cluster, or per service. +- Choose **Auto (AI)** when the connections aren't obvious from labels (a slow database dragging down three services that don't share a tag) and you want the engine to surface them for you. + +## Why alerts group in Auto mode + +Auto grouping analyzes each burst of alerts and links them on the strongest connection it can find, in this order: + +1. **Same rule:** every firing of one rule (always together, as above). +2. **Same identity label:** firings that share a `key=value`, like `pod=api-7`. +3. **Fire together often:** alerts with a track record of going off at the same time. +4. **Cause and effect:** a known pattern where trouble in one resource sets off another. +5. **Connected resources:** the two are linked in the dependency graph. +6. **AI grouping:** a machine-learning model makes the call on whatever's left. +7. **On its own:** nothing links it, so it stays a group of one. + +However alerts link, the group shows the reason under [**Why these were grouped**](../alert-groups/#why-these-were-grouped), naming the strongest connection that covers the whole group. The engine also learns continuously: a new workspace starts with less history, and grouping gets more precise over the first few days as the model sees which alerts travel together. See [Cold start](../alert-groups/#cold-start). + +## How Auto grouping behaves + +Auto grouping follows a few predictable rules: + +**One rule, one group.** Auto grouping always keeps a single rule's firings together, across every host it fires on. To split them, use **Static** grouping with `host_name` as the Group-by key. + +**Labels match on key and value.** A shared label links alerts only when the key and the value both match, and Static grouping keys on the value. Two different hosts group together because they share a rule, not because `host_name=minikube` and `host_name=fedora` count as the same. + +**Grouping is per instance.** Every firing is placed on its own, by its own labels. When one rule fills a group, that's Auto keeping its firings together, one instance at a time. + +## Related + +- [Routing Rules](../routing-rules/): set the grouping mode and Group-by keys. +- [Alert Groups](../alert-groups/): the group you get, and where the grouping reason shows. diff --git a/src/content/docs/docs/alerts/images/alert-group-blast-radius.png b/src/content/docs/docs/alerts/images/alert-group-blast-radius.png new file mode 100644 index 00000000..82722b79 Binary files /dev/null and b/src/content/docs/docs/alerts/images/alert-group-blast-radius.png differ diff --git a/src/content/docs/docs/alerts/images/alert-group-detail.png b/src/content/docs/docs/alerts/images/alert-group-detail.png index 8481f096..2870afac 100644 Binary files a/src/content/docs/docs/alerts/images/alert-group-detail.png and b/src/content/docs/docs/alerts/images/alert-group-detail.png differ diff --git a/src/content/docs/docs/alerts/images/alert-group-rca-drawer.png b/src/content/docs/docs/alerts/images/alert-group-rca-drawer.png new file mode 100644 index 00000000..dc166fa2 Binary files /dev/null and b/src/content/docs/docs/alerts/images/alert-group-rca-drawer.png differ diff --git a/src/content/docs/docs/alerts/images/alert-groups-list.png b/src/content/docs/docs/alerts/images/alert-groups-list.png index c2e5ae28..11c53983 100644 Binary files a/src/content/docs/docs/alerts/images/alert-groups-list.png and b/src/content/docs/docs/alerts/images/alert-groups-list.png differ diff --git a/src/content/docs/docs/alerts/images/auto-rca-single-investigation.png b/src/content/docs/docs/alerts/images/auto-rca-single-investigation.png new file mode 100644 index 00000000..789c1d82 Binary files /dev/null and b/src/content/docs/docs/alerts/images/auto-rca-single-investigation.png differ diff --git a/src/content/docs/docs/alerts/images/routing-rules-alert-grouping.png b/src/content/docs/docs/alerts/images/routing-rules-alert-grouping.png new file mode 100644 index 00000000..c1cab2db Binary files /dev/null and b/src/content/docs/docs/alerts/images/routing-rules-alert-grouping.png differ diff --git a/src/content/docs/docs/alerts/index.mdx b/src/content/docs/docs/alerts/index.mdx index 2c7e5c78..0bca0f3d 100644 --- a/src/content/docs/docs/alerts/index.mdx +++ b/src/content/docs/docs/alerts/index.mdx @@ -8,15 +8,15 @@ sidebar: import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -Alerts tell you when something needs attention — an error-rate spike, a service going quiet, a budget threshold crossed. You write the rule once; KloudMate evaluates it on a schedule, groups related firings together, and routes notifications to the channels you care about. +Alerts tell you when something needs attention: an error-rate spike, a service going quiet, a budget threshold crossed. You write the rule once; KloudMate evaluates it on a schedule, groups related firings together, and routes notifications to the channels you care about. -This page covers the concepts behind the Alerts module. Once they click, the rest of the section is a tour of each surface. +This page covers the concepts behind the Alerts module. The rest of this section covers each feature in detail. ## Key concepts ### Alert rule -An alert rule contains the evaluation criteria — one or more queries, expressions, and a condition. It also specifies how often KloudMate evaluates the rule, how long the condition must hold before the rule fires, and how long it must stay clear before the rule resolves. +An alert rule contains the evaluation criteria: one or more queries, expressions, and a condition. It also specifies how often KloudMate evaluates the rule, how long the condition must hold before the rule fires, and how long it must stay clear before the rule resolves. ### Alert query @@ -28,7 +28,7 @@ Expressions transform query output with math, reductions, or conditions. They re ### Alert labels -Labels are key-value pairs attached to each firing alert. They come from query dimensions and any folder the rule lives in, and they're what [Routing Rules](./routing-rules/) match against to decide where notifications go. KloudMate generates labels automatically — you don't enter them by hand. +Labels are key-value pairs attached to each firing alert. They come from query dimensions and any folder the rule lives in, and they're what [Routing Rules](./routing-rules/) match against to decide where notifications go. KloudMate generates labels automatically. You don't enter them by hand. ### Annotations and severity @@ -36,11 +36,11 @@ Annotations carry human-readable context (summary, runbook URL, dashboard link) ### Folders -Folders group related alert rules and let member rules inherit shared defaults — evaluation interval, no-data state, and eval-error state. See [Folders](./folders/). +Folders group related alert rules and let member rules inherit shared defaults: evaluation interval, no-data state, and eval-error state. See [Folders](./folders/). ### Alert groups -When matching alerts fire close together, KloudMate bundles them into a single **Alert Group** — a durable, deduplicated container that updates as higher-severity signals join, holds the auto-RCA if you've enabled one, and provides a single thread to notify against. See [Alert Groups](./alert-groups/). +When matching alerts fire close together, KloudMate bundles them into a single **Alert Group**: a durable, deduplicated container that updates as higher-severity alerts join, holds the auto-RCA if you've enabled one, and gives you a single thread to notify against. See [Alert Groups](./alert-groups/). ### Routing rules @@ -48,13 +48,13 @@ Routing rules decide which channels get notified for which alerts. Each rule mat ### Silences -A silence suppresses notifications for matching labels for a bounded time window — useful when you know an alert will be noisy and don't want to flood your channels. See [Silences](./silences/). +A silence suppresses notifications for matching labels for a bounded time window. Use it when you know an alert will be noisy and you don't want to flood your channels. See [Silences](./silences/). ### Multi-dimensional alerts A single rule can produce multiple alert instances, one per dimension. A rule watching Lambda throttling generates one instance per throttled function. -## Workflow of KloudMate Alerts +## The alert workflow 1. A rule retrieves data from its source using queries. 2. Expressions reduce or transform query results. @@ -63,16 +63,16 @@ A single rule can produce multiple alert instances, one per dimension. A rule wa 5. The grouping engine matches the alert against [Routing Rules](./routing-rules/), opens or appends to an [Alert Group](./alert-groups/), and dispatches notifications to the rule's destination channels. 6. Active [Silences](./silences/) and [Maintenance Windows](./maintenance-windows/) can suppress the outbound notification at this step. The rule keeps evaluating in the background and the state change still lands in history. -## KloudMate alert states +## Alert states -- **Firing / Alerting** — the rule's condition has held longer than the pending duration. -- **Pending** — the condition is currently true but hasn't held long enough yet. -- **Recovering** — the condition has cleared, but the rule is still firing while it waits out the recovery period before resolving. -- **Normal** — everything's quiet. -- **Error** — the evaluation hit an error (bad query, source unreachable). -- **No Data** — the query returned no data for the configured window. +- **Firing / Alerting:** the rule's condition has held longer than the pending duration. +- **Pending:** the condition is currently true but hasn't held long enough yet. +- **Recovering:** the condition has cleared, but the rule is still firing while it waits out the recovery period before resolving. +- **Normal:** everything's quiet. +- **Error:** the evaluation hit an error (bad query, source unreachable). +- **No Data:** the query returned no data for the configured window. -For how these states transition — and how the pending duration, recovery period, and no-data / error settings shape them — see [Alert Lifecycle & States](./alert-lifecycle/). +For how these states transition, and how the pending duration, recovery period, and no-data or error settings shape them, see [Alert Lifecycle & States](./alert-lifecycle/). ## Next steps @@ -82,6 +82,7 @@ For how these states transition — and how the pending duration, recovery perio + ## Related resources diff --git a/src/content/docs/docs/alerts/instance-absence-detection.mdx b/src/content/docs/docs/alerts/instance-absence-detection.mdx new file mode 100644 index 00000000..51b184d0 --- /dev/null +++ b/src/content/docs/docs/alerts/instance-absence-detection.mdx @@ -0,0 +1,114 @@ +--- +title: "Instance Absence Detection" +description: "Turn on absence detection to catch any instance that stops reporting, tune how long silent instances stay tracked, and dismiss the ones that were terminated on purpose." +sidebar: + order: 2.7 + label: "Absence Detection" +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +A threshold alert can only judge the datapoints that arrive. It has nothing to say when the datapoints stop. **Alert when an instance stops reporting** covers that case: turn it on for a rule and KloudMate remembers every instance the rule has seen, then alerts when any of them goes silent. + +An **instance** is a unique series identified by its distinct labels. A heartbeat metric grouped by `host_name` produces one instance per host; group by `serviceName` and `pod_name` and you get one per pod. Absence detection tracks each instance individually: it has its own state, its own history, its own row on the **Instances** tab, and its own dismissal. + +The setting is off by default and opt-in per rule, so existing rules don't change until you enable it. It works with any datasource, whether the data comes from the KloudMate agent or not. + +## How it works + +On every evaluation, KloudMate compares the query results against the set of instances the rule has seen. An instance missing from the results is evaluated as if the query had returned no data for it, and it moves through the normal [lifecycle](../alert-lifecycle/): **Pending** for the pending duration, then **Firing**, with the reason `instance stopped reporting; last seen `. The moment the instance reports again, it recovers through the usual path, including any recovery period. + +Absence detection changes what fires, not how notifications are threaded. Threading comes from the [routing rule](../routing-rules/) that matches, through its **Group-by** keys, exactly as it does for threshold alerts: + +- **No grouping dimension** (empty, or only `alarm_id`): one thread per alert rule. Every silent instance is listed in that thread, and instances that go silent later arrive as updates to it rather than as separate notifications. +- **Grouped by an instance dimension**, such as `host_name`: one thread per host. Two hosts going silent gives you two threads and two [Alert Groups](../alert-groups/), which is what grouping by host means. Each of those threads also collects every other alert rule firing for that host, not just this one. + +Routing itself works as usual, since each instance carries its own labels. + +## Turn it on + +1. Create or edit an alert rule and go to **Configure evaluation settings**. +2. Turn on **Alert when an instance stops reporting**. +3. Optionally set **Auto-close after**: how long a silent instance stays tracked, and firing, before it closes on its own, which happens only while other instances are still reporting. Leave it blank to inherit the folder default, or the system default of 24 hours. It accepts durations from `5m` to `72h`, for example `30m` or `48h`. +4. Save the rule. + +Turning the toggle on also sets **Alert state if No data** to **Firing**. Both are needed for absence detection, so the form does it in one step. Turning the toggle off leaves the No-data state alone, in case you still want the rule to fire when the whole query returns nothing. + +Both settings inherit from [folders](../folders/) exactly like the other evaluation defaults: the rule's own value wins, a blank field falls back to the folder default, and the system default applies last. The folder edit dialog carries the matching fields, **Instance stops reporting** (**Alert** / **Don't alert**) and **Auto-close after**. + +## Make the query window longer than the reporting interval + +The query window and the pending duration together decide when an absence fires: + +- **The query's time window** decides when an instance counts as missing. As long as the window still contains old datapoints, the instance counts as reporting; it goes missing only after the window slides past its last datapoint. +- **Pending duration** then runs as usual before the instance fires. + +So an instance alerts roughly **query window + pending duration** after its last datapoint. Leave the pending duration empty and it fires about one query window after the data stops. + +Make the window several times the reporting interval. If instances report every 60 seconds and the window is also 60 seconds, one slightly late datapoint makes a healthy instance flap between reporting and missing. A 5-minute window over a 60-second heartbeat absorbs the jitter and still alerts within minutes. + +## Example: a fleet heartbeat + +Say every host emits a heartbeat metric once a minute, and you want to know when any of them stops. One rule covers the whole fleet: + +1. Query the heartbeat metric (a count of datapoints works well) with **Group By** `host_name` and a 5-minute time window. +2. Set **Evaluate every** to `1m` and **Pending duration** to `2m`. +3. Turn on **Alert when an instance stops reporting** and save. + +Here's what happens when `web-7` drops off the network at 10:03, right after its last heartbeat: + +- Until about 10:08, the 5-minute window still contains datapoints from `web-7`, so it counts as reporting. +- At 10:09 the window is empty for `web-7`, and the instance enters **Pending**. +- At 10:11, after the 2-minute pending duration, `web-7` starts **Firing** with the reason `instance stopped reporting; last seen 2026-08-03T10:08:12Z`. With a routing rule that doesn't group, the rule's notification thread lists `host_name=web-7`. +- If `web-9` goes quiet at 10:20, it fires the same way. It joins the same thread under that ungrouped rule, or opens its own under a rule grouped by `host_name`. +- When `web-7` comes back, its instance resolves automatically. Nothing to clean up. + +That's about eight minutes after the last datapoint: the 5-minute window plus the 2-minute pending duration, rounded up to the next evaluation. New hosts need no registration; each is tracked from the first heartbeat it sends. + +## The Instances tab + +The alert detail page's **Instances** tab shows every tracked instance with a **Last seen** column: a relative time, with the absolute timestamp on hover. It records when the instance last appeared in query results, so with a lookback window it can read up to one window later than the instance's final datapoint. While an instance is firing for absence, its row shows the reason `instance stopped reporting; last seen `. + +### Dismiss an instance that was terminated on purpose + +Autoscalers scale in and hosts get decommissioned. When an instance went away on purpose, close its alert with one click instead of waiting out the auto-close window. Admins see a **Dismiss** action on any instance that stopped reporting; the confirmation reads: + +> Stop tracking this instance? Its alert closes and it won't fire again unless it reports again. Use this when the instance was terminated on purpose. + +Dismissal forgets the instance; it isn't a mute. If the instance ever reports again, tracking resumes automatically, so a dismissal can't permanently hide a live host. The instance's alert closes right away, and the alert group and any linked ticket close within one evaluation interval. Dismissing an instance that's already gone shows "Already dismissed." and changes nothing. + +If the instance is coming back later, after planned maintenance or a reboot, use a [silence](../silences/) or [maintenance window](../maintenance-windows/) instead. Both apply to absence alerts like any other: the instance keeps its state, notifications are withheld, and it recovers on its own when it reports again. + +## Auto-close and history reasons + +Auto-close applies to partial absence, when the rule can still see other instances. A silent instance sitting among reporting ones is genuinely gone, so if it's neither dismissed nor heard from again, it closes on its own once the **Auto-close after** window passes. Auto-close and dismissal both forget the instance, so if it ever reports again it's tracked again from scratch. + +If the query returns nothing at all, no instance is auto-closed. Every tracked instance is held, firing, keeping its own labels and grouping, until data returns or you dismiss it. A completely dark query proves nothing about any individual instance, because the collection pipeline itself may be broken, so it isn't safe to conclude that any host is gone. + +On a rule that watches a single instance, auto-close therefore never fires in practice: that one instance going quiet is what makes the query dark. Use **Dismiss** for a host you decommissioned on purpose. + +Every transition lands in the alert's **History** tab with a reason: + +| Reason | What happened | +|---|---| +| `instance stopped reporting; last seen ` | The instance went silent and started firing. | +| `auto-closed: instance silent past absence retention` | The instance stayed silent past the **Auto-close after** window while other instances kept reporting, and closed on its own. | +| `dismissed: instance deregistered by user` | An admin dismissed the instance. | + +## Other behaviors you might notice + +- **Tracking starts at first sight.** An instance is tracked from the first time it appears in results after the setting is on. Anything that stopped reporting earlier is never tracked, so enable the toggle while the fleet is healthy. +- **New instances get a short grace period.** An instance appearing for the first time can't fire for absence during its first few evaluations, so a host that reports once mid-provisioning doesn't alert while it settles. +- **Editing the query resets tracking.** Changing the rule's queries clears the tracked set, which is relearned from the next results. Changing the toggle or the auto-close window resets nothing. +- **A failing datasource doesn't mark instances missing.** If the query errors, the rule follows **Alert state if Error** and tracked instances are left as they were. +- **If the whole query goes dark, every tracked instance fires and none of them auto-close.** They stay firing, each with its own labels and grouping, until data returns or you dismiss them. Auto-close only steps in when other instances are still reporting. +- **Some datasources never drop a series.** CloudWatch, with dimensions listed explicitly in the rule, keeps returning the series with empty values, so those instances always count as reporting. The **Alert state if No data** setting is what catches them. + +## Related + + + + + + + diff --git a/src/content/docs/docs/alerts/maintenance-windows.mdx b/src/content/docs/docs/alerts/maintenance-windows.mdx index 41dd34ff..32c6f649 100644 --- a/src/content/docs/docs/alerts/maintenance-windows.mdx +++ b/src/content/docs/docs/alerts/maintenance-windows.mdx @@ -4,9 +4,10 @@ description: "Schedule recurring or one-time maintenance windows that suppress a sidebar: order: 9 --- -A Maintenance Window defines a scheduled period during which matching alerts continue to evaluate but their notifications are suppressed. State changes are still recorded in alert history (marked with `silenced_at` / `silenced_by_*` so you can audit what happened), but no notification fires. Like a [Silence](../silences/), a maintenance window is a **notification gate**: it mutes notifications for the matching alerts but never changes their state or resolves them. -Use a maintenance window when you have a calendar-driven quiet period: a planned deployment, a recurring batch job window, a vendor outage you've been told to expect. For one-off, ad-hoc suppression in the moment, use [Silences](../silences/) instead. +A Maintenance Window is a scheduled period during which matching alerts keep evaluating but their notifications are suppressed. State changes are still recorded in alert history (marked with `silenced_at` / `silenced_by_*` so you can audit what happened), but no notification fires. Like a [Silence](../silences/), a maintenance window is a **notification gate**: it mutes notifications for the matching alerts but never changes their state or resolves them. + +Use a maintenance window when you have a calendar-driven quiet period: a planned deployment, a recurring batch job window, or a vendor outage you've been told to expect. For one-off, ad-hoc suppression in the moment, use [Silences](../silences/) instead. ## Purpose @@ -16,40 +17,40 @@ Use a maintenance window when you have a calendar-driven quiet period: a planned ## Overview page -The overview page displays all created maintenance windows with these details: +The overview page lists every maintenance window with these details: -- **Name** — Name of the maintenance window. -- **Description** — Description of the maintenance window. -- **Alert rule targeted** — Alert rule to which the maintenance window applies. -- **Matching label** — Labels assigned to the maintenance window. -- **Schedule** — Start and end time period for the maintenance window. -- **Status** — Whether the maintenance window is active or paused. +- **Name:** name of the maintenance window. +- **Description:** description of the maintenance window. +- **Alert rule targeted:** the alert rule the maintenance window applies to. +- **Matching label:** labels assigned to the maintenance window. +- **Schedule:** start and end time for the maintenance window. +- **Status:** whether the maintenance window is active or paused. ![Maintenance window overview](./images/alarm-maintenance-window-1.png) ## Creating a maintenance window -1. Navigate to the **Maintenance Windows** page in the Alerts module. +1. Open the **Maintenance Windows** page in the Alerts module. 2. Click **Create Maintenance Window**. ![Create Maintenance Window](./images/alarm-maintenance-window-1.jpeg) 3. Configure these details: -- **Name** — Name for the maintenance window. -- **Description** — Short description of the maintenance window. -- **Timezone** — Timezone for the scheduled maintenance window. -- **One time** — A one-time maintenance window schedules a maintenance period only once for a specific date and time. -- **Recurring** — A recurring maintenance window automatically repeats on a regular cycle, eliminating the need to configure it each time. +- **Name:** a name for the maintenance window. +- **Description:** a short description of the maintenance window. +- **Timezone:** the timezone for the scheduled window. +- **One time:** schedules the maintenance period once, for a specific date and time. +- **Recurring:** repeats automatically on a regular cycle, so you don't reconfigure it each time. ![Maintenance window timing options](./images/alarm-maintenance-window-2.jpeg) Additional settings: -- **Start time** — Date and time when the maintenance window begins. -- **End time** — Date and time when the maintenance window ends. -- **Frequency** — Available only for recurring windows; select daily, weekly, or monthly schedules. -- **Label Matchers** — Target the window to specific alerts by their labels (e.g. service name, host, region). Each matcher is a label key, an operator, and a value, combined with AND. The operators are the same four used across the alert module — **Equals**, **Not equals**, **Matches regex**, **Doesn't match regex** — matched against alert rule labels and alert instance labels. +- **Start time:** date and time when the maintenance window begins. +- **End time:** date and time when the maintenance window ends. +- **Frequency:** available only for recurring windows; select a daily, weekly, or monthly schedule. +- **Label Matchers:** target the window to specific alerts by their labels (for example, service name, host, or region). Each matcher is a label key, an operator, and a value, combined with AND. The operators are the same four used across the alert module (**Equals**, **Not equals**, **Matches regex**, **Doesn't match regex**), matched against alert rule labels and alert instance labels. 4. Click **Save** to create the new maintenance window. diff --git a/src/content/docs/docs/alerts/routing-rules.mdx b/src/content/docs/docs/alerts/routing-rules.mdx index cf3b71c0..842922ee 100644 --- a/src/content/docs/docs/alerts/routing-rules.mdx +++ b/src/content/docs/docs/alerts/routing-rules.mdx @@ -7,42 +7,42 @@ sidebar: import { Steps } from '@astrojs/starlight/components'; -Routing Rules decide which channels get notified for which alerts. They govern **every** alert notification — both **individual alerts** and **correlated alert groups**. Each rule matches alerts by their labels; the **first matching rule wins** and sends those alerts to its destination channels. Whether the matched alerts notify one-by-one or fold into a single [Alert Group](../alert-groups/) is decided by the rule's **group-by** keys. +Routing rules decide which channels get notified for which alerts. They govern **every** alert notification, both **individual alerts** and **correlated alert groups**. Each rule matches alerts by their labels; the **first matching rule wins** and sends those alerts to its destination channels. Whether the matched alerts notify one by one or fold into a single [Alert Group](../alert-groups/) comes down to the rule's **Alert grouping** setting: deterministic group-by keys in **Static** mode, or the correlation engine in **Auto (AI)** mode. :::note[Not the incident routing rules] -This page routes alert *notifications* to channels. To control how an alert becomes an incident inside Incident Management — its service, escalation policy, and severity — see [Incident Management → Routing Rules](../../incident-management/routing-rules/) instead. +This page routes alert *notifications* to channels. To control how an alert becomes an incident inside Incident Management (its service, escalation policy, and severity), see [Incident Management → Routing Rules](../../incident-management/routing-rules/) instead. ::: -Routing Rules replace the older **Notification Policies** flow. If you previously routed by tags, your existing policies were migrated to routing rules during upgrade — the matching logic now keys off [labels](../create-alerts/#4-folder-severity-and-labels) instead of free-form tags. +Routing rules replace the older **Notification Policies** flow. If you previously routed by tags, KloudMate migrated those policies to routing rules during the upgrade; the matching logic now keys off [labels](../create-alerts/#4-notifications) instead of free-form tags. ## Where to find them -Open **Alerts → Routing rules** in the left navigation. Rules are evaluated in priority order (lowest number wins). The first rule an alert matches decides which channels notify and how often — and, through its group-by keys, whether the alert notifies on its own or correlates with others into one [Alert Group](../alert-groups/). +Open **Alerts → Routing rules** in the left navigation. Rules are evaluated in priority order (lowest number wins). The first rule an alert matches decides which channels notify and how often, and whether the alert notifies on its own or correlates with others into one [Alert Group](../alert-groups/). ![Routing Rules list](./images/routing-rules-list.png) -The list shows every routing rule in the workspace, sorted by priority (lower wins). The **default passthrough rule** sits at the bottom — it matches everything that didn't already match a higher-priority rule and can't be deleted. +The list shows every routing rule in the workspace, sorted by priority (lower wins). The **default passthrough rule** sits at the bottom. It matches everything that didn't already match a higher-priority rule, and it can't be deleted. ## Anatomy of a rule -A routing rule has six parts: +A routing rule has these parts: | Field | What it does | |---|---| | **Name** | How the rule appears in the list and audit log. | | **Priority** | Lower numbers win. When an alert matches multiple rules, the highest-priority (lowest-numbered) rule routes it. | | **Matchers** | Label conditions that pick which alerts this rule applies to. | -| **Group-by** | Whether and how matched alerts correlate. Leave it empty (or set only `alarm_id`) for **one notification per alarm rule** — no grouping. Add a real label key (e.g. `service`) to **correlate** matching alerts that share that value into one [Alert Group](../alert-groups/). `severity` can't be used here. | -| **Cadence** | `group_wait` (how long to wait after the first signal before sending the first notification) and `group_interval` (how often to re-notify on an open group). | +| **Group-by** | In **Static** grouping, whether and how matched alerts correlate. Leave it empty (or set only `alarm_id`) for **one notification per alert rule**, with no grouping. Add a real label key (for example `service`) to **correlate** matching alerts that share that value into one [Alert Group](../alert-groups/). `severity` can't be used here. This field is ignored when the rule uses **Auto (AI)** grouping. | +| **Cadence** | `group_wait` (how long to wait after the first alert before sending the first notification) and `group_interval` (how often to re-notify on an open group). | | **Destination channels** | One or more notification channels that receive the dispatched payload. | Optional: -- **Auto-RCA** — toggle to run KloudMate's AI investigator on every group this rule opens. See [Auto-RCA](../auto-rca/). +- **Auto-RCA:** toggle to run KloudMate's AI investigator on every group this rule opens. See [Auto-RCA](../auto-rca/). ### Matchers -Each matcher is one label condition — a label key, an operator, and a value. An alert has to satisfy **every** matcher you add (they combine with AND) before this rule handles it. Leave the list empty and the rule becomes a catch-all: it handles every alert no higher-priority rule already caught. +Each matcher is one label condition: a label key, an operator, and a value. An alert has to satisfy **every** matcher you add (they combine with AND) before this rule handles it. Leave the list empty and the rule becomes a catch-all: it handles every alert no higher-priority rule already caught. The available operators: @@ -53,47 +53,62 @@ The available operators: | Matches regex | `matches_regex` | Label value matches the regular expression. | | Doesn't match regex | `not_matches_regex` | Label value doesn't match the regex (also matches alerts without the label). | -A matcher value is always a single string. To match any of several values, use a regex alternation — for example `service` **Matches regex** `api|web` matches either service. (There's no `in`/`not_in` operator; the regex form replaces it.) +A matcher value is always a single string. To match any of several values, use a regex alternation. For example, `service` **Matches regex** `api|web` matches either service. (There's no `in`/`not_in` operator; the regex form replaces it.) -As you add or edit matchers, the editor shows a live **Affected Alert Instances** preview — the alert instances this rule currently matches, with a running match count — so you can confirm the rule's scope before saving. +As you add or edit matchers, the editor shows a live **Affected Alert Instances** preview: the alert instances this rule currently matches, with a running count, so you can confirm the rule's scope before saving. + +### Alert grouping: Static or Auto (AI) + +Each rule has an **Alert grouping** control, and it defaults to **Static**. + +- **Static** groups matching alerts by the label keys you choose, the deterministic behavior described under [Group-by keys](#group-by-keys) below. Alerts that share those key values fold into one group. +- **Auto (AI)** hands grouping to the correlation engine. It correlates related alerts from this rule into a single incident on its own, and learns which alerts tend to fire together over time. In this mode the Group-by keys picker is hidden, so you don't pick grouping keys at all. + +In-product, the Auto option reads: *"AI and ML automatically correlate related alerts from this rule into a single incident, learning which alerts tend to fire together over time. No grouping keys needed."* + +Leave a rule on **Static** when you want predictable grouping along keys you control. Switch it to **Auto (AI)** when related alerts don't share a clean label to group on, or when you'd rather let the engine find the connections. Either mode produces the same [Alert Group](../alert-groups/); that page shows what the grouped incident looks like and how an Auto-correlated group explains itself. + +For how each mode decides what belongs together, including why one rule firing on many hosts stays a single Auto group, see [How Alert Grouping Works](../how-alert-grouping-works/). + +![Routing rule editor with Alert grouping set to Auto (AI) and no Group-by keys picker shown](./images/routing-rules-alert-grouping.png) ### Group-by keys -Group-by decides **whether** matched alerts correlate and, if so, along **which** dimension: +Group-by keys apply in **Static** mode. They decide **whether** matched alerts correlate and, if so, along **which** dimension: -- **Leave it empty — or set only `alarm_id`** — and the rule doesn't correlate anything: each alarm rule notifies on its own, one notification per rule. These notifications surface through the normal alerts UI, not the [Alert Groups](../alert-groups/) page. -- **Add a real label key** and matching alerts that share that key's value fold into a single alert group. The editor suggests common keys — `service`, `env`, `host`, `region`, `team`, `issue_id`, `check_id`, `alarm_rule_folder_id` — and you can type any other label key. +- **Leave it empty, or set only `alarm_id`,** and the rule doesn't correlate anything: each alert rule notifies on its own, one notification per rule. These notifications surface through the normal alerts UI, not the [Alert Groups](../alert-groups/) page. +- **Add a real label key** and matching alerts that share that key's value fold into a single alert group. The editor suggests common keys (`service`, `env`, `host`, `region`, `team`, `issue_id`, `check_id`, `alarm_rule_folder_id`), and you can type any other label key. -Two constraints the editor enforces: +The editor enforces two constraints: -- If you add a real dimension alongside `alarm_id`, the `alarm_id` is dropped automatically — pinning each group to a single alarm would defeat the correlation. +- If you add a real dimension alongside `alarm_id`, KloudMate drops the `alarm_id` automatically. Pinning each group to a single alert rule would defeat the correlation. - `severity` can't be a grouping dimension. It's a per-instance property fixed when the group opens, not an axis to group on. -You can combine keys (e.g. `service` + `env`) to scope each group tightly. Pick keys based on which dimension changes the most: aggressive grouping (only `service`) reduces notification noise but sacrifices precision in the group title. +You can combine keys (for example, `service` + `env`) to scope each group tightly. Pick keys by which dimension changes the most: aggressive grouping (only `service`) reduces notification noise but sacrifices precision in the group title. ### Cadence -- **group_wait** — default `30s`. The grouping engine holds the first notification for this long after a group opens, so closely related signals fold in before paging. -- **group_interval** — default `5m`. Once a group is open, KloudMate re-notifies at this cadence if new signals continue to arrive. +- **group_wait:** default `30s`. The grouping engine waits this long after a group opens before sending the first notification, so closely related alerts have time to fold in. +- **group_interval:** default `5m`. Once a group is open, KloudMate re-notifies at this cadence if new alerts continue to arrive. ## The default passthrough rule Every workspace has a default passthrough rule pinned to the bottom of the list. It matches everything that didn't match a higher-priority rule. You can edit its destination channels and cadence, but you can't delete it or change its matchers. -If you want a clean "everything else goes to Slack" path, point the default passthrough at your fallback Slack channel. +For a clean "everything else goes to Slack" path, point the default passthrough at your fallback Slack channel. ## Create a routing rule 1. **Open Alerts → Routing rules** and click **Create rule**. -2. **Name the rule** and set its priority. Lower numbers win — give specific rules low numbers (10, 20, 30) and generic ones higher numbers. The default passthrough sits at priority 1,000,000 so unmatched alerts always have somewhere to land. +2. **Name the rule** and set its priority. Lower numbers win. Give specific rules low numbers (10, 20, 30) and generic ones higher numbers. The default passthrough sits at priority 1,000,000 so unmatched alerts always have somewhere to land. -3. **Add matchers.** Pick a label key, an operator, and a value. Add as many rows as needed; all matchers are combined with AND. +3. **Add matchers.** Pick a label key, an operator, and a value. Add as many rows as needed; all matchers combine with AND. -4. **Set group-by keys.** Leave this empty to notify once per alarm rule, or add a real label key (start with `service`) to correlate matching alerts into one group. `severity` isn't allowed here. +4. **Choose how alerts group.** Leave **Alert grouping** on **Static** and set group-by keys: empty to notify once per alert rule, or a real label key (start with `service`) to correlate matching alerts into one group (`severity` isn't allowed here). Or switch to **Auto (AI)** to let the correlation engine group related alerts for you, with no keys to set. -5. **Tune cadence.** Leave the defaults (`30s` wait, `5m` interval) for most cases. Tighten them for paging rules, or loosen them for digest-style summaries. +5. **Tune cadence.** Leave the defaults (`30s` wait, `5m` interval) for most cases. Tighten them for time-sensitive notifications, or loosen them for digest-style summaries. 6. **Pick destination channels.** Add one or more channels. If you don't have a [KloudMate Incidents](../../platform/settings/notification-channels/#kloudmate-incidents) channel yet, use the inline **+ Add KloudMate Incidents channel** link in the picker. @@ -106,32 +121,32 @@ If you want a clean "everything else goes to Slack" path, point the default pass The kebab menu on each row offers: -- **Edit** — opens the rule editor. -- **Duplicate** — opens the editor pre-filled with the rule's settings under a "(copy)" name so you can adjust matchers or priority and save as a new rule. -- **Delete** — removes the rule. The delete is **blocked while open alert groups still route through this rule** — resolve or re-route those groups first, then delete. The default passthrough rule can't be deleted at all. +- **Edit:** opens the rule editor. +- **Duplicate:** opens the editor pre-filled with the rule's settings under a "(copy)" name, so you can adjust matchers or priority and save as a new rule. +- **Delete:** removes the rule. The delete is **blocked while open alert groups still route through this rule**; resolve or re-route those groups first, then delete. The default passthrough rule can't be deleted at all. -The **Enabled** toggle in each row disables the rule without deleting it — useful for staging a rule before enabling it. +The **Enabled** toggle disables a rule without deleting it. Use it to stage a rule before it goes live. ## Suggest rules -The list header has a **Suggest rules** button (visible to admins only). Clicking it opens a drawer with up to five AI-suggested rules based on your workspace's recent alert traffic. +The list header has a **Suggest rules** button (visible to admins only). It opens a drawer with up to five AI-suggested rules based on your workspace's recent alert traffic. ![Suggest rules drawer](./images/routing-rules-suggest.png) Each suggestion shows: -- A title describing the proposed grouping (e.g. "Group by service + env"). -- A short rationale — how much notification noise the rule would have collapsed over the lookback window. +- A title describing the proposed grouping (for example, "Group by service + env"). +- A short rationale: how much notification noise the rule would have collapsed over the lookback window. - A **preview matches** count for that window. - **Accept** opens the create-rule form pre-filled with the suggestion's matchers, group-by keys, and cadence, under a name derived from what the rule targets. **Dismiss** drops the card. -Some suggestions are grouping-only: they propose group-by keys with no matchers. Accepting one starts the rule as a catch-all that groups everything by those keys — add matchers before saving if you want to scope it to a subset of alerts. +Some suggestions are grouping-only: they propose group-by keys with no matchers. Accepting one starts the rule as a catch-all that groups everything by those keys. Add matchers before saving to scope it to a subset of alerts. -Suggestions don't persist — closing the drawer discards them. Re-run the suggester whenever you want fresh proposals. +Suggestions don't persist; closing the drawer discards them. Re-run the suggester whenever you want fresh proposals. ## Related -- [Alert Groups](../alert-groups/) — what routing rules produce. -- [Silences](../silences/) — temporarily suppress matching alerts. -- [Notification Channels](../../platform/settings/notification-channels/) — set up Slack, KloudMate Incidents, email, and more. -- [Auto-RCA](../auto-rca/) — automatic investigations on group open. +- [Alert Groups](../alert-groups/): what routing rules produce. +- [Silences](../silences/): temporarily suppress matching alerts. +- [Notification Channels](../../platform/settings/notification-channels/): set up Slack, KloudMate Incidents, email, and more. +- [Auto-RCA](../auto-rca/): automatic investigations on group open. diff --git a/src/content/docs/docs/alerts/silences.mdx b/src/content/docs/docs/alerts/silences.mdx index 1a1ce902..3462be87 100644 --- a/src/content/docs/docs/alerts/silences.mdx +++ b/src/content/docs/docs/alerts/silences.mdx @@ -5,17 +5,17 @@ sidebar: order: 7 --- -A **Silence** suppresses notifications for alerts whose labels match the silence's matchers, for a bounded time window. Use silences when you know an alert will be noisy and you don't want it paging anyone — a deployment expected to spike error rates, a known third-party outage, or a maintenance window you can't fully scope. +A **Silence** suppresses notifications for alerts whose labels match the silence's matchers, for a bounded time window. Use silences when you know an alert will be noisy and you don't want it notifying anyone: a deployment expected to spike error rates, a known third-party outage, or a maintenance window you can't fully scope. -Silences cap at 30 days. The cap exists so silences don't get forgotten about and quietly mask real problems — re-create one explicitly if you need it longer. +Silences cap at 30 days. The cap keeps a silence from being forgotten and quietly masking real problems. Re-create one explicitly if you need it longer. ## Where to find silences -Open **Alerts → Silences** in the left navigation. Silences are ad-hoc — for recurring quiet windows use [Maintenance Windows](../maintenance-windows/) instead. +Open **Alerts → Silences** in the left navigation. Silences are ad-hoc; for recurring quiet windows, use [Maintenance Windows](../maintenance-windows/) instead. ![Silences list](./images/silences-list.png) -The list has three tabs with live counts: **Active (N)** (default), **Expired (N)**, and **All (N)**. Columns: **Matchers** (chips, with overflow tooltip), **Expires** (countdown like e.g. *"In 4h"* or *"Expired 12m ago"*), **Scope** (a `Standalone` or `Group-bound` chip), **Reason**, **Created by**. +The list has tabs with live counts: **Active (N)** (default), **Expired (N)**, and **All (N)**. Columns: **Matchers** (chips, with an overflow tooltip), **Expires** (a countdown such as *"In 4h"* or *"Expired 12m ago"*), **Scope** (a `Standalone` or `Group-bound` chip), **Reason**, and **Created by**. ## Anatomy of a silence @@ -23,18 +23,18 @@ The list has three tabs with live counts: **Active (N)** (default), **Expired (N |---|---| | **Matchers** | Label conditions that pick which alerts the silence applies to. Same operator set as [Routing Rules](../routing-rules/#matchers): Equals, Not equals, Matches regex, Doesn't match regex. | | **Expires at** | Date and time the silence stops. Capped at 30 days from creation. | -| **Reason** | Free-form text — recommended so others know why the silence exists. | +| **Reason** | Free-form text; recommended so others know why the silence exists. | | **Scope** | Either **Group-bound** (auto-expires when a specific [Alert Group](../alert-groups/) resolves) or **Standalone** (lives until its `expires_at`). | ## Create a silence -There are a few ways to create a silence: +Create a silence from the Silences page, an alert group, or an alert's Pause Notifications. ### From Alerts → Silences 1. Open **Alerts → Silences** and click **New silence**. -2. Add one or more matcher rows. For each: pick a label key, an operator, and a value. -3. Pick an **Expires at** date/time. The form caps duration at 30 days and disables submit otherwise. +2. Add one or more matcher rows. For each, pick a label key, an operator, and a value. +3. Pick an **Expires at** date and time. The form caps duration at 30 days and disables submit otherwise. 4. Add a **Reason** describing why this silence exists. 5. Click **Save**. The silence appears in the **Active** tab. @@ -44,33 +44,33 @@ On any [Alert Group](../alert-groups/) detail page, click **Silence this group** - The group's labels populated as matchers. - A hidden `auto_expire_group_id` set to the current group, so the silence stops automatically when the group resolves. -- A banner above the form reads e.g. "This silence will auto-expire when group `` resolves." +- A banner above the form that reads something like "This silence will auto-expire when group `<title>` resolves." Save it from here. The silence shows up as **Group-bound** on the list. ### From an alert's Pause Notifications -On an alert rule's **more options (⋯) → Pause Notifications**, KloudMate creates a silence scoped to that alert (an `alarm_id` matcher) with an expiry — the alert keeps evaluating, only its notifications are suppressed. While paused, the rule is marked **Silenced** on the alerts list and its detail page, and the silence appears here in the **Active** tab, where you view or end it. Add matchers to narrow the pause to specific instances. +On an alert rule's **more options (⋯) → Pause Notifications**, KloudMate creates a silence scoped to that alert (an `alarm_id` matcher) with an expiry: the alert keeps evaluating, and only its notifications are suppressed. While paused, the rule is marked **Silenced** on the alerts list and its detail page, and the silence appears here in the **Active** tab, where you view or end it. Add matchers to narrow the pause to specific instances. ## Silence vs. maintenance window -Both features are a **notification gate, not a state change** — the alert still evaluates on its schedule and its state transitions are still recorded in history; only the notification is withheld. Silencing every firing instance of an alert group keeps the group **Open** (shown as **Muted**) — it does **not** resolve the group, and muting never emits a *resolved* notification. The differences between the two are operational, not behavioral: +Both features are a **notification gate, not a state change**: the alert still evaluates on its schedule and its state transitions are still recorded in history; only the notification is withheld. Silencing every firing instance of an alert group keeps the group **Open** (shown as **Muted**); it does **not** resolve the group, and muting never emits a *resolved* notification. The differences between the two are operational, not behavioral: - **Silences** are ad-hoc, label-matcher driven, capped at 30 days, and can be auto-bound to a specific alert group so they expire when the group resolves. -- **Maintenance Windows** are scheduled — one-time (RFC3339 start/end) or recurring (`DAILY` / `WEEKLY` / `MONTHLY` with a timezone-aware time-of-day). +- **Maintenance Windows** are scheduled: one-time (RFC3339 start/end) or recurring (`DAILY` / `WEEKLY` / `MONTHLY` with a timezone-aware time-of-day). -Both use the same four label-matcher operators — **Equals**, **Not equals**, **Matches regex**, **Doesn't match regex** — matched against alert rule labels and alert instance labels. +Both use the same four label-matcher operators (**Equals**, **Not equals**, **Matches regex**, **Doesn't match regex**), matched against alert rule labels and alert instance labels. -Both leave the alert evaluating. Suppressed state transitions are marked with `silenced_at` and `silenced_by_*` columns visible on the alert state history, so you can always tell after the fact whether a quiet stretch was a real recovery or a suppressed notification. +Both leave the alert evaluating. Suppressed state transitions are marked with `silenced_at` and `silenced_by_*` columns on the alert state history, so you can always tell after the fact whether a quiet stretch was a real recovery or a suppressed notification. Pick a silence for a reactive, one-off suppression you'll re-evaluate within 30 days. Pick a [Maintenance Window](../maintenance-windows/) for planned downtime, recurring quiet hours, or anything that needs a calendar schedule. ## Viewing and ending a silence -Each row's kebab menu has **View** and **End silence**. Ending an active silence removes suppression for matching alerts immediately — new firings notify as normal. +Each row's kebab menu has **View** and **End silence**. Ending an active silence removes suppression for matching alerts immediately; new firings notify as normal. ## Related -- [Alert Groups](../alert-groups/) — Silence-this-group action. -- [Maintenance Windows](../maintenance-windows/) — scheduled, recurring counterpart to silences. -- [Routing Rules](../routing-rules/) — same matcher model. +- [Alert Groups](../alert-groups/): the Silence-this-group action. +- [Maintenance Windows](../maintenance-windows/): the scheduled, recurring counterpart to silences. +- [Routing Rules](../routing-rules/): the same matcher model. diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/index.mdx b/src/content/docs/docs/apm-and-tracing/apm-views/index.mdx index 7167adfe..2933a24c 100644 --- a/src/content/docs/docs/apm-and-tracing/apm-views/index.mdx +++ b/src/content/docs/docs/apm-and-tracing/apm-views/index.mdx @@ -24,8 +24,8 @@ It is built from tracing data and RED-style operational metrics. Use APM when yo ## Setup Paths +- [Auto-Instrumentation](../auto-instrumentation/) to have the KloudMate Agent instrument each service with eBPF or an SDK, with no code change - [KloudMate Agent eBPF Observability](../../kloudmate-agent/ebpf-observability/) for kernel-level capture without code changes or restarts -- [Kubernetes APM with KloudMate Agent](../auto-instrumentation/kubernetes-apm/) for the preferred Kubernetes APM flow through the KloudMate Agent - [Manual Instrumentation](../manual-instrumentation/) for SDK-based tracing and custom spans ## Where Metrics Fit diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/dotnet.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/dotnet.mdx deleted file mode 100644 index 3b4bae24..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/dotnet.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: ".NET Auto Instrumentation" -description: "Send auto-instrumented .NET traces to KloudMate." -sidebar: - label: ".NET" ---- - -KloudMate natively supports OpenTelemetry (OTel) for auto-instrumenting .NET applications. Because OpenTelemetry actively maintains robust auto-instrumentation agents for .NET, we rely on their standard process to ensure you always have the most up-to-date and compatible tracing capabilities. - -## Routing Data to KloudMate - -To send auto-instrumented data to KloudMate, you only need to configure the OpenTelemetry agent with KloudMate's OTLP endpoint and your API key using environment variables. - -When running your .NET application, provide the following environment variables: - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-dotnet-service-name" -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -## Setup Instructions - -For the actual installation and usage instructions of the OpenTelemetry .NET auto-instrumentation tools, please refer to the official OpenTelemetry documentation. - -[**View Official OpenTelemetry .NET Zero-code Instrumentation Docs →**](https://opentelemetry.io/docs/zero-code/net/) diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/ebpf.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/ebpf.mdx deleted file mode 100644 index b7345c69..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/ebpf.mdx +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: "Zero-Downtime Observability (eBPF)" -description: "Get APM metrics, traces, and a service map from the Linux kernel with eBPF, with no code changes and no application restarts." -sidebar: - label: "eBPF (Zero-Downtime)" ---- - -Traditional APM agents usually need a code change or an application restart before they collect anything. KloudMate's eBPF approach avoids both: it runs in the Linux kernel and observes the system's behavior without altering your user-level applications. - -Enable the eBPF receiver on the KloudMate Agent and you get metrics, traces, and a service map for your infrastructure with no code changes, no per-service configuration, and no restarts. - -## What you get - -The eBPF receiver reads activity from the kernel and turns it into telemetry. As soon as it's running: - -### Universal RED metrics -Request Rate, Error Rate, and Duration (latency) for every observed service. -- **Protocol-aware:** HTTP, HTTP/2, gRPC, MySQL, PostgreSQL, Redis, MongoDB, Kafka, and Elasticsearch. -- **Operational context:** each metric carries markers such as HTTP status codes, gRPC status flags, and database operation type. - -### Distributed tracing -- Links an incoming request to the outbound calls it makes, for example an HTTP handler querying a database. -- Produces OpenTelemetry-compliant spans, which build the service map. - -### Service inventory and metadata -- Detects each process's language (`km.apm.runtime.language`) without touching the binary. -- Adds host IDs, process IDs, and cloud-provider metadata to traces. -- On Kubernetes, correlates telemetry with `namespace`, `pod_name`, `deployment`, and `node_name`. - -### Network observability -- Captures L3/L4 flow metrics: bytes transferred, TCP retransmits, connection-state changes, and packet drops. -- Gives visibility into service-to-service communication for dependency mapping and security review. - -## eBPF compared with SDK instrumentation - -eBPF and language SDKs solve different problems. eBPF gives broad, no-touch coverage at network boundaries; SDKs give deep, in-process detail. They work well together. - -| Feature | KloudMate eBPF receiver | SDK / auto-instrumentation | -| :--- | :--- | :--- | -| **Code changes** | None. Deploy the agent to the Linux node. | SDK dependencies, or agents attached via env vars. | -| **Application restarts** | None. | Rolling restarts to inject the instrumentation. | -| **Setup** | One DaemonSet or VM process per host. | Per-service configuration and library updates. | -| **Language support** | Any language, including Go, Rust, and C++. | Per-language SDKs; compiled languages are harder. | -| **Overhead** | Low; runs in kernel space. | Higher, especially with rich library instrumentation. | -| **Coverage** | Every service the kernel sees on the network. | Un-instrumented services are blind spots. | - -### What eBPF cannot do - -eBPF works at network boundaries and system calls, so it does not see inside your application: -- **Business logic:** it cannot read custom values like `user_id` or `cart_value`, or trace specific function paths. -- **In-process detail:** it does not produce deep stack traces or internal function timing the way an SDK does. - -## Recommended approach - -1. **Start with eBPF.** Deploy the KloudMate Agent to get a service map, network metrics, and RED metrics across every language with no per-service work. -2. **Add SDKs where you need depth.** For the services where you need business-logic context, such as a specific user ID or transaction state, add [Manual Instrumentation](../../manual-instrumentation/). The eBPF and SDK data join into one view. - -## Configuration and setup - -To install the agent and enable eBPF, see the [eBPF observability guide](../../../kloudmate-agent/ebpf-observability/). diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/go.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/go.mdx deleted file mode 100644 index 0bbf26e4..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/go.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Go Auto Instrumentation" -description: "Send auto-instrumented Go traces to KloudMate." -sidebar: - label: "Go" ---- - -KloudMate natively supports OpenTelemetry (OTel) for auto-instrumenting Go applications. Because OpenTelemetry actively maintains robust auto-instrumentation agents for Go, we rely on their standard process to ensure you always have the most up-to-date and compatible tracing capabilities. - -## Routing Data to KloudMate - -To send auto-instrumented data to KloudMate, you only need to configure the OpenTelemetry agent with KloudMate's OTLP endpoint and your API key using environment variables. - -When running your Go application with the OpenTelemetry Go auto-instrumentation agent, provide the following environment variables: - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-go-service-name" -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -## Setup Instructions - -For the actual installation and usage instructions of the OpenTelemetry Go auto-instrumentation tools, please refer to the official OpenTelemetry documentation. - -[**View Official OpenTelemetry Go Zero-code Instrumentation Docs →**](https://opentelemetry.io/docs/zero-code/go/) diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/index.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/index.mdx index ed9ed728..7159eb99 100644 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/index.mdx +++ b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/index.mdx @@ -1,37 +1,73 @@ --- -title: "Auto Instrumentation" -description: "Enable APM for your applications using zero-code or auto-instrumentation methods." +title: "Auto-Instrumentation" +description: "Get traces and RED metrics into KloudMate with no code changes: the KloudMate Agent instruments each service with eBPF or an OpenTelemetry SDK." sidebar: - label: "Overview" - order: 1 + label: "Auto-Instrumentation" + order: 2 --- -Auto-instrumentation (or zero-code instrumentation) allows you to collect telemetry data from your applications without modifying their source code. KloudMate supports multiple pathways for instant observability, ranging from kernel-level eBPF to framework-specific OpenTelemetry agents. +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -Choose the path that fits your environment: +Auto-instrumentation gets traces and RED metrics into the views on this page without you touching your application code. The KloudMate Agent discovers each service and instruments it for you, so you turn tracing on per service instead of editing and redeploying every app. -## Zero-Downtime eBPF (Recommended Foundation) +You pick one of two modes for each service: -If your applications run on Linux (VMs, bare metal, or Kubernetes), the preferred foundational layer for APM is eBPF. It requires no code changes and **no application restarts**, providing universal coverage across all languages. +- **eBPF** captures Rate, Errors, and Duration (RED) metrics and trace spans from the Linux kernel. It works for any language, including Go, and needs **no restart**. +- **SDK** attaches the matching OpenTelemetry (OTel) agent in-process for deeper distributed traces, with spans for your frameworks, database calls, and outbound requests. It needs a **one-time restart**, so you turn it on one service at a time. -- [Zero-Downtime Observability (eBPF)](./ebpf/) +The two share one trace. When a service runs on SDK, the agent stops eBPF from tracing that service's own requests, so you get the richer application spans while eBPF keeps tracing the services around it, all connected. -## Kubernetes Workloads +## eBPF or SDK -If your applications run on Kubernetes, you can use the KloudMate Agent to deploy APM. +eBPF gives broad, no-touch coverage at network and system boundaries. SDKs give deep, in-process detail. Most setups use both: eBPF everywhere, SDK on the services where you need application-level context. -- [Kubernetes APM with KloudMate Agent](./kubernetes-apm/) +| | eBPF | SDK auto-instrumentation | +| :--- | :--- | :--- | +| **Code changes** | None. | None — the agent attaches the OTel agent for you. | +| **Application restarts** | None. | One restart per service, with your consent. | +| **Language support** | Any language, including Go, Rust, and C++. | Java, Node.js, Python, .NET, and PHP. | +| **Overhead** | Low; runs in kernel space. | Higher, with richer library instrumentation. | +| **What you see** | RED metrics, the service map, and spans at request boundaries. | Deep spans: frameworks, database queries, and outbound calls. | -## OpenTelemetry Auto-Instrumentation +### What eBPF cannot do -For richer framework-specific attributes, or environments where eBPF is not an option, KloudMate uses standard OpenTelemetry auto-instrumentation agents. This approach requires attaching an agent to your runtime and **restarting your application**. +eBPF works at network boundaries and system calls, so it does not see inside your application: -Select your language to get started: +- **Business logic:** it cannot read custom values like `user_id` or `cart_value`, or trace a specific function path. +- **In-process detail:** it does not produce internal function timing the way an SDK does. -- [Java](./java/) -- [Python](./python/) -- [Node.js](./node-js/) -- [.NET](./dotnet/) -- [Go](./go/) -- [PHP](./php/) -- [Ruby](./ruby/) +For those, add [manual instrumentation](../manual-instrumentation/) or move the service to SDK mode. + +## Set it up + +Auto-instrumentation runs through the KloudMate Agent. Install the agent, then instrument services from the **Application APM** experience: + +<CardGrid> + <LinkCard title="Application APM overview" href="../../kloudmate-agent/auto-instrumentation/overview/" description="Turn tracing on per service: Off, eBPF, or SDK, with restart consent and automatic rollback." /> + <LinkCard title="Java" href="../../kloudmate-agent/auto-instrumentation/java/" description="systemd, runtime attach, Kubernetes, ECS, and Windows." /> + <LinkCard title="Node.js" href="../../kloudmate-agent/auto-instrumentation/nodejs/" description="systemd, PM2, Kubernetes, ECS, and Windows." /> + <LinkCard title="Python" href="../../kloudmate-agent/auto-instrumentation/python/" description="systemd, PM2, Kubernetes, ECS, and Windows." /> + <LinkCard title=".NET" href="../../kloudmate-agent/auto-instrumentation/dotnet/" description="The CLR profiler on Windows and Linux, Kubernetes, and ECS." /> + <LinkCard title="PHP" href="../../kloudmate-agent/auto-instrumentation/php/" description="PHP 7 and 8 on Linux and Docker; eBPF on Kubernetes." /> + <LinkCard title="Go" href="../../kloudmate-agent/auto-instrumentation/go/" description="Covered by eBPF, no injection." /> + <LinkCard title="Ruby" href="../../kloudmate-agent/auto-instrumentation/ruby/" description="eBPF out of the box, plus deep tracing with the OTel gems." /> + <LinkCard title="eBPF observability" href="../../kloudmate-agent/ebpf-observability/" description="Enable and tune eBPF monitoring across your fleet." /> + <LinkCard title="Sampling" href="../../kloudmate-agent/auto-instrumentation/sampling/" description="Balance trace fidelity against ingestion cost." /> +</CardGrid> + +## Without the agent + +If you run where you can't install a host agent — serverless, or a managed platform — instrument the app yourself and send the traces straight to KloudMate's cloud endpoint. Run your language's OpenTelemetry agent or SDK and point it at `otel.kloudmate.com` with your API key: + +```bash +export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" +export OTEL_SERVICE_NAME="your-service-name" +``` + +Replace `YOUR_API_KEY` with your KloudMate API key. The step-by-step [OpenTelemetry guides](../../../guides/opentelemetry/java/) walk through this for each language, and [Manual Instrumentation](../manual-instrumentation/) covers doing it in code. + +## Related paths + +- [Manual Instrumentation](../manual-instrumentation/) for custom spans, business-logic attributes, and application metrics. +- [APM Views](../apm-views/) and [Trace Explorer](../trace-explorer/) to see the data once it flows. diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/java.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/java.mdx deleted file mode 100644 index 8fc9b777..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/java.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Java Auto Instrumentation" -description: "Send auto-instrumented Java traces to KloudMate." -sidebar: - label: "Java" ---- - -KloudMate natively supports OpenTelemetry (OTel) for auto-instrumenting Java applications. Because OpenTelemetry actively maintains robust auto-instrumentation agents for Java, we rely on their standard process to ensure you always have the most up-to-date and compatible tracing capabilities. - -## Routing Data to KloudMate - -To send auto-instrumented data to KloudMate, you only need to configure the OpenTelemetry agent with KloudMate's OTLP endpoint and your API key using environment variables. - -When running your Java application, provide the following environment variables: - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-java-service-name" -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -## Setup Instructions - -For the actual download and usage instructions of the OpenTelemetry Java agent (`opentelemetry-javaagent.jar`), please refer to the official OpenTelemetry documentation. - -[**View Official OpenTelemetry Java Zero-code Instrumentation Docs →**](https://opentelemetry.io/docs/zero-code/java/agent/) diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/kubernetes-apm.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/kubernetes-apm.mdx deleted file mode 100644 index 57b5967c..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/kubernetes-apm.mdx +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: "Kubernetes APM with KloudMate Agent" -description: "Use the KloudMate Agent to enable APM for Kubernetes workloads, which is the preferred auto-instrumentation path in KloudMate." -sidebar: - label: "Kubernetes APM" - order: 1 ---- -If your applications run on Kubernetes, the preferred way to turn on APM in KloudMate is through the [KloudMate Agent Kubernetes installation guide](../../../kloudmate-agent/installation/kubernetes-agent/#apm-setup-instructions). - -That guide walks you through the supported setup flow for: - -- enabling APM from the KloudMate Agent install experience -- turning on automatic runtime detection -- patching workloads with the correct instrumentation annotations when needed -- collecting traces alongside Kubernetes metrics, logs, and events - -## Preferred Setup Path - -Use [APM Setup Instructions](../../../kloudmate-agent/installation/kubernetes-agent/#apm-setup-instructions) when you want to: - -- enable APM during KloudMate Agent installation -- auto-instrument supported runtimes in Kubernetes -- patch deployments manually when auto-detection does not cover a workload - -## Related Paths - -- [Kubernetes Agent](../../../kloudmate-agent/installation/kubernetes-agent/) -- [APM](../../) -- [Manual Instrumentation](../../manual-instrumentation/) diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/node-js.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/node-js.mdx deleted file mode 100644 index 9359d3bb..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/node-js.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Node.js Auto Instrumentation" -description: "Send auto-instrumented Node.js traces to KloudMate." -sidebar: - label: "Node.js" ---- - -KloudMate natively supports OpenTelemetry (OTel) for auto-instrumenting Node.js applications. Because OpenTelemetry actively maintains robust auto-instrumentation agents for Node.js, we rely on their standard process to ensure you always have the most up-to-date and compatible tracing capabilities. - -## Routing Data to KloudMate - -To send auto-instrumented data to KloudMate, you only need to configure the OpenTelemetry agent with KloudMate's OTLP endpoint and your API key using environment variables. - -When running your Node.js application, provide the following environment variables: - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-nodejs-service-name" -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -## Setup Instructions - -For the actual installation and usage instructions of the OpenTelemetry Node.js tools, please refer to the official OpenTelemetry documentation. - -[**View Official OpenTelemetry Node.js Zero-code Instrumentation Docs →**](https://opentelemetry.io/docs/zero-code/js/) diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/php.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/php.mdx deleted file mode 100644 index 43ba9e99..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/php.mdx +++ /dev/null @@ -1,259 +0,0 @@ ---- -title: "PHP Auto Instrumentation" -description: "Send auto-instrumented PHP traces to KloudMate." -sidebar: - label: "PHP" ---- - -KloudMate natively supports OpenTelemetry (OTel) for auto-instrumenting PHP applications. Because OpenTelemetry actively maintains robust auto-instrumentation agents for PHP, we rely on their standard process to ensure you always have the most up-to-date and compatible tracing capabilities. - -:::tip -For a detailed, step-by-step tutorial covering both Slim and Laravel, see the [Instrument a PHP App](../../../../guides/opentelemetry/php/) guide. -::: - -## Routing Data to KloudMate - -To send auto-instrumented data to KloudMate, you only need to configure the OpenTelemetry agent with KloudMate's OTLP endpoint and your API key using environment variables. - -When running your PHP application, provide the following environment variables (or set them in your environment configuration): - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-php-service-name" -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -## Setup Instructions - -For the actual installation and usage instructions of the OpenTelemetry PHP auto-instrumentation extension, please refer to the official OpenTelemetry documentation. - -[**View Official OpenTelemetry PHP Zero-code Instrumentation Docs →**](https://opentelemetry.io/docs/zero-code/php/) - -## Laravel - -This section provides step-by-step instructions to integrate **OpenTelemetry zero-code instrumentation** into an existing Laravel application and route the data to KloudMate. - -### Prerequisites - -- PHP 8.0+ (PHP 8.2 recommended) -- Composer installed -- An existing Laravel application -- A running KloudMate Agent or OTLP-compatible backend - -### Step 1: Install the OpenTelemetry PHP Extension - -Zero-code instrumentation requires the `opentelemetry` PHP extension to intercept and instrument PHP function calls automatically. - -#### Linux (Ubuntu/Debian) - -```bash -sudo apt-get install php-dev php-pear -sudo pecl install opentelemetry -``` - -#### macOS - -```bash -pecl install opentelemetry -``` - -#### Enable the Extension - -Add the following line to your `php.ini` file: - -```ini -extension=opentelemetry.so -``` - -> **Tip:** Find your `php.ini` location by running `php --ini`. - -#### Verify Installation - -```bash -php -m | grep opentelemetry -``` - -You should see `opentelemetry` in the output. - -### Step 2: Install OpenTelemetry Packages via Composer - -Run the following commands in your Laravel project root directory: - -```bash -composer require open-telemetry/opentelemetry-auto-laravel -composer require open-telemetry/sdk -composer require open-telemetry/exporter-otlp -composer require open-telemetry/api -``` - -#### What each package does - -| Package | Purpose | -|---------|---------| -| `open-telemetry/opentelemetry-auto-laravel` | Automatic zero-code instrumentation hooks for Laravel | -| `open-telemetry/sdk` | OpenTelemetry SDK for trace/metric/log export | -| `open-telemetry/exporter-otlp` | OTLP protocol exporter to send data to collectors/agents | -| `open-telemetry/api` | OpenTelemetry API contracts | - -### Step 3: Configure Environment Variables - -Add the following variables to your Laravel `.env` file: - -```ini -# ============================================ -# OpenTelemetry Configuration -# ============================================ - -# Enable auto-instrumentation (REQUIRED) -OTEL_PHP_AUTOLOAD_ENABLED=true - -# Service identification -OTEL_SERVICE_NAME=my-laravel-app -OTEL_SERVICE_VERSION=1.0.0 - -# Exporter configuration -OTEL_TRACES_EXPORTER=otlp -OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.kloudmate.com:4318 -OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf - -# Also export logs and metrics (optional) -OTEL_LOGS_EXPORTER=otlp -OTEL_METRICS_EXPORTER=otlp - -# Resource attributes -OTEL_RESOURCE_ATTRIBUTES=deployment.environment=development,host.name=localhost -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -#### Important Notes - -| Variable | Description | -|----------|-------------| -| `OTEL_PHP_AUTOLOAD_ENABLED=true` | **Critical.** Enables the auto-loader to register instrumentations automatically. | -| `OTEL_SERVICE_NAME` | Name that identifies your application in the observability backend. | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | URL of your KloudMate OTLP endpoint. | -| `OTEL_EXPORTER_OTLP_HEADERS` | Your KloudMate API key for authentication. | -| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` (default) or `grpc`. Must match what your collector supports. | - -#### Docker to Host Machine (KM Agent on Host) - -If your Laravel app runs in Docker and the KM Agent runs on the host machine, use: - -```ini -OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318 -``` - -> **Note:** Ensure your KM Agent is listening on `0.0.0.0` (all interfaces), not just `127.0.0.1`, otherwise the container cannot reach it. - -### Step 4: Understanding Zero-Code Instrumentation - -The `opentelemetry-auto-laravel` package uses the `opentelemetry` PHP extension to automatically instrument your application **without any code changes**. - -#### What gets automatically instrumented - -- **HTTP Requests** — Incoming requests to controllers and middleware -- **Database Queries** — Eloquent and Query Builder operations -- **Cache Operations** — Redis, Memcached, and file cache calls -- **Queue Jobs** — Job dispatching and processing (sync, database, redis drivers) -- **Exceptions & Errors** — Automatic error tracking and stack traces -- **Outgoing HTTP Requests** — Guzzle HTTP client calls - -#### What you do NOT need to do - -- No manual span creation -- No middleware additions -- No controller modifications -- No service provider registration -- No `use` statements or imports - -### Step 5: Verify the Integration - -#### 1. Confirm the PHP Extension is Active - -```bash -php artisan tinker -``` - -Inside Tinker, run: - -```php -echo extension_loaded('opentelemetry') ? 'YES' : 'NO'; -``` - -Expected output: `YES` - -#### 2. Generate Some Traffic - -Make a request to your application: - -```bash -curl http://localhost:8000/ -``` - -Or visit any route in your browser. - -#### 3. Check Your Observability Backend - -- If using **KM Agent** — check the agent logs or the KloudMate dashboard. -- If using **OTel Collector** — check the collector container/service logs. -- If using **Jaeger / Tempo / Zipkin** — open the UI and search for traces with your service name (`OTEL_SERVICE_NAME`). - -You should see traces, spans, and metadata automatically captured from your Laravel app. - -### Optional: Fine-Tuning & Debugging - -#### Disable Specific Instrumentations - -If you want to disable Laravel auto-instrumentation: - -```ini -OTEL_PHP_DISABLED_INSTRUMENTATIONS=laravel -``` - -#### Switch to gRPC Protocol - -```ini -OTEL_EXPORTER_OTLP_PROTOCOL=grpc -OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.kloudmate.com:4317 -``` - -#### Enable Debug Logging - -If traces are not appearing, enable SDK debug mode: - -```ini -OTEL_LOG_LEVEL=debug -``` - -#### Add Custom Resource Attributes - -```ini -OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,host.name=prod-server-01,service.namespace=my-org -``` - -### Quick Reference: Minimal Configuration - -The absolute minimum required to get zero-code instrumentation working with KloudMate: - -```ini -OTEL_PHP_AUTOLOAD_ENABLED=true -OTEL_SERVICE_NAME=my-laravel-app -OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.kloudmate.com:4318 -OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -``` - -### Summary - -| Step | Action | -|------|--------| -| 1 | Install the `opentelemetry` PHP extension via `pecl` | -| 2 | Install `opentelemetry-auto-laravel` + SDK + Exporter via Composer | -| 3 | Add `OTEL_*` environment variables to your `.env` file | -| 4 | Ensure your KloudMate Agent / Collector is running and accessible | -| 5 | Run your application — instrumentation is fully automatic | - -That's it! Zero-code instrumentation means **zero code changes** in your application. diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/python.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/python.mdx deleted file mode 100644 index 436de444..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/python.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Python Auto Instrumentation" -description: "Send auto-instrumented Python traces to KloudMate." -sidebar: - label: "Python" ---- - -KloudMate natively supports OpenTelemetry (OTel) for auto-instrumenting Python applications. Because OpenTelemetry actively maintains robust auto-instrumentation agents for Python, we rely on their standard process to ensure you always have the most up-to-date and compatible tracing capabilities. - -## Routing Data to KloudMate - -To send auto-instrumented data to KloudMate, you only need to configure the OpenTelemetry agent with KloudMate's OTLP endpoint and your API key using environment variables. - -When running your Python application (e.g., using `opentelemetry-instrument`), provide the following environment variables: - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-python-service-name" -``` - -*Replace `YOUR_API_KEY` with your actual KloudMate API key.* - -## Setup Instructions - -For the actual installation and usage instructions of the OpenTelemetry Python tools (`opentelemetry-bootstrap`, `opentelemetry-instrument`), please refer to the official OpenTelemetry documentation. - -[**View Official OpenTelemetry Python Zero-code Instrumentation Docs →**](https://opentelemetry.io/docs/zero-code/python/) diff --git a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/ruby.mdx b/src/content/docs/docs/apm-and-tracing/auto-instrumentation/ruby.mdx deleted file mode 100644 index c4c53d47..00000000 --- a/src/content/docs/docs/apm-and-tracing/auto-instrumentation/ruby.mdx +++ /dev/null @@ -1,69 +0,0 @@ ---- -title: "Ruby Auto Instrumentation" -description: "Auto-instrument a Ruby or Rails app with OpenTelemetry and send the traces to KloudMate — what gets captured out of the box, and where the SQL comes from." -sidebar: - label: "Ruby" ---- - -KloudMate uses OpenTelemetry to auto-instrument Ruby applications. The instrumentation gems are maintained by the OpenTelemetry project; KloudMate ingests whatever they emit over OTLP, so your coverage tracks the upstream gems. - -## Route data to KloudMate - -Point the OpenTelemetry exporter at KloudMate's OTLP endpoint and your API key through environment variables. When you run your app, set: - -```bash -export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.kloudmate.com:4318" -export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_API_KEY" -export OTEL_SERVICE_NAME="your-ruby-service-name" -``` - -Replace `YOUR_API_KEY` with your KloudMate API key. - -## Install the gems - -Add the SDK, the all-in-one instrumentation bundle, and the OTLP exporter to your `Gemfile`: - -```ruby -gem 'opentelemetry-sdk' -gem 'opentelemetry-instrumentation-all' -gem 'opentelemetry-exporter-otlp' -``` - -Enable them once at boot. For Rails, use an initializer: - -```ruby -# config/initializers/opentelemetry.rb -require 'opentelemetry/sdk' -require 'opentelemetry/instrumentation/all' -require 'opentelemetry/exporter/otlp' - -OpenTelemetry::SDK.configure do |c| - c.use_all # turn on every instrumentation whose library is loaded -end -``` - -The `opentelemetry-exporter-otlp` gem is required. Without it the SDK has no exporter, logs `otlp exporter cannot be configured` at startup, and drops every span. - -## What gets instrumented automatically - -`use_all` turns on each instrumentation whose underlying library is loaded. For a Rails app it covers the whole request path with no extra code: - -| Layer | Gem | What you get | -|---|---|---| -| HTTP server | `rack`, `action_pack` | One span per request — the root of the trace. `action_pack` adds the matched route, controller, and action, and names the span `GET /users/:id` (Rails 7.1+). | -| Database (SQL) | `pg`, `mysql2`, `trilogy` | One span per query, carrying the SQL as `db.statement`. This is the span that shows the query text. | -| Database (ORM) | `active_record` | Timing spans for model operations — `User#save`, `User.create`, `User query`. They show the Rails call, not the SQL. | -| View rendering | `action_view` | Spans for template, partial, collection, and layout renders, tagged with the template path. | -| Background jobs | `active_job`, plus `sidekiq`, `delayed_job`, `resque`, `que` | An enqueue span and a perform span per job, linked across the two processes. | -| Outbound HTTP | `net_http`, `faraday`, `http`, `httpx`, `excon` | A client span for each call your app makes to another service. | -| Caches and stores | `redis`, `dalli`, `mongo` | A span per cache or store operation. | - -The bundle also covers GraphQL, gRPC, Sinatra, Grape, the AWS SDK, and Kafka and RabbitMQ clients when your app uses them. - -:::note -The SQL query text (`db.statement`) comes from the database driver instrumentation (`pg`, `mysql2`), not from `active_record`. SQLite has no such instrumentation, so a SQLite app shows the model operation but no SQL — PostgreSQL and MySQL show the query. -::: - -## Reference - -For setup, configuration options, and the full list of instrumentations, see the [OpenTelemetry Ruby zero-code instrumentation docs](https://opentelemetry.io/docs/zero-code/ruby/). diff --git a/src/content/docs/docs/apm-and-tracing/index.mdx b/src/content/docs/docs/apm-and-tracing/index.mdx index f478672c..7b336332 100644 --- a/src/content/docs/docs/apm-and-tracing/index.mdx +++ b/src/content/docs/docs/apm-and-tracing/index.mdx @@ -18,17 +18,15 @@ The `APM & Tracing` section in KloudMate gives you end-to-end visibility into ho Start in **APM Views** when you need higher-level service health and dependency analysis. Move to **Trace Explorer** to find the trace you care about, then open it in **Trace Detail** to find the root cause. -## Instrumentation Paths +## Get your data in -KloudMate supports multiple ways to generate APM and tracing data: +These views are built from traces and RED metrics. There are two ways to produce them: -- [KloudMate Agent eBPF Observability](../kloudmate-agent/ebpf-observability/) for kernel-level tracing and RED metrics without code changes or restarts -- [KloudMate Agent Application APM](../kloudmate-agent/auto-instrumentation/overview/) to add full distributed tracing to a service, one at a time, and view the results here -- [Auto Instrumentation](./auto-instrumentation/) for the preferred Kubernetes APM flow through the KloudMate Agent -- [Manual Instrumentation](./manual-instrumentation/) for SDK-based tracing, application metrics, and custom metrics with the most control. +- **[Auto-Instrumentation](./auto-instrumentation/)** — the KloudMate Agent instruments each service for you with no code change, using eBPF (no restart, any language) or an OpenTelemetry SDK (deep traces, one restart per service). This is the fastest path to a populated service map. Set it up from the agent's [Application APM](../kloudmate-agent/auto-instrumentation/overview/) experience. +- **[Manual Instrumentation](./manual-instrumentation/)** — use the OpenTelemetry SDK in your own code when you need custom spans, business-logic attributes, and application metrics. :::note -Because Application Metrics and Custom Metrics are emitted through the exact same OpenTelemetry SDK flow as traces, they are configured together. See [Manual Instrumentation](./manual-instrumentation/) for language-specific guides. +Application Metrics and Custom Metrics go through the same OpenTelemetry SDK flow as traces, so they are configured together. See [Manual Instrumentation](./manual-instrumentation/) for language-specific guides. ::: ## Related Paths diff --git a/src/content/docs/docs/apm-and-tracing/manual-instrumentation/ruby.mdx b/src/content/docs/docs/apm-and-tracing/manual-instrumentation/ruby.mdx index 7c0731c5..6a6a17ce 100644 --- a/src/content/docs/docs/apm-and-tracing/manual-instrumentation/ruby.mdx +++ b/src/content/docs/docs/apm-and-tracing/manual-instrumentation/ruby.mdx @@ -32,6 +32,51 @@ export OTEL_SERVICE_NAME="your-ruby-service" ``` *(Replace `KM_AGENT_HOST` with the IP address or hostname of your KloudMate Agent).* +## Instrument the whole request path + +Ruby needs its instrumentation gems added to the app, because Ruby only loads gems listed in the `Gemfile`. Once you add them, the all-in-one bundle traces the whole request path with no per-library code. Add the SDK, the instrumentation bundle, and the OTLP exporter: + +```ruby +gem 'opentelemetry-sdk' +gem 'opentelemetry-instrumentation-all' +gem 'opentelemetry-exporter-otlp' +``` + +Enable them once at boot. For Rails, use an initializer: + +```ruby +# config/initializers/opentelemetry.rb +require 'opentelemetry/sdk' +require 'opentelemetry/instrumentation/all' +require 'opentelemetry/exporter/otlp' + +OpenTelemetry::SDK.configure do |c| + c.use_all # turn on every instrumentation whose library is loaded +end +``` + +The `opentelemetry-exporter-otlp` gem is required. Without it the SDK has no exporter, logs `otlp exporter cannot be configured` at startup, and drops every span. + +### What gets instrumented automatically + +`use_all` turns on each instrumentation whose underlying library is loaded. For a Rails app it covers the whole request path with no extra code: + +| Layer | Gem | What you get | +|---|---|---| +| HTTP server | `rack`, `action_pack` | One span per request — the root of the trace. `action_pack` adds the matched route, controller, and action, and names the span `GET /users/:id` (Rails 7.1+). | +| Database (SQL) | `pg`, `mysql2`, `trilogy` | One span per query, carrying the SQL as `db.statement`. This is the span that shows the query text. | +| Database (ORM) | `active_record` | Timing spans for model operations — `User#save`, `User.create`, `User query`. They show the Rails call, not the SQL. | +| View rendering | `action_view` | Spans for template, partial, collection, and layout renders, tagged with the template path. | +| Background jobs | `active_job`, plus `sidekiq`, `delayed_job`, `resque`, `que` | An enqueue span and a perform span per job, linked across the two processes. | +| Outbound HTTP | `net_http`, `faraday`, `http`, `httpx`, `excon` | A client span for each call your app makes to another service. | +| Caches and stores | `redis`, `dalli`, `mongo` | A span per cache or store operation. | + +The bundle also covers GraphQL, gRPC, Sinatra, Grape, the AWS SDK, and Kafka and RabbitMQ clients when your app uses them. + +:::note +The SQL query text (`db.statement`) comes from the database driver instrumentation (`pg`, `mysql2`), not from `active_record`. SQLite has no such instrumentation, so a SQLite app shows the model operation but no SQL — PostgreSQL and MySQL show the query. +::: + ## Official Documentation OpenTelemetry is continuously evolving. For the most up-to-date SDK instructions, advanced configurations, and custom instrumentation details, refer to the official OpenTelemetry documentation: diff --git a/src/content/docs/docs/aws-integration/account-setup.mdx b/src/content/docs/docs/aws-integration/account-setup.mdx index 91326ccb..c6bf9ab3 100644 --- a/src/content/docs/docs/aws-integration/account-setup.mdx +++ b/src/content/docs/docs/aws-integration/account-setup.mdx @@ -36,4 +36,5 @@ Your AWS account is now connected to KloudMate. - [AWS Integration](../) - [CloudWatch Logs](../cloudwatch-logs/) - [AWS Lambda Monitoring](../lambda-monitoring/) +- [Troubleshoot AWS sync](../../platform/settings/troubleshoot-aws-sync/) - [Sending Data to KloudMate](../../getting-started/sending-data-to-kloudmate/) diff --git a/src/content/docs/docs/aws-integration/images/step-functions-configuration.png b/src/content/docs/docs/aws-integration/images/step-functions-configuration.png new file mode 100644 index 00000000..08da1452 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-configuration.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-execution-flow.png b/src/content/docs/docs/aws-integration/images/step-functions-execution-flow.png new file mode 100644 index 00000000..e62c63a6 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-execution-flow.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-failed-execution.png b/src/content/docs/docs/aws-integration/images/step-functions-failed-execution.png new file mode 100644 index 00000000..d0d955ac Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-failed-execution.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-machine-executions.png b/src/content/docs/docs/aws-integration/images/step-functions-machine-executions.png new file mode 100644 index 00000000..a8e68206 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-machine-executions.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-overview.png b/src/content/docs/docs/aws-integration/images/step-functions-overview.png new file mode 100644 index 00000000..89d6a903 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-overview.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-recent-executions.png b/src/content/docs/docs/aws-integration/images/step-functions-recent-executions.png new file mode 100644 index 00000000..6a136bf6 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-recent-executions.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-state-machines.png b/src/content/docs/docs/aws-integration/images/step-functions-state-machines.png new file mode 100644 index 00000000..32220e50 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-state-machines.png differ diff --git a/src/content/docs/docs/aws-integration/images/step-functions-timeline-log.png b/src/content/docs/docs/aws-integration/images/step-functions-timeline-log.png new file mode 100644 index 00000000..be0f40b5 Binary files /dev/null and b/src/content/docs/docs/aws-integration/images/step-functions-timeline-log.png differ diff --git a/src/content/docs/docs/aws-integration/index.mdx b/src/content/docs/docs/aws-integration/index.mdx index ab0696b7..dd89e278 100644 --- a/src/content/docs/docs/aws-integration/index.mdx +++ b/src/content/docs/docs/aws-integration/index.mdx @@ -29,3 +29,4 @@ Explore the following guides to set up and configure your AWS integrations: - **[CloudWatch Logs](./cloudwatch-logs/)**: Instructions for subscribing to CloudWatch Log groups and making them fully searchable inside KloudMate. - **[AWS Lambda Monitoring](./lambda-monitoring/)**: Explore how KloudMate aggregates Lambda metrics, invocations, and issues in a unified console. - **[Lambda Telemetry Extension](./lambda-telemetry-extension/)**: Send Lambda logs, metrics, and traces to KloudMate with a layer, skipping the cross-account CloudFormation setup. +- **[AWS Step Functions Monitoring](./step-functions-monitoring/)**: Watch state-machine executions, spot failures, and replay the workflow graph to see which state failed. diff --git a/src/content/docs/docs/aws-integration/step-functions-monitoring.mdx b/src/content/docs/docs/aws-integration/step-functions-monitoring.mdx new file mode 100644 index 00000000..17d28c97 --- /dev/null +++ b/src/content/docs/docs/aws-integration/step-functions-monitoring.mdx @@ -0,0 +1,152 @@ +--- +title: "AWS Step Functions Monitoring" +description: "Monitor Step Functions executions across your connected AWS accounts, find failures, and see which state failed." +sidebar: + label: "Step Functions Monitoring" + order: 6 +--- + +Step Functions monitoring shows your state-machine executions, which ones failed, and where they failed. Open a failed run to read the error and follow the path it took through the workflow. + +Open **Step Functions** from the sidebar, under **Serverless**. + +Some of this works the moment you connect an AWS account. The rest needs execution logging turned on: + +- **Inventory and execution counts work automatically.** KloudMate discovers your state machines and reads their execution counts from CloudWatch. There's nothing to configure. +- **Per-execution timeline and workflow graph need execution logging.** To see the events inside a run and the path it took, turn on Step Functions logging in AWS and connect the log group to KloudMate. See [Turn on execution logging](#turn-on-execution-logging). + +## What you can monitor + +- **Execution outcomes** for an account: how many runs started, succeeded, failed, aborted, timed out, or were throttled in the last 24 hours. +- **Per-state-machine health**: executions, errors, and error rate for every machine KloudMate discovers. +- **A single execution end to end**: its status, duration, the states it ran, and the input and output of each one. +- **Failure details**: the failing state, its error code, and its cause, marked on the workflow graph. + +## Before you start + +- A connected AWS account. If you don't have one yet, see [AWS Account Setup](../account-setup/). Inventory and the execution counts start working on their own once an account is connected. +- For the execution timeline and workflow graph: Step Functions **execution logging** at level **ALL** with execution data included, connected in KloudMate. This works the same for Standard and Express workflows. See [Turn on execution logging](#turn-on-execution-logging). + +## The Overview dashboard + +Start on the **Overview** tab to see whether your executions are healthy. It summarizes the last 24 hours across the workspace. + +![The Step Functions Overview tab: an Account and Region picker, six outcome tiles for Started, Succeeded, Failed, Aborted, Timed out, and Throttled, and an Executions chart](./images/step-functions-overview.png) + +A workspace can span several AWS accounts, so set the scope with the **Account** and **Region** menus at the top left. KloudMate reads CloudWatch for that account and region, and the page remembers your choice next time. Use **Refresh** to update the numbers. + +The tab shows: + +- **Outcome tiles**: **Started**, **Succeeded**, **Failed**, **Aborted**, **Timed out**, and **Throttled** counts for the selected scope. +- **Executions chart**: Started, Succeeded, and Failed plotted across the day. +- **Recent executions**: the latest 10 runs, filtered by **All**, **Running**, **Succeeded**, **Failed**, **Aborted**, or **Timed out**. Click a row to open the execution. + +![The Recent executions list with a status filter and columns for execution name, state machine, status, start time, and duration](./images/step-functions-recent-executions.png) + +## The State machines table + +Use the **State machines** tab to find a specific machine or spot the ones that are failing. It lists every state machine KloudMate has discovered in the workspace. Search by name from the box at the top right. + +![The State machines tab: a searchable table with Name, Region, Executions, Errors, and Error Rate columns](./images/step-functions-state-machines.png) + +Each row shows the machine's **Name** and **Region**, with its **Executions**, **Errors**, and **Error Rate** over the last 24 hours. Sort by **Error Rate** to bring the machines with the most errors to the top, then open one to see which runs failed. + +## Inside a state machine + +Opening a state machine shows its type and region at the top, with tabs for Executions, Metrics, and Configuration. + +### Executions + +The **Executions** tab lists that machine's recent runs, each with its status, start time, and duration. Click a run to open its [detail view](#inside-an-execution). + +![A state machine's Executions tab, showing recent runs with status, start time, and duration](./images/step-functions-machine-executions.png) + +### Metrics + +The **Metrics** tab charts the machine's health from CloudWatch: executions by outcome, throttles, and execution time (average, p90, and p99). Adjust the time range from the picker at the top right. + +Express workflows report different metrics. Instead of the outcome and duration charts, they show **Billed duration** and **Billed memory**. See [Standard and Express workflows](#standard-and-express-workflows). + +### Configuration + +The **Configuration** tab shows the machine's ARN, region, type, and account. It's also where you connect its execution logs to KloudMate. + +![The Configuration tab: state machine details above the Execution log ingestion control, which shows a connected vended log group](./images/step-functions-configuration.png) + +Under **Execution log ingestion**, KloudMate looks for the machine's vended log group (`/aws/vendedlogs/states/<name>`) and shows whether it's connected. Click **Connect** to start reading its logs, or **Disconnect** to stop. The timeline and workflow graph are built from these logs. If no vended log group appears, logging isn't on yet in AWS; see [Turn on execution logging](#turn-on-execution-logging). + +## Inside an execution + +Open an execution to see what happened in a single run: the path it took, every event, and, when it failed, why. Each execution has its own URL, so you can share a link to a specific run. + +The header shows the status, start and end times, duration, event count, and the execution ARN. When a run fails, a banner names the failing state, its error code, and the cause. + +![A failed execution: an error banner naming the failing state and cause, an Ask AI button, and the Execution flow graph with the failing state outlined in red](./images/step-functions-failed-execution.png) + +### Execution flow + +The **Execution flow** graph shows the path the run took, built from its logs. A failing state is outlined in red. Click any state to see its input and output. Parallel branches and Map iterations show as separate paths. + +![The Execution flow graph for a successful run with Parallel branches and a Map state, above the event timeline](./images/step-functions-execution-flow.png) + +### Timeline + +The **Timeline** lists every execution event in order. Each row has a **Show log** toggle that reveals the raw event log, with the input, output, error, and cause where present. + +![The Timeline with one event expanded through Show log, revealing the raw event JSON](./images/step-functions-timeline-log.png) + +### Ask AI + +On a failed execution, **Ask AI** opens the [KloudMate Assistant](/kloudmate-assistant/) with the failing state, error, and cause already filled in, so you can start debugging without retyping the context. + +## Turn on execution logging + +Outcome counts and metrics come from CloudWatch and need no per-machine setup. The execution **timeline** and **workflow graph** come from the state machine's execution logs, so they need logging turned on in AWS and the log group connected in KloudMate. + +### 1. Enable logging in AWS + +In the AWS Step Functions console, open the state machine and edit its logging settings: + +1. Set the **Log level** to **ALL**. Lower levels don't record the per-state events the timeline and graph need. +2. Turn on **Include execution data** so the logs carry each state's input and output. +3. Save. + +AWS writes these logs to a vended log group named `/aws/vendedlogs/states/<state-machine-name>`, creating it on the first logged run. + +:::note +This works for both Standard and Express workflows. Express keeps its execution history only in these logs, so Express runs don't show up until you turn logging on. +::: + +### 2. Connect the log group in KloudMate + +Open the state machine in KloudMate, go to the **Configuration** tab, and under **Execution log ingestion** click **Connect**. KloudMate detects the `/aws/vendedlogs/states/<name>` group for you. The next run shows up with its timeline and workflow graph. + +## Standard and Express workflows + +KloudMate supports both Standard and Express workflows. The type shows next to the machine's name, and it changes what the **Metrics** tab plots: + +- **Standard** workflows report execution outcomes (started, succeeded, failed, aborted, timed out), throttles, and execution time (average, p90, and p99). +- **Express** workflows don't report per-outcome execution metrics to CloudWatch. Their **Metrics** tab shows **Billed duration** and **Billed memory** instead, and their execution counts come from the logs. Turn on execution logging to see Express runs at all. + +## Troubleshooting + +### I see metrics but no executions + +The outcome tiles and charts are filled in, but the **Recent executions** list or a machine's **Executions** tab is empty. + +The counts come from CloudWatch, which needs no logging. Individual executions come from the logs, so they appear only after you turn on execution logging and connect it in KloudMate. See [Turn on execution logging](#turn-on-execution-logging). + +### The graph or timeline is empty + +An execution opens, but its **Execution flow** graph or **Timeline** has nothing in it. + +This usually means the logs aren't detailed enough, or aren't reaching KloudMate. Check both of these: + +- **Log level is ALL, with execution data included.** At lower levels, or with execution data turned off, the logs don't carry the per-state events the graph and timeline need. +- **The log group is connected.** In the machine's **Configuration** tab, confirm **Execution log ingestion** shows **Connected**. A message about no `/aws/vendedlogs/states/` log group means logging hasn't started in AWS yet. + +## Related + +- [AWS Account Setup](../account-setup/): connect an AWS account so KloudMate can discover your state machines. +- [AWS Lambda Monitoring](../lambda-monitoring/): monitor your Lambda functions the same way. +- [KloudMate Assistant](/kloudmate-assistant/): the assistant that Ask AI opens with the failure details. diff --git a/src/content/docs/docs/infrastructure/integrations/kafka-jmx-exporter-integration.mdx b/src/content/docs/docs/infrastructure/integrations/kafka-jmx-exporter-integration.mdx new file mode 100644 index 00000000..8c9ff050 --- /dev/null +++ b/src/content/docs/docs/infrastructure/integrations/kafka-jmx-exporter-integration.mdx @@ -0,0 +1,135 @@ +--- +title: "Kafka JMX Exporter Integration" +description: "Collect Kafka broker, topic, and consumer group metrics from a Strimzi Kafka cluster on Kubernetes through the JMX Prometheus Exporter and the OpenTelemetry Collector's Prometheus receiver." +sidebar: + order: 3.5 +--- + +Use this integration when your Kafka cluster runs on Kubernetes through Strimzi and already exposes JMX metrics in Prometheus format. It's a Kubernetes-native alternative to [Kafka Integration](../kafka-integration/)'s `kafkametrics` receiver: instead of connecting to the Kafka protocol directly, the OpenTelemetry Collector scrapes the JMX Prometheus Exporter endpoint that Strimzi already exposes on each broker and exporter pod. + +## Prerequisites + +1. A Strimzi Kafka cluster. +2. The JMX Prometheus Exporter enabled on the cluster. +3. Metrics exposed on port `9404`. + +## Step 1: Install the Agent + +Install and run the [KloudMate Agent](../../../kloudmate-agent/) in your Kubernetes cluster. See [Kubernetes](../../../kloudmate-agent/installation/kubernetes-agent/) for the Helm install steps. + +## Step 2: Define the Prometheus Receiver for Kafka JMX Metrics + +Add the following configuration to the Agent's collector configuration. + +```yaml +extensions: + health_check: + endpoint: 0.0.0.0:13133 + +receivers: + prometheus: + config: + global: + scrape_interval: 30s + + scrape_configs: + - job_name: kafka-resources + kubernetes_sd_configs: + - role: pod + + relabel_configs: + - source_labels: + - __meta_kubernetes_namespace + action: keep + regex: kafka-prod + + - source_labels: + - __meta_kubernetes_pod_label_strimzi_io_component_type + action: keep + regex: kafka|kafka-exporter + + - source_labels: + - __meta_kubernetes_pod_container_port_name + action: keep + regex: tcp-prometheus +``` + +This uses Kubernetes service discovery to find every pod in the cluster, then narrows the scrape targets down through three `relabel_configs` stages: the first keeps only pods in the `kafka-prod` namespace, the second keeps only pods whose Strimzi component-type label is `kafka` or `kafka-exporter`, and the third keeps only the `tcp-prometheus` container port those pods expose. Replace the `kafka-prod` regex with your own Kafka namespace before applying the configuration. + +## Step 3: Add to Pipeline + +Add the `prometheus` receiver to the metrics pipeline in your Agent configuration. + +```yaml +service: + extensions: + - health_check + telemetry: + metrics: + level: none + pipelines: + metrics/kafka-jmx: + receivers: + - prometheus + processors: + - batch + exporters: + - otlphttp +``` + +## Step 4: Apply the Configuration + +If you run the OpenTelemetry Collector through the KloudMate Agent in managed mode, add this configuration through [Custom config override](../../../kloudmate-agent/custom-config-override/) instead of editing the deployment's YAML directly — the override merges it into the running configuration without switching the agent to manual mode. If you operate your own OpenTelemetry Collector, merge this block into its existing configuration instead. + +## Step 5: Restart the Collector + +If the Agent runs as a Kubernetes workload, restart whichever one runs your Collector: + +```bash +kubectl rollout restart deployment <otel-collector> -n <namespace> +``` + +or + +```bash +kubectl rollout restart daemonset <otel-collector> -n <namespace> +``` + +## Step 6: Validate in KloudMate + +After the Agent restarts: + +- Open [Explore](../../../visualize-data/explore/) in KloudMate. +- Query one of the scraped metrics, for example: + +``` +kafka_server_brokertopicmetrics_messagesin_total + +kafka_server_replicamanager_leadercount + +kafka_consumergroup_lag +``` + +- Build dashboards or alerts once the data is visible. + +## Metrics Collected + +This integration collects: + +- Kafka broker metrics +- Kafka topic metrics +- Kafka consumer group metrics +- Kafka Exporter metrics +- JVM metrics +- Broker health metrics +- Replication metrics +- Network metrics +- Request metrics +- Controller metrics + +## Related + +- [Kafka Integration](../kafka-integration/) — the `kafkametrics` receiver alternative, for brokers reachable directly over the Kafka protocol instead of through JMX. +- [Prometheus Integration](../prometheus-integration/) — the general-purpose pattern this integration builds on. +- [Custom config override](../../../kloudmate-agent/custom-config-override/) — how to add this receiver to a managed KloudMate Agent. +- [Explore](../../../visualize-data/explore/) — where to query the collected metrics. diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx new file mode 100644 index 00000000..84140c9f --- /dev/null +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx @@ -0,0 +1,86 @@ +--- +title: ".NET" +description: "How the KloudMate agent traces .NET with the OpenTelemetry CLR profiler on Windows (IIS and services) and Linux, plus Kubernetes, Amazon ECS, and Docker." +sidebar: + order: 5 +--- + +import { Tabs, TabItem, LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The agent traces your .NET applications with the OpenTelemetry .NET auto-instrumentation, a Common Language Runtime (CLR) profiler, with no code change. Windows is the most common home for .NET, so it's covered first, then Linux, Kubernetes, Amazon ECS, and Docker. For the shared opt-in flow, restart consent, and automatic rollback, see the [Application APM overview](../overview/). + +The profiler ships inside the KloudMate agent package and upgrades with it. When you instrument a .NET service, the agent stops eBPF monitoring from tracing that service's own requests, so you get the richer profiler spans without duplicates, on one connected trace. + +## On Windows + +For .NET on Windows, the agent sets the profiler environment (`CORECLR_ENABLE_PROFILING=1` for .NET Core and modern .NET, `COR_ENABLE_PROFILING=1` for .NET Framework, plus the profiler identifier and path), then recycles the workload. It writes these entries straight to the Windows registry or to the IIS `applicationHost.config`, so it doesn't rely on PowerShell or an administrator running commands by hand. + +### Supported runtimes + +Coverage depends on the **installed CLR runtime**, not the framework your application targets. So an application built for .NET 4.0 or 4.5 is instrumented as long as it runs on a supported runtime: + +| Installed runtime | Result | +|---|---| +| .NET Framework 4.6.2 and newer (CLR 4.x) | Full application traces through the CLR profiler | +| .NET Core and modern .NET | Full application traces through the CLR profiler | +| .NET Framework 3.5 (CLR 2.0) | No profiler; covered by ETW monitoring | + +A .NET Framework 3.5 (CLR 2.0) pool has no profiler, but ETW still gives it server spans and RED metrics. See [Windows platform notes](../../platform-notes/windows/). + +### IIS and Windows services + +<Tabs> +<TabItem label="IIS 10 and newer"> + +Instrumentation is **per application pool**. The agent writes the profiler environment into that pool's `<environmentVariables>` in `applicationHost.config`, then recycles the pool. Other pools are unaffected. + +</TabItem> +<TabItem label="IIS 8.5"> + +IIS 8.5 has no per-pool environment, so the agent writes to the global registry. The change is host-wide and affects all application pools rather than a single one. + +</TabItem> +<TabItem label="Windows service"> + +For a .NET Windows service, the agent writes the profiler environment to the service's registry environment at `HKLM\SYSTEM\CurrentControlSet\Services\<service>\Environment`, then restarts the service. + +</TabItem> +</Tabs> + +:::caution +An ASP.NET Core application pool must be set to "No Managed Code" for the profiler to attach, and applying instrumentation requires an IIS recycle. +::: + +## On Linux + +On Linux, the agent instruments .NET through the same CLR profiler, loaded through the `CORECLR_*` and `DOTNET_STARTUP_HOOKS` variables in a systemd startup config, then restarts the service. This is less common than .NET on Windows. .NET under PM2 isn't handled automatically; run a .NET service under systemd to instrument it. + +## On Kubernetes + +Choose **SDK** for the workload and the agent adds the OpenTelemetry Operator annotation `instrumentation.opentelemetry.io/inject-dotnet`. The Operator injects the profiler, and that workload's pods roll to pick it up. + +If a workload isn't detected automatically, add the annotation yourself, then restart the deployment: + +```bash +kubectl patch deployment <name> -n <ns> -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-dotnet":"km-agent/km-agent-instrumentation-crd"}}}}}' +``` + +See [Kubernetes setup](../kubernetes/) and the [Kubernetes installation guide](../../installation/kubernetes-agent/). + +## On Amazon ECS + +On Amazon ECS, the agent injects the .NET instrumentation into the services you name, on both the EC2 and Fargate launch types, by rewriting the task definition. List your .NET services in `KM_ECS_SERVICES` when you install. The bundled .NET instrumentation is glibc-only, so use a glibc-based image; on Alpine or other musl-based images it won't load. See the [Amazon ECS install guide](../../installation/ecs-agent/). + +## On Docker + +When the agent runs in Docker mode, eBPF traces your .NET containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. + +The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add OpenTelemetry to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On Windows, systemd, Kubernetes, or Amazon ECS, the agent injects the profiler for you instead. + +## Next steps + +<CardGrid> + <LinkCard title="Windows platform notes" href="../../platform-notes/windows/" description="ETW, IIS application pools, and discovery on Windows." /> + <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate." /> + <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> +</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/go-ebpf.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/go.mdx similarity index 61% rename from src/content/docs/docs/kloudmate-agent/auto-instrumentation/go-ebpf.mdx rename to src/content/docs/docs/kloudmate-agent/auto-instrumentation/go.mdx index 8b45b201..20cd30f8 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/go-ebpf.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/go.mdx @@ -1,13 +1,13 @@ --- -title: "Go (eBPF)" -description: "Why the KloudMate agent covers Go services through eBPF uprobes instead of code injection, and how Go traces connect to instrumented neighbors." +title: "Go" +description: "Why the KloudMate agent covers Go services through eBPF instead of code injection, and how Go traces connect to instrumented neighbors." sidebar: - order: 6 + order: 7 --- import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -Go is the one runtime the agent does not inject. It compiles to a static binary with no runtime to attach an agent to. So instead of code injection, the agent covers Go services through extended Berkeley Packet Filter (eBPF) monitoring. There is no **Instrument** toggle to turn on for a Go service. It is covered automatically wherever eBPF monitoring is active: on **Linux, Kubernetes, and Docker**, but not on **Windows**, which has no eBPF. +Go is the one runtime the agent doesn't inject. It compiles to a static binary with no runtime to attach an agent to, so instead of code injection, the agent covers Go services through eBPF monitoring. There's no **SDK** option for a Go service. It's covered by **eBPF** wherever eBPF monitoring runs: on **Linux, Kubernetes, and Docker**, and on **Amazon ECS EC2**. Go isn't traced on **Windows** or on **ECS Fargate**, neither of which runs eBPF, though the service still gets logs and metrics there. ## Rich eBPF coverage for Go @@ -27,7 +27,7 @@ The result is close to what a language SDK would produce, without changing your eBPF monitoring and the language agents both propagate W3C trace context (`traceparent`). So when an instrumented service calls a Go service, the Go service's eBPF span joins the caller's trace. You see one connected trace with the correct parent-child links, not two separate views. :::note -In the Discovered Services list, a Go service is marked as covered by eBPF rather than as one you can instrument, and its Instrument toggle is disabled. This is expected: eBPF monitoring is already tracing it. +In the Discovered Services list, a Go service offers **Off** and **eBPF** but not **SDK**. eBPF is already tracing it at the kernel level, so there's no in-process agent to add. ::: ## Next steps diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/images/discovered-services-status-chips.png b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/images/discovered-services-status-chips.png new file mode 100644 index 00000000..b724941b Binary files /dev/null and b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/images/discovered-services-status-chips.png differ diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx new file mode 100644 index 00000000..14106f37 --- /dev/null +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx @@ -0,0 +1,74 @@ +--- +title: "Java" +description: "How the KloudMate agent traces Java services with the OpenTelemetry Java agent on systemd, under PM2, on Kubernetes, and on Windows, with no code change." +sidebar: + order: 2 +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The agent traces your Java services with the OpenTelemetry Java agent, without a code change. How it attaches depends on where the service runs, so each environment is covered below. For the shared opt-in flow, restart consent, and automatic rollback, see the [Application APM overview](../overview/). + +The Java agent ships inside the KloudMate agent package and upgrades with it, so there's no separate download to manage. When you instrument a Java service, the agent stops eBPF monitoring from tracing that service's own requests, so you get the richer Java-agent spans without duplicates, on one connected trace. + +## On Linux + +### systemd services + +This is the main method on Linux. When you turn on SDK tracing for a systemd service, the agent writes a small startup config that loads the Java agent through `JAVA_TOOL_OPTIONS`, then restarts the service after you confirm. Because the service restarts, the agent sets everything up front: a fixed `service.name` (the name shown in Discovered Services), the sampling rate, and where to send traces. + +Turning tracing off removes the config and restarts the service, leaving it as it was. Running the same action twice changes nothing, so it's safe to repeat. + +### PM2 and standalone processes (runtime attach) + +A Java process that isn't a systemd service can still be traced, whether it runs as a plain `java -jar` process, under PM2, or from a script. The agent attaches the Java agent to the already-running process, with no restart. This is tested against common frameworks, including Spring Boot and Tomcat. + +The agent sets `service.name` automatically, trying these in order: + +1. The jar name for a `java -jar app.jar` launch, so `app.jar` reports as `app`. +2. `spring.application.name`, when the app sets it. +3. Otherwise `unknown_service:java`. + +Attaching to a running process has these consequences: + +- **You can't force a specific `service.name`.** For a fixed name, set `spring.application.name` in the app, or run it as a systemd service. +- **It can't be undone while the app runs.** Once attached, the agent stays until the process restarts. Turning tracing off doesn't remove it from a running process; that takes effect on the next restart. + +## On Kubernetes + +Choose **SDK** for the workload and the agent adds the OpenTelemetry Operator annotation `instrumentation.opentelemetry.io/inject-java`. The Operator injects the Java agent, and that workload's pods roll to pick it up. + +If a workload isn't detected automatically, add the annotation yourself, then restart the deployment so the Operator injects the agent into new pods: + +```bash +kubectl patch deployment <name> -n <ns> -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"km-agent/km-agent-instrumentation-crd"}}}}}' +``` + +The Operator and its Cert Manager prerequisite are installed with the agent. For that shared setup, see [Kubernetes setup](../kubernetes/) and the [Kubernetes installation guide](../../installation/kubernetes-agent/). + +## On Amazon ECS + +On Amazon ECS, the agent injects the OpenTelemetry Java agent into the services you name, on both the EC2 and Fargate launch types. It rewrites the task definition to load the Java agent and send traces to the collector: the per-instance daemon on EC2, or a sidecar on Fargate. List your Java services in `KM_ECS_SERVICES` when you install. See the [Amazon ECS install guide](../../installation/ecs-agent/). + +## On Docker + +When the agent runs in Docker mode, eBPF traces your Java containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. + +The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add the OpenTelemetry Java agent to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On systemd, Kubernetes, or Amazon ECS, the agent injects the Java agent for you instead. + +## On Windows + +The agent injects the Java agent through the per-service registry environment at `HKLM\SYSTEM\CurrentControlSet\Services\<service>\Environment`, the Windows equivalent of a systemd startup config. It sets `JAVA_TOOL_OPTIONS`, then restarts the service. This is the same Java agent used on Linux. For how Windows discovers services and traces IIS, see [Windows platform notes](../../platform-notes/windows/). + +## Versions and limitations + +- **JDK 21 and newer print a dynamic-agent warning.** Runtime attach triggers the JVM warning `A future release of the JVM may disallow dynamic loading of agents by default` ([JEP 451](https://openjdk.org/jeps/451)). It's harmless today. A future JVM may need you to start the app with `-XX:+EnableDynamicAgentLoading` for runtime attach to keep working. This applies only to runtime attach (PM2 and standalone), not to systemd, Kubernetes, or Windows, where the agent loads at startup. +- **Runtime attach doesn't cover the JDK's built-in `com.sun.net.httpserver`.** That server sets up its handler before the agent attaches, and it can't be changed afterward. Spring Boot, Tomcat, and other common frameworks are fully covered; only apps that serve directly from the JDK's built-in server are affected. + +## Next steps + +<CardGrid> + <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate for Java services." /> + <LinkCard title="Application APM overview" href="../overview/" description="The per-service opt-in flow, consent, and rollback." /> + <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> +</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/kubernetes.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/kubernetes.mdx index a4968ecb..7a582949 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/kubernetes.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/kubernetes.mdx @@ -1,16 +1,17 @@ --- -title: "Kubernetes" -description: "How the KloudMate agent traces Kubernetes workloads one at a time — Off, eBPF, or SDK — with the OpenTelemetry Operator handling SDK injection." +title: "Kubernetes setup" +description: "How the KloudMate agent traces Kubernetes workloads: Off, eBPF, or SDK, with the OpenTelemetry Operator handling SDK injection." sidebar: - order: 3 + label: "Kubernetes setup" + order: 10 --- import { Tabs, TabItem, LinkCard, CardGrid } from '@astrojs/starlight/components'; -On Kubernetes, application performance monitoring (APM) works the same opt-in, per-workload way as on other platforms: for each workload you choose **Off**, **eBPF**, or **SDK**. eBPF tracing runs in the DaemonSet collector and needs no code change and no restart. SDK tracing uses the OpenTelemetry (OTel) Operator — the agent adds an annotation, the Operator injects the matching language agent, and that workload's pods restart with tracing on. Off is the default, so nothing is traced until you turn it on. For the shared flow, consent, and status behavior, see the [Application APM overview](../overview/). +On Kubernetes, application performance monitoring (APM) works the same opt-in, per-workload way as on other platforms: for each workload you choose **Off**, **eBPF**, or **SDK**. eBPF tracing runs in the DaemonSet collector and needs no code change and no restart. SDK tracing uses the OpenTelemetry (OTel) Operator: the agent adds an annotation, the Operator injects the matching language agent, and that workload's pods restart with tracing on. Off is the default, so nothing is traced until you turn it on. For the shared flow, consent, and status behavior, see the [Application APM overview](../overview/). :::note[Which modes a workload offers] -Workloads with a supported SDK language — Java, Node.js, Python, or .NET — offer all three modes. Go, Ruby, and unrecognized runtimes such as Hasura have no SDK injector, so they offer **Off** or **eBPF** only. eBPF traces those at the kernel level with no injector and no restart, which is the only way to get traces from a workload no language agent can attach to. +Workloads with a supported SDK language (Java, Node.js, Python, or .NET) offer all three modes. Go, Ruby, PHP, and unrecognized runtimes such as Hasura have no SDK injector, so they offer **Off** or **eBPF** only. eBPF traces those at the kernel level with no injector and no restart, which is the only way to get traces from a workload no language agent can attach to. ::: ## Prerequisites @@ -19,7 +20,7 @@ Auto-instrumentation on Kubernetes uses the OpenTelemetry Operator, which the Kl ## Enable APM -There are two ways to instrument workloads. +Instrument workloads from the dashboard, or add the annotation yourself. ### From the dashboard @@ -29,7 +30,7 @@ After the agent is installed, open the cluster's APM configuration. The agent li ### With deployment annotations -If a workload isn't detected automatically, you can turn on **SDK** tracing by hand: add the OpenTelemetry Operator annotation for its language and restart the workload. This is the same annotation the SDK control applies. eBPF tracing has no annotation — the DaemonSet drives it, so you select it from the dashboard. +If a workload isn't detected automatically, you can turn on **SDK** tracing by hand: add the OpenTelemetry Operator annotation for its language and restart the workload. This is the same annotation the SDK control applies. eBPF tracing has no annotation; the DaemonSet drives it, so you select it from the dashboard. <Tabs> <TabItem label="Java"> @@ -66,14 +67,14 @@ After adding the annotation, restart the deployment so the Operator injects the ## How each mode applies -The opt-in flow is the same as on [Linux](../linux-systemd/): discover a workload, pick Off, eBPF, or SDK, and let traces flow. What differs is how Kubernetes applies each mode: +The opt-in flow is the same as on any platform: discover a workload, pick Off, eBPF, or SDK, and let traces flow. What differs is how Kubernetes applies each mode: - **eBPF** changes take effect on the next agent check-in with **no pod restart**. The DaemonSet reloads its configuration in place and starts tracing the workloads you selected, so switching a workload on or off doesn't roll any pods. - **SDK** changes go through the OTel Operator's annotation, so **only that workload's pods** roll to pick up the injected agent. Linux, by contrast, applies SDK through systemd config files and a service restart. ## Monitored namespaces -APM discovery applies to the namespaces you monitor, which you set at install time with `monitoredNamespaces`. Leave it empty to discover workloads in every namespace except the system ones: `kube-system`, `kube-public`, `kube-node-lease`, `cert-manager`, and anything starting with `kube-` are excluded by default. To also skip your own cluster-infrastructure namespaces — ingress controllers, `vpa`, `external-secrets`, and the like — list them in the `excludedNamespaces` Helm value. See the [Kubernetes installation guide](../../installation/kubernetes-agent/). +APM discovery applies to the namespaces you monitor, which you set at install time with `monitoredNamespaces`. Leave it empty to discover workloads in every namespace except the system ones: `kube-system`, `kube-public`, `kube-node-lease`, `cert-manager`, and anything starting with `kube-` are excluded by default. To also skip your own cluster-infrastructure namespaces (ingress controllers, `vpa`, `external-secrets`, and the like), list them in the `excludedNamespaces` Helm value. See the [Kubernetes installation guide](../../installation/kubernetes-agent/). ## Next steps diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/language-support.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/language-support.mdx deleted file mode 100644 index 7e60a8c1..00000000 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/language-support.mdx +++ /dev/null @@ -1,139 +0,0 @@ ---- -title: "Languages and process managers" -description: "Which languages the KloudMate agent auto-instruments on Linux, how the method changes with systemd, PM2, or a standalone process, and the known limitations." -sidebar: - label: "Languages & process managers" - order: 2.5 ---- - -import { LinkCard, CardGrid } from '@astrojs/starlight/components'; - -On a Linux host, how the agent adds tracing to a service depends on two things: the language it is written in, and how it is started (as a systemd service, under PM2, or on its own). The agent picks the right method for each combination. This page lists what is supported and what to watch out for. - -Every method follows the same steps to turn tracing on, ask before it restarts anything, and roll back if needed. See the [Application APM overview](../overview/). - -## Support matrix - -| Language | systemd service | PM2 app | Standalone process | -|---|---|---|---| -| Java | Startup config, then restart | Attaches live, no restart | Attaches live, no restart | -| Node.js | Startup config, then restart | Restart with tracing on | Run under systemd or PM2 | -| Python | Startup config, then restart | Restart with tracing on | Run under systemd or PM2 | -| .NET | Startup config, then restart | Run under systemd | Run under systemd | -| PHP | PHP config + graceful reload | PHP config + graceful reload | PHP config + graceful reload | -| Go | eBPF, no code change | eBPF, no code change | eBPF, no code change | -| Ruby | eBPF, no code change | eBPF, no code change | eBPF, no code change | - -With any method, the traces go to the agent's local collector, so an instrumented service shares a single trace with the [eBPF monitoring](../../ebpf-observability/) around it. - -## systemd services - -This is the main method on Linux. When you instrument a systemd service, the agent adds a small startup config file for it and, after you confirm, restarts the service so the change takes effect: - -- **Java** loads the OpenTelemetry Java agent through `JAVA_TOOL_OPTIONS`. -- **Node.js** loads the instrumentation through `NODE_OPTIONS`. -- **Python** loads it through `PYTHONPATH`. -- **.NET** loads the CLR profiler through the `CORECLR_*` and `DOTNET_STARTUP_HOOKS` variables. - -Because the service restarts, the agent can set everything up front: a fixed `service.name` (the name shown in the Discovered Services list), the sampling rate, and where to send the traces. Turning instrumentation off removes the config file and restarts the service, so it goes back to how it was. Running the same action twice changes nothing, so it is safe to repeat. - -## PM2 apps - -PM2 apps are not systemd services, so the agent uses a different method. For **Node.js and Python** apps under PM2, it restarts each selected app with tracing switched on (`pm2 restart <app> --update-env`). Each app keeps its PM2 process name as its `service.name`, so apps under the same PM2 manager stay separate. - -The agent covers PM2 for every user on the host, not just root. If an app does not come back up after the change, the agent undoes it and restarts the app, so a failed attempt never leaves your app down. - -Java under PM2 uses the runtime attach method described below. .NET under PM2 is uncommon and is not handled automatically; run a .NET service under systemd to instrument it. - -## Java without systemd (runtime attach) - -A Java process that is not a systemd service can still be instrumented, whether it runs as a plain `java -jar` process, under PM2, or from a script. The agent attaches the OpenTelemetry Java agent to the **already-running** process, with no restart needed. This is tested against common frameworks, including Spring Boot and Tomcat. - -The agent sets `service.name` automatically, trying these in order: - -1. The jar name for a `java -jar app.jar` launch (so `app.jar` reports as `app`). -2. `spring.application.name`, when the app sets it. -3. Otherwise `unknown_service:java`. - -Attaching to a running process has two consequences: - -- **You cannot force a specific `service.name`.** If you need a fixed name, set `spring.application.name` in the app, or run it as a systemd service. -- **It cannot be undone while the app runs.** Once attached, the agent stays until the process restarts. Turning instrumentation off does not remove it from a running process; it takes effect the next time the process restarts. - -## Standalone Node.js and Python - -The agent can instrument Node.js and Python only when they run under systemd or PM2. A process started on its own, under neither, cannot be instrumented automatically. Run it under systemd or PM2 first. The Discovered Services list flags these processes and tells you to do this. - -## Python versions - -SDK injection supports **Python 3.10 through 3.13**. Other versions, such as 3.14, are covered by [eBPF monitoring](../../ebpf-observability/) instead. - -## PHP - -PHP is instrumented through its own configuration, not a restart. The agent adds a config file for the target's PHP build, then reloads PHP gracefully (a php-fpm or Apache reload) so no requests are dropped. It uses the native OpenTelemetry extension on **PHP 8**, and falls back to a tracer that also covers **PHP 7**. See [PHP instrumentation](../php/). - -## Go - -Go does not use code injection. It is covered by [eBPF monitoring](../go-ebpf/), which gives Go services RED metrics (request rate, errors, and duration), the service map, and traces from the kernel, with no change to the app. - -## Ruby - -A Ruby app has two levels of coverage: - -- **eBPF monitoring, with no code change.** [eBPF](../../ebpf-observability/) gives a Ruby service HTTP server traces, RED metrics (request rate, errors, and duration), and the service map from the kernel, with no gem and on any Ruby version. This is the same as Go. -- **Deep Ruby tracing** needs the OpenTelemetry Ruby gems added to the app's `Gemfile`. The agent cannot add them from outside, because Ruby only loads gems listed in the app's `Gemfile`. With them, `use_all` traces the whole request path — HTTP handling, database queries, view rendering, background jobs, and outbound HTTP calls. - - ```ruby - # Gemfile - gem 'opentelemetry-sdk' - gem 'opentelemetry-instrumentation-all' - gem 'opentelemetry-exporter-otlp' - ``` - - ```ruby - # config/initializers/opentelemetry.rb - require 'opentelemetry/sdk' - require 'opentelemetry/instrumentation/all' - require 'opentelemetry/exporter/otlp' - OpenTelemetry::SDK.configure { |c| c.use_all } - ``` - - Set the service name and where to send traces through the app's environment: - - ```text - OTEL_SERVICE_NAME=my-rails-app - OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 - ``` - - The SQL query text on a database span comes from the driver instrumentation (`pg`, `mysql2`), not from ActiveRecord. A database with no OpenTelemetry driver — SQLite — shows the model operation (`User#save`) but no SQL. For the full breakdown of what these gems capture, see [Ruby auto-instrumentation](../../../apm-and-tracing/auto-instrumentation/ruby/). - - The current OpenTelemetry Ruby gems need **Ruby 3.3 or newer**. On Ruby 3.0 to 3.2 you have to pin older gem versions, and on Ruby 2.x they will not install at all. On those versions, use eBPF monitoring, which does not depend on the Ruby version. - -## How discovered services are named and filtered - -The agent keeps the Discovered Services list focused on real applications: - -- **Infrastructure and the agent's own processes are left out.** The agent never lists or instruments itself, cloud-provider agents (such as the AWS SSM agent, the Google guest agent, or the Azure agent), or container runtimes (such as `containerd`, `dockerd`, and `kubelet`). Left in, these would clutter the list and could be traced as if they were your services. -- **A plain Java process is listed by its jar name.** A `java -jar app.jar` process shows as `app` (matching its `service.name`), not the generic `java`. - -For the full discovery model, see [Discovery](../../concepts/discovery/). - -## Limitations - -Keep these in mind, especially for Java runtime attach: - -- **Java runtime attach does not cover the JDK's built-in `com.sun.net.httpserver`.** That server sets up its handler before the agent attaches, and it cannot be changed afterward. Common frameworks like Spring Boot and Tomcat are fully covered; only apps that serve directly from the JDK's built-in server are affected. -- **JDK 21 and newer print a warning about dynamic agent loading.** Runtime attach triggers the JVM warning `A future release of the JVM may disallow dynamic loading of agents by default` ([JEP 451](https://openjdk.org/jeps/451)). It is harmless today. A future JVM may need you to start the app with `-XX:+EnableDynamicAgentLoading` for runtime attach to keep working. -- **Standalone Node.js and Python need systemd or PM2.** Without one of them, there is no automatic way to instrument these apps. -- **Python SDK injection covers Python 3.10 through 3.13.** Other versions, such as 3.14, are covered by eBPF monitoring. -- **Go has no code injection.** Its coverage comes from eBPF monitoring. -- **Ruby has no code injection.** For no-code coverage it uses eBPF. Deep Ruby tracing (database queries, view rendering, background jobs, outbound HTTP) needs the OpenTelemetry Ruby gems added to the app, which require Ruby 3.3 or newer. - -## Related - -<CardGrid> - <LinkCard title="Application APM overview" href="../overview/" description="How to turn APM on per service, restarts, and rollback." /> - <LinkCard title="Linux (systemd)" href="../linux-systemd/" description="The systemd method in detail." /> - <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate." /> - <LinkCard title="Discovery" href="../../concepts/discovery/" description="How services are found and named." /> -</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/linux-systemd.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/linux-systemd.mdx deleted file mode 100644 index bcca015a..00000000 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/linux-systemd.mdx +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: "Linux (systemd)" -description: "How the KloudMate agent instruments Java, Node.js, Python, and .NET services on Linux, per service, without changing your application code." -sidebar: - order: 2 ---- - -import { LinkCard, CardGrid } from '@astrojs/starlight/components'; - -On Linux, the agent instruments a service by attaching the matching language agent to it and restarting it. It changes only the service you chose, not the whole system. For the flow, consent, and rollback behavior, see the [Application APM overview](../overview/). - -## How it works - -When you instrument a service, the agent adds the tracing settings to that service and, with your consent, restarts it so the settings take effect. Turning instrumentation off reverses the change and restarts the service, leaving it as it was. Applying the same choice twice does nothing, so it is safe to re-run. - -## Per-runtime support - -The agent uses the standard tracing mechanism for each runtime, so the behavior matches what each language's OpenTelemetry (OTel) agent documents. - -| Runtime | How it is attached | -|---|---| -| Java | The Java agent, through the standard `JAVA_TOOL_OPTIONS` startup option | -| Node.js | The Node.js instrumentation, loaded at startup with `NODE_OPTIONS` | -| Python | The Python auto-instrumentation | -| .NET | The .NET (CLR) profiler (less common on Linux than on Windows) | - -:::note -PHP and Go work differently. PHP is reloaded gracefully instead of restarted; see [PHP instrumentation](../php/). Go is not attached at all; it is covered by [eBPF monitoring](../go-ebpf/). -::: - -## Bundled with the agent - -The language agents come inside the KloudMate agent package and are versioned with it, so they upgrade when the agent does and there is no separate download to manage. - -## Restart behavior - -The agent restarts one service at a time, only with your consent, and can wait for the service to come back up before reporting success. If a restart fails, the agent undoes the change and restarts the service, returning it to its previous state. See the [overview](../overview/) for details. - -## Other process managers - -systemd is the main method on Linux, but not the only one. Node.js and Python apps under **PM2** are instrumented by restarting them with the tracing environment applied. A **Java** process that is not a systemd service is instrumented by runtime attach, with no restart, whether it runs as a bare `java -jar` process or under another supervisor. A bare Node.js or Python process still has to run under systemd or PM2 first. See [Languages and process managers](../language-support/) for the full matrix and the limits of each method. - -## Next steps - -<CardGrid> - <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate for instrumented services." /> - <LinkCard title="PHP" href="../php/" description="Instrument PHP services on the host." /> - <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> -</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx new file mode 100644 index 00000000..e0140204 --- /dev/null +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx @@ -0,0 +1,64 @@ +--- +title: "Node.js" +description: "How the KloudMate agent traces Node.js services with OpenTelemetry on systemd, under PM2, on Kubernetes, on Amazon ECS, and on Windows." +sidebar: + order: 3 +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The agent traces your Node.js services with the OpenTelemetry Node.js instrumentation, loaded at startup with no code change. How it attaches depends on where the service runs, so each environment is covered below. For the shared opt-in flow, restart consent, and automatic rollback, see the [Application APM overview](../overview/). + +The instrumentation ships inside the KloudMate agent package and upgrades with it. When you instrument a Node.js service, the agent stops eBPF monitoring from tracing that service's own requests, so you get the richer application spans without duplicates, on one connected trace. + +## On Linux + +### systemd services + +When you turn on SDK tracing for a systemd service, the agent writes a startup config that loads the instrumentation through `NODE_OPTIONS`, then restarts the service after you confirm. It sets a fixed `service.name`, the sampling rate, and where to send traces up front. Turning tracing off removes the config and restarts the service. Running the same action twice changes nothing. + +### PM2 apps + +For a Node.js app under PM2, the agent restarts it with tracing switched on (`pm2 restart <app> --update-env`). Each app keeps its PM2 process name as its `service.name`, so apps under the same PM2 manager stay separate. The agent covers PM2 for every user on the host, not just root. If an app doesn't come back up after the change, the agent undoes it and restarts the app. + +### Standalone processes + +The agent can instrument a Node.js process only when it runs under systemd or PM2. A process started on its own can't be instrumented automatically. Run it under systemd or PM2 first; the Discovered Services list flags these and tells you so. + +## On Kubernetes + +Choose **SDK** for the workload and the agent adds the OpenTelemetry Operator annotation `instrumentation.opentelemetry.io/inject-nodejs`. The Operator injects the instrumentation, and that workload's pods roll to pick it up. + +If a workload isn't detected automatically, add the annotation yourself, then restart the deployment: + +```bash +kubectl patch deployment <name> -n <ns> -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-nodejs":"km-agent/km-agent-instrumentation-crd"}}}}}' +``` + +The Operator and its Cert Manager prerequisite are installed with the agent. See [Kubernetes setup](../kubernetes/) and the [Kubernetes installation guide](../../installation/kubernetes-agent/). + +## On Amazon ECS + +On Amazon ECS, the agent injects the Node.js instrumentation into the services you name, on both the EC2 and Fargate launch types. It rewrites the task definition and sends traces to the collector: the per-instance daemon on EC2, or a sidecar on Fargate. List your Node.js services in `KM_ECS_SERVICES` when you install. See the [Amazon ECS install guide](../../installation/ecs-agent/). + +## On Docker + +When the agent runs in Docker mode, eBPF traces your Node.js containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. + +The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add OpenTelemetry to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On systemd, Kubernetes, or Amazon ECS, the agent injects the instrumentation for you instead. + +## On Windows + +The agent injects the instrumentation through the per-service registry environment at `HKLM\SYSTEM\CurrentControlSet\Services\<service>\Environment`, the Windows equivalent of a systemd startup config. It sets `NODE_OPTIONS`, then restarts the service. For how Windows discovers services and traces IIS, see [Windows platform notes](../../platform-notes/windows/). + +## Limitations + +- **Standalone Node.js needs systemd or PM2.** Without one of them, there's no automatic way to instrument the app. + +## Next steps + +<CardGrid> + <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate for Node.js services." /> + <LinkCard title="Application APM overview" href="../overview/" description="The per-service opt-in flow, consent, and rollback." /> + <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> +</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/overview.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/overview.mdx index 95d1ecb0..5e132862 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/overview.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/overview.mdx @@ -9,9 +9,9 @@ import { LinkCard, CardGrid } from '@astrojs/starlight/components'; Application performance monitoring (APM) is opt-in and per service. For each service the agent discovers, you choose how it is traced: -- **Off** — not traced. This is the default, so nothing is traced (and nothing costs) until you turn it on. -- **eBPF** — Rate, Errors, and Duration (RED) metrics and trace spans, captured out of process with no code change and no restart. -- **SDK** — OpenTelemetry (OTel) auto-instrumentation attached in-process for full distributed traces, with spans for your frameworks, database calls, and outbound requests. It needs a one-time process restart, so you turn it on one service at a time. +- **Off:** not traced. This is the default, so nothing is traced (and nothing costs) until you turn it on. +- **eBPF:** Rate, Errors, and Duration (RED) metrics and trace spans, captured out of process with no code change and no restart. +- **SDK:** OpenTelemetry (OTel) auto-instrumentation attached in-process for full distributed traces, with spans for your frameworks, database calls, and outbound requests. It needs a one-time process restart, so you turn it on one service at a time. ## The per-service flow @@ -26,14 +26,14 @@ The flow is the same across every deployment mode: Each service selected for SDK instrumentation shows a status chip so you always know where it stands: -- **Pending:** the service is selected for SDK and the agent is wiring it up — applying the change and restarting the app. It clears to **Instrumented** once the agent confirms, usually within a minute. +- **Pending:** the service is selected for SDK and the agent is wiring it up, applying the change and restarting the app. It clears to **Instrumented** once the agent confirms, usually within a minute. - **Instrumented:** the SDK is applied and the service is sending application traces. - **Failed:** the restart failed, so the agent rolled back the change (see rollback below) and the service reverts to its previous mode. -- **Unsupported:** the runtime version has no bundled instrumentation — for example a Python minor release with no bundled cell. The service stays covered by eBPF; move to a supported version to add SDK traces. +- **Unsupported:** the runtime version has no bundled instrumentation, for example a Python minor release with no bundled cell. The service stays covered by eBPF; move to a supported version to add SDK traces. -Services covered by eBPF (Go, Ruby, or anything without an SDK injector) don't restart and carry no chip — eBPF traces them as soon as you turn them on. A separate **Restart pending** marker on the *agents list* is unrelated: it means the agent's own collector configuration is queued for a restart, not a per-service instrumentation change. +Services covered by eBPF (Go, Ruby, or anything without an SDK injector) don't restart and carry no chip. eBPF traces them as soon as you turn them on. A separate **Restart pending** marker on the *agents list* is unrelated: it means the agent's own collector configuration is queued for a restart, not a per-service instrumentation change. -TODO(screenshot): the Discovered Services list with per-service Off / eBPF / SDK controls and status chips. +![Discovered Services list with per-service Off / eBPF / SDK controls and status chips](./images/discovered-services-status-chips.png) ## Restart consent and health gating @@ -57,6 +57,7 @@ If an instrumented service fails to restart or falls into a crash loop, the agen | .NET | OpenTelemetry .NET auto-instrumentation (CLR profiler) | | PHP | Attached per service, covering both PHP 7 and PHP 8 | | Go | Covered by eBPF monitoring, not injected | +| Ruby | Covered by eBPF monitoring; deep tracing with the OpenTelemetry Ruby gems | The language agents ship inside the KloudMate agent package and are versioned with it, so there is no separate download to manage. @@ -68,14 +69,33 @@ When you instrument a service, the agent tells eBPF monitoring to stop tracing t Instrumenting per service is the default and the recommended approach. For automated rollouts, the agent also has an autonomous mode that instruments every eligible service without per-service toggles. This is an advanced option. See [Advanced configuration](../../advanced-configuration/). -## Per-mode guides +## Instrument by language + +Pick your language for the full setup, on every environment it runs on: Linux, Kubernetes, Amazon ECS, Docker, and Windows. + +<CardGrid> + <LinkCard title="Java" href="../java/" description="systemd, runtime attach, Kubernetes, ECS, and Windows." /> + <LinkCard title="Node.js" href="../nodejs/" description="systemd, PM2, Kubernetes, ECS, and Windows." /> + <LinkCard title="Python" href="../python/" description="systemd, PM2, Kubernetes, ECS, and Windows; versions 3.10 to 3.13." /> + <LinkCard title=".NET" href="../dotnet/" description="The CLR profiler on Windows and Linux, plus Kubernetes and ECS." /> + <LinkCard title="PHP" href="../php/" description="PHP 7 and 8 on Linux and Docker; eBPF on Kubernetes." /> + <LinkCard title="Go" href="../go/" description="Covered by eBPF, no injection." /> + <LinkCard title="Ruby" href="../ruby/" description="eBPF out of the box, plus deep tracing with the OTel gems." /> +</CardGrid> + +## Environment specifics + +Shared setup that applies to every language on that platform: + +<CardGrid> + <LinkCard title="Kubernetes setup" href="../kubernetes/" description="The OpenTelemetry Operator, Cert Manager, and monitored namespaces." /> + <LinkCard title="Windows platform notes" href="../../platform-notes/windows/" description="ETW, IIS application pools, and service discovery." /> + <LinkCard title="Docker" href="../../platform-notes/docker/" description="eBPF per container, and manual OpenTelemetry for full SDK depth." /> + <LinkCard title="Amazon ECS" href="../../installation/ecs-agent/" description="Language-aware SDK injection on EC2 and Fargate." /> +</CardGrid> + +## Set the sampling rate <CardGrid> - <LinkCard title="Linux (systemd)" href="../linux-systemd/" description="Instrument Java, Node.js, Python, and .NET on systemd hosts." /> - <LinkCard title="Languages & process managers" href="../language-support/" description="The support matrix across systemd, PM2, and bare processes, and the limitations." /> - <LinkCard title="Kubernetes" href="../kubernetes/" description="Instrument workloads with the OpenTelemetry Operator." /> - <LinkCard title="Windows and .NET" href="../windows-dotnet/" description="Instrument .NET on IIS and Windows services." /> - <LinkCard title="PHP" href="../php/" description="Trace PHP 7 and PHP 8 applications." /> - <LinkCard title="Go (eBPF)" href="../go-ebpf/" description="How Go is covered without injection." /> <LinkCard title="Sampling" href="../sampling/" description="Balance trace fidelity against ingestion cost." /> </CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx index ef16f2d3..93c1a2d1 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx @@ -1,32 +1,36 @@ --- title: "PHP" -description: "How the KloudMate agent adds route-level tracing to PHP 7 and 8 applications on Linux and Docker." +description: "How the KloudMate agent adds route-level tracing to PHP 7 and 8 on Linux and Docker, and how PHP is covered on Kubernetes." sidebar: - order: 5 + order: 6 --- import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -The agent traces PHP **7.0 through 8.x** applications on **Linux** (host PHP-FPM) and **Docker**. It does **not** apply on **Windows**. +The agent traces PHP **7.0 through 8.x** applications with no code change, by attaching tracing to the PHP runtime and reloading it gracefully. PHP works on **Linux** and **Docker**. On **Kubernetes** it's covered by eBPF, and it isn't traced on **Windows**. For the shared opt-in flow, see the [Application APM overview](../overview/). -## How it works +## What you get -When you instrument a PHP service, the agent attaches tracing to it automatically, with **no application code changes**. It reloads the PHP process manager gracefully, so requests keep being served. If tracing fails to load, it rolls back. If a PHP build is not supported, the agent skips it and tells you why. +The agent produces route-level server spans, database spans (for example, PDO calls), and error spans. It propagates W3C trace context (`traceparent`), so a PHP span shares its trace with eBPF monitoring and with any upstream or downstream service. -## What you get +## On Linux + +When you instrument a PHP service, the agent adds a config file for the target's PHP build, then reloads the process manager gracefully (a php-fpm or Apache reload) so no requests are dropped. It uses the native OpenTelemetry extension on **PHP 8**, and falls back to a tracer that also covers **PHP 7**. If a PHP build isn't supported, the agent skips it and tells you why. If tracing fails to load, it rolls back. + +## On Docker -The agent produces route-level server spans, database spans (for example, PDO calls), and error spans. It propagates W3C trace context (`traceparent`), so a PHP span shares its trace identifier with eBPF monitoring and with any upstream or downstream service. +In Docker mode, the agent instruments PHP containers automatically, in place, and matches the container's PHP build. It reloads the SAPI (Apache graceful or php-fpm) with no redeploy, so this is separate from the **Off / eBPF** toggle. Narrow or turn it off with the `KM_CONTAINER_INSTRUMENT_*` environment variables. The instrumented application sends traces to the agent through the container's host gateway. -The overhead is small: a fixed cost of roughly 1.6 milliseconds per request, which is a low single-digit percentage on realistic requests. +The built-in PHP server (`php -S`) and Alpine (musl) images aren't supported: the built-in server has no graceful reload, and the bundled tracer assets target glibc. For the details, including the PHP 7 build-time recipe, see [Docker platform notes](../../platform-notes/docker/). -## PHP in containers +## On Kubernetes -In Docker, the agent instruments PHP containers without a redeploy, and matches the container's PHP build automatically. The instrumented application sends its traces to the agent through the container's host gateway. See [Docker platform notes](../../platform-notes/docker/) for the container details. +On Kubernetes, PHP is covered by [eBPF monitoring](../../ebpf-observability/), not SDK injection, because there's no OpenTelemetry Operator injector for PHP. Select **eBPF** for the workload to get HTTP server traces, RED metrics, and the service map from the kernel, with no code change. ## Next steps <CardGrid> - <LinkCard title="Application APM overview" href="../overview/" description="The per-service opt-in flow, consent, and rollback." /> + <LinkCard title="Docker platform notes" href="../../platform-notes/docker/" description="PHP containers, supported servers, and the PHP 7 recipe." /> <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate." /> <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> </CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx new file mode 100644 index 00000000..7187cd1e --- /dev/null +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx @@ -0,0 +1,66 @@ +--- +title: "Python" +description: "How the KloudMate agent traces Python services with OpenTelemetry on systemd, under PM2, on Kubernetes, on Amazon ECS, and on Windows, including the supported versions." +sidebar: + order: 4 +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The agent traces your Python services with OpenTelemetry auto-instrumentation, loaded at startup with no code change. How it attaches depends on where the service runs, so each environment is covered below. For the shared opt-in flow, restart consent, and automatic rollback, see the [Application APM overview](../overview/). + +The instrumentation ships inside the KloudMate agent package and upgrades with it. When you instrument a Python service, the agent stops eBPF monitoring from tracing that service's own requests, so you get the richer application spans without duplicates, on one connected trace. + +## On Linux + +### systemd services + +When you turn on SDK tracing for a systemd service, the agent writes a startup config that loads the instrumentation through `PYTHONPATH`, then restarts the service after you confirm. It sets a fixed `service.name`, the sampling rate, and where to send traces up front. Turning tracing off removes the config and restarts the service. Running the same action twice changes nothing. + +### PM2 apps + +For a Python app under PM2, the agent restarts it with tracing switched on (`pm2 restart <app> --update-env`). Each app keeps its PM2 process name as its `service.name`. The agent covers PM2 for every user on the host, not just root. If an app doesn't come back up after the change, the agent undoes it and restarts the app. + +### Standalone processes + +The agent can instrument a Python process only when it runs under systemd or PM2. A process started on its own can't be instrumented automatically. Run it under systemd or PM2 first; the Discovered Services list flags these and tells you so. + +## On Kubernetes + +Choose **SDK** for the workload and the agent adds the OpenTelemetry Operator annotation `instrumentation.opentelemetry.io/inject-python`. The Operator injects the instrumentation, and that workload's pods roll to pick it up. + +If a workload isn't detected automatically, add the annotation yourself, then restart the deployment: + +```bash +kubectl patch deployment <name> -n <ns> -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-python":"km-agent/km-agent-instrumentation-crd"}}}}}' +``` + +The Operator and its Cert Manager prerequisite are installed with the agent. See [Kubernetes setup](../kubernetes/) and the [Kubernetes installation guide](../../installation/kubernetes-agent/). + +## On Amazon ECS + +On Amazon ECS, the agent injects the Python instrumentation into the services you name, on both the EC2 and Fargate launch types. It rewrites the task definition and sends traces to the collector: the per-instance daemon on EC2, or a sidecar on Fargate. List your Python services in `KM_ECS_SERVICES` when you install. See the [Amazon ECS install guide](../../installation/ecs-agent/). + +## On Docker + +When the agent runs in Docker mode, eBPF traces your Python containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. + +The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add OpenTelemetry to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On systemd, Kubernetes, or Amazon ECS, the agent injects the instrumentation for you instead. + +## On Windows + +The agent injects the instrumentation through the per-service registry environment at `HKLM\SYSTEM\CurrentControlSet\Services\<service>\Environment`. It sets `PYTHONPATH`, then restarts the service. For how Windows discovers services and traces IIS, see [Windows platform notes](../../platform-notes/windows/). + +## Versions and limitations + +- **SDK injection covers Python 3.10 through 3.13.** Other versions, such as 3.14, are covered by [eBPF monitoring](../../ebpf-observability/) instead. +- **The bundled instrumentation is glibc-only.** On Alpine or other musl-based images it won't load; use a glibc-based image. +- **Standalone Python needs systemd or PM2.** Without one of them, there's no automatic way to instrument the app. + +## Next steps + +<CardGrid> + <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate for Python services." /> + <LinkCard title="Application APM overview" href="../overview/" description="The per-service opt-in flow, consent, and rollback." /> + <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> +</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/ruby.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/ruby.mdx new file mode 100644 index 00000000..4e72b89d --- /dev/null +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/ruby.mdx @@ -0,0 +1,54 @@ +--- +title: "Ruby" +description: "How the KloudMate agent covers Ruby with eBPF out of the box, and how to add deep OpenTelemetry tracing with the Ruby gems." +sidebar: + order: 8 +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +A Ruby service gets eBPF monitoring out of the box with no code change, and deeper OpenTelemetry tracing once you add the Ruby gems to the app. The agent can't inject a Ruby SDK from outside, because Ruby only loads gems listed in the app's `Gemfile`. For the shared opt-in flow, see the [Application APM overview](../overview/). + +## eBPF monitoring, no code change + +[eBPF monitoring](../../ebpf-observability/) gives a Ruby service HTTP server traces, RED metrics (request rate, errors, and duration), and the service map from the kernel, with no gem and on any Ruby version. This is the same coverage Go gets, and it works on Linux, Kubernetes, and Docker. On Kubernetes it's the only trace path for Ruby, since there's no OpenTelemetry Operator injector for it. + +## Deep Ruby tracing with the OpenTelemetry gems + +For database queries, view rendering, background jobs, and outbound HTTP calls, add the OpenTelemetry Ruby gems to the app's `Gemfile`. With them, `use_all` traces the whole request path. + +```ruby +# Gemfile +gem 'opentelemetry-sdk' +gem 'opentelemetry-instrumentation-all' +gem 'opentelemetry-exporter-otlp' +``` + +```ruby +# config/initializers/opentelemetry.rb +require 'opentelemetry/sdk' +require 'opentelemetry/instrumentation/all' +require 'opentelemetry/exporter/otlp' +OpenTelemetry::SDK.configure { |c| c.use_all } +``` + +Set the service name and where to send traces through the app's environment: + +```text +OTEL_SERVICE_NAME=my-rails-app +OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 +``` + +The SQL query text on a database span comes from the driver instrumentation (`pg`, `mysql2`), not from ActiveRecord. A database with no OpenTelemetry driver, such as SQLite, shows the model operation (`User#save`) but no SQL. For the full breakdown of what these gems capture, see [Ruby manual instrumentation](../../../apm-and-tracing/manual-instrumentation/ruby/). + +## Versions + +The current OpenTelemetry Ruby gems need **Ruby 3.3 or newer**. On Ruby 3.0 to 3.2 you have to pin older gem versions, and on Ruby 2.x they won't install at all. On those versions, use eBPF monitoring, which doesn't depend on the Ruby version. + +## Next steps + +<CardGrid> + <LinkCard title="eBPF observability" href="../../ebpf-observability/" description="The kernel-level monitoring that covers Ruby." /> + <LinkCard title="Ruby manual instrumentation" href="../../../apm-and-tracing/manual-instrumentation/ruby/" description="What the OpenTelemetry Ruby gems capture." /> + <LinkCard title="Application APM overview" href="../overview/" description="How opt-in tracing works." /> +</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/sampling.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/sampling.mdx index 1fc179f8..f6ba9c3e 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/sampling.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/sampling.mdx @@ -2,7 +2,7 @@ title: "Sampling" description: "Set a global sampling default and per-service overrides for the KloudMate agent to balance trace fidelity against ingestion cost." sidebar: - order: 7 + order: 9 --- import { LinkCard, CardGrid } from '@astrojs/starlight/components'; diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/windows-dotnet.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/windows-dotnet.mdx deleted file mode 100644 index 3e023bd1..00000000 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/windows-dotnet.mdx +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: "Windows and .NET" -description: "How the KloudMate agent instruments .NET on IIS and Windows services with the CLR profiler, plus Java, Node.js, and Python through per-service registry environment." -sidebar: - order: 4 ---- - -import { Tabs, TabItem, LinkCard, CardGrid } from '@astrojs/starlight/components'; - -Windows has no `LD_PRELOAD` and no production eBPF. But its main workload, .NET, can be traced with no code changes through the Common Language Runtime (CLR) profiler. The agent uses the profiler for .NET. For Java, Node.js, and Python it sets per-service environment variables, and it monitors all IIS traffic through Event Tracing for Windows (ETW). The opt-in flow, consent, and rollback are the same as in the [overview](../overview/). - -## .NET applications - -For .NET, the agent uses the OpenTelemetry .NET auto-instrumentation CLR profiler. It sets the profiler environment (`CORECLR_ENABLE_PROFILING=1` for .NET Core and modern .NET, `COR_ENABLE_PROFILING=1` for .NET Framework, plus the profiler identifier and path), then recycles the workload. It writes these entries straight to the Windows registry or to the Internet Information Services (IIS) `applicationHost.config`, so it does not rely on PowerShell or an administrator running commands by hand. - -### Supported runtimes - -Coverage depends on the **installed CLR runtime**, not the framework your application targets. So most older applications are covered, as long as they run on a supported runtime: - -| Installed runtime | Result | -|---|---| -| .NET Framework 4.6.2 and newer (CLR 4.x) | Full application traces through the CLR profiler | -| .NET Core and modern .NET | Full application traces through the CLR profiler | -| .NET Framework 3.5 (CLR 2.0) | No profiler; covered by ETW monitoring below | - -An application built for .NET 4.0 or 4.5 but running on a 4.6.2 or newer runtime is instrumented, because the profiler goes by the runtime, not the build target. - -### IIS and Windows services - -<Tabs> -<TabItem label="IIS 10 and newer"> - -Instrumentation is **per application pool**. The agent writes the profiler environment into that pool's `<environmentVariables>` in `applicationHost.config`, then recycles the pool. Other pools are unaffected. - -</TabItem> -<TabItem label="IIS 8.5"> - -IIS 8.5 has no per-pool environment, so the agent writes to the global registry. The change is host-wide and affects all application pools rather than a single one. - -</TabItem> -<TabItem label="Windows service"> - -For a .NET Windows service, the agent writes the profiler environment to the service's registry environment at `HKLM\SYSTEM\CurrentControlSet\Services\<service>\Environment`, then restarts the service. - -</TabItem> -</Tabs> - -:::caution -An ASP.NET Core application pool must be set to "No Managed Code" for the profiler to attach, and applying instrumentation requires an IIS recycle. -::: - -## Java, Node.js, and Python on Windows - -These runtimes use the same language agents as Linux. The agent injects them through the per-service registry environment at `HKLM\SYSTEM\CurrentControlSet\Services\<service>\Environment`, which is the Windows equivalent of a systemd startup config. It sets the standard variable for the runtime (`JAVA_TOOL_OPTIONS`, `NODE_OPTIONS`, or `PYTHONPATH`), then restarts the service to apply it. - -## ETW monitoring - -Because Windows has no eBPF, the agent monitors without any code changes through a custom ETW receiver. It taps the HTTP.sys provider to produce: - -- **Server spans** for every request, at the request-path level. -- **RED metrics** (Rate, Errors, and Duration) grouped by application pool. - -This works for any IIS traffic without injection. That includes workloads that cannot use the profiler, such as .NET Framework 3.5 (CLR 2.0) pools and non-.NET applications hosted on IIS. The ETW receiver only reads events. It needs no credentials and changes nothing in your applications, and it runs under the Local System account the agent uses. When a pool is also instrumented with the profiler, the agent stops generating ETW spans for that pool but keeps its RED metrics, so you do not get duplicate spans. - -## Discovery on Windows - -To build the Discovered Services list, the agent reads the Service Control Manager, IIS sites and application pools, and listening ports through Windows Management Instrumentation (WMI). Services appear with a kind of `iis` or `windows_service`. See [Discovery](../../concepts/discovery/) and [Windows platform notes](../../platform-notes/windows/). - -## Next steps - -<CardGrid> - <LinkCard title="Windows platform notes" href="../../platform-notes/windows/" description="Discovery, the ETW receiver, and privileges in depth." /> - <LinkCard title="Sampling" href="../sampling/" description="Set the trace sampling rate." /> - <LinkCard title="View traces" href="../../../apm-and-tracing/" description="See the distributed traces in KloudMate." /> -</CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx b/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx index bde91b49..18c5b2c6 100644 --- a/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx +++ b/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx @@ -101,7 +101,7 @@ The agent installs as a Windows service running as Local System. It needs that p - Host metrics and the Windows Event Log start flowing automatically. See [Host metrics and logs](../../baseline/host-metrics-and-logs/). - eBPF does not apply on Windows. The agent gives you equivalent monitoring through Event Tracing for Windows (ETW). See [Windows platform notes](../../platform-notes/windows/). -- To add distributed tracing to .NET applications, see [Windows and .NET application APM](../../auto-instrumentation/windows-dotnet/). +- To add distributed tracing to .NET applications, see [.NET application APM](../../auto-instrumentation/dotnet/). ## **Next Steps** diff --git a/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx b/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx index 78e95cc2..3e2b2b5b 100644 --- a/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx +++ b/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx @@ -7,7 +7,7 @@ sidebar: import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -How the agent runs on Windows. For installation, see the [Windows installation guide](../../installation/windows-agent/). For how .NET and other runtimes are instrumented, see [Windows and .NET application APM](../../auto-instrumentation/windows-dotnet/). +How the agent runs on Windows. For installation, see the [Windows installation guide](../../installation/windows-agent/). For how each runtime is instrumented, see the [Application APM](../../auto-instrumentation/overview/) language pages, starting with [.NET](../../auto-instrumentation/dotnet/). ## A different mechanism, the same model @@ -25,14 +25,14 @@ Discovered services appear with a kind of `iis` or `windows_service`. See [Disco ## ETW monitoring -Because Windows has no eBPF, the agent gives you equivalent zero-code monitoring through a custom Event Tracing for Windows (ETW) receiver. The receiver taps the HTTP.sys provider to emit server spans (at the request-path level) and Rate, Errors, and Duration (RED) metrics for all IIS traffic, with no injection. This covers workloads that cannot use the .NET profiler, including legacy .NET Framework 3.5 pools and non-.NET applications on IIS. The receiver reads events only and makes no changes to your applications. See [Windows and .NET application APM](../../auto-instrumentation/windows-dotnet/) for how ETW monitoring and the CLR profiler work together. +Because Windows has no eBPF, the agent gives you equivalent zero-code monitoring through a custom Event Tracing for Windows (ETW) receiver. The receiver taps the HTTP.sys provider to emit server spans (at the request-path level) and Rate, Errors, and Duration (RED) metrics for all IIS traffic, with no injection. This covers workloads that cannot use the .NET profiler, including legacy .NET Framework 3.5 pools and non-.NET applications on IIS. The receiver reads events only and makes no changes to your applications. See [.NET](../../auto-instrumentation/dotnet/) for how the CLR profiler and ETW monitoring work together. ## Injection mechanisms - **.NET** is instrumented with the Common Language Runtime (CLR) profiler. The agent writes the profiler's environment directly to the registry or to the IIS `applicationHost.config`. On IIS 10 and newer, it can do this per application pool. - **Java, Node.js, and Python** are instrumented through the per-service registry environment. -See [Windows and .NET application APM](../../auto-instrumentation/windows-dotnet/) for the details. +See the [Application APM](../../auto-instrumentation/overview/) language pages for the details. ## Privileges @@ -50,6 +50,6 @@ Alongside application instrumentation, the agent automatically collects Windows <CardGrid> <LinkCard title="Install on Windows" href="../../installation/windows-agent/" description="MSI installation and setup." /> - <LinkCard title="Windows and .NET APM" href="../../auto-instrumentation/windows-dotnet/" description="Instrument .NET and other runtimes." /> + <LinkCard title=".NET application APM" href="../../auto-instrumentation/dotnet/" description="Instrument .NET on IIS and Windows services." /> <LinkCard title="Host metrics and logs" href="../../baseline/host-metrics-and-logs/" description="Metrics and logs collected automatically on Windows." /> </CardGrid> diff --git a/src/content/docs/docs/kloudmate-agent/reference/glossary.mdx b/src/content/docs/docs/kloudmate-agent/reference/glossary.mdx index d67cf226..7bd79c48 100644 --- a/src/content/docs/docs/kloudmate-agent/reference/glossary.mdx +++ b/src/content/docs/docs/kloudmate-agent/reference/glossary.mdx @@ -29,7 +29,7 @@ Definitions of the terms used across the agent documentation. **Profile.** A template that provides one coverage for one database engine. When you select a profile, the agent turns it into the right receiver configuration. See [Database monitoring overview](../../database-monitoring/overview/). -**Profiler (CLR profiler).** The Common Language Runtime profiler used to instrument .NET applications on Windows without code changes. See [Windows and .NET application APM](../../auto-instrumentation/windows-dotnet/). +**Profiler (CLR profiler).** The Common Language Runtime profiler used to instrument .NET applications on Windows without code changes. See [.NET application APM](../../auto-instrumentation/dotnet/). **Receiver.** A collector component that collects telemetry from a source, such as `hostmetrics`, `filelog`, or a database receiver like `postgresql`. diff --git a/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx b/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx index 390dae19..7d182214 100644 --- a/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx +++ b/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx @@ -37,13 +37,9 @@ The agent reports these `kmagent.*` metrics about itself, separately from the co |---|---| | `kmagent.up` | Heartbeat | | `kmagent.uptime` | Process uptime | -| `kmagent.goroutines` | Internal thread count | | `kmagent.memory.rss` | Process memory | | `kmagent.cpu.utilization` | Process CPU | | `kmagent.collector.up` | Collector liveness, with `state` and `reason` | -| `kmagent.collector.restarts` | Collector restart count | -| `kmagent.config.apply.success` | Last configuration apply succeeded | -| `kmagent.config.apply.errors` | Cumulative configuration-apply failures | See [Agent health metrics](../../self-observability/health-metrics/) for how to use them. When collector self-telemetry is enabled, the agent also produces the standard `otelcol_*` metrics. See [Collector self-telemetry](../../self-observability/self-telemetry/). diff --git a/src/content/docs/docs/kloudmate-agent/self-observability/health-metrics.mdx b/src/content/docs/docs/kloudmate-agent/self-observability/health-metrics.mdx index 7ff0633a..e55ac475 100644 --- a/src/content/docs/docs/kloudmate-agent/self-observability/health-metrics.mdx +++ b/src/content/docs/docs/kloudmate-agent/self-observability/health-metrics.mdx @@ -17,13 +17,9 @@ The agent reports these metrics, all carrying the resource attributes below: |---|---| | `kmagent.up` | Heartbeat. Its absence is the signal that the agent, host, or network is down. | | `kmagent.uptime` | Time since the agent process started. | -| `kmagent.goroutines` | Internal thread count, useful for spotting a leak. | | `kmagent.memory.rss` | Agent process memory. | | `kmagent.cpu.utilization` | Agent process CPU. | | `kmagent.collector.up` | Whether the embedded collector is running (1 or 0), with `state` and `reason` attributes. | -| `kmagent.collector.restarts` | How many times the collector has been restarted. | -| `kmagent.config.apply.success` | Whether the last configuration push applied cleanly (1 or 0). | -| `kmagent.config.apply.errors` | Cumulative configuration-apply failures. | Resource attributes: `service.name` is `kmagent` (so you can target it directly in dashboards and alerts), plus `host.name`, `agent.version`, `collector.version`, `os.type`, and `host.arch`. @@ -35,19 +31,21 @@ Resource attributes: `service.name` is `kmagent` (so you can target it directly |---|---| | Healthy | `kmagent.collector.up=1`, `state=running` | | A bad configuration was pushed | `kmagent.collector.up=0`, `reason=config_invalid` | -| The collector exited with an error | `kmagent.collector.up=0`, `reason=run_error`, restart count increases | -| The collector panicked | `kmagent.collector.up=0`, `reason=panic`, restart count increases (then it auto-restarts) | +| The collector exited with an error | `kmagent.collector.up=0`, `reason=run_error` | +| The collector panicked | `kmagent.collector.up=0`, `reason=panic` (then it auto-restarts) | | A graceful shutdown | `reason=shutdown` | -The agent restarts the collector automatically after a crash, and the restart count records that it happened. +The agent restarts the collector automatically after a crash. ## Alerting on a silent agent The one failure the agent cannot report is total silence. If the agent, host, or network goes down, nothing is emitted. Detect this with a dead-man's-switch alert in KloudMate that fires when `kmagent.up` stops arriving for a host. -## Enabling and tuning +## On by default, and tuning -Health reporting is controlled by the agent's settings. You can turn it on or off and set its interval through the agent configuration. These are the `health-enabled` and `health-interval` settings, which map to the `KM_HEALTH_ENABLED` and `KM_HEALTH_INTERVAL` environment variables. The default interval matches the agent's check-in interval. +Health reporting is on by default, so a freshly installed agent starts emitting these metrics without any extra step. To change how often it reports, or to turn it off, use the `health-enabled` and `health-interval` settings — either in the agent's configuration file or as the `KM_HEALTH_ENABLED` and `KM_HEALTH_INTERVAL` environment variables. Set `health-enabled: false` (or `KM_HEALTH_ENABLED=false`) to stop reporting. The default interval is 60 seconds. + +In autonomous auto-instrument mode the agent reports its health regardless of this setting. ## Next steps diff --git a/src/content/docs/docs/kloudmate-agent/troubleshooting.mdx b/src/content/docs/docs/kloudmate-agent/troubleshooting.mdx index 10c77a0c..c3f9bebc 100644 --- a/src/content/docs/docs/kloudmate-agent/troubleshooting.mdx +++ b/src/content/docs/docs/kloudmate-agent/troubleshooting.mdx @@ -35,7 +35,7 @@ Start with two agent features. The [health metrics](../self-observability/health ## Windows-specific -- **The .NET profiler did not attach.** Confirm the application pool runs a supported runtime (.NET Framework 4.6.2 or newer), that an ASP.NET Core pool is set to "No Managed Code," and that the pool was recycled. Legacy CLR 2.0 pools use ETW monitoring instead. See [Windows and .NET application APM](../auto-instrumentation/windows-dotnet/). +- **The .NET profiler did not attach.** Confirm the application pool runs a supported runtime (.NET Framework 4.6.2 or newer), that an ASP.NET Core pool is set to "No Managed Code," and that the pool was recycled. Legacy CLR 2.0 pools use ETW monitoring instead. See [.NET application APM](../auto-instrumentation/dotnet/). - **No ETW spans.** ETW monitoring needs the Local System privileges the agent runs with. See [Windows platform notes](../platform-notes/windows/). ## Kubernetes-specific diff --git a/src/content/docs/docs/kloudmate-agent/web-server-monitoring.mdx b/src/content/docs/docs/kloudmate-agent/web-server-monitoring.mdx index bd615967..535baf3c 100644 --- a/src/content/docs/docs/kloudmate-agent/web-server-monitoring.mdx +++ b/src/content/docs/docs/kloudmate-agent/web-server-monitoring.mdx @@ -7,7 +7,7 @@ sidebar: import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -The agent monitors nginx and Apache web servers by reading their built-in status modules. As with [database monitoring](../database-monitoring/overview/), it finds the servers and sets up collection for you. It applies to nginx and Apache on Linux hosts. It does **not** apply on **Windows**, where Internet Information Services (IIS) is covered by Event Tracing for Windows (ETW) monitoring instead. See [Windows and .NET application APM](../auto-instrumentation/windows-dotnet/). +The agent monitors nginx and Apache web servers by reading their built-in status modules. As with [database monitoring](../database-monitoring/overview/), it finds the servers and sets up collection for you. It applies to nginx and Apache on Linux hosts. It does **not** apply on **Windows**, where Internet Information Services (IIS) is covered by Event Tracing for Windows (ETW) monitoring instead. See [Windows platform notes](../platform-notes/windows/). ## What it collects diff --git a/src/content/docs/docs/platform/settings/data-sources.mdx b/src/content/docs/docs/platform/settings/data-sources.mdx index 7bc2d783..38e738cc 100644 --- a/src/content/docs/docs/platform/settings/data-sources.mdx +++ b/src/content/docs/docs/platform/settings/data-sources.mdx @@ -5,7 +5,8 @@ sidebar: label: "Data Sources" order: 5 --- -Use **Data Sources** to connect AWS accounts, review sync settings, and control which resources are imported into KloudMate. + +Use **Data Sources** to connect AWS accounts, review their sync settings, and check sync health across regions. ## Connect an AWS Account @@ -15,59 +16,87 @@ To connect a new account, follow [AWS Account Setup](../../../aws-integration/ac ## Manage Connected Accounts -Open **Settings -> Data Sources** to view all connected AWS accounts. +Open **Settings → Data Sources** to view all connected AWS accounts. + +Each connected account appears as a card showing the AWS logo, a connection-status indicator, and the account ID with a copy button. An **Edit** action opens the connection, and a menu gives you **Sync settings** and **CloudWatch Logs**. + +The connection-status indicator also flags access problems. When the delegation role KloudMate uses is missing a required permission, the account shows a **Permission error** state with the reason. That means the whole account's access is failing, not a single region. Open the account's **Edit** screen to see the same error alongside the connection details. To fix it, update the delegation role's permissions, then let KloudMate re-check on the next sync. See [Troubleshoot AWS sync](../troubleshoot-aws-sync/). From this page, you can: -- Add additional AWS accounts -- Open account details +- Add another AWS account +- Open account details and edit the connection - Review or update sync settings - Access CloudWatch log subscription controls - Remove an AWS account -![image](./images/data-sources-1.png) +![Connected AWS accounts on the Data Sources page](./images/data-sources-1.png) Enter the AWS account name and follow the on-screen setup flow when adding a new account. -![image](./images/data-sources-2.png) +![Adding an AWS account](./images/data-sources-2.png) -## Review Sync Settings +## Sync Settings -The data sources page shows every connected AWS account. +Sync Settings control what KloudMate inventories for a connected account. Open an account's menu and choose **Sync settings**. -![image](./images/data-sources-3.png) +Two account-level controls decide what gets synced: **Primary regions** and an optional **tag allowlist**. The set of resource types KloudMate inventories is platform-managed and the same for every account. See [Covered services](#covered-services). The **Region sync status** section on the same page surfaces any regions that have stopped syncing and lets an admin turn them back on. -Open the account details screen for the AWS account you want to manage. +![The Sync Settings page, showing Region sync status, Primary regions, Tag Filtering, and Covered services](./images/data-sources-3.png) -![image](./images/data-sources-4.png) +### Primary regions -Use the account settings view to control sync behavior and review CloudWatch log options. +**Primary regions** are the regions where the account's main workloads run. KloudMate syncs primary regions most frequently, and other enabled regions refresh about once a day. Set them to keep your busiest regions the freshest. -![image](./images/data-sources-5.png) +Primary regions must be a subset of the regions enabled for sync, so the picker offers only regions that are currently syncing. Leave the field empty and KloudMate detects the primary regions for you, based on where the account's resources actually live. -Enable or disable sync for individual services and regions based on what you want KloudMate to monitor. +### Region sync status -![image](./images/data-sources-6.png) +KloudMate normally syncs every region that's enabled for an account. If a sync hits an AWS permission error in one region, KloudMate turns that region off so it stops retrying a region it can't read, and the rest of the account keeps syncing. The **Region sync status** section lists any regions that are currently turned off and not syncing. -## Tag Filtering +For each region, the panel shows why it stopped: -Tag filtering lets you limit syncing to AWS resources that match specific tags. +- If a permission error disabled it, the recorded reason appears, for example an `AccessDenied` message naming the missing permission. +- If it was turned off some other way, it shows **Disabled. No recent error recorded.** -1. Open the sync settings screen. -2. Click **Add Tags** under **Tag Filtering**. +A workspace admin can select **Re-enable** on a region. That turns the region back on, clears its stored error, and starts a sync right away, so its resources return without waiting for the next cycle. Non-admins see the status but can't re-enable. + +Most permission problems recover on their own once you fix the underlying access. To find the cause and confirm the fix, see [Troubleshoot AWS sync](../troubleshoot-aws-sync/). + +### Tag filtering + +The tag allowlist limits syncing to resources that carry a tag you list. Add one or more `name = value` tags and KloudMate syncs only resources matching at least one of them. Leave the list empty to sync everything. + +- Tags combine with **OR**: a resource matches if it carries any one of the listed tags. +- Tag matching is **case-sensitive**, so `env = Prod` and `env = prod` are different. -![image](./images/data-sources-7.png) +To set tags: -3. Enter the tags you want to use. +1. Open the account's **Sync settings**. +2. Click **Add Tags** under **Tag Filtering**. +3. Enter the tags you want to match. 4. Save the configuration. -![image](./images/data-sources-8.png) +![Tags entered under Tag Filtering, ready to save](./images/data-sources-8.png) If tags are already configured, use the edit action to update or remove them. -![image](./images/data-sources-9.png) +![Editing configured tags](./images/data-sources-9.png) + +### Covered services + +KloudMate inventories a fixed set of about 52 AWS resource types, the same for every account. Sync Settings shows them as a read-only **Covered services** list, grouped by AWS category (Compute, Storage, Database, Networking, and so on) with each service's icon. + +This list is platform-managed and applies to every connected account. There's no per-service on/off switch. You scope what's inventoried with regions and tags, not by toggling individual services. + +## Sync now + +To refresh an account without waiting for the scheduled cycle, use **Sync now**. It refreshes the cloud inventory and the per-region resource sync together, so one action brings everything for the account up to date. + +Reach for it after you re-enable a region or fix a permission problem and want the account's resources to reappear right away. ## Related Resources - [AWS Account Setup](../../../aws-integration/account-setup/) +- [Troubleshoot AWS sync](../troubleshoot-aws-sync/) - [CloudWatch Logs](../../../aws-integration/cloudwatch-logs/) diff --git a/src/content/docs/docs/platform/settings/images/data-sources-1.png b/src/content/docs/docs/platform/settings/images/data-sources-1.png index bd0999ab..4a1a2baf 100644 Binary files a/src/content/docs/docs/platform/settings/images/data-sources-1.png and b/src/content/docs/docs/platform/settings/images/data-sources-1.png differ diff --git a/src/content/docs/docs/platform/settings/images/data-sources-2.png b/src/content/docs/docs/platform/settings/images/data-sources-2.png index c2362087..27427e1e 100644 Binary files a/src/content/docs/docs/platform/settings/images/data-sources-2.png and b/src/content/docs/docs/platform/settings/images/data-sources-2.png differ diff --git a/src/content/docs/docs/platform/settings/images/data-sources-3.png b/src/content/docs/docs/platform/settings/images/data-sources-3.png index 6f282a77..6c9b0130 100644 Binary files a/src/content/docs/docs/platform/settings/images/data-sources-3.png and b/src/content/docs/docs/platform/settings/images/data-sources-3.png differ diff --git a/src/content/docs/docs/platform/settings/images/data-sources-7.png b/src/content/docs/docs/platform/settings/images/data-sources-7.png deleted file mode 100644 index f02080f4..00000000 Binary files a/src/content/docs/docs/platform/settings/images/data-sources-7.png and /dev/null differ diff --git a/src/content/docs/docs/platform/settings/images/data-sources-8.png b/src/content/docs/docs/platform/settings/images/data-sources-8.png index b29169be..77e7979e 100644 Binary files a/src/content/docs/docs/platform/settings/images/data-sources-8.png and b/src/content/docs/docs/platform/settings/images/data-sources-8.png differ diff --git a/src/content/docs/docs/platform/settings/images/data-sources-9.png b/src/content/docs/docs/platform/settings/images/data-sources-9.png index 8ff5936b..51097f56 100644 Binary files a/src/content/docs/docs/platform/settings/images/data-sources-9.png and b/src/content/docs/docs/platform/settings/images/data-sources-9.png differ diff --git a/src/content/docs/docs/platform/settings/index.mdx b/src/content/docs/docs/platform/settings/index.mdx index 09bececb..a752e363 100644 --- a/src/content/docs/docs/platform/settings/index.mdx +++ b/src/content/docs/docs/platform/settings/index.mdx @@ -15,6 +15,7 @@ The Settings area is where you manage workspaces, users, security, billing, and - [Single Sign-On (SSO)](./sso/) - [Profile & Security](./profile-security/) - [Data Sources](./data-sources/) +- [Troubleshoot AWS sync](./troubleshoot-aws-sync/) - [Remove an AWS Account](./deleting-your-aws-account-from-kloudmate/) - [Notification Channels](./notification-channels/) - [Subscription & Billing](./subscription-billing/) diff --git a/src/content/docs/docs/platform/settings/troubleshoot-aws-sync.mdx b/src/content/docs/docs/platform/settings/troubleshoot-aws-sync.mdx new file mode 100644 index 00000000..c13de603 --- /dev/null +++ b/src/content/docs/docs/platform/settings/troubleshoot-aws-sync.mdx @@ -0,0 +1,49 @@ +--- +title: "Troubleshoot AWS sync" +description: "Fix AWS accounts and regions that stop syncing in KloudMate after a permission error, and understand how they recover on their own." +sidebar: + label: "Troubleshoot AWS sync" + order: 5.5 +--- + +When KloudMate can't read part of an AWS account, it narrows the problem instead of failing the whole account. A single region that hits a permission error gets turned off, while an account whose delegation role loses access is flagged outright. This page explains why that happens, how to fix it, and how KloudMate recovers once access is restored. + +Start at **Settings → Data Sources**. The account card shows the connection status, and each account's **Sync settings** has a [Region sync status](../data-sources/#region-sync-status) section that lists any regions that stopped syncing. + +## A region stopped syncing + +When a sync job for a region hits an AWS permission error (an `AccessDenied` or "not authorized" response), KloudMate disables that region for the account. This is deliberate. It stops KloudMate from retrying a region it can't read, so one broken region doesn't hold back the rest of the account. The region then shows up in **Region sync status** with the recorded reason. + +Two causes are common: + +- **The delegation role is missing a permission.** The cross-account IAM role KloudMate uses doesn't grant an action it needs for a resource type in that region. +- **The region isn't enabled in the AWS account.** KloudMate tried a region the account hasn't turned on, so AWS rejects the call. + +To fix a missing permission, update the KloudMate delegation stack: + +1. Sign in to the [AWS Console](https://console.aws.amazon.com/) and open **CloudFormation → Stacks**. +2. Redeploy or update the `kloudmate-delegation` stack so the role grants the permission named in the error. Reconnecting the account through [AWS Account Setup](../../../aws-integration/account-setup/) gives you the current template. +3. Back in KloudMate, either wait for automatic recovery (below) or open **Region sync status** and select **Re-enable** on the region. + +## Regions recover automatically + +Once the underlying permission is fixed, you usually don't need to do anything. KloudMate re-probes the regions it disabled for a recorded error and re-enables them on the next successful sync. Their resources come back on their own, with no manual step. + +Automatic recovery applies only to regions that carry a recorded error. A region that's off with **Disabled. No recent error recorded.** won't come back by itself, because there's nothing for KloudMate to re-probe. Re-enable those manually from **Region sync status**. + +To bring a region back immediately instead of waiting for the next cycle, select **Re-enable**, or use **Sync now** on the account after you've corrected the permission. + +## The whole account shows a permission error + +A **Permission error** on the account itself is different from a single region. It means the delegation role is missing permissions KloudMate needs across the board, so the account's access is failing rather than one region's. You'll see it on the account card in **Data Sources** and on the account's **Edit** screen, both with the reason. + +The fix is the same kind of change as a region permission error, applied to the whole account: + +1. Update the `kloudmate-delegation` CloudFormation stack so the role has the permissions it's missing. +2. KloudMate re-checks access on the next sync. When the role can read the account again, the **Permission error** state clears on its own. + +## Related Resources + +- [Data Sources](../data-sources/) +- [AWS Account Setup](../../../aws-integration/account-setup/) +- [Remove an AWS Account](../deleting-your-aws-account-from-kloudmate/) diff --git a/src/content/docs/docs/reliability/service-reliability-tab.mdx b/src/content/docs/docs/reliability/service-reliability-tab.mdx index cea21022..57f7fda5 100644 --- a/src/content/docs/docs/reliability/service-reliability-tab.mdx +++ b/src/content/docs/docs/reliability/service-reliability-tab.mdx @@ -43,17 +43,6 @@ Clicking a card opens the SLO detail page. Matching is **service-only** in v1 — every incident-based SLO on the affected service appears, even if the SLO's severity / metadata filters might exclude this specific incident. Slight over-reporting is the deliberate trade-off; re-evaluating SLO logic in the browser would be fragile. -## Alert detail — Define an SLO - -The alert detail page header now includes a **Define an SLO** outlined button (admin-only) next to the existing actions (Ask AI, Jira integration, Edit Alert). Clicking it opens the SLO create wizard at `/<workspaceId>/slos/create`. - -:::note[Placeholder image] -Screenshot pending — alert detail page header showing the Define an SLO CTA next to Edit Alert and Ask AI. -::: -![Alert detail Define an SLO CTA](../_placeholders/screenshot-placeholder.png) - -The CTA is a discovery prompt — consider defining an SLO covering the same condition as this alert. It does **not** auto-prefill the wizard because alert configurations don't map cleanly to SLI / target / window choices. - ## Related - [Reliability Overview](../) — workspace-wide hub. diff --git a/src/content/docs/guides/opentelemetry/react.mdx b/src/content/docs/guides/opentelemetry/react.mdx index 4e5d55b3..4f12ddbd 100644 --- a/src/content/docs/guides/opentelemetry/react.mdx +++ b/src/content/docs/guides/opentelemetry/react.mdx @@ -65,7 +65,7 @@ const provider = new WebTracerProvider({ } }))], resource: resourceFromAttributes({ - 'service.name': 'TODO-Frontend', + 'service.name': 'YOUR_SERVICE_NAME', 'service.version': '1.0', }), })