Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"displayName": "PostHog",
"source": "./",
"description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from your AI coding tool. Optionally capture Claude Code sessions to PostHog LLM Analytics.",
"version": "1.1.63",
"version": "1.1.64",
"author": {
"name": "PostHog",
"email": "hey@posthog.com",
Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "posthog",
"description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from your AI coding tool. Optionally capture Claude Code sessions to PostHog LLM Analytics.",
"version": "1.1.63",
"version": "1.1.64",
"author": {
"name": "PostHog",
"email": "hey@posthog.com",
Expand Down
2 changes: 1 addition & 1 deletion .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "posthog",
"version": "1.0.61",
"version": "1.0.62",
"description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from Codex",
"author": {
"name": "PostHog",
Expand Down
2 changes: 1 addition & 1 deletion .cursor-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "posthog",
"displayName": "PostHog",
"version": "1.1.57",
"version": "1.1.58",
"description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from Cursor",
"author": {
"name": "PostHog",
Expand Down
2 changes: 1 addition & 1 deletion gemini-extension.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "posthog",
"version": "1.0.59",
"version": "1.0.60",
"description": "Access PostHog analytics, feature flags, experiments, error tracking, and insights directly from Gemini CLI",
"mcpServers": {
"posthog": {
Expand Down
10 changes: 3 additions & 7 deletions skills/.sync-manifest
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@ authoring-scouts
building-a-dashboard
building-canvases
building-html-canvases
building-loops
building-react-quill-canvases
building-workflows
checking-deploy-timing
checking-member-access
choosing-trend-or-slope-view
cleaning-up-stale-feature-flags
composing-grid-canvases
Expand All @@ -34,10 +36,8 @@ creating-box-plot-insights
creating-experiments
creating-online-evaluations
creating-replay-vision-scanners
creating-surveys
debugging-experiments
debugging-local-replay
debugging-mcp-analytics
debugging-signals-pipeline
debugging-surveys
designing-email-templates
diagnosing-ci-and-merge-bottlenecks
Expand Down Expand Up @@ -72,7 +72,6 @@ finding-replay-for-issue
finding-sessions-to-watch
formatting-insight-axes
grouping-noisy-errors
improving-mcp-tools
inbox-exploration
instrument-error-tracking
instrument-feature-flags
Expand All @@ -99,7 +98,6 @@ modeling-dimension-tables
modeling-product-usage-metrics
modeling-revenue-metrics
modeling-warehouse-foundations
organizing-conversations-code
planning-voice-agent-user-interviews
querying-canvas-data
querying-posthog-data
Expand All @@ -115,7 +113,6 @@ scanning-experiments-with-replay-vision
setting-up-a-custom-rest-source
setting-up-a-data-warehouse-source
setting-up-data-catalog
setting-up-support-slack-locally
setting-up-warehouse-properties
signals
signals-scout-ai-observability
Expand Down Expand Up @@ -150,7 +147,6 @@ skills-store
suggesting-data-imports
suggesting-path-cleaning-rules
suppressing-noisy-errors
testing-mcp-tools-locally
triaging-error-issues
triaging-visual-review-runs
tuning-incremental-sync-config
Expand Down
2 changes: 1 addition & 1 deletion skills/adding-warehouse-person-properties/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ the `external_data_schema` value the source needs. A table name alone is not eno

```sql
select column_name, data_type
from information_schema.columns
from system.information_schema.columns
where table_name = '<table name>'
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,16 +50,15 @@ Verifies the experiment's linked feature flag is valid and correctly configured.
- Report: "The linked feature flag has been deleted."
- Action: Create a new flag and re-link it, or archive the experiment.

- **Uneven variant split**: The linked flag's variant rollout percentages differ from the experiment's expected split by more than 5 percentage points.
Compare the flag's `filters.multivariate.variants` rollout percentages to the experiment's `parameters.feature_flag_variants`.
- **Incomplete variant split**: The rollout percentages in the flag's `filters.multivariate.variants` don't add up to 100.
- Severity: WARNING · Category: Correctness
- Report: "Variant rollout percentages on the flag don't match the experiment's expected split."
- Action: Adjust the flag's variant percentages to match the experiment configuration.
- Report: "The flag's variant rollout percentages don't add up to 100%. Some matched users get no variant."
- Action: Adjust the variant percentages on the flag so they total 100%.

- **Variant mismatch**: The variant keys in the experiment's `parameters.feature_flag_variants` don't match the variant keys in the flag's `filters.multivariate.variants`.
- **Too few variants**: The flag's `filters.multivariate.variants` holds fewer than two variants.
- Severity: CRITICAL · Category: Correctness
- Report: "Variant keys differ between the experiment and its linked flag."
- Action: Align variant keys between the experiment and its flag.
- Report: "The linked feature flag has fewer than two variants. There is nothing to compare."
- Action: Add a test variant to the flag alongside the baseline.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ Phase 1 is read-only — all actions require the user to make changes manually.
| Missing feature flag | Create and link a feature flag to this experiment. |
| Inactive (paused) flag | Re-enable the linked feature flag, or end the experiment if it's no longer needed. |
| Deleted flag | The experiment's flag was deleted. Create a new flag and re-link, or archive the experiment. |
| Uneven variant split | Adjust variant rollout percentages on the linked flag to match the experiment's expected split. |
| Variant mismatch | Align the variants between the experiment and its linked flag — they must use the same variant keys. |
| Incomplete variant split | Adjust the variant rollout percentages on the linked flag so they total 100%. |
| Too few variants | Add a test variant to the linked flag, so it has a baseline and at least one variant to compare. |
| Conclusion contradicts shipped variant | Review the experiment conclusion and the flag's current state. Either update the conclusion or change the flag to match. |
| Concluded but still splitting | The experiment has a conclusion but the flag is still splitting traffic. Roll out the winning variant or disable the flag. |
| Stale draft | This experiment has been in draft for over 7 days. Either launch it or delete it. |
Expand Down
59 changes: 49 additions & 10 deletions skills/authoring-data-quality-checks/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
name: authoring-data-quality-checks
description: >
Adds and runs data quality checks (dbt-test style assertions) on a project's warehouse tables and
saved-query views: not-null, uniqueness, accepted values, referential integrity, row-count bounds,
freshness, and custom HogQL. Use when asked to test a model, validate a view, check for nulls or
saved-query views, and HogQL catalog metrics: not-null, uniqueness, accepted values, referential
integrity, row-count bounds, freshness, and custom HogQL. Metrics support custom SQL checks only.
Use when asked to test a model, validate a view, check for nulls or
duplicates, add data quality checks, find out why a number looks wrong, or judge whether a warehouse
table is trustworthy before using it in an analysis. To describe what data *means* (metrics,
certifications, joins), see setting-up-data-catalog instead. Trigger terms: data quality, data test,
Expand All @@ -13,8 +14,8 @@ description: >

# Authoring data quality checks

A check is one assertion about one warehouse table or view. It compiles to a count-only HogQL query
and **passes when it finds zero failing rows** — the same semantics as `dbt test`. Failing rows are
A check is one assertion about one warehouse table, view, or HogQL catalog metric. It compiles to a
count-only HogQL query and **passes when it finds zero failing rows**, like `dbt test`. Failing rows are
never stored; only counts and the compiled query are, so to see the offending rows you re-run the
stored query yourself.

Expand Down Expand Up @@ -45,9 +46,8 @@ WHERE table_name = 'orders'
Re-creating a byte-identical check is a harmless no-op — checks are keyed by a fingerprint of the
subject, type, column, and config, so an identical create upserts. A _near_-duplicate is not
harmless: it doubles the noise for whoever reads the results. If an existing check's assertion is
close but wrong, create the corrected check and delete the old one — the assertion (type, column,
config) is immutable and the subject is fixed by the URL, so an update that tries to change them is
rejected. Update is only for metadata, severity, and ownership.
close but wrong, edit the existing check. Updates preserve its identity and history; the subject
stays fixed by the URL. An edit that duplicates another check's assertion is rejected.

## Choosing checks

Expand All @@ -73,19 +73,57 @@ Call `posthog:data-quality-check-types` for each type's exact config schema rath
Checks live on the subject they audit: create them with `data-quality-check-create-on-view`
(`saved_query_id` path parameter) or `data-quality-check-create-on-table` (`table_id`).

## Checks on catalog metrics

Only metrics with a saved `HogQLQuery` definition support checks. Markdown, Trends, Funnels, event
series, and metrics without definitions do not. A metric can return any number of rows and columns.

Create a `custom_sql` check with an empty `column_name`. Include `{metric}` exactly once as a relation:

```sql
SELECT *
FROM {metric}
WHERE orders < 100
```

The check queries the owning metric's current saved output. The example assumes that output has an
`orders` column. Every returned row is a failure; zero rows passes. Query `{metric}` directly or
through a subquery. Metric check SQL cannot define CTEs, including nested CTEs and scalar `WITH`
bindings. CTEs and saved parameters inside the metric definition remain supported. Other placeholders
are not accepted in the check.

Use the metric's Tests tab or the nested REST endpoints under
`/api/projects/{project_id}/data_catalog/metrics/{metric_id}/checks/`. The catalog metric detail
endpoint uses the metric name, but nested check endpoints use its UUID. The metric check-type
endpoint offers only Custom SQL. Do not assume the table/view MCP tools accept metric subjects.

Saving validates SQL composition without executing it. Run the check to verify column names and
results. Every run reloads the saved metric: if an edit removes a column used by the check, the next
run errors. Fix the check SQL or restore the expected metric output.

## Severity and triggers

**Severity** is a decision about consequences, not about confidence. Use `error` when the failure
means downstream numbers should not be trusted — those failures mark the subject `failing` and
notify. Use `warn` for things worth surfacing that nobody would act on today. When unsure, `warn` is
the safer default: an `error` check that cries wolf gets everything ignored.

**Triggers** — there is nothing to schedule. A check runs when its subject's data changes: a
**Table and view triggers:** A check runs when its subject's data changes: a
materialized view's checks run as part of its refresh (and, when the team turns the gate on, a
refresh whose error-severity checks fail is not published), a source table's checks run after each
completed sync, and a plain view's checks run when its DAG runs. Checks on a view outside any DAG
only run on demand.

**Metric schedules:** The first saved check creates an enabled daily schedule for all checks on the
metric. The Tests tab lets you change the interval or turn automatic runs off. Manual runs remain
available. Scheduled checks use the latest definition author's access, falling back to the creator;
manual runs use the initiating user's access. Underlying and additional tables must be readable.

Temporal owns each metric's cadence and pause state. Paused schedules have no next execution time.
Reload after an unavailable schedule response before retrying an edit; the edit may have succeeded.
Automatic runs skip overlaps and catch up missed occurrences only within 15 minutes.
Disabling or deleting every check preserves the schedule preferences. Deleting the metric removes its schedule.

## Verify what you wrote

Author, run once, read the result. A check nobody has run is a guess.
Expand Down Expand Up @@ -119,8 +157,9 @@ FROM system.information_schema.data_quality_health
- `unknown` / absent — no checks, or none have run. Absence of failures is not evidence of health.

For the history behind a verdict, `system.information_schema.data_quality_check_runs` carries recent
executions with `observed_value` recorded on passes too, so you can see when a number started
drifting rather than just that it is wrong now.
executions with status, failing-row count, and errors. For metric checks, the failing-row count
describes the assertion result, not a scalar metric value. Open the failing-row query from the
Tests tab to inspect the current rows that violate the check.

## Related

Expand Down
25 changes: 20 additions & 5 deletions skills/authoring-error-tracking-alerts/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ actually act on.

| Tool | Job | Where it fits |
| ---------------------------------------------- | ---------------------------------------------------------------- | ---------------------------- |
| `posthog:error-tracking-alerts-list` | List existing alerts; dedupe before creating. | Step 2 — dedupe. |
| `posthog:error-tracking-alerts-list` | List existing alerts (paginated); dedupe before creating. | Step 2 — dedupe. |
| `posthog:integrations-list` | Find the user's Slack workspace id (filter by `kind=slack`). | Step 3 — pick channel. |
| `posthog:integrations-channels-retrieve` | List Slack channels for a workspace. | Step 3 — pick channel. |
| `posthog:error-tracking-alerts-create` | Create the alert (HogFunction with `type=internal_destination`). | Step 4 — ship. |
Expand Down Expand Up @@ -70,15 +70,28 @@ You need three things from the user before creating anything:

### 2. Dedupe against existing alerts

Call `posthog:error-tracking-alerts-list`. Filter the response client-side by `filters.events[].id`.
Scan **every** destination before you decide. The endpoint is paginated and defaults to 100 rows per
page, so a single default call misses older alerts on a project with hundreds of destinations.

- If an alert exists for the **same event** delivering to the **same channel**, stop. Tell the user it
already exists and ask whether they want to change anything (in which case use
Call `posthog:error-tracking-alerts-list` with `type: ["internal_destination"]` and `limit: 1000`. If the
response still carries a non-null `next`, keep paging with `offset` until `next` is null. Then filter the
collected rows client-side by `filters.events[].id` and by any per-issue scope in `filters.properties`.

The list response carries no `inputs`, so it does not tell you which channel or URL a row delivers to.
Read the destination off `name` and `description` instead — the naming convention below puts the channel
in the name. When a row matches on event and scope but its name does not say where it delivers, do not
guess: show the user the matching names and ask whether one of them is the alert they mean.

- If an alert exists for the **same event** delivering to the **same channel** with the **same scope**,
stop. Tell the user it already exists and ask whether they want to change anything (in which case use
`error-tracking-alerts-partial-update`) or skip.
- Multiple alerts on the same event for the same channel produce duplicate Slack messages — the user
almost never wants this.
- Multiple alerts on the same event for **different** channels (e.g. one for `#oncall`, one for the
oncall webhook) is fine and sometimes intentional. Confirm.
- Multiple alerts on the same event and channel but **different scopes** (e.g. two `_reopened` alerts
each pinned to a different `$exception_issue_id`) are not duplicates. Confirm before adding. Note that
a scoped alert and an all-issues alert to the same channel both fire on the scoped issue.

PostHog's "alerts configured" recommendation only inspects `filters.events` — adding per-issue
`filters.properties` does not affect the status the recommendations card reports.
Expand Down Expand Up @@ -156,7 +169,9 @@ the issue evolves.

## Token-economy rules

- One `posthog:error-tracking-alerts-list` call up front, not per candidate.
- One complete `posthog:error-tracking-alerts-list` scan up front, not per candidate. The scan is
complete only when `next` is null — page through it once and reuse the collected rows for every
candidate.
- Reuse a single integration lookup for multiple alerts going to the same workspace.
- Confirm the channel / URL with the user **before** creating each alert. Never batch-create alerts to a
destination the user has not explicitly named.
Expand Down
Loading
Loading