Skip to content

docs(analytics): document --include-external and refresh the report doc #529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yanaSelin merged 1 commit into from
Sep 4, 2026
Merged

docs(analytics): document --include-external and refresh the report doc #529

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 68 additions & 11 deletions docs/ANALYTICS-REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,23 @@ codemie analytics --report --report-output ~/reports/weekly.html

# Also export the underlying data as JSON
codemie analytics --report --report-format both

# Include ALL local agent usage — also the sessions you ran outside CodeMie
codemie analytics --report --open --include-external
```

> **If your question is "what did AI actually cost us?", you probably want `--include-external`.**
> By default the report counts only sessions CodeMie launched. Any time you ran `claude`,
> `codex`, `gemini`, `pi`, or `copilot` directly, that spend is discovered but withheld.
> See [Session provenance](#session-provenance) for why, and for the trade-off.

---

## What the Report Covers

The dashboard reads every AI session CodeMie has tracked — Claude Code, Gemini, Codex, OpenCode, and the built-in agent — plus native agent logs it discovers automatically. It builds a single portable HTML file with **nine interactive views**.
The dashboard reads every AI session CodeMie has tracked — Claude Code, Codex, Gemini, OpenCode, Pi, GitHub Copilot CLI, and the built-in agent — plus native agent logs it discovers automatically on disk. It builds a single portable HTML file with **nine interactive views**, grouped in the sidebar as *Insights*, *Spend*, and *Raw*.

Discovered sessions that CodeMie did not launch are **excluded by default**; see [Session provenance](#session-provenance).

---

Expand Down Expand Up @@ -151,7 +161,9 @@ Key elements:

![Sessions](assets/analytics-report-sessions.png)

Full paginated table of every session (up to 300 shown; searchable). Columns: date, agent, project, branch, turns, net lines, input/output/cached tokens, cost.
Full paginated table of every session (up to 300 shown; searchable). Columns: date, first prompt, agent, project, branch, source, turns, net lines, input/output/cached tokens, cost.

The **Source** column carries the framework classification described under [Frameworks · Compare](#frameworks--compare), so you can scan tooling adoption without leaving the raw table.

Click any row to open the **session detail modal**.

Expand All @@ -170,6 +182,7 @@ Clicking a session anywhere in the report opens a detail overlay with:
- **Token & cost growth chart** — cumulative cost and token usage per turn (from the native log; shown when data is available)
- **Dispatch timeline** — Interactive Gantt of every top-level dispatch. A session bar spans the full activity window across the top, then each agent, skill, and slash-command dispatch is rendered as its own bar positioned by wall-clock start time, so each sits where it actually ran. The window is the union of the tracked session span and the dispatch span, so dispatches from a resumed or compacted session still place correctly. Short skills and zero-duration commands fall back to a minimum bar width so they stay visible as markers. Click any bar to open a detail panel on the right showing wall-clock duration, start offset from session start, top tool call counts, and — for agent and skill dispatches where usage can be attributed to their time window — estimated cost and token breakdown (input / output / cache read / cache write). Slash-command dispatches are point events with no window to attribute usage from, so they show timing only, never a cost.
- **Skills / Agent subtypes / Slash commands** — chip lists of what was invoked and how many times
- **Copy buttons** — copy the session ID or the transcript's file location to the clipboard, for pasting into a bug report or opening the raw log directly

---

Expand All @@ -193,18 +206,25 @@ Filters apply to every view simultaneously. The URL does not update, so share th
|---|---|---|
| HTML dashboard | `--report` or `--report-format html` | Self-contained `.html` with all charts and data embedded |
| JSON data | `--report-format json` | The cost-enriched session payload — useful for further analysis in notebooks or BI tools |
| Both | `--report-format both` | Writes `.html` and `.json` with a shared base name |
| Both | `--report-format both` | Writes the `.html` and the `.report.json` side by side with a shared base name |

Default output paths, in the current directory:

- HTML — `./codemie-analytics-<email-slug>-YYYY-MM-DD.html`
- JSON — `./codemie-analytics-<email-slug>-YYYY-MM-DD.report.json`

Default output path: `./codemie-analytics-YYYY-MM-DD.html` in the current directory. Override with `--report-output <path>`.
The JSON report deliberately ends in `.report.json` rather than `.json` so it can never collide with the very different file `--export json` writes. Override either with `--report-output <path>`.

**Report metadata and your email.** Reports embed the reporting user's email plus the period covered. The address is read from your CodeMie config; if it is missing and you're on an interactive terminal, report generation warns and prompts for it once, then saves it for future runs. Declining the prompt cancels report generation. The `<email-slug>` segment is dropped from the filenames when no email is available.

---

## Automatic Per-Session Report on Exit

When a `codemie-claude`, `codemie-codex`, or `codemie-opencode` session exits, CodeMie automatically writes a JSON analytics report for that session to `./docs/codemie/analytics/codemie-analytics-<session_id>.json`.
When an interactive agent session exits — `codemie-claude`, `codemie-codex`, `codemie-opencode`, `codemie-pi`, or `codemie-copilot` — CodeMie automatically writes a JSON analytics report for that session to `./docs/codemie/analytics/codemie-analytics-<session_id>.json`.

- **Enabled by default** — it runs in-process as part of session finalization, not as a separate command.
- **Disable per run** with `--no-analytics-report`, e.g. `codemie claude --no-analytics-report ...` (same for `codemie codex` and `codemie opencode`).
- **Disable per run** with `--no-analytics-report`, e.g. `codemie claude --no-analytics-report ...` (the flag is available on every agent that supports the feature).
- **Non-fatal** — a failure never blocks session exit, and the report is skipped when the session produced no analytics data.

---
Expand All @@ -213,13 +233,44 @@ When a `codemie-claude`, `codemie-codex`, or `codemie-opencode` session exits, C

CodeMie merges two sources to give the most complete picture:

1. **Tracked sessions** — metrics written by the CodeMie hooks during your sessions
2. **Native agent logs** — transcripts left by `claude`, `gemini`, and other agents that ran outside of CodeMie (discovered automatically; deduped against tracked sessions)
1. **Tracked sessions** — metrics written by the CodeMie hooks during sessions CodeMie launched
2. **Native agent logs** — transcripts left on disk by `claude`, `codex`, `gemini`, `pi`, and `copilot`, discovered automatically and deduped against tracked sessions

Pass `--no-scan-native` to disable native-log discovery and use only CodeMie-tracked sessions.

Discovery looks back as far as your date filter requires: with `--from` or `--last` the window is that range, and with no lower bound it is effectively unlimited.

Cost enrichment requires the native log to read per-turn token data. Sessions where the log has already been rotated or deleted will appear with `—` cost; the **Coverage** section in the Cost view shows exactly which sessions are priced.

<a id="session-provenance"></a>

### Session provenance — and why some sessions are hidden

Finding a native log is not the same as counting it. Every discovered session is tagged with a **provenance**, recorded on the session's start event as `provider`:

| Provenance | Meaning | In the report by default? |
|---|---|---|
| `native` | CodeMie launched this session (`codemie-claude`, `codemie-codex`, …). Its transcript carries a `codemie_session_start` ownership marker. | ✅ Yes |
| `native-external` | The same agent, run **directly** — `claude`, `codex`, `gemini`, `pi`, `copilot` — with no CodeMie involvement, so no ownership marker. | ❌ **No** — opt in with `--include-external` |
| `native-unmanaged` | Reserved for agents CodeMie can only ever read analytics for and never launches. No agent currently carries this tag. | ✅ Yes |

The default exists so that a report titled "CodeMie usage" measures CodeMie usage: without the ownership gate, every unmanaged run of an agent CodeMie *can* manage would be silently folded into CodeMie's numbers.

That default is the right one for adoption reporting and the **wrong** one for consumption reporting. If you want total local AI spend across every agent on the machine, ask for it:

```bash
codemie analytics --report --open --include-external
```

**This is the flag that shows all of your local agent usage.** GitHub Copilot CLI sessions are included in the gate, so they too are absent from the default report.

Two things to know before you rely on the wider number:

- **It is broader but less precise.** An external session has no CodeMie run to attribute it to — no profile, no managed provider — and its cost depends entirely on a native log that may already have been rotated away. Expect a lower priced-session ratio in the Cost view's **Coverage** banner than you'd see for CodeMie-launched sessions.
- **`--include-external` needs native scanning.** External sessions *are* discovered natives, so `--no-scan-native --include-external` adds nothing — the first flag suppresses the very sessions the second one asks for.

`--include-external` applies to the default local-session source only. The `analytics otel` subcommand does not accept it — an OTEL events file has no notion of CodeMie ownership.

### OTEL events file (`analytics otel`)

As an alternative to the local-session sources above, the `analytics otel` subcommand builds the same report from a **flattened OTEL events file** (`otel-events.jsonl`) — for example, telemetry exported from a fleet or CI environment rather than the current machine's history.
Expand Down Expand Up @@ -256,14 +307,20 @@ Filter flags:
--branch <name> Filter by git branch
--session <id> Filter to a single session

Source flags:
--no-scan-native Skip native-log discovery (CodeMie-tracked sessions only)
--include-external Also count local sessions CodeMie did not launch
(see "Session provenance"; requires native scanning)

Other flags:
-v, --verbose Session-level breakdown in the terminal output
--export <fmt> Export terminal data to json or csv file
-o, --output <path> Output path for --export
--no-scan-native Skip native-log discovery
```

All filter flags work for both the terminal output and the HTML report. The date filters control which sessions are **embedded** in the report; the client-side range presets (Today / 7d / 30d / 90d) then let the report viewer narrow further within that data.
**Every filter and source flag governs the terminal output and the HTML report alike.** There is no report-only or terminal-only filtering: `--include-external`, `--no-scan-native`, and the date/project/agent filters all decide which sessions the command sees, and both outputs are rendered from that same set.

The date filters control which sessions are **embedded** in the report; the client-side range presets (Today / 7d / 30d / 90d) then let the report viewer narrow further within that data.

### OTEL source subcommand

Expand All @@ -274,4 +331,4 @@ codemie analytics otel --file <path> [options]
--user <id> Scope to one user (native user.email or user.id)
```

All filter, report, and export flags from the base command also apply to `analytics otel`. Cost is read from each event's native `cost_usd`, so `--no-scan-native` is not used by this source.
All filter, report, and export flags from the base command also apply to `analytics otel`. The source flags do not: an OTEL events file is neither scanned for native logs nor gated on CodeMie ownership, so `--no-scan-native` and `--include-external` have no meaning here. Cost is read from each event's native `cost_usd`.