diff --git a/docs/ANALYTICS-REPORT.md b/docs/ANALYTICS-REPORT.md index 03e9d0f44..04c43c77c 100644 --- a/docs/ANALYTICS-REPORT.md +++ b/docs/ANALYTICS-REPORT.md @@ -24,13 +24,23 @@ codemie analytics --report --report-output ~/reports/weekly.html # Also export the underlying data as JSON codemie analytics --report --report-format both + +# Include ALL local agent usage — also the sessions you ran outside CodeMie +codemie analytics --report --open --include-external ``` +> **If your question is "what did AI actually cost us?", you probably want `--include-external`.** +> By default the report counts only sessions CodeMie launched. Any time you ran `claude`, +> `codex`, `gemini`, `pi`, or `copilot` directly, that spend is discovered but withheld. +> See [Session provenance](#session-provenance) for why, and for the trade-off. + --- ## What the Report Covers -The dashboard reads every AI session CodeMie has tracked — Claude Code, Gemini, Codex, OpenCode, and the built-in agent — plus native agent logs it discovers automatically. It builds a single portable HTML file with **nine interactive views**. +The dashboard reads every AI session CodeMie has tracked — Claude Code, Codex, Gemini, OpenCode, Pi, GitHub Copilot CLI, and the built-in agent — plus native agent logs it discovers automatically on disk. It builds a single portable HTML file with **nine interactive views**, grouped in the sidebar as *Insights*, *Spend*, and *Raw*. + +Discovered sessions that CodeMie did not launch are **excluded by default**; see [Session provenance](#session-provenance). --- @@ -151,7 +161,9 @@ Key elements: ![Sessions](assets/analytics-report-sessions.png) -Full paginated table of every session (up to 300 shown; searchable). Columns: date, agent, project, branch, turns, net lines, input/output/cached tokens, cost. +Full paginated table of every session (up to 300 shown; searchable). Columns: date, first prompt, agent, project, branch, source, turns, net lines, input/output/cached tokens, cost. + +The **Source** column carries the framework classification described under [Frameworks · Compare](#frameworks--compare), so you can scan tooling adoption without leaving the raw table. Click any row to open the **session detail modal**. @@ -170,6 +182,7 @@ Clicking a session anywhere in the report opens a detail overlay with: - **Token & cost growth chart** — cumulative cost and token usage per turn (from the native log; shown when data is available) - **Dispatch timeline** — Interactive Gantt of every top-level dispatch. A session bar spans the full activity window across the top, then each agent, skill, and slash-command dispatch is rendered as its own bar positioned by wall-clock start time, so each sits where it actually ran. The window is the union of the tracked session span and the dispatch span, so dispatches from a resumed or compacted session still place correctly. Short skills and zero-duration commands fall back to a minimum bar width so they stay visible as markers. Click any bar to open a detail panel on the right showing wall-clock duration, start offset from session start, top tool call counts, and — for agent and skill dispatches where usage can be attributed to their time window — estimated cost and token breakdown (input / output / cache read / cache write). Slash-command dispatches are point events with no window to attribute usage from, so they show timing only, never a cost. - **Skills / Agent subtypes / Slash commands** — chip lists of what was invoked and how many times +- **Copy buttons** — copy the session ID or the transcript's file location to the clipboard, for pasting into a bug report or opening the raw log directly --- @@ -193,18 +206,25 @@ Filters apply to every view simultaneously. The URL does not update, so share th |---|---|---| | HTML dashboard | `--report` or `--report-format html` | Self-contained `.html` with all charts and data embedded | | JSON data | `--report-format json` | The cost-enriched session payload — useful for further analysis in notebooks or BI tools | -| Both | `--report-format both` | Writes `.html` and `.json` with a shared base name | +| Both | `--report-format both` | Writes the `.html` and the `.report.json` side by side with a shared base name | + +Default output paths, in the current directory: + +- HTML — `./codemie-analytics--YYYY-MM-DD.html` +- JSON — `./codemie-analytics--YYYY-MM-DD.report.json` -Default output path: `./codemie-analytics-YYYY-MM-DD.html` in the current directory. Override with `--report-output `. +The JSON report deliberately ends in `.report.json` rather than `.json` so it can never collide with the very different file `--export json` writes. Override either with `--report-output `. + +**Report metadata and your email.** Reports embed the reporting user's email plus the period covered. The address is read from your CodeMie config; if it is missing and you're on an interactive terminal, report generation warns and prompts for it once, then saves it for future runs. Declining the prompt cancels report generation. The `` segment is dropped from the filenames when no email is available. --- ## Automatic Per-Session Report on Exit -When a `codemie-claude`, `codemie-codex`, or `codemie-opencode` session exits, CodeMie automatically writes a JSON analytics report for that session to `./docs/codemie/analytics/codemie-analytics-.json`. +When an interactive agent session exits — `codemie-claude`, `codemie-codex`, `codemie-opencode`, `codemie-pi`, or `codemie-copilot` — CodeMie automatically writes a JSON analytics report for that session to `./docs/codemie/analytics/codemie-analytics-.json`. - **Enabled by default** — it runs in-process as part of session finalization, not as a separate command. -- **Disable per run** with `--no-analytics-report`, e.g. `codemie claude --no-analytics-report ...` (same for `codemie codex` and `codemie opencode`). +- **Disable per run** with `--no-analytics-report`, e.g. `codemie claude --no-analytics-report ...` (the flag is available on every agent that supports the feature). - **Non-fatal** — a failure never blocks session exit, and the report is skipped when the session produced no analytics data. --- @@ -213,13 +233,44 @@ When a `codemie-claude`, `codemie-codex`, or `codemie-opencode` session exits, C CodeMie merges two sources to give the most complete picture: -1. **Tracked sessions** — metrics written by the CodeMie hooks during your sessions -2. **Native agent logs** — transcripts left by `claude`, `gemini`, and other agents that ran outside of CodeMie (discovered automatically; deduped against tracked sessions) +1. **Tracked sessions** — metrics written by the CodeMie hooks during sessions CodeMie launched +2. **Native agent logs** — transcripts left on disk by `claude`, `codex`, `gemini`, `pi`, and `copilot`, discovered automatically and deduped against tracked sessions Pass `--no-scan-native` to disable native-log discovery and use only CodeMie-tracked sessions. +Discovery looks back as far as your date filter requires: with `--from` or `--last` the window is that range, and with no lower bound it is effectively unlimited. + Cost enrichment requires the native log to read per-turn token data. Sessions where the log has already been rotated or deleted will appear with `—` cost; the **Coverage** section in the Cost view shows exactly which sessions are priced. + + +### Session provenance — and why some sessions are hidden + +Finding a native log is not the same as counting it. Every discovered session is tagged with a **provenance**, recorded on the session's start event as `provider`: + +| Provenance | Meaning | In the report by default? | +|---|---|---| +| `native` | CodeMie launched this session (`codemie-claude`, `codemie-codex`, …). Its transcript carries a `codemie_session_start` ownership marker. | ✅ Yes | +| `native-external` | The same agent, run **directly** — `claude`, `codex`, `gemini`, `pi`, `copilot` — with no CodeMie involvement, so no ownership marker. | ❌ **No** — opt in with `--include-external` | +| `native-unmanaged` | Reserved for agents CodeMie can only ever read analytics for and never launches. No agent currently carries this tag. | ✅ Yes | + +The default exists so that a report titled "CodeMie usage" measures CodeMie usage: without the ownership gate, every unmanaged run of an agent CodeMie *can* manage would be silently folded into CodeMie's numbers. + +That default is the right one for adoption reporting and the **wrong** one for consumption reporting. If you want total local AI spend across every agent on the machine, ask for it: + +```bash +codemie analytics --report --open --include-external +``` + +**This is the flag that shows all of your local agent usage.** GitHub Copilot CLI sessions are included in the gate, so they too are absent from the default report. + +Two things to know before you rely on the wider number: + +- **It is broader but less precise.** An external session has no CodeMie run to attribute it to — no profile, no managed provider — and its cost depends entirely on a native log that may already have been rotated away. Expect a lower priced-session ratio in the Cost view's **Coverage** banner than you'd see for CodeMie-launched sessions. +- **`--include-external` needs native scanning.** External sessions *are* discovered natives, so `--no-scan-native --include-external` adds nothing — the first flag suppresses the very sessions the second one asks for. + +`--include-external` applies to the default local-session source only. The `analytics otel` subcommand does not accept it — an OTEL events file has no notion of CodeMie ownership. + ### OTEL events file (`analytics otel`) As an alternative to the local-session sources above, the `analytics otel` subcommand builds the same report from a **flattened OTEL events file** (`otel-events.jsonl`) — for example, telemetry exported from a fleet or CI environment rather than the current machine's history. @@ -256,14 +307,20 @@ Filter flags: --branch Filter by git branch --session Filter to a single session +Source flags: + --no-scan-native Skip native-log discovery (CodeMie-tracked sessions only) + --include-external Also count local sessions CodeMie did not launch + (see "Session provenance"; requires native scanning) + Other flags: -v, --verbose Session-level breakdown in the terminal output --export Export terminal data to json or csv file -o, --output Output path for --export - --no-scan-native Skip native-log discovery ``` -All filter flags work for both the terminal output and the HTML report. The date filters control which sessions are **embedded** in the report; the client-side range presets (Today / 7d / 30d / 90d) then let the report viewer narrow further within that data. +**Every filter and source flag governs the terminal output and the HTML report alike.** There is no report-only or terminal-only filtering: `--include-external`, `--no-scan-native`, and the date/project/agent filters all decide which sessions the command sees, and both outputs are rendered from that same set. + +The date filters control which sessions are **embedded** in the report; the client-side range presets (Today / 7d / 30d / 90d) then let the report viewer narrow further within that data. ### OTEL source subcommand @@ -274,4 +331,4 @@ codemie analytics otel --file [options] --user Scope to one user (native user.email or user.id) ``` -All filter, report, and export flags from the base command also apply to `analytics otel`. Cost is read from each event's native `cost_usd`, so `--no-scan-native` is not used by this source. +All filter, report, and export flags from the base command also apply to `analytics otel`. The source flags do not: an OTEL events file is neither scanned for native logs nor gated on CodeMie ownership, so `--no-scan-native` and `--include-external` have no meaning here. Cost is read from each event's native `cost_usd`.