Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions src/pages/docs/error-feed/guides/run-root-cause-analysis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ The **Fix** tab is Error Feed's agentic root-cause chat thread, with a follow-up

Start from an issue's detail page, with the failure pattern already confirmed via [Investigate an issue](/docs/error-feed/guides/investigate-an-issue).

From the issue's detail page, click into the Fix tab. If nothing has run yet it shows an empty state, **No analysis yet**, with a button labeled **Analyze this cluster**. You can also start it from the issue's headline card, whose button reads **Debug this cluster** before a run exists. Both start the same run, and each uses 1 credit, taken when the run starts and refunded if the run fails.
From the issue's detail page, click into the Fix tab. If nothing has run yet it shows an empty state, **No analysis yet**, with a button labeled **Analyze this cluster**. You can also start it from the Cluster analysis card on the Overview tab, whose button reads **Debug this cluster** before a run exists. Both start the same run, and each uses 1 credit, taken when the run starts and refunded if the run fails.

<img src="/images/docs/error-feed/guides/fix-tab-empty-state-annotated.png" alt="Fix tab empty state with the Analyze this cluster button labeled, alongside the headline card's Debug this cluster button" style={{ borderRadius: '5px' }} />
*The Fix tab's empty state, before any run has started*
<img src="/images/docs/error-feed/guides/fix-tab-empty-state-annotated.png" alt="Fix tab empty state showing No analysis yet with the Analyze this cluster button, and the Re-run button on the headline card above it" style={{ borderRadius: '5px' }} />
*The Fix tab before any run has started. **Re-run** sits in the headline card from the outset, so it isn't a sign that something already ran*

The run starts immediately and the tab keeps checking until it lands or fails, with a one-hour cut-off. The result arrives as a single written message in the thread: a root cause explaining what's going wrong, and a proposed fix for it, based on the calls Falcon sampled. To probe the reasoning further, type into the composer at the bottom (placeholder: **Ask Falcon a follow-up...**), and **Falcon is investigating...** shows while a reply streams in.

Expand Down
7 changes: 5 additions & 2 deletions src/pages/docs/error-feed/guides/turn-on-error-feed.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@ Error Feed ships off. A project's scanner starts at a 0% sampling rate, so nothi

## Turn on scanning

Open the Observe project you want issues for, then click the settings gear icon, tooltipped **Settings**, in the project header. A drawer titled **Configure Project** opens, carrying the project's settings including sampling.
Open the Observe project you want issues for, then click the settings gear icon, tooltipped **Settings**, in the project header. A dialog titled **Configure Project** opens, carrying the project's name and its sampling rate.

Find the sampling rate control in that drawer and raise it above 0. 100% is a safe default while you're trying it out; see [Choosing a rate](#choosing-a-rate) for the cost tradeoff once volume climbs. Click **Update** to apply it.
**Sampling rate** is a slider running 0% to 100%, with the current value in a box beside it. Drag it above 0. 100% is a safe default while you're trying it out; see [Choosing a rate](#choosing-a-rate) for the cost tradeoff once volume climbs. Click **Update** to apply it.

<img src="/images/docs/error-feed/guides/turn-on-error-feed.png" alt="The Configure Project dialog for the ava-production project, with the Sampling rate slider at 100% and Cancel, Delete, and Update buttons below" style={{ borderRadius: '5px' }} />
*The same dialog also deletes the project, so read the buttons before clicking*

Then wait before checking for results, because a new rate only reaches traces that arrive after you save it. Send your agent through a request, give scanning a moment, and go to **Error Feed** in the left sidebar. A row appearing in the list confirms scanning is live. An upgrade prompt instead of the Feed means the workspace lacks the Error Feed capability.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,10 @@ Fix: wait at least 60 seconds after the trace lands before assuming scanning isn
The table's empty state tells you which situation you're actually in.

- If your filters exclude everything currently in the Feed, it shows **"No errors match your filters"** / **"Try adjusting your search or filter criteria."**
- If there genuinely are no issues, it shows **"No errors - everything looks good!"** / **"Errors captured by Future AGI will appear here."**
- If there genuinely are no issues, it shows **"No errors — everything looks good!"** / **"Errors captured by Future AGI will appear here."**

<img src="/images/docs/error-feed/troubleshooting/no-issues-in-the-feed.png" alt="The Error Feed list page in its unfiltered empty state, showing No errors — everything looks good! above Errors captured by Future AGI will appear here., with the filter bar unset above it" style={{ borderRadius: '5px' }} />
*The unfiltered case: every filter still at its default, and the count beside the title reads 0*

Fix: if you're looking at the first message, clear or widen your [filters](/docs/error-feed/guides/triage-issues). The **Clear** control:

Expand Down
12 changes: 3 additions & 9 deletions src/pages/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,9 @@ Future AGI is an end-to-end platform for building **reliable AI agents**. It bri

It's built for the whole team shipping AI (engineers, product managers, and domain experts working from one source of truth), and it works with the stack you already use. If you use it, we probably support it. You can start with a single line of code

## The Learning Loop

Every part of Future AGI feeds the next. You [**simulate**](/docs/simulation) an agent before launch, [**evaluate**](/docs/evaluation) its outputs against built-in and custom metrics, [**observe**](/docs/observe) real traffic once it's live, and [**optimize**](/docs/optimization) from what you learn, then the cycle repeats

![Future AGI platform](/images/agi2.webp)

Because every product shares the same **traces, datasets, and scores**, the work compounds: a trace you capture becomes evaluation data, an evaluation result becomes an optimization signal, and a dataset feeds simulations and experiments alike. That shared spine is the mental model for everything below

{/* TODO: embed the Future AGI overview video here once the URL is ready. */}
<div style={{ position: 'relative', paddingBottom: '56.25%', height: 0, margin: '1.5rem 0 2rem', borderRadius: '12px', overflow: 'hidden' }}>
<iframe src="https://www.youtube-nocookie.com/embed/6keOTAOUUWI?rel=0&modestbranding=1&playsinline=1" title="Future AGI overview" style={{ position: 'absolute', top: 0, left: 0, width: '100%', height: '100%', border: 0 }} allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowFullScreen></iframe>
</div>

## Explore the platform

Expand Down
5 changes: 4 additions & 1 deletion src/pages/docs/simulation/guides/edit-evals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ This page assumes a run test that already exists, from [Create a simulation](/do

## Open the run's evals

Under **Simulate** in the sidebar, open **Run Simulation** and click into the run. The **Evals** chip on its page opens the **All Evaluations** panel, which lists every eval currently attached, each row carrying its own edit and delete controls.
Under **Simulate** in the sidebar, open **Run Simulation** and click into the run. The **Evals** chip on its page opens the **All Evaluations** panel, which lists every eval currently attached, each row carrying its own edit and delete controls. Each row also shows the mapping it's reading under its name, so you can see what every eval is pointed at without opening any of them.

<img src="/images/docs/simulation/guides/edit-evals/run-evals-tab.png" alt="The All Evaluations panel for the SDR - Chat Test run, listing thirteen attached evals with their mappings, the Add button in the header, and the Enable Tool Call Evaluation checkbox at the bottom" style={{ borderRadius: '5px' }} />
*Thirteen evals on one run. Most read the whole conversation; prompt conformance also reads the system prompt*

**Enable tool call evaluation** doesn't live in this list, it's a checkbox at the bottom of this same panel. Turn it on or off from there and it saves straight onto the run. [Evaluate tool calls](/docs/simulation/guides/evaluate-tool-calls) covers what it scores.

Expand Down
3 changes: 3 additions & 0 deletions src/pages/docs/simulation/guides/evaluate-tool-calls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ Tool call evaluation scores the tool calls your agent made during a conversation

The switch lives on the **Select evaluations** step of the [simulation wizard](/docs/simulation/guides/create-simulation), labelled **Enable tool call evaluation**. It sits above the eval library, off by default, and turning it on doesn't count as one of the evals that step requires you to add. It's set once, when you create the run.

<img src="/images/docs/simulation/guides/evaluate-tool-calls/enable-tool-eval.png" alt="The Select evaluations step of the simulation wizard with Enable tool call evaluation switched on, above an empty eval panel offering Add Evaluations" style={{ borderRadius: '5px' }} />
*Switched on, with no evals added yet. The step still won't let you past until at least one is*

Flip it only for a run test whose agent actually calls tools during the scenarios you've attached. A scenario that never reaches a tool leaves nothing for it to evaluate.

<Warning>
Expand Down
56 changes: 37 additions & 19 deletions src/pages/docs/simulation/guides/explore-results/analytics.mdx
Original file line number Diff line number Diff line change
@@ -1,37 +1,55 @@
---
title: "Analytics & metrics"
description: "Read the performance summary, KPIs, and eval summary on a run's Analytics tab"
description: "Judge a whole execution at once, then line several up against each other"
---

A run test's **Analytics** tab rolls every call in the run into three reads: a performance summary, a set of KPI cards, and an eval summary. None of them replace opening an individual call, covered in [Calls & transcripts](/docs/simulation/guides/explore-results/calls-and-transcripts); they're for judging the run as a whole before you drop into one conversation.
Opening calls one at a time, the way [Calls & transcripts](/docs/simulation/guides/explore-results/calls-and-transcripts) does, tells you what happened in one conversation. It won't tell you whether the agent is getting better. This guide covers the three places that will: a summary panel above the call table, an **Analytics** tab that scores one attempt as a whole, and a second Analytics tab that stacks attempts against each other.

## Read the performance summary
All of it is scored by the [evals](/docs/evaluation) attached to the run, so a run created without any will show these surfaces empty.

One read is the performance summary, **Test Run Performance Metrics**, three cards:
## Execution-level or run-level

- **Pass Rate**: the share of calls in the run that came back passing. A number here below what you'd expect from a stable agent is the first sign something regressed
- **Total Test Runs**: how many times this run test has been executed. One card, not per-call, so it tells you how many attempts you have to compare, not how any single attempt went
- **Latest Fail Rate**: the fail rate of the most recent execution specifically. Read it next to Pass Rate to tell a run that's always been shaky from one that just got worse
Start from **Simulate** in the sidebar, then **Run Simulation**, and click the run you want. You land on **Simulated runs**, which lists every attempt that run has made. Each attempt is an [execution](/docs/simulation/concepts/runs-and-results), and clicking one opens it.

Alongside the cards, **Top Performing Scenarios** lists each scenario with a score chip: red under 5, orange under 7, green at 7 and above. A scenario sitting in red is where the agent is struggling hardest, and it's the one worth opening first.
That gives you two levels, and mixing them up is the easiest mistake to make here. An execution has its own summary and its own Analytics tab, answering "how did this attempt go". The run that owns those executions has an Analytics tab of its own, answering "is this getting better than last time".

## Read the KPIs
Both tabs are called **Analytics**, so read what's above them to tell which one you're on: an execution's tabs sit under an execution ID, and the run's sit under the run's name. A run only has more than one execution if it's been started more than once, which is what **Run New Simulation** on the Simulated runs tab does.

The KPI cards differ by channel, because a voice call and a chat conversation are measured differently.
## Read Performance Metrics

A voice run shows CSAT (the header label for the call's overall score), Agent Latency in milliseconds, the agent's words-per-minute pace, how quickly the agent stops talking when the caller cuts in, average turn count, and a talk ratio comparing how much of the call the agent spent talking against the caller. A low CSAT alongside high latency or a lopsided talk ratio usually points at the same root cause: the agent is talking too much, or too slowly, to keep the caller satisfied.
**Performance Metrics** sits at the top of **Call Details** on a voice run, or **Chat Details** on a chat one, above the table of individual calls. It covers one execution, not one call, and it's the fastest read on the page.

A chat run shows CSAT again, average latency in milliseconds, average turn count, and three token counts: total, input, and output. Token counts matter here beyond cost. A run whose output tokens climb without a matching lift in CSAT is spending more per reply without the conversation actually getting better.
It groups into three panels. The first two differ by channel, because the two channels fail in different ways:

Every field behind these cards, including the ones not surfaced as KPI cards, is listed in [Call metrics](/docs/simulation/reference/call-metrics).
| Panel | Voice | Chat |
|---|---|---|
| Throughput | Calls placed, how many connected, connection rate | Chats started, how many completed, completion rate |
| System metrics | Pace and timing: latency, words per minute, how fast the agent stops when the caller cuts in, talk ratio | Cost and length: token counts, latency, turn count |

## Read the eval summary
The third panel is the same on both: an average for each eval attached to the run.

The eval summary is one card per eval attached to the run, graphing how that eval scored across every call. This is where you catch an eval that's consistently weak across the whole run rather than failing on one unlucky call, and it's the signal that tells you which eval to chase into individual transcripts. Eval types and how each one is configured belong to [Evaluation](/docs/evaluation); this tab only shows the scores it produced.
Read the throughput panel first, because it can settle the question the eval scores can't. **A voice execution reporting 20 calls placed and 10 connected has a delivery problem, not a quality one**, and no eval score is going to tell you that. [Simulation FAQ & fixes](/docs/simulation/troubleshooting) covers what to do about calls that never connect.

<Note>
Rerunning a call, from [Calls & transcripts](/docs/simulation/guides/explore-results/calls-and-transcripts), updates the eval summary and KPIs the next time you load this tab.
</Note>
An eval that returns a category rather than a percentage shows its split there instead, so a conversation-quality eval reads as its distribution across the calls rather than as one number. **View all metrics** expands the panel in place and flips to **Minimize**. The full field list for either channel is in [Call metrics](/docs/simulation/reference/call-metrics).

## Score one execution

The execution's **Analytics** tab takes the same evals and goes deeper. A radar chart plots every eval against each other, with each one's score listed beside it, so a single weak axis stands out against the rest.

Each eval then gets its own card below. The **Table** and **Column Chart** toggle switches between reading that eval's scores as numbers and reading them spread across percentile buckets, which is where you separate an eval that's mediocre on every call from one that's fine on most and falls apart on a few. The **All** selector on each card draws every scoring variant of that eval at once, or one at a time.

<img src="/images/docs/simulation/guides/explore-results/analytics-tab.png" alt="The Analytics tab of a chat execution, with the radar chart and its eval scores on the left, the Critical issues panel on the right, and a per-eval card below showing the Table and Column Chart toggle" style={{ borderRadius: '5px' }} />
*One execution's Analytics tab. Context retention at 14% is the axis pulling the radar in*

**Critical issues (How to solve it)** sits to the right of the radar. It names the failure patterns it found across this execution and gives numbered fixes for each, stamps when it last updated, and re-runs on **Refresh**. Generating it takes a few minutes, and it says so while it works. When it finds nothing it says that too: "Our analysis didn't find any clusters of similar failures. This may mean issues are rare, inconsistent, or below the current threshold." [Fix My Agent](/docs/simulation/guides/fix-my-agent) is where findings turn into an actual change.

## Compare executions

The run's own **Analytics** tab is the one you reach from the run without opening any execution, and it's where a regression shows up.

Check the execution list before you read anything into it. **Executions (N)** in the header opens a searchable checklist of every attempt, all ticked by default, and the **Compare** panel labels the ones you keep as A, B, C and so on, with **A being the most recent**. Attempts that failed before scoring stay in that list and report 0%, so a column of zeros beside one healthy execution usually means those attempts never ran rather than that the agent scored nothing.

Every eval then reports each execution's score in one block, so a number that moved between two attempts is visible without opening either. The per-eval cards and the percentile view follow underneath, this time layered across the executions you kept ticked. Only eval scores are compared here; latency, tokens and connection rates stay on each execution's own Performance Metrics panel.

## Dive deeper

Expand All @@ -43,6 +61,6 @@ Rerunning a call, from [Calls & transcripts](/docs/simulation/guides/explore-res
Every metric a call carries, field by field
</Card>
<Card title="Fix My Agent" icon="wand-magic-sparkles" href="/docs/simulation/guides/fix-my-agent">
Turn a weak eval summary into concrete fixes
Turn a weak eval score into concrete fixes
</Card>
</CardGroup>
Loading
Loading