diff --git a/public/images/docs/error-feed/guides/fix-tab-empty-state-annotated.png b/public/images/docs/error-feed/guides/fix-tab-empty-state-annotated.png new file mode 100644 index 000000000..a01581e5c Binary files /dev/null and b/public/images/docs/error-feed/guides/fix-tab-empty-state-annotated.png differ diff --git a/public/images/docs/error-feed/guides/overview-tab-annotated.png b/public/images/docs/error-feed/guides/overview-tab-annotated.png new file mode 100644 index 000000000..39fb14714 Binary files /dev/null and b/public/images/docs/error-feed/guides/overview-tab-annotated.png differ diff --git a/public/images/docs/error-feed/guides/turn-on-error-feed.png b/public/images/docs/error-feed/guides/turn-on-error-feed.png new file mode 100644 index 000000000..18c572ca7 Binary files /dev/null and b/public/images/docs/error-feed/guides/turn-on-error-feed.png differ diff --git a/public/images/docs/error-feed/troubleshooting/no-issues-in-the-feed.png b/public/images/docs/error-feed/troubleshooting/no-issues-in-the-feed.png new file mode 100644 index 000000000..b77f52732 Binary files /dev/null and b/public/images/docs/error-feed/troubleshooting/no-issues-in-the-feed.png differ diff --git a/public/images/docs/simulation/guides/edit-evals/run-evals-tab.png b/public/images/docs/simulation/guides/edit-evals/run-evals-tab.png new file mode 100644 index 000000000..cafb7e9f4 Binary files /dev/null and b/public/images/docs/simulation/guides/edit-evals/run-evals-tab.png differ diff --git a/public/images/docs/simulation/guides/evaluate-tool-calls/enable-tool-eval.png b/public/images/docs/simulation/guides/evaluate-tool-calls/enable-tool-eval.png new file mode 100644 index 000000000..9f3d2ef26 Binary files /dev/null and b/public/images/docs/simulation/guides/evaluate-tool-calls/enable-tool-eval.png differ diff --git a/public/images/docs/simulation/guides/explore-results/analytics-tab.png b/public/images/docs/simulation/guides/explore-results/analytics-tab.png new file mode 100644 index 000000000..b143ef8b8 Binary files /dev/null and b/public/images/docs/simulation/guides/explore-results/analytics-tab.png differ diff --git a/public/images/docs/simulation/guides/explore-results/call-log-details.png b/public/images/docs/simulation/guides/explore-results/call-log-details.png new file mode 100644 index 000000000..cb057f032 Binary files /dev/null and b/public/images/docs/simulation/guides/explore-results/call-log-details.png differ diff --git a/public/images/docs/simulation/guides/explore-results/results-page.png b/public/images/docs/simulation/guides/explore-results/results-page.png new file mode 100644 index 000000000..c7ea50928 Binary files /dev/null and b/public/images/docs/simulation/guides/explore-results/results-page.png differ diff --git a/public/images/docs/simulation/guides/optimization-runs/optimization-steps.png b/public/images/docs/simulation/guides/optimization-runs/optimization-steps.png new file mode 100644 index 000000000..a6bb84f79 Binary files /dev/null and b/public/images/docs/simulation/guides/optimization-runs/optimization-steps.png differ diff --git a/public/images/docs/simulation/guides/running-optimizations/optimize-drawer.png b/public/images/docs/simulation/guides/running-optimizations/optimize-drawer.png new file mode 100644 index 000000000..45f38f50c Binary files /dev/null and b/public/images/docs/simulation/guides/running-optimizations/optimize-drawer.png differ diff --git a/src/pages/docs/error-feed/guides/run-root-cause-analysis.mdx b/src/pages/docs/error-feed/guides/run-root-cause-analysis.mdx index 4591eea15..9bb6b29d5 100644 --- a/src/pages/docs/error-feed/guides/run-root-cause-analysis.mdx +++ b/src/pages/docs/error-feed/guides/run-root-cause-analysis.mdx @@ -9,10 +9,10 @@ The **Fix** tab is Error Feed's agentic root-cause chat thread, with a follow-up Start from an issue's detail page, with the failure pattern already confirmed via [Investigate an issue](/docs/error-feed/guides/investigate-an-issue). -From the issue's detail page, click into the Fix tab. If nothing has run yet it shows an empty state, **No analysis yet**, with a button labeled **Analyze this cluster**. You can also start it from the issue's headline card, whose button reads **Debug this cluster** before a run exists. Both start the same run, and each uses 1 credit, taken when the run starts and refunded if the run fails. +From the issue's detail page, click into the Fix tab. If nothing has run yet it shows an empty state, **No analysis yet**, with a button labeled **Analyze this cluster**. You can also start it from the Cluster analysis card on the Overview tab, whose button reads **Debug this cluster** before a run exists. Both start the same run, and each uses 1 credit, taken when the run starts and refunded if the run fails. -Fix tab empty state with the Analyze this cluster button labeled, alongside the headline card's Debug this cluster button -*The Fix tab's empty state, before any run has started* +Fix tab empty state showing No analysis yet with the Analyze this cluster button, and the Re-run button on the headline card above it +*The Fix tab before any run has started. **Re-run** sits in the headline card from the outset, so it isn't a sign that something already ran* The run starts immediately and the tab keeps checking until it lands or fails, with a one-hour cut-off. The result arrives as a single written message in the thread: a root cause explaining what's going wrong, and a proposed fix for it, based on the calls Falcon sampled. To probe the reasoning further, type into the composer at the bottom (placeholder: **Ask Falcon a follow-up...**), and **Falcon is investigating...** shows while a reply streams in. diff --git a/src/pages/docs/error-feed/guides/turn-on-error-feed.mdx b/src/pages/docs/error-feed/guides/turn-on-error-feed.mdx index d421201af..db27ce4fd 100644 --- a/src/pages/docs/error-feed/guides/turn-on-error-feed.mdx +++ b/src/pages/docs/error-feed/guides/turn-on-error-feed.mdx @@ -12,9 +12,12 @@ Error Feed ships off. A project's scanner starts at a 0% sampling rate, so nothi ## Turn on scanning -Open the Observe project you want issues for, then click the settings gear icon, tooltipped **Settings**, in the project header. A drawer titled **Configure Project** opens, carrying the project's settings including sampling. +Open the Observe project you want issues for, then click the settings gear icon, tooltipped **Settings**, in the project header. A dialog titled **Configure Project** opens, carrying the project's name and its sampling rate. -Find the sampling rate control in that drawer and raise it above 0. 100% is a safe default while you're trying it out; see [Choosing a rate](#choosing-a-rate) for the cost tradeoff once volume climbs. Click **Update** to apply it. +**Sampling rate** is a slider running 0% to 100%, with the current value in a box beside it. Drag it above 0. 100% is a safe default while you're trying it out; see [Choosing a rate](#choosing-a-rate) for the cost tradeoff once volume climbs. Click **Update** to apply it. + +The Configure Project dialog for the ava-production project, with the Sampling rate slider at 100% and Cancel, Delete, and Update buttons below +*The same dialog also deletes the project, so read the buttons before clicking* Then wait before checking for results, because a new rate only reaches traces that arrive after you save it. Send your agent through a request, give scanning a moment, and go to **Error Feed** in the left sidebar. A row appearing in the list confirms scanning is live. An upgrade prompt instead of the Feed means the workspace lacks the Error Feed capability. diff --git a/src/pages/docs/error-feed/troubleshooting/no-issues-in-the-feed.mdx b/src/pages/docs/error-feed/troubleshooting/no-issues-in-the-feed.mdx index f6297c667..0fe0e80cb 100644 --- a/src/pages/docs/error-feed/troubleshooting/no-issues-in-the-feed.mdx +++ b/src/pages/docs/error-feed/troubleshooting/no-issues-in-the-feed.mdx @@ -50,7 +50,10 @@ Fix: wait at least 60 seconds after the trace lands before assuming scanning isn The table's empty state tells you which situation you're actually in. - If your filters exclude everything currently in the Feed, it shows **"No errors match your filters"** / **"Try adjusting your search or filter criteria."** -- If there genuinely are no issues, it shows **"No errors - everything looks good!"** / **"Errors captured by Future AGI will appear here."** +- If there genuinely are no issues, it shows **"No errors — everything looks good!"** / **"Errors captured by Future AGI will appear here."** + +The Error Feed list page in its unfiltered empty state, showing No errors — everything looks good! above Errors captured by Future AGI will appear here., with the filter bar unset above it +*The unfiltered case: every filter still at its default, and the count beside the title reads 0* Fix: if you're looking at the first message, clear or widen your [filters](/docs/error-feed/guides/triage-issues). The **Clear** control: diff --git a/src/pages/docs/index.mdx b/src/pages/docs/index.mdx index 7fb6500c6..c8f26b367 100644 --- a/src/pages/docs/index.mdx +++ b/src/pages/docs/index.mdx @@ -7,15 +7,9 @@ Future AGI is an end-to-end platform for building **reliable AI agents**. It bri It's built for the whole team shipping AI (engineers, product managers, and domain experts working from one source of truth), and it works with the stack you already use. If you use it, we probably support it. You can start with a single line of code -## The Learning Loop - -Every part of Future AGI feeds the next. You [**simulate**](/docs/simulation) an agent before launch, [**evaluate**](/docs/evaluation) its outputs against built-in and custom metrics, [**observe**](/docs/observe) real traffic once it's live, and [**optimize**](/docs/optimization) from what you learn, then the cycle repeats - -![Future AGI platform](/images/agi2.webp) - -Because every product shares the same **traces, datasets, and scores**, the work compounds: a trace you capture becomes evaluation data, an evaluation result becomes an optimization signal, and a dataset feeds simulations and experiments alike. That shared spine is the mental model for everything below - -{/* TODO: embed the Future AGI overview video here once the URL is ready. */} +
+ +
## Explore the platform diff --git a/src/pages/docs/simulation/guides/edit-evals.mdx b/src/pages/docs/simulation/guides/edit-evals.mdx index e75370b13..81339761b 100644 --- a/src/pages/docs/simulation/guides/edit-evals.mdx +++ b/src/pages/docs/simulation/guides/edit-evals.mdx @@ -11,7 +11,10 @@ This page assumes a run test that already exists, from [Create a simulation](/do ## Open the run's evals -Under **Simulate** in the sidebar, open **Run Simulation** and click into the run. The **Evals** chip on its page opens the **All Evaluations** panel, which lists every eval currently attached, each row carrying its own edit and delete controls. +Under **Simulate** in the sidebar, open **Run Simulation** and click into the run. The **Evals** chip on its page opens the **All Evaluations** panel, which lists every eval currently attached, each row carrying its own edit and delete controls. Each row also shows the mapping it's reading under its name, so you can see what every eval is pointed at without opening any of them. + +The All Evaluations panel for the SDR - Chat Test run, listing thirteen attached evals with their mappings, the Add button in the header, and the Enable Tool Call Evaluation checkbox at the bottom +*Thirteen evals on one run. Most read the whole conversation; prompt conformance also reads the system prompt* **Enable tool call evaluation** doesn't live in this list, it's a checkbox at the bottom of this same panel. Turn it on or off from there and it saves straight onto the run. [Evaluate tool calls](/docs/simulation/guides/evaluate-tool-calls) covers what it scores. diff --git a/src/pages/docs/simulation/guides/evaluate-tool-calls.mdx b/src/pages/docs/simulation/guides/evaluate-tool-calls.mdx index 73230d180..f52de7a4c 100644 --- a/src/pages/docs/simulation/guides/evaluate-tool-calls.mdx +++ b/src/pages/docs/simulation/guides/evaluate-tool-calls.mdx @@ -9,6 +9,9 @@ Tool call evaluation scores the tool calls your agent made during a conversation The switch lives on the **Select evaluations** step of the [simulation wizard](/docs/simulation/guides/create-simulation), labelled **Enable tool call evaluation**. It sits above the eval library, off by default, and turning it on doesn't count as one of the evals that step requires you to add. It's set once, when you create the run. +The Select evaluations step of the simulation wizard with Enable tool call evaluation switched on, above an empty eval panel offering Add Evaluations +*Switched on, with no evals added yet. The step still won't let you past until at least one is* + Flip it only for a run test whose agent actually calls tools during the scenarios you've attached. A scenario that never reaches a tool leaves nothing for it to evaluate. diff --git a/src/pages/docs/simulation/guides/explore-results/analytics.mdx b/src/pages/docs/simulation/guides/explore-results/analytics.mdx index dfc7b65d0..4a33d776a 100644 --- a/src/pages/docs/simulation/guides/explore-results/analytics.mdx +++ b/src/pages/docs/simulation/guides/explore-results/analytics.mdx @@ -1,37 +1,55 @@ --- title: "Analytics & metrics" -description: "Read the performance summary, KPIs, and eval summary on a run's Analytics tab" +description: "Judge a whole execution at once, then line several up against each other" --- -A run test's **Analytics** tab rolls every call in the run into three reads: a performance summary, a set of KPI cards, and an eval summary. None of them replace opening an individual call, covered in [Calls & transcripts](/docs/simulation/guides/explore-results/calls-and-transcripts); they're for judging the run as a whole before you drop into one conversation. +Opening calls one at a time, the way [Calls & transcripts](/docs/simulation/guides/explore-results/calls-and-transcripts) does, tells you what happened in one conversation. It won't tell you whether the agent is getting better. This guide covers the three places that will: a summary panel above the call table, an **Analytics** tab that scores one attempt as a whole, and a second Analytics tab that stacks attempts against each other. -## Read the performance summary +All of it is scored by the [evals](/docs/evaluation) attached to the run, so a run created without any will show these surfaces empty. -One read is the performance summary, **Test Run Performance Metrics**, three cards: +## Execution-level or run-level -- **Pass Rate**: the share of calls in the run that came back passing. A number here below what you'd expect from a stable agent is the first sign something regressed -- **Total Test Runs**: how many times this run test has been executed. One card, not per-call, so it tells you how many attempts you have to compare, not how any single attempt went -- **Latest Fail Rate**: the fail rate of the most recent execution specifically. Read it next to Pass Rate to tell a run that's always been shaky from one that just got worse +Start from **Simulate** in the sidebar, then **Run Simulation**, and click the run you want. You land on **Simulated runs**, which lists every attempt that run has made. Each attempt is an [execution](/docs/simulation/concepts/runs-and-results), and clicking one opens it. -Alongside the cards, **Top Performing Scenarios** lists each scenario with a score chip: red under 5, orange under 7, green at 7 and above. A scenario sitting in red is where the agent is struggling hardest, and it's the one worth opening first. +That gives you two levels, and mixing them up is the easiest mistake to make here. An execution has its own summary and its own Analytics tab, answering "how did this attempt go". The run that owns those executions has an Analytics tab of its own, answering "is this getting better than last time". -## Read the KPIs +Both tabs are called **Analytics**, so read what's above them to tell which one you're on: an execution's tabs sit under an execution ID, and the run's sit under the run's name. A run only has more than one execution if it's been started more than once, which is what **Run New Simulation** on the Simulated runs tab does. -The KPI cards differ by channel, because a voice call and a chat conversation are measured differently. +## Read Performance Metrics -A voice run shows CSAT (the header label for the call's overall score), Agent Latency in milliseconds, the agent's words-per-minute pace, how quickly the agent stops talking when the caller cuts in, average turn count, and a talk ratio comparing how much of the call the agent spent talking against the caller. A low CSAT alongside high latency or a lopsided talk ratio usually points at the same root cause: the agent is talking too much, or too slowly, to keep the caller satisfied. +**Performance Metrics** sits at the top of **Call Details** on a voice run, or **Chat Details** on a chat one, above the table of individual calls. It covers one execution, not one call, and it's the fastest read on the page. -A chat run shows CSAT again, average latency in milliseconds, average turn count, and three token counts: total, input, and output. Token counts matter here beyond cost. A run whose output tokens climb without a matching lift in CSAT is spending more per reply without the conversation actually getting better. +It groups into three panels. The first two differ by channel, because the two channels fail in different ways: -Every field behind these cards, including the ones not surfaced as KPI cards, is listed in [Call metrics](/docs/simulation/reference/call-metrics). +| Panel | Voice | Chat | +|---|---|---| +| Throughput | Calls placed, how many connected, connection rate | Chats started, how many completed, completion rate | +| System metrics | Pace and timing: latency, words per minute, how fast the agent stops when the caller cuts in, talk ratio | Cost and length: token counts, latency, turn count | -## Read the eval summary +The third panel is the same on both: an average for each eval attached to the run. -The eval summary is one card per eval attached to the run, graphing how that eval scored across every call. This is where you catch an eval that's consistently weak across the whole run rather than failing on one unlucky call, and it's the signal that tells you which eval to chase into individual transcripts. Eval types and how each one is configured belong to [Evaluation](/docs/evaluation); this tab only shows the scores it produced. +Read the throughput panel first, because it can settle the question the eval scores can't. **A voice execution reporting 20 calls placed and 10 connected has a delivery problem, not a quality one**, and no eval score is going to tell you that. [Simulation FAQ & fixes](/docs/simulation/troubleshooting) covers what to do about calls that never connect. - -Rerunning a call, from [Calls & transcripts](/docs/simulation/guides/explore-results/calls-and-transcripts), updates the eval summary and KPIs the next time you load this tab. - +An eval that returns a category rather than a percentage shows its split there instead, so a conversation-quality eval reads as its distribution across the calls rather than as one number. **View all metrics** expands the panel in place and flips to **Minimize**. The full field list for either channel is in [Call metrics](/docs/simulation/reference/call-metrics). + +## Score one execution + +The execution's **Analytics** tab takes the same evals and goes deeper. A radar chart plots every eval against each other, with each one's score listed beside it, so a single weak axis stands out against the rest. + +Each eval then gets its own card below. The **Table** and **Column Chart** toggle switches between reading that eval's scores as numbers and reading them spread across percentile buckets, which is where you separate an eval that's mediocre on every call from one that's fine on most and falls apart on a few. The **All** selector on each card draws every scoring variant of that eval at once, or one at a time. + +The Analytics tab of a chat execution, with the radar chart and its eval scores on the left, the Critical issues panel on the right, and a per-eval card below showing the Table and Column Chart toggle +*One execution's Analytics tab. Context retention at 14% is the axis pulling the radar in* + +**Critical issues (How to solve it)** sits to the right of the radar. It names the failure patterns it found across this execution and gives numbered fixes for each, stamps when it last updated, and re-runs on **Refresh**. Generating it takes a few minutes, and it says so while it works. When it finds nothing it says that too: "Our analysis didn't find any clusters of similar failures. This may mean issues are rare, inconsistent, or below the current threshold." [Fix My Agent](/docs/simulation/guides/fix-my-agent) is where findings turn into an actual change. + +## Compare executions + +The run's own **Analytics** tab is the one you reach from the run without opening any execution, and it's where a regression shows up. + +Check the execution list before you read anything into it. **Executions (N)** in the header opens a searchable checklist of every attempt, all ticked by default, and the **Compare** panel labels the ones you keep as A, B, C and so on, with **A being the most recent**. Attempts that failed before scoring stay in that list and report 0%, so a column of zeros beside one healthy execution usually means those attempts never ran rather than that the agent scored nothing. + +Every eval then reports each execution's score in one block, so a number that moved between two attempts is visible without opening either. The per-eval cards and the percentile view follow underneath, this time layered across the executions you kept ticked. Only eval scores are compared here; latency, tokens and connection rates stay on each execution's own Performance Metrics panel. ## Dive deeper @@ -43,6 +61,6 @@ Rerunning a call, from [Calls & transcripts](/docs/simulation/guides/explore-res Every metric a call carries, field by field - Turn a weak eval summary into concrete fixes + Turn a weak eval score into concrete fixes diff --git a/src/pages/docs/simulation/guides/explore-results/calls-and-transcripts.mdx b/src/pages/docs/simulation/guides/explore-results/calls-and-transcripts.mdx index 4c77c562f..697f5a60c 100644 --- a/src/pages/docs/simulation/guides/explore-results/calls-and-transcripts.mdx +++ b/src/pages/docs/simulation/guides/explore-results/calls-and-transcripts.mdx @@ -3,17 +3,20 @@ title: "Calls & transcripts" description: "Open one call and read its transcript, evals, cost, and recording" --- -A [run test](/docs/simulation/concepts/runs-and-results) fans out into calls, one per scenario row, and each call is the full record of one conversation. This guide opens a single call and reads through everything it carries: the transcript, the recording, the evals it scored, and what it cost. +A [run test](/docs/simulation/concepts/runs-and-results) fans out into calls, one per [scenario](/docs/simulation/concepts/scenarios) row, and each call is the full record of one conversation. This guide opens a single call and reads through everything it carries: the transcript, the recording, the evals it scored, and what it cost. ## Open a call Every call is a row in the run's **Call Details** tab (**Chat Details** for a chat run). Click a row and its detail drawer opens over the grid. -The drawer opens on **Call Log Details**, with chips for the [persona](/docs/simulation/concepts/personas) or customer name, the [scenario](/docs/simulation/concepts/scenarios) the call ran, when it started, how long it ran, and a status badge. A **View Docs** link in the header points back to the simulation docs. +The drawer is headed by the call's own ID, with arrows beside it that step to the previous or next call without going back to the grid. A row of chips underneath carries the call's type and status, how long it ran, its average latency, the number it dialled, the voice provider that placed it, when it started, and how it ended. + +The call drawer open on one completed inbound voice call, with the recording waveform and transcript on the left, the call's chips and Call Analytics on the right, and the cost breakdown beneath them +*One inbound voice call end to end: the recording, the transcript beside it, and what the call cost* ## Follow the transcript -The transcript runs turn by turn, and each turn carries a speaker role. Three of them show up in the transcript you read: `USER` is the simulated persona's turn, `ASSISTANT` is your agent's, and `SYSTEM` is a turn that came from a system-level instruction rather than either side of the conversation. +The transcript runs turn by turn, and each turn carries a speaker role. Three of them show up in the transcript you read: `USER` is the simulated [persona](/docs/simulation/concepts/personas)'s turn, `ASSISTANT` is your agent's, and `SYSTEM` is a turn that came from a system-level instruction rather than either side of the conversation. If your agent calls tools mid-conversation, those turns exist too, under two further roles kept out of the transcript view: one holds the name of the tool that was called, the other the result it returned. They aren't something you'd otherwise see here; [Evaluate tool calls](/docs/simulation/guides/evaluate-tool-calls) covers scoring them directly. diff --git a/src/pages/docs/simulation/guides/explore-results/index.mdx b/src/pages/docs/simulation/guides/explore-results/index.mdx index 68e7652fd..666f2bffe 100644 --- a/src/pages/docs/simulation/guides/explore-results/index.mdx +++ b/src/pages/docs/simulation/guides/explore-results/index.mdx @@ -13,16 +13,8 @@ Under **Simulate** in the sidebar, **Run Simulation** lists every run in the wor ## The page at a glance - H["Header
name, status, export, rerun, stop"] - R --> T1["Call Details / Chat Details
one row per call"] - R --> T2["Analytics
aggregated eval scores"] - R --> T3["Optimization Runs
optimization attempts"] - T1 -.-> G1(["Calls and transcripts guide"]) - T2 -.-> G2(["Analytics and metrics guide"]) -`} /> +One execution of the SDR - Chat Test run, showing Export Data and Re-run simulation in the header, the Chat Details, Analytics and Optimization Runs tabs, the Performance Metrics cards, and the table of individual chats +*A finished chat execution. A voice one reads the same, with Call Details in place of Chat Details* ## The header diff --git a/src/pages/docs/simulation/guides/optimization-runs.mdx b/src/pages/docs/simulation/guides/optimization-runs.mdx index 8caabc375..c5af79cd7 100644 --- a/src/pages/docs/simulation/guides/optimization-runs.mdx +++ b/src/pages/docs/simulation/guides/optimization-runs.mdx @@ -5,16 +5,6 @@ description: "Track an optimization run through its steps and trials, and apply An **optimization run** is what you get after [Fix My Agent](/docs/simulation/guides/fix-my-agent) points the search at a finished [run](/docs/simulation/concepts/runs-and-results), covered in [Running optimizations](/docs/simulation/guides/running-optimizations). This page covers reading one you've already started: its steps, its trials, the score behind each trial, and what to do with the one that wins. - ST["Steps
initializing, baseline, trials, finalizing"] - OR --> TR["Trials"] - TR --> BASE["Baseline trial
your current prompt, unchanged"] - TR --> CAND["Candidate trials
one prompt each"] - CAND --> BEST(["Best trial
highest score, flagged for you"]) -`} /> - ## Find a run's optimization history Open a run's results page and switch to the **Optimization Runs** tab, one of the three tabs covered in [Explore results](/docs/simulation/guides/explore-results). It lists every optimization attempt made against that execution, one row per attempt, with its name, how many trials it ran, which optimizer it used, and its status. A run with none yet shows **No optimization runs found**. @@ -33,7 +23,10 @@ Once a run is completed or failed, a **Rerun Optimization** button appears in th Every optimization run produces trials. The first is always the baseline: it scores your prompt exactly as it stands today, before the search changes anything, and gives every later trial a line to beat. Each trial after it is a candidate the search tried, and carries its own prompt text, its own average score, and how that score moved against the baseline. -The best-performing trial, the one with the highest average score among everything the search actually tried, is flagged so you don't have to hunt for it. Open it to read its full prompt text, plus which [evals](/docs/evaluation) scored it and which [scenarios](/docs/simulation/concepts/scenarios) it ran against, the same ones your original run used. +The best-performing trial, the one with the highest average score among everything the search actually tried, is flagged with a crown so you don't have to hunt for it. + +A completed GEPA optimization run, showing the Optimization Results chart across the baseline and seven trials, and the trials table with an improvement percentage against each one +*Three trials matched the best score here, and the crown falls on the first of them* Open it to read its full prompt text, plus which [evals](/docs/evaluation) scored it and which [scenarios](/docs/simulation/concepts/scenarios) it ran against, the same ones your original run used. ## Apply the winning configuration diff --git a/src/pages/docs/simulation/guides/running-optimizations.mdx b/src/pages/docs/simulation/guides/running-optimizations.mdx index adc809843..b98175158 100644 --- a/src/pages/docs/simulation/guides/running-optimizations.mdx +++ b/src/pages/docs/simulation/guides/running-optimizations.mdx @@ -37,6 +37,9 @@ Below them, the parameter fields change with the algorithm: Every algorithm ends on the same field, **Optimization Objective**, multiline: write in your own words what you want this run to fix or improve. +The Choose optimization type dialog filled in with a name, Random Search as the optimizer, gpt-4.1 as the language model, three variations, and an optimization objective written out +*The objective is free text, so it's worth naming the specific failure rather than asking for better output* + ## Start it Click **Start Optimizing your agent** at the bottom of the drawer. A toast confirms it, "Optimization Created Successfully," and the run appears in the execution's **Optimization Runs** tab right away, moving from **pending** to **running** as it works through its trials.