From e2df3fd1b22a8768d89fffb0d0bd5c57d0cdec7d Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:07:32 +0000 Subject: [PATCH 1/5] docs(langchain): Document TypeSafeClassifier evaluation tracing Co-Authored-By: Andrei Borza --- .../configuration/integrations/langchain.mdx | 45 +++++++++++++++++++ .../configuration/integrations/langgraph.mdx | 36 +++++++++++++++ .../configuration/integrations/typesafe.mdx | 2 + 3 files changed, 83 insertions(+) diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index 2affbeabd9428f..d02fd9e1893278 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -56,6 +56,45 @@ The integration captures `gen_ai.embeddings` spans for `embedQuery()` and `embed - `@langchain/mistralai` - `@langchain/google-vertexai` + + +## Trace Classifier Evaluations + + + +[`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe)'s `TypeSafeClassifier` asks TypeSafe's Jev model typed questions about your input. The `langChainIntegration` traces these calls for you, whether you call the classifier yourself, from a LangGraph node, or through the `modelRouterMiddleware` and `autoModeMiddleware` of `@langchain/typesafe/middleware` inside a `createAgent` run. No additional integration or setup is needed. + +```javascript +import { TypeSafeClassifier } from "@langchain/typesafe"; + +const classifier = new TypeSafeClassifier({ + questions: { + department: { + type: "choice", + instructions: "Which team should handle this?", + criteria: { + billing: "Charges and refunds", + technical: "Bugs and outages", + }, + }, + }, +}); + +const result = await classifier.invoke( + "I was charged twice. Please refund the extra charge." +); + +console.log(result.answers.department.choice); +``` + +Each `invoke()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them. + +Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other LangChain calls. Disable these options if evaluation state or answers contain sensitive data. + +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the TypeSafe or Vercel AI integration. + + + ## Supported Versions - `langchain`: `>=0.1.0 <2.0.0` @@ -71,6 +110,12 @@ Automatic embeddings instrumentation supports these package versions: For `@langchain/google-vertexai`, the version requirement applies to its `@langchain/google-common` dependency, which implements the embedding methods. + + +Classifier evaluation tracing supports `@langchain/typesafe` `>=0.0.1 <1.0.0`. + + + ## Options ### `recordInputs` diff --git a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx index 63331cfbbdafe0..a7c1bae47a5655 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx @@ -36,6 +36,42 @@ By default, tracing support is added to the following LangGraph operations: - **Agent Creation** (`gen_ai.create_agent`) - Captures spans when compiling a StateGraph into an executable agent - **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()` + + +## Trace Classifier Evaluations + + + +When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the LangChain integration records the call as a `gen_ai.evaluate` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed. + +```javascript +import { Annotation, END, START, StateGraph } from "@langchain/langgraph"; +import { TypeSafeClassifier } from "@langchain/typesafe"; + +const classifier = new TypeSafeClassifier({ + questions: { + urgent: { type: "noul", instructions: "Is this urgent?" }, + }, +}); + +const State = Annotation.Root({ ticket: Annotation(), urgent: Annotation() }); + +const graph = new StateGraph(State) + .addNode("triage", async (state) => { + const result = await classifier.invoke(state.ticket); + return { urgent: result.nouls.urgent.noul }; + }) + .addEdge(START, "triage") + .addEdge("triage", END) + .compile({ name: "triage_graph" }); + +await graph.invoke({ ticket: "My payouts have been failing for 3 days." }); +``` + +The evaluation state, questions, and answers follow the LangChain integration's `recordInputs` and `recordOutputs` options. + + + ## Supported Versions - `@langchain/langgraph`: `>=0.2.0 <2.0.0` diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx index e496bc386773a6..c8917b012b1cb6 100644 --- a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -23,6 +23,8 @@ The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https:// If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). +If you call Jev through LangChain's `TypeSafeClassifier`, including from LangGraph nodes and the `@langchain/typesafe` middleware, see the [LangChain integration](/platforms/javascript/guides/node/configuration/integrations/langchain/#trace-classifier-evaluations). + ```javascript Sentry.init({ dsn: "___PUBLIC_DSN___", From 0071ddd1a93791e8cd5b133243dabc8f2d237a28 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:11:21 +0000 Subject: [PATCH 2/5] docs(langchain): Read the choice answer through result.choices --- .../javascript/common/configuration/integrations/langchain.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index d02fd9e1893278..5a9e71353eb5aa 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -84,7 +84,7 @@ const result = await classifier.invoke( "I was charged twice. Please refund the extra charge." ); -console.log(result.answers.department.choice); +console.log(result.choices.department.choice); ``` Each `invoke()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them. From cc4cd1a5bfe348e9b47594d29de764aa91a58496 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:55:08 +0000 Subject: [PATCH 3/5] docs(langchain): List classifier evaluations under Supported Operations Co-Authored-By: Andrei Borza --- .../common/configuration/integrations/langchain.mdx | 6 ++++++ .../common/configuration/integrations/langgraph.mdx | 2 ++ 2 files changed, 8 insertions(+) diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index 5a9e71353eb5aa..85e59c5252da29 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -38,6 +38,12 @@ By default, tracing support is added to the following LangChain operations: - **Chain executions** - Captures spans for chain invocations - **Tool executions** - Captures spans for tool calls + + +- **Classifier evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` runs from `@langchain/typesafe`. See [Trace Classifier Evaluations](#trace-classifier-evaluations). + + + The integration automatically instruments the `invoke()`, `stream()`, and `batch()` runnable methods, across the following LangChain provider packages: - `@langchain/anthropic` diff --git a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx index a7c1bae47a5655..51df74acd07848 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx @@ -38,6 +38,8 @@ By default, tracing support is added to the following LangGraph operations: +- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls made inside graph nodes, through the LangChain integration. See [Trace Classifier Evaluations](#trace-classifier-evaluations). + ## Trace Classifier Evaluations From e461011a7e56caf6be2478f4f1a4c89dc4ff9503 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 12:14:46 +0000 Subject: [PATCH 4/5] docs(langchain): Inline classifier evaluations in Supported Operations Co-Authored-By: Andrei Borza --- .../common/configuration/integrations/langchain.mdx | 7 +------ .../common/configuration/integrations/langgraph.mdx | 3 +-- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index 85e59c5252da29..6d8fd28599f068 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -37,12 +37,7 @@ By default, tracing support is added to the following LangChain operations: - **LLM invocations** - Captures spans for LLM pipeline executions - **Chain executions** - Captures spans for chain invocations - **Tool executions** - Captures spans for tool calls - - - -- **Classifier evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` runs from `@langchain/typesafe`. See [Trace Classifier Evaluations](#trace-classifier-evaluations). - - +- **Classifier evaluations** - Captures spans for `TypeSafeClassifier` runs The integration automatically instruments the `invoke()`, `stream()`, and `batch()` runnable methods, across the following LangChain provider packages: diff --git a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx index 51df74acd07848..31e0437dde9616 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx @@ -35,11 +35,10 @@ By default, tracing support is added to the following LangGraph operations: - **Agent Creation** (`gen_ai.create_agent`) - Captures spans when compiling a StateGraph into an executable agent - **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()` +- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls in graph nodes -- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls made inside graph nodes, through the LangChain integration. See [Trace Classifier Evaluations](#trace-classifier-evaluations). - ## Trace Classifier Evaluations From c096b1cf50883affc13e5864e5492f8e872d839a Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Fri, 9 Oct 2026 12:26:31 +0000 Subject: [PATCH 5/5] docs(langchain): Show classifier evaluation tracing on Cloudflare Co-Authored-By: Andrei Borza --- .../common/configuration/integrations/langchain.mdx | 8 -------- .../common/configuration/integrations/langgraph.mdx | 4 ---- 2 files changed, 12 deletions(-) diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index 6d8fd28599f068..e7e2639392103c 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -57,8 +57,6 @@ The integration captures `gen_ai.embeddings` spans for `embedQuery()` and `embed - `@langchain/mistralai` - `@langchain/google-vertexai` - - ## Trace Classifier Evaluations @@ -94,8 +92,6 @@ Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) an If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the TypeSafe or Vercel AI integration. - - ## Supported Versions - `langchain`: `>=0.1.0 <2.0.0` @@ -111,12 +107,8 @@ Automatic embeddings instrumentation supports these package versions: For `@langchain/google-vertexai`, the version requirement applies to its `@langchain/google-common` dependency, which implements the embedding methods. - - Classifier evaluation tracing supports `@langchain/typesafe` `>=0.0.1 <1.0.0`. - - ## Options ### `recordInputs` diff --git a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx index 31e0437dde9616..872c7ae5c3eb34 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx @@ -37,8 +37,6 @@ By default, tracing support is added to the following LangGraph operations: - **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()` - **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls in graph nodes - - ## Trace Classifier Evaluations @@ -71,8 +69,6 @@ await graph.invoke({ ticket: "My payouts have been failing for 3 days." }); The evaluation state, questions, and answers follow the LangChain integration's `recordInputs` and `recordOutputs` options. - - ## Supported Versions - `@langchain/langgraph`: `>=0.2.0 <2.0.0`