Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ By default, tracing support is added to the following LangChain operations:
- **LLM invocations** - Captures spans for LLM pipeline executions
- **Chain executions** - Captures spans for chain invocations
- **Tool executions** - Captures spans for tool calls
- **Classifier evaluations** - Captures spans for `TypeSafeClassifier` runs

The integration automatically instruments the `invoke()`, `stream()`, and `batch()` runnable methods, across the following LangChain provider packages:

Expand All @@ -56,6 +57,41 @@ The integration captures `gen_ai.embeddings` spans for `embedQuery()` and `embed
- `@langchain/mistralai`
- `@langchain/google-vertexai`

## Trace Classifier Evaluations

<AvailableSince version="11.6.0" />

[`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe)'s `TypeSafeClassifier` asks TypeSafe's Jev model typed questions about your input. The `langChainIntegration` traces these calls for you, whether you call the classifier yourself, from a <PlatformLink to="/configuration/integrations/langgraph/#trace-classifier-evaluations">LangGraph</PlatformLink> node, or through the `modelRouterMiddleware` and `autoModeMiddleware` of `@langchain/typesafe/middleware` inside a `createAgent` run. No additional integration or setup is needed.

```javascript
import { TypeSafeClassifier } from "@langchain/typesafe";

const classifier = new TypeSafeClassifier({
questions: {
department: {
type: "choice",
instructions: "Which team should handle this?",
criteria: {
billing: "Charges and refunds",
technical: "Bugs and outages",
},
},
},
});

const result = await classifier.invoke(
"I was charged twice. Please refund the extra charge."
);

console.log(result.choices.department.choice);
```

Each `invoke()` call creates a `gen_ai.evaluate` span named `evaluate <model>`, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them.

Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other LangChain calls. Disable these options if evaluation state or answers contain sensitive data.

If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the <PlatformLink to="/configuration/integrations/typesafe/">TypeSafe</PlatformLink> or <PlatformLink to="/configuration/integrations/vercelai/#trace-jev-evaluations">Vercel AI</PlatformLink> integration.

## Supported Versions

- `langchain`: `>=0.1.0 <2.0.0`
Expand All @@ -71,6 +107,8 @@ Automatic embeddings instrumentation supports these package versions:

For `@langchain/google-vertexai`, the version requirement applies to its `@langchain/google-common` dependency, which implements the embedding methods.

Classifier evaluation tracing supports `@langchain/typesafe` `>=0.0.1 <1.0.0`.

## Options

### `recordInputs`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,39 @@ By default, tracing support is added to the following LangGraph operations:

- **Agent Creation** (`gen_ai.create_agent`) - Captures spans when compiling a StateGraph into an executable agent
- **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()`
- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls in graph nodes
Comment thread
sentry-junior[bot] marked this conversation as resolved.

## Trace Classifier Evaluations

<AvailableSince version="11.6.0" />

When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the <PlatformLink to="/configuration/integrations/langchain/#trace-classifier-evaluations">LangChain integration</PlatformLink> records the call as a `gen_ai.evaluate` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed.

```javascript
import { Annotation, END, START, StateGraph } from "@langchain/langgraph";
import { TypeSafeClassifier } from "@langchain/typesafe";

const classifier = new TypeSafeClassifier({
questions: {
urgent: { type: "noul", instructions: "Is this urgent?" },
},
});

const State = Annotation.Root({ ticket: Annotation(), urgent: Annotation() });

const graph = new StateGraph(State)
.addNode("triage", async (state) => {
const result = await classifier.invoke(state.ticket);
return { urgent: result.nouls.urgent.noul };
})
.addEdge(START, "triage")
.addEdge("triage", END)
.compile({ name: "triage_graph" });

await graph.invoke({ ticket: "My payouts have been failing for 3 days." });
```

The evaluation state, questions, and answers follow the LangChain integration's `recordInputs` and `recordOutputs` options.

## Supported Versions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://

If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations).

If you call Jev through LangChain's `TypeSafeClassifier`, including from LangGraph nodes and the `@langchain/typesafe` middleware, see the [LangChain integration](/platforms/javascript/guides/node/configuration/integrations/langchain/#trace-classifier-evaluations).

```javascript
Sentry.init({
dsn: "___PUBLIC_DSN___",
Expand Down
Loading