diff --git a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx index 2affbeabd9428..e7e2639392103 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langchain.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langchain.mdx @@ -37,6 +37,7 @@ By default, tracing support is added to the following LangChain operations: - **LLM invocations** - Captures spans for LLM pipeline executions - **Chain executions** - Captures spans for chain invocations - **Tool executions** - Captures spans for tool calls +- **Classifier evaluations** - Captures spans for `TypeSafeClassifier` runs The integration automatically instruments the `invoke()`, `stream()`, and `batch()` runnable methods, across the following LangChain provider packages: @@ -56,6 +57,41 @@ The integration captures `gen_ai.embeddings` spans for `embedQuery()` and `embed - `@langchain/mistralai` - `@langchain/google-vertexai` +## Trace Classifier Evaluations + + + +[`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe)'s `TypeSafeClassifier` asks TypeSafe's Jev model typed questions about your input. The `langChainIntegration` traces these calls for you, whether you call the classifier yourself, from a LangGraph node, or through the `modelRouterMiddleware` and `autoModeMiddleware` of `@langchain/typesafe/middleware` inside a `createAgent` run. No additional integration or setup is needed. + +```javascript +import { TypeSafeClassifier } from "@langchain/typesafe"; + +const classifier = new TypeSafeClassifier({ + questions: { + department: { + type: "choice", + instructions: "Which team should handle this?", + criteria: { + billing: "Charges and refunds", + technical: "Bugs and outages", + }, + }, + }, +}); + +const result = await classifier.invoke( + "I was charged twice. Please refund the extra charge." +); + +console.log(result.choices.department.choice); +``` + +Each `invoke()` call creates a `gen_ai.evaluate` span named `evaluate `, with the model, provider, latency, and token usage. Inside an agent, the span also records the agent name. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures the `state` and `questions` sent to Jev as input messages and `answers` as output messages, including probabilities returned by the model. LangChain messages in the input are recorded as transcript lines, the same way the classifier sends them. + +Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other LangChain calls. Disable these options if evaluation state or answers contain sensitive data. + +If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the TypeSafe or Vercel AI integration. + ## Supported Versions - `langchain`: `>=0.1.0 <2.0.0` @@ -71,6 +107,8 @@ Automatic embeddings instrumentation supports these package versions: For `@langchain/google-vertexai`, the version requirement applies to its `@langchain/google-common` dependency, which implements the embedding methods. +Classifier evaluation tracing supports `@langchain/typesafe` `>=0.0.1 <1.0.0`. + ## Options ### `recordInputs` diff --git a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx index 63331cfbbdafe..872c7ae5c3eb3 100644 --- a/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/langgraph.mdx @@ -35,6 +35,39 @@ By default, tracing support is added to the following LangGraph operations: - **Agent Creation** (`gen_ai.create_agent`) - Captures spans when compiling a StateGraph into an executable agent - **Agent Invocation** (`gen_ai.invoke_agent`) - Captures spans for agent execution via `invoke()` +- **Classifier Evaluations** (`gen_ai.evaluate`) - Captures spans for `TypeSafeClassifier` calls in graph nodes + +## Trace Classifier Evaluations + + + +When a graph node calls a `TypeSafeClassifier` from [`@langchain/typesafe`](https://www.npmjs.com/package/@langchain/typesafe), the LangChain integration records the call as a `gen_ai.evaluate` span. The span is a child of the graph's `invoke_agent` span and records the graph name as the agent name. No additional setup is needed. + +```javascript +import { Annotation, END, START, StateGraph } from "@langchain/langgraph"; +import { TypeSafeClassifier } from "@langchain/typesafe"; + +const classifier = new TypeSafeClassifier({ + questions: { + urgent: { type: "noul", instructions: "Is this urgent?" }, + }, +}); + +const State = Annotation.Root({ ticket: Annotation(), urgent: Annotation() }); + +const graph = new StateGraph(State) + .addNode("triage", async (state) => { + const result = await classifier.invoke(state.ticket); + return { urgent: result.nouls.urgent.noul }; + }) + .addEdge(START, "triage") + .addEdge("triage", END) + .compile({ name: "triage_graph" }); + +await graph.invoke({ ticket: "My payouts have been failing for 3 days." }); +``` + +The evaluation state, questions, and answers follow the LangChain integration's `recordInputs` and `recordOutputs` options. ## Supported Versions diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx index e496bc386773a..c8917b012b1cb 100644 --- a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -23,6 +23,8 @@ The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https:// If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). +If you call Jev through LangChain's `TypeSafeClassifier`, including from LangGraph nodes and the `@langchain/typesafe` middleware, see the [LangChain integration](/platforms/javascript/guides/node/configuration/integrations/langchain/#trace-classifier-evaluations). + ```javascript Sentry.init({ dsn: "___PUBLIC_DSN___",