Proposal: contrix-openeval-adapter — export runtime results as portable EvalPort ResultSets

[adhabnr-ux]

Hi! I really like the contract-first shape of Contrix — the Comparison Workflow (same endpoint contract run across prompt/model/provider variants) plus the structured CallLogRecord / RuntimeResponse data model in @contrix/runtime-core is genuinely more complete than most "wrapper" tools in this space.

I maintain EvalPort, an open interchange format (Suite + ResultSet JSON documents, spec at spec/SPEC.md) for portable LLM eval data — the idea is that results produced by one tool (Contrix, promptfoo, DeepEval, OpenAI Evals, …) can be diffed, archived, or re-graded by another without a bespoke converter each time.

I'd like to propose a small contrix-openeval-adapter package that converts a Contrix run into an EvalPort ResultSet, and would be happy to build/PR it myself if that's welcome — opening this first per the "open an issue for major feature changes" note in docs/contributing.md.

Why Contrix is a good fit: @contrix/runtime-core already exports a real, typed result object per call — RuntimeResponse = RuntimeSuccessResponse | RuntimeFailureResponse — plus CallLogRecord, ValidationResult/ValidationIssue, and RetryAttemptMeta[]. That's essentially an EvalPort Result + GraderResult already, just under different field names. The Comparison Workflow (one endpoint contract, many prompt/model/provider variants) maps naturally to multiple EvalPort ResultSets sharing one suite_id, which is exactly the "compare fairly under one contract" scenario EvalPort's diffing is meant for.

Sketch of the conversion (types below are the real exports from @contrix/runtime-core and @contrix/spec-core):

import type {
  RuntimeResponse,
  RuntimeSuccessResponse,
  RuntimeFailureResponse,
  CallLogRecord,
  ValidationResult
} from '@contrix/runtime-core';
import type { EndpointSchemaDocument } from '@contrix/spec-core';

// EvalPort types (spec/SPEC.md §4)
interface GraderResult {
  grader_id: string;
  type: string;
  score: number;
  passed: boolean;
  reason?: string;
  metadata?: Record<string, unknown>;
}
interface Result {
  test_case_id: string;
  actual_output?: string;
  grader_results: GraderResult[];
  passed: boolean;
  duration_ms?: number;
  error?: { message: string; type: string };
  metadata?: Record<string, unknown>;
}

function toGraderResult(v: ValidationResult, graderId: string): GraderResult {
  return {
    grader_id: graderId,
    type: 'json_schema',
    score: v.success ? 1 : 0,
    passed: v.success,
    reason: v.errors.map((e) => e.message).join('; ') || undefined,
    metadata: { errors: v.errors }
  };
}

// one CallLogRecord + its RuntimeResponse -> one EvalPort Result
function toResult(log: CallLogRecord, response: RuntimeResponse): Result {
  if (response.success) {
    const r = response as RuntimeSuccessResponse;
    const lastAttempt = r.attempts[r.attempts.length - 1];
    const graderResults = lastAttempt?.validationResult
      ? [toGraderResult(lastAttempt.validationResult, 'gr_json_schema')]
      : [];
    return {
      test_case_id: log.endpointKey ?? log.requestId,
      actual_output: r.finalOutput,
      grader_results: graderResults,
      passed: graderResults.every((g) => g.passed),
      duration_ms: log.latencyMs ?? undefined,
      metadata: {
        model: log.model,
        provider: log.providerKey,
        outputSource: r.outputSource,
        attemptCount: r.attemptCount,
        usage: r.usage
      }
    };
  }
  const f = response as RuntimeFailureResponse;
  return {
    test_case_id: log.endpointKey ?? log.requestId,
    grader_results: [],
    passed: false,
    duration_ms: log.latencyMs ?? undefined,
    error: { message: f.error.message, type: f.error.type }
  };
}

An EndpointSchemaDocument.outputSchema (from @contrix/spec-core) would compile 1:1 into an EvalPort Grader of type: "json_schema" with params.schema set to that schema, so the endpoint contract itself becomes the Suite's grader definition, and each logged call becomes a Result against it.

Happy to scope this down further or adjust the mapping if you see a cleaner seam (e.g. hanging the converter off CallLogRecord + a debug snapshot instead of the full RuntimeResponse). Let me know if this is something you'd want in-repo, as a separate package, or not at all — no worries either way, just wanted to float it since the runtime-core types made the mapping unusually clean.