Skip to content

Add artifact validation, fidelity gates, and a capability conformance matrix #360

Description

@szmyty

Outcome

Require Renderflow to prove that an output is structurally usable and policy-compliant before it is reported as a successful artifact.

Format support should mean a tested capability, not merely an enum variant or an external command that happened to exit zero.

Scope

Validation model

Define structured validation outcomes that can represent:

  • valid
  • valid with warnings
  • invalid
  • validator unavailable
  • validation skipped by explicit policy

Validation evidence should identify the validator implementation/version and preserve raw diagnostics safely where useful.

Fidelity/loss policy

Make transform loss characteristics enforceable:

  • lossless
  • partial loss
  • lossy
  • path-dependent

Allow specs/profiles to set minimum fidelity requirements or reject particular loss classes.

Capability conformance

Generate a machine-readable matrix for each declared format/capability/transform covering:

  • implemented executor
  • required provider/tool
  • fixture coverage
  • validator
  • supported platforms
  • deterministic status
  • loss profile
  • current support status

Do not advertise a capability as production-supported when its required fixture or validator evidence is absent.

Initial validation adapters

Use narrow, replaceable validators appropriate to each family. Evaluate existing/native parsers first and wrap external validators through #356/#359 where required. Examples may include structural probes for media, document/PDF, EPUB, images, archives, and HTML.

Do not make a specific third-party validator canonical without evaluating licensing, portability, maintenance, and failure semantics.

Acceptance criteria

  • Every terminal artifact has an explicit validation state in execution evidence.
  • A transform exiting successfully cannot produce a complete artifact when required validation fails.
  • Validators are capability/tool providers with versions and structured diagnostics.
  • Specs/profiles can require validation and reject configured loss classes.
  • Validation failure can be fatal or branch-local according to explicit execution policy.
  • The generated capability matrix distinguishes implemented, experimental, unavailable, and planned behavior.
  • Documentation and product-site capability claims can be derived from the conformance matrix.
  • Representative fixtures cover document, image, audio/video, archive, and structured-data validation.
  • Corrupt/truncated outputs are caught by tests even when the producing fake command exits zero.
  • Validation evidence participates in checkpoint/resume compatibility.
  • Formatting, Clippy, tests, docs, and CI pass.

Dependencies

Depends on #352 and #359; execution evidence should integrate with #355.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions