Roadmap: universal artifact forest, maximal Renderflow profile, and Flow orchestration readiness

[szmyty]

North-star outcome

Make Renderflow a trustworthy universal derivative engine that can:

1. accept an arbitrary source artifact or source collection,
2. inspect and classify it safely,
3. discover every configured/available derivative capability,
4. resolve one shared execution DAG,
5. wrap many replaceable local or remote tools through explicit adapters,
6. generate a bounded "everything" artifact forest when requested,
7. validate and provenance every output,
8. survive interruption and resume deterministically, and
9. expose a stable provider seam that egohygiene/flow can orchestrate without owning Renderflow's domain logic.

The concrete proving workload is a magazine/publication build that can fan reviewed source into print, digital/web, e-book, editorial, image/preview, accessibility, metadata, archive, and optional enrichment derivatives.

Deep-audit summary

Strong foundations already present

Renderflow is not starting from scratch. The repository already contains substantial architecture and implementation for:

• a modular Rust workspace and embeddable core SDK
• graph planning and multi-target DAG execution
• optimization modes
• transform registries and command transforms
• plugin SDK boundaries
• AI provider abstraction
• caching
• execution-plan serialization/visualization
• broad format/capability modeling and magic-byte detection
• documentation/product site/release infrastructure
• existing derivative roadmap issues

These pieces should be preserved and evolved rather than replaced wholesale.

Critical architectural gaps found

1. The "universal" graph is binary-incompatible underneath

The core transform/executor contracts still move String values, graph source loading uses read_to_string, and command output can be read as UTF-8 text. Binary-native documents/media/archives therefore cannot truthfully share the same execution substrate yet.

2. Configuration is still single-source and family-gated

The current config has one input and hard-coded document/image/audio compatibility rules. That prevents arbitrary source collections, cross-family derivatives, and declarative artifact-forest profiles.

3. There are multiple execution paths

The SDK dispatches between legacy build and graph-build paths instead of one canonical plan/execute lifecycle. This creates different semantics depending on invocation mode.

4. Public execution evidence is not authoritative yet

ExecutionResult exposes useful fields but currently populates cache reuse, skipped transforms, and diagnostics as empty values. ArtifactManifest is currently path/name oriented rather than provenance-complete artifact evidence.

5. External tool execution is duplicated and under-bounded

Multiple command runners execute arbitrary tools without one shared timeout/cancellation/output-bound/environment/provenance policy. This becomes a major reliability and security concern when the maximal profile wraps many tools.

6. Tool availability is not a complete planner model

The capability model currently knows a small built-in external-tool set while arbitrary commands/plugins can exist. Provider identity/version/platform/capability must become canonical planner input.

7. Format breadth is ahead of validation evidence

Many formats are declared, but the long-term support contract needs to require fixtures, validators, provider evidence, and honest loss/fidelity classification before a capability is called production-supported.

8. Flow integration needs resumable evidence, not just an SDK call

Flow already defines federated ownership, versioned contracts, artifact identity, redacted diagnostics, and resumable orchestration. Renderflow needs checkpoint compatibility and structured run/result contracts while Flow retains cross-holon state/recovery ownership.

9. Maximal publication expands the privacy/rights blast radius

The audit found no first-class publication redaction/sanitization stage. Generating many derivatives can multiply embedded metadata, paths, credentials, geolocation, rights ambiguity, or PII unless publication hygiene is explicit.

10. Current default-branch CI is red

The latest audited main run fails web setup because pnpm caching is requested before pnpm exists and fails Snapcraft validation because Snapcraft 9 rejects the current override-install key. These should be fixed before architecture expansion is treated as verified.

---

Dependency-aware quest line

Wave 0 — Restore truth

#351 — Restore the default-branch CI and packaging quality gate

Make main green and keep package/docs claims evidence-backed.

Exit: default-branch CI is completely green.

---

Wave 1 — Build the universal artifact substrate

#352 — Introduce a binary-safe artifact kernel and content-addressed artifact store

Replace String -> String as the universal substrate with binary-safe artifact identity, storage, collections, and lineage.

#356 — Centralize and harden external tool execution with bounded process policies

Create the one process execution service every wrapped tool will use.

#353 — Define Renderflow spec v2 for arbitrary sources, artifact forests, profiles, and execution policy

Make source collections, target selectors, profiles, budgets, and policy declarative/versioned.

#354 — Unify legacy build and graph build behind one canonical planner and executor

Every operation should become inspect -> plan -> execute -> validate -> evidence.

#359 — Create a runtime tool capability registry and reproducible toolchain fingerprint

Make installed provider/tool identity and versions planner/cache/provenance inputs.

#365 — Build universal input intake, deep inspection, and safe recursive artifact extraction

Turn arbitrary files and compound artifacts into trustworthy graph seeds without pretending unknown formats are supported.

Wave 1 exit: a binary source can be safely inspected, represented, planned, and transformed through the same canonical engine as text.

---

Wave 2 — Make execution trustworthy and resumable

#355 — Make execution results provenance-complete and align artifact evidence with Flow contracts

Populate actual outputs, cache reuse, skips, failures, validation, digests, producers, and source relationships.

#357 — Stabilize Transform v2 and the plugin SDK around typed artifact I/O

Give built-in and third-party transforms the same artifact-native runtime contract.

#360 — Add artifact validation, fidelity gates, and a capability conformance matrix

An exit code is not enough: outputs must carry validation evidence and honest support status.

#358 — Add checkpointed execution and a resumable Renderflow provider seam for Flow

Implement interruption-safe Renderflow work and the structured provider contract Flow will consume.

#366 — Establish a golden artifact corpus and end-to-end artifact-forest conformance suite

Build this incrementally throughout Waves 1-4; it becomes the release-grade proof that the contracts work together.

Wave 2 exit: a failed run can be assessed/resumed without repeating compatible work and every result explains exactly what happened.

---

Wave 3 — Expand the replaceable tool ecosystem

#362 — Establish first-class adapter packs for the high-value open tool ecosystem

Evaluate and wrap document/PDF, image, media, office, archive/data, OCR/extraction, subtitle, and related providers through the common adapter contracts.

Existing #344 — Evaluate and implement EPUB and KEPUB derivative support

Integrate as e-book provider capabilities rather than a special pipeline.

Existing #345 — Define the HandBrake transform adapter and Aniflow boundary

Use the bounded process/provider contracts and preserve Aniflow's temporal-video ownership.

Wave 3 exit: the maximal profile has broad real provider coverage without core/provider coupling.

---

Wave 4 — Build the maximal derivative product

#361 — Add publication hygiene for metadata sanitization, secret detection, privacy redaction, and rights gates

Make public/commercial release a deliberate state transition rather than "files exist, therefore ship them."

#363 — Add derivative profiles and an everything artifact-forest expansion mode

This is the "jacked on steroids" switch: expand every reachable, allowed, available, validated branch under explicit storage/runtime/network/AI budgets.

#364 — Add a magazine publication profile and complete multi-channel release bundle

Use the generic profile system to produce a magazine issue's print, digital, editorial, preview, metadata, provenance, checksum, and package derivatives.

Existing #348 — Add a rights-aware coloring-book publication profile and line-art pipeline

Optional specialized publication branch.

Existing #349 — Evaluate repository briefings and podcast derivatives from canonical content

Optional audio/podcast enrichment branch.

Existing #350 — Add a Slidev-backed product presentation publication profile

Optional presentation branch.

Wave 4 exit: one reviewed source publication can produce a validated multi-channel release forest in one shared DAG, with unavailable/blocked branches explicitly explained.

---

Suggested execution order

For small, reviewable PRs, use this order unless implementation evidence changes a dependency:

#351
  ↓
#352 ─────── #356
  ↓            ↓
#353         #359
  ↓            ↓
#354         #365
  ↓
#355 ─────── #357
  ↓            ↓
#360 ──────────┘
  ↓
#358
  ↓
#362 → #344 / #345
  ↓
#361
  ↓
#363
  ↓
#364

#366 should grow continuously alongside the implementation and become the completion proof rather than being postponed until the end.

Architecture guardrails

• Flow orchestrates; Renderflow transforms.
• Renderflow does not import mutable sibling source.
• Canonical source is immutable; derivatives are artifacts.
• Formats are domain types; external tools are replaceable providers.
• No capability is "supported" without implementation + fixture + availability semantics + validation evidence.
• No human console text is an orchestration contract.
• No hidden network/AI execution in deterministic/local profiles.
• No maximal build is unbounded; budgets and policies are explicit.
• No public/commercial bundle bypasses configured privacy/rights/validation gates.
• Generated artifacts and docs are projections, not competing sources of truth.
• Every new adapter should fit the same artifact/process/provider contracts rather than creating a bespoke pipeline.

Completion definition

This roadmap is complete when a synthetic magazine fixture and at least one real reviewed publication can be driven through the public Renderflow contract—directly or through Flow—to produce the configured artifact forest, interrupt and resume safely, validate every terminal artifact, emit provenance-complete manifests/checksums, and truthfully report unavailable, excluded, blocked, failed, reused, and produced branches.