Conversation
Multiple resolvers on one extraction artifact can now partition its
mentions by label: an event-labeled resolver promotes events into an
events table while a catch-all resolver keeps entity labels
open-vocabulary by skipping sibling-claimed labels at runtime. Labeled
resolvers sharing an artifact must claim disjoint label sets
(admission-enforced); label changes trigger re-resolution backfill.
Adds a replay-stable {{ hash }} key-template helper (xxhash64) for
minting canonical event keys from normalized text.
See zig/AUTOSCHEMA.md (AutoSchemaKG integration, arXiv:2505.23628).
Seeded PPR (AUTOSCHEMA.md engine gap 2): pageRank kernel gains a
personalized teleport vector (uniform over seed nodes, sink mass
restarts through the teleport, deterministic across executor widths).
GraphMetricRead accepts seed_nodes (max 128) + damping, fresh-only:
personalization with published freshness fails closed, as does the
serverless published-segment reader. Absent seed keys are skipped and
counted, empty seeds degenerate to global PageRank.
Neighbor context (engine gap 1): asset enrichments may declare
neighbor_context {graph_index, edge_types, direction, limit} to sample
local graph adjacency into the producer input as a deterministic JSON
block - the conceptualizer's context for abstraction phrases. The block
lands before the skip-state hash so unchanged adjacency skips and edge
changes re-run the producer. Admission closes the graph-index
reference; runtime fails open to empty neighbors. Generator/extractor
producers only (reader/transcriber treat source_text as a media
locator).
See zig/AUTOSCHEMA.md (AutoSchemaKG integration, arXiv:2505.23628).
…l agent graph_metric and graph_metric_rerank accept seed_nodes (max 128) and damping on the wire; seeds route to fresh personalized top-k/column readers, never published reads. Fail-closed boundaries: published freshness with seeds and non-pagerank seeding map to 400, serverless and cross-shard personalized requests to 422 (shard-local personalized scores are not globally comparable; single-group forwarding carries seeds losslessly). The retrieval agent auto-seeds a fresh graph_metric_rerank from the literal traversal start keys of the query's graph searches - the HippoRAG-style personalization seeds for autoschema retrieval - and degrades to unseeded behavior when no literal start keys exist. See zig/AUTOSCHEMA.md engine gap 3.
labels on GraphResolverConfig, seed_nodes/damping on GraphMetricQuery and GraphMetricRerank, EnrichmentNeighborContextConfig - across the joined public spec and the Go/Python/TypeScript SDKs and generated Zig types. antfly build, api, and graph suites green against the regenerated types.
Catalog registration enforces label disjointness, but registration runs during asynchronous shard provisioning, so an ambiguous partition surfaced as a provisioning stall instead of a create/update error. Close it in validateArtifactIndexReferences so the conflict is a synchronous 4xx, checked across every graph index in the request because resolvers are table-scoped. Proven by e2e: test_overlapping_labeled_resolvers_rejected_at_admission now passes alongside the label-routing promotion e2e.
…xonomy --autoschema provisions the full AutoSchemaKG pipeline (zig/AUTOSCHEMA.md, arXiv:2505.23628): three forced-tool-call generator enrichments on the docs table (entity-entity open-vocabulary; entity-event with events as label-"event" mentions and participates_in relations; event-event with enum-constrained temporal/causal types, prompts ported from the MIT reference implementation), one knowledge_graph index with three extraction_graph sources and label-routed resolvers promoting events and entities into their own tables, and a recursive taxonomy autograph on the entities table: conceptualize_v1 emits >=3 concept phrases per entity with is_a relations, resolved into a concepts table, with self-referential neighbor_context so later passes ground in previously promoted is_a adjacency. Also adapts the example to the regenerated SDK (iter.Seq linear merge, IndexEmbedderConfig).
…nifest The storage shard audit requires every test source under storage/ to be imported by the manifest; the neighbor-context module's tests otherwise never run in the unit gate.
ajroetker
force-pushed
the
feat/autoschema-kg
branch
from
September 20, 2026 04:51
f203791 to
9c701e9
Compare
Three defects found by running --autoschema against a real local model (gemma-4-E4B on Metal, 5-document epstein smoke corpus): 1. Generator producers have no prompt config field: the canonical generator contract rejects it with UnknownField at enrichment time (it was verified against the reader contract by mistake). The extraction and conceptualizer instructions now travel in the enrichment source template, whose rendered text is the producer's prompt. 2. A malformed model reply crashed the storage kernel: MissingToolCall (invalid tool arguments on a successful generation response) propagates as a retryable request error with no recorded failure fingerprint, which activeRequestRetryBudgetAllowsYield explicitly admits but restoreDeferredRequestRetryAuthorization asserted against. Fingerprint 0 now means no authorization to restore. 3. Multi-source graphs pair every changed resolution key with every mention-edge source; a resolution artifact owned by a different source's resolver failed closed as MissingResolverArtifactContract, stalling knowledge_graph journal apply. Sibling ownership is now a benign skip; only an unowned resolution artifact fails closed. Also declares a named ppr pagerank metric on knowledge_graph so query-seeded personalized reads work out of the box. Validated live: full pipeline (extraction -> label-routed event/entity promotion -> conceptualization -> is_a taxonomy with cross-table hydration) completes with zero panics, and seeded PPR returns teleport-concentrated scores. Known follow-up: relation edges materialize with unresolved local-id endpoints (e0/e1) absent node mapping templates; mention edges resolve correctly.
Closes the two issues from the first live autoschema evaluation, plus two more engine defects the fixes uncovered: 1. Relation endpoints canonicalize through resolution artifacts. A relation endpoint referencing an extraction entity by local id now renders the resolver-minted canonical key: the replay materializer injects a per-document resolution map (local_id -> doc_ref) into the parsed artifact, and a landed resolution artifact re-renders the owning extraction artifact (depth-one recursion in materializeGraphSourceArtifactsForIndex). Before resolution the endpoint is dropped rather than rendered - a local mention id is not a node, and rendering it stranded orphan edges no replay retired. Endpoints matching no extraction entity keep the external-node string passthrough. Verified live: seeded PPR over the epstein corpus lists only canonical person/event/concept/doc nodes, zero local-id leaks. 2. Extraction prompts refuse fabrication. Empty or unreadable pages under a forced tool call made the model invent canned content (Musk/Tesla triples from blank scanned pages). All three passes now instruct extracting only what the passage states and calling the tool with empty arrays when there is nothing. Verified live: empty pages yield empty extractions, promoted entities are all genuine corpus content. 3. Generator asset producers keep their enrichment lease alive. A local-LLM generation exceeding the lease TTL lost the fence, was discarded and retried until the worker retired permanently - the OCR/transcription paths already guard against exactly this; flushAssetProducerBatchItems now uses the same RuntimeLeaseHeartbeatGuard. Verified live: a run that previously stalled at zero artifacts completes with no worker retirement. 4. No store reads during batch apply: the endpoint-resolution lookup is replay-only; the batch graph-artifact path renders without it and relies on the resolution replay for canonical convergence. Suites: antfly-storage-db-test 1389 passed (single pre-existing failure unchanged), enrichment 283/283, autoschema e2e 2/2, epstein go tests green.
# Conflicts: # go/pkg/sdk/oapi/client.gen.go
Contributor
Author
|
/ci run 97aef2c |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements native AutoSchemaKG-style knowledge graph construction (arXiv:2505.23628) inside the autograph pipeline. Design doc:
zig/AUTOSCHEMA.md(committed here, gap table tracks as-built status).What this adds
Label-routed resolvers —
GraphResolverConfig.labelslets multiple resolvers partition one extraction artifact by mention label: anevent-labeled resolver promotes events into aneventstable while a catch-all keeps entity labels open-vocabulary by skipping sibling-claimed labels at runtime. Labeled siblings sharing an artifact must claim disjoint sets — enforced synchronously at admission (4xx) and again at catalog registration. New{{ hash }}key-template helper (xxhash64) mints replay-stable event keys from normalized text.Seeded personalized PageRank — the pagerank kernel gains a personalized teleport vector (uniform over seeds, sink mass restarts through the teleport, deterministic across executor widths).
graph_metricandgraph_metric_rerankacceptseed_nodes(max 128) +dampingend-to-end: wire parse, fresh personalized top-k/column readers, single-group distributed forwarding. Fail-closed everywhere personalization can't be honest: published freshness → 400, serverless → 422, cross-shard → 422.Producer neighbor context — asset enrichments may declare
neighbor_context {graph_index, edge_types, direction, limit}to sample local graph adjacency into the producer input as a deterministic JSON block (the conceptualizer's grounding for abstraction phrases). Participates in the skip-state hash; admission closes the graph-index reference; runtime fails open to empty neighbors.Retrieval agent seeding —
graph_searchauto-seeds a freshgraph_metric_rerankfrom the query's literal traversal start keys (HippoRAG-style personalization), degrading to unseeded when none resolve.Epstein
--autoschemamode — the paper's full pipeline as configuration: three forced-tool-call extraction enrichments (entity-entity; entity-event with events aslabel:"event"mentions; event-event with enum-constrained temporal/causal types, prompts ported from the MIT reference implementation), a knowledge graph with four label-routed resolvers over the three artifacts, and a recursive taxonomy autograph promoting ≥3is_aconcept phrases per entity into aconceptstable with self-referential neighbor context.Validation
zig/e2e/antfly/test_autoschema.py: label-routed event/entity promotion partition on a live 3-node cluster, and synchronous admission rejection of ambiguous label claims (which caught a real gap — conflicts previously surfaced as an async provisioning stall).relational columnar maintenance survives unrelated artifact corruption and backoff) is pre-existing — identical failure on a clean checkout of the merge base viaantfly-storage-db-test; likewise 17 retrieval-agent test failures are byte-identical on base.go build/vet/testgreen; SDKs regenerated (Go/Python/TS) and canonical after rebasing onto the Exa retrieval changes.🤖 Generated with Claude Code