feat: relational indexes, typed row APIs, and distributed integrity - #784
Merged
Merged
Conversation
…, and TTL Unify staged document and relational restoration, preserve schema migrations, add durable standby recovery protections and strict observed-row predicates. Add bounded coordinated TTL and constrained topology handoff coverage. Generation and formatting complete; final multi-node validation and main integration remain in progress.
…ners Preserve coordinated restore and integrity lifecycles across the compiled storage boundary, including exact backup seals, metadata standby outboxes, hidden owners, and bounded TTL callbacks. Route unpublished transition reads through exact metadata authority and keep distributed HTTP on the owning std.Io runtime. Validated production build, linked owner/source regressions, mixed native/portable and standalone restore suites, three-node merge/split/restart history, generated sources, formatting, and boundary audits.
…d schema rewrites Unify relational index maintenance and generated APIs; add deferred FK and MATCH PARTIAL coordination, typed partial-index implication, and retained-effects online topology and rewrite workflows. Harden snapshot recovery, staged restore activation, callback error transport, and prepared outcome lifetimes with component and distributed fault regressions.
…dexes # Conflicts: # go/pkg/sdk/oapi/client.gen.go # scripts/test_openapi_build_paths.py # ts/packages/sdk/test/index-config.test.ts # zig/pkg/antfly/antfarm/assets/index-BH7KrwjM.js # zig/pkg/antfly/antfarm/assets/index-CVp2L_j0.js # zig/pkg/antfly/antfarm/assets/index-DdHoRU6k.js # zig/pkg/antfly/antfarm/index.html # zig/pkg/antfly/build/storage_owner_tests.zig # zig/pkg/antfly/build/tests.zig # zig/pkg/antfly/src/api/http_client.zig # zig/pkg/antfly/src/api/http_server.zig # zig/pkg/antfly/src/api/httpx_handler.zig # zig/pkg/antfly/src/api/internal_group_operations.zig # zig/pkg/antfly/src/api/kernel_owner_source.zig # zig/pkg/antfly/src/api/table_reads.zig # zig/pkg/antfly/src/api/table_write_source.zig # zig/pkg/antfly/src/api/table_writes.zig # zig/pkg/antfly/src/capi/db.zig # zig/pkg/antfly/src/data/runtime.zig # zig/pkg/antfly/src/metadata/table_manager.zig # zig/pkg/antfly/src/openapi/generated/antfly_client_openapi/types.zig # zig/pkg/antfly/src/openapi/generated/antfly_metadata_openapi/server.zig # zig/pkg/antfly/src/openapi/generated/antfly_public_openapi/server.zig # zig/pkg/antfly/src/runtime_error_abi.zig # zig/pkg/antfly/src/runtime_failure_abi.zig # zig/pkg/antfly/src/runtime_failure_identity.zig # zig/pkg/antfly/src/runtime_native_abi.zig # zig/pkg/antfly/src/standalone/runtime.zig # zig/pkg/antfly/src/storage/db/db.zig # zig/pkg/antfly/src/storage/hot_standby/seed_topology.zig # zig/pkg/antfly/src/storage/kernel_owner_abi.zig
Keep yielded restore successors queued until worker release, publish constraint failures with source and claim guards, and cache bounded snapshot-local row bindings. Cover queue races, parent repairs, partial-witness diagnostics, schema churn and allocation failures.
…dexes # Conflicts: # go/pkg/sdk/oapi/client.gen.go # zig/pkg/antfly/antfarm/assets/index-BH7KrwjM.js # zig/pkg/antfly/antfarm/assets/index-DlSyhrKc.js # zig/pkg/antfly/antfarm/assets/index-Dzdabs15.js # zig/pkg/antfly/antfarm/index.html
…dexes # Conflicts: # zig/pkg/antfly/src/data/runtime.zig # zig/pkg/antfly/src/runtime_error_abi.zig # zig/pkg/antfly/src/runtime_failure_abi.zig
… paths Integrate PR 691 with one native standalone catalog authority, stable FK bindings, scoped generated relational APIs, and atomic catalog-aware staged restore. Preserve released main protocol and ABI identities. Accept explicit schema epoch zero, bound admitted index keys consistently with cursors and maintenance, and retain unaffected LSM sources during monotone seeks. Cover catalog upgrade, cancellation lock ordering, qualified restore copies, and long catalog identities with regressions.
…dexes # Conflicts: # go/pkg/sdk/oapi/client.gen.go
…dexes # Conflicts: # go/pkg/sdk/oapi/client.gen.go
…dexes # Conflicts: # zig/e2e/antfly/test_backup_restore.py # zig/pkg/antfly/src/api/kernel_owner_source.zig # zig/pkg/antfly/src/api/table_catalog.zig # zig/pkg/antfly/src/runtime_error_abi.zig # zig/pkg/antfly/src/runtime_failure_abi.zig # zig/pkg/antfly/src/storage/kernel_owner_abi.zig
Contributor
Author
|
/ci run faf3c8c |
…dexes # Conflicts: # go/pkg/sdk/oapi/client.gen.go # scripts/test_openapi_build_paths.py # zig/pkg/antfly/build/imports.zig
Contributor
Author
|
/ci run fa9776f |
1 similar comment
Contributor
Author
|
/ci run fa9776f |
Contributor
Author
|
/ci run 01224a3 |
…dexes # Conflicts: # go/pkg/sdk/oapi/client.gen.go # zig/pkg/antfly/src/api/retrieval_agent.zig # zig/pkg/antfly/src/storage/db/db.zig
# Conflicts: # scripts/ci/test_zig_e2e_shard.py # zig/e2e/antfly/README.md
Contributor
Author
|
/ci run 2340e21 |
Contributor
Author
|
/ci run acd89eb |
Contributor
Author
|
/ci run ccbc006 |
…dexes # Conflicts: # zig/pkg/antfly/src/standalone/runtime.zig
Contributor
Author
|
/ci run af79b7d |
ajroetker
added a commit
that referenced
this pull request
Sep 21, 2026
schema/table_schema_impl.zig now takes the public table storage mode and the relational wire types from antfly_schema_openapi (#784), but the embedded module graph (embedded_support and the embedded db/api roots) and its wasm twin were never handed that module, so embedded-test no longer compiled. Add it to configureModule next to the other generated OpenAPI modules and to both callers' dependency tuples. CI does not run embedded-test, which is why main stayed green. The wasm bundle itself still does not build, for reasons that predate this change: 64-bit atomics in hbc_vector_directory, index_manager, core, search_exec, relational_columns, source_pin and hbc_adapter, and std.Io.Dir/std.Thread reached from the same graph, none of which the freestanding target implements.
This was referenced Sep 21, 2026
ajroetker
added a commit
that referenced
this pull request
Sep 22, 2026
…gfood knowledge-graph example (#793) * inference: pull GLiNER2.5 boundary encoder config and report unqualified models clearly Add encoder_config/config.json to the always-downloaded file list so `antfly inference pull fastino/gliner2.5-base-v1` produces a loadable model directory, with a unit test. Map error.UnsupportedGlinerBoundaryRuntime to a 400 MODEL_NOT_QUALIFIED response instead of the generic 413 MODEL_RESOURCE_LIMIT fallback. * libantfly: always link the standalone inference runtime libantfly now embeds inference the way the antfly executable does: the inference archive is linked into the C ABI shared library, the link anchor no longer traps antfly_standalone_inference_get_function_table, and Lite advertises local_inference_runtime by default. The release size gate moves from 20 MiB to 60 MiB (measured 53 MiB unstripped, 48 MiB stripped); no inference symbols are exported. COMPILATION.md and LITE.md record the decision. * lite: provision the default full-text index and run enrichment producers Creating a Lite database now provisions full_text_index_v0 like a server table, empty index names resolve to it, and a second full-text index no longer makes no-name queries or replay ambiguous; portable import treats a DB holding only the default index as empty. Native Lite handles now configure the managed enrichment runtime: remote api_url producers (chunker, embedder, extractor asset) execute through the existing provider interfaces, dense index configs are translated the way the server does before addIndex, and semantic_search embeds the query through the index's embedder. Handles opened with LocalRuntimeConfigured own an embedded inference provider with a resource-budget owner. The chunk and extraction HTTP clients normalize a bare api_url to /ai/v1. Go binding tests cover the default index, remote-provider embedding and query-time embedding with a fake server, and the embedded capability. * examples: add dogfood, a knowledge graph over the design docs in Antfly Lite Ingests the Zig design docs and work-log into an embedded .aflite with Qwen3 embeddings over fixed chunks and a GLiNER extraction-fed graph index, then runs hybrid search plus graph traversal. Inference runs in-process by default with -inference-url as a remote override. The README records measured throughput and the currently known gaps. * lite: wire the new embedded tests into the embedded-test filter * lite: fix double free in antfly_db_get_edges_json graphFreeEdges called GraphIndex.freeEdges, which already frees the edge slice, and then freed the slice again. With no edges the second free was a no-op, so the bug stayed hidden until a graph index had real edges, after which the corrupted heap aborted at an unrelated later free. Add a Zig regression test under the testing allocator and a Go binding test that builds a graph from a fake extraction provider and reads it back. * lite: resolve the inference worker for library hosts and batch enrichment The sandboxed inference worker was spawned from argv[0], which only works when the host is the antfly binary. Resolve it explicitly instead: ANTFLY_INFERENCE_WORKER, then an antfly binary beside the loaded libantfly (dladdr), then PATH, with a clear error naming the variable. The worker stays a separate process because Metal, CUDA, and PJRT calls require process isolation for crash containment. The embedded node keeps all models resident, the enrichment runtime batches embeddings 32 at a time and extraction 16 at a time, and runUntilIdle logs a throughput summary. The Go binding test steps set ANTFLY_INFERENCE_WORKER to the build's own antfly so they cannot fall through to an unrelated antfly on PATH. * lite: stop rescanning the catalog on every mutation and drain without a hidden deadline pageAllocatorFromFreeMap validated free pages against every checkpoint slot, the full catalog, and the document tree on every mutation, making each commit cost proportional to everything written so far; an ingest of a few thousand vectors stalled at seconds per commit. Verify the cross-checkpoint invariant once per open handle (and in check()), which still catches reopen-after-corruption. runUntilIdle inherited the 5-minute sync-wait timeout when no deadline was supplied and then failed with an internal error. Give its own drain path an unbounded wait while leaving every other caller's deadline behavior alone. * examples/dogfood: record the in-process Metal run and remaining gaps * Lite: concurrent enrichment, GLiNER2.5 qualification, artifact-sourced vectors, and surface parity (#806) * enrichment: run the asset and dense lanes concurrently with independent replay cursors flushDeferredGeneratedWork ran extraction and then embedding strictly one after the other per preparation window, so drain time was the sum of both streams. Split the work into an asset lane and a dense lane, each with its own replay window and cursor scope (generated.assets, generated.dense), run them via Io.concurrent when the runtime supports it and sequentially otherwise, and let each publish and checkpoint independently. Resume skips a group only when both cursors cover it; the applied_sequence watermark is unchanged. ENRICHMENTS.md documents the two-stream model. * lite: accept artifact-sourced dense-vector indexes and surface config errors antfly_db_add_index_json never registered the nested enrichments array inside an index's config_json, so a dense_vector index sourced from a chunk artifact declared a producer that was never installed and failed deep in the runtime. Harvest and dependency-sort those enrichments with the server's own logic before the index is translated and admitted, so the two-stage chunk-artifact pattern works in Lite exactly as on the server, with hierarchy projection back to the parent document. The C ABI error map also had no entries for index and enrichment validation errors, which all fell through to ANTFLY_INTERNAL; they now surface as ANTFLY_INVALID_ARGUMENT, and a missing embedding probe as ANTFLY_BUSY. * lite: one create routine and one identity policy across the CLI, C ABI, and embedded surfaces antfly lite init went through a third creation path that did not provision full_text_index_v0, and the CLI pinned every file to the embedded root identity while the C ABI and embedded package left the namespace unset, so a database created from Go could not be opened by the CLI (IdentityNamespaceMismatch). storage/lite/connection.zig now owns identityOpenOptions and provisionDefaultFullTextIndex and every surface calls them: new files are pinned to the embedded root identity, existing files keep whatever identity they carry. Also fixes two pre-existing test failures at their root: the enrichment runtime refresh after open/addIndex rebuilt its config from the index catalog alone and tore down a resumed generated-replay runtime whose chunk enrichment was referenced by name; and dense-index translation never reached the embedder dimension probe when the config omitted field. The embedded db module's own tests are now wired into embedded-test, which surfaced and fixed a stale format literal, a double-pointer in importPortable, and a vacuum/status race. * inference: qualify GLiNER2.5 base for production extraction Add the production qualification row for fastino/gliner2.5-base-v1 (native and Metal) backed by verified artifact digests against the oracle manifest, the pinned pipeline fixtures now run against the real checkpoint, a real-tokenizer geometry probe for the length contract, and the checked-in CPU benchmark. runtime_available is on; pull-time manifest synthesis hashes the downloaded weights and emits extract tasks and capabilities only for a qualified identity, so unqualified digests and variants stay refused. Two further gates only surfaced by running the real request: a blanket name-based denial in the capability listing, and a routing gap that sent a plain (schema_version-less) request to the legacy handler; boundary requests now auto-upgrade to the schema_version 2 path. Adds a direct vs HTTP embedding benchmark test showing the in-process entry point has no per-batch overhead of its own. GLINER25.md records the evidence and re-qualification steps; EXTRACT.md lists model support. * inference: upgrade plain boundary requests on the shared extraction entry The schema_version upgrade for GLiNER boundary models ran only in the HTTP handler, so the in-process provider operation used by Lite's enrichment runtime still reached the legacy dispatch and failed with BoundaryExtractionRequiresSchema. Apply the upgrade in extractWithAdmission, which both entries share, and cover the provider entry with a test against the qualified checkpoint. Widen the qualified length contract from the measured requirement of a production-sized schema (11 entity labels, 6 relation types) on a realistic paragraph; longer inputs still fail closed. * inference: qualify GLiNER2.5 long-document windowed extraction The windowed boundary executor was implemented but never reviewed, so every document beyond a single window failed closed. Add production rows (native and Metal) carrying the long_document feature with a length contract measured on this repository's real sections up to the corpus maximum (99 KB, four windows), verify the canonical schema_version 2 envelope survives the multi-window merge through both the HTTP handler and the provider entry, and record the evidence in GLINER25.md. * inference: fast defaults and batched windows for long-document boundary extraction The 4096-word default window made attention cost dominate; a sweep on real sections picks 1024 words (overlap 32) as the default, with 4096 still allowed by explicit option. The long executor now runs windows in groups of four through one forward pass for entity and relation schemas, so peak memory follows the fixed group size rather than window count. A short document through the windowed path costs the same as the single-window path. GLINER25.md records the sweep and an fp16-encoder diagnostic that matched fp32 on every fixture but is not yet qualified. * lite: host-derived inference budgets for the embedded node and accepted extraction upgrades The Lite-owned embedded inference node was created with every memory budget at zero, so boundary-model extraction failed admission in-process while antfly inference run served the same inputs. Resolve the same host-detected policy the CLI uses and derive the host, backend, combined, KV, and scratch budgets with host-clamped defaults; expose all of them as overrides on the Lite open options (C ABI and Go) and report them in the inference status. Graph edges from a boundary model were silently dropped because the extraction response validator pinned the expected schema_version to the config default and rejected the model's legitimate upgrade to version 2; only an explicitly requested version is now enforced. Also fixes a real memory-safety bug where batch response items were mutated with the wrong allocator when relation payloads outgrew their parsed capacity. * enrichment: converge artifact-sourced dense lanes and fail a stuck drain by name The dense lane matched written embedding artifacts against a dense index's singular embedding_name only, never the plural embedding_names that a sources-based index populates, so its durable target counter never moved and the drain deferred to an artifact maintenance pass that only the structural index-creation path runs. Match embedding_names like the DB counter catalog already does. runUntilIdle now has a per-index no-progress guard (open option, default 60 s) that fails with RunUntilIdleNoProgress naming the index and its indexed/expected counters, surfaced through the C ABI as ANTFLY_STALLED with the diagnostic in the JSON result, instead of spinning. * inference: never batch GLiNER boundary requests and name every length rejection The nine refused sections were pairs of documents: the server advertised native batching for the boundary executor while the qualification's request_items contract is one, so the enrichment batcher grouped documents into a request the contract rejected. Advertise max_items=1 for the boundary kind. Length rejections now return a per-dimension error mapped to HTTP 413 EXTRACTION_LIMIT_EXCEEDED with the limit in the message; the long-document contract is widened to 182 KB / 28,275 words / 29 windows on synthetic probes built from real sections; a window group that runs out of memory retries at a halved group size down to one window. * examples/dogfood: two-stage chunk artifacts, GLiNER2.5 windowed extraction, measured run Use the server's chunk-artifact pattern (chunk enrichment plus artifact-sourced embeddings index) so chunks are a queryable artifact, select the token-aware chunker through chunker_json because a chunk enrichment's chunk_size is bytes, default the extractor to the qualified GLiNER2.5 base with windowed long documents, and record the measured in-process run and what remains. * examples/dogfood: GLiNER2.5 is the extractor, no fallback model * enrichment: treat deterministic extraction rejections as terminal and admit qualified documents Invocation-memory and boundary length rejections are pure functions of the request, so retrying them wasted five attempts per document; they are now terminal. The local-extraction allocator ceiling gets a 256 MiB floor so a document at the qualified 182 KB bound is never rejected by the response-envelope arithmetic. An executor that advertises no batch mode now falls back to sequential production instead of failing the batch path with an invalid-capabilities error. * Fix stale gliner_boundary_parity_test import missed by the merge extractors/gliner_boundary_qualification.zig was co-modified by both branch and origin/main and merged without a textual conflict, but it still imported architectures/gliner_boundary_parity_test.zig under its pre-#776 path. #776 moved that fixture to architectures/gliner/boundary_parity_test.zig; the stale import broke zig build capi with FileNotFound until fixed here. * Rename the recognizer model type to extractor and record the GLiNER2.5 fp16 result The extraction model type is now spelled extractor across the manifest, registry, session factory, model manager, capabilities and server; the legacy "recognizer" spelling is still accepted from previously written model_manifest.json sidecars. taskMatchesModelListing gained a guard so an unqualified boundary artifact cannot pass through the pluralized fallback after the rename. The fp16-encoder GLiNER2.5 bundle converts deterministically but one of 62 pinned confidence values misses the 5e-4 tolerance by about 18% on both native and Metal, so it stays unqualified; the pipeline tests now run the pinned fixtures through the real session path for both bundles with named tolerances. Direct in-process extraction measures the same rate as the HTTP server on a 40-section corpus sample, which points the remaining ingest gap at GPU contention with concurrent embedding rather than the call path. * Pipeline the asset and dense enrichment lanes across quanta The two lanes were dispatched and awaited within a single quantum, so a slow extraction quantum held back every dense quantum behind it. Each lane now keeps one quantum in flight and only drains its own previous quantum before dispatching the next. A dispatched quantum owns what it reads: queued generated requests are deep-copied off the request plan cache, which the earlier attempt freed underneath a running lane, and the chunk cache moves wholesale into the dense lane. Both the Lite runUntilIdle drive and the background catch-up pass share one scan loop and always drain both lanes after it, so cancellation and guard timeouts never abandon an in-flight future and a terminal lane error still surfaces. * Point the GLiNER2.5 pipeline tests at the moved boundary architecture files * Advertise one item per request for boundary-architecture GLiNER checkpoints resolvedExecutorKind only recognized the split encoder+head GGUF bundle as the concrete GLiNER executor, so a GLiNER2.5 checkpoint served from safetensors (a boundary-architecture manifest with no head file) fell through to the generic serial batch contract of 128 items. The antfly asset-producer batcher then grouped up to eight sections into one request, which the boundary executor's one-item LengthContract rejected with GlinerBoundaryRequestItemsLimitExceeded for the whole group; in the dogfood ingest that cost five sections per run. Both concrete GLiNER executors now share the kind, and its batch contract stays mode=none, max_items=1. * Floor the asset producer plan-resolution budget Resolving an invocation memory plan parses the producer configuration into a JSON value tree, whose allocations do not shrink with the source text, but the budget for that resolution was eight times the request's own bytes plus 4 KB. Six markdown sections of 150 to 560 bytes with the ordinary GLiNER extractor config therefore failed during planning with InferenceInvocationMemoryExceeded, a terminal disposition, before any provider ran. The resolution budget now has a fixed 1 MiB floor; it is a planning-only bound, the invocation itself is still bounded by the resolved plan. * Name the error behind ANTFLY_INTERNAL and the document behind a failed enrichment runUntilIdle logs the error it fails with, the C ABI logs any error it collapses to the generic internal code, isolated enrichment failures log the document key, and an asset producer invocation that exhausts its memory plan logs the limit and peak. Embedded callers only see the mapped code, so the process log is the only place these names can live. * Qualify GLiNER2.5 down to the corpus-minimum document examples/dogfood's smallest real section, zig/SCHEMA.md's Related Docs list as docsaf emits it, is the 20-byte string TODO.mdSERVERLESS.md. Both production rows had a 26-byte document floor taken from the shortest canonical fixture, so that one section was refused with GlinerBoundaryDocumentBytesLimitExceeded and, because an explicit drain reports terminal request failures, the whole ingest exited with EnrichmentWorkerFailed. The floors are lowered by measurement: both geometry tests gained that section and a one-character document (20 bytes / 5 words / 6 window words / 110 padded tokens and 1 / 1 / 2 / 103 with the dogfood schema), the rows now admit document_bytes >= 1, document_words >= 1 and window_words >= 2, with the long-document padded floor at 103, and two corpus-minimum provider tests run both documents through Node.extractDirect on native and Metal requiring a canonical, finite single-window response. GLINER25.md section 14 records this together with the serial batch contract and plan-resolution floor fixes. * dogfood: record the 2026-09-19 in-process run with no lost sections * Give the blocked-dense-lane test the slow poll budget The asset lane's first publish is a provider round trip plus a durable checkpoint on a background worker; on a loaded host it exceeded the default one-second poll budget and the test failed once in a full suite run while passing minutes earlier on the same tree. The assertion is unchanged. * Make the two enrichment lanes safe to run concurrently with the scanner Three defects in the cross-quantum lane pipeline, all reported in review. Replay checkpoints could skip unfinished extraction. The scanner's mid-scan flush of its synchronous window checkpointed both stream cursors at the current group even while earlier documents' asset requests were still queued for the lane or running in a parked quantum; a restart then skipped those documents. The scanner now checkpoints a stream only when it has nothing outstanding (no queued requests, no in-flight quantum); a lane with outstanding work checkpoints itself once that work is durable. Failure identity was shared across concurrent lanes. The runtime's active_failure_fingerprint drove retry budgets, attempt numbers, durable retry-progress credit and the supervisor's retry scope, and both lanes kept writing it while running concurrently with the scanner and with each other. Each lane quantum now carries a FailureScope (the request or provider batch being attempted and the first retryable error it deferred, with that error's identity); lane-side helpers take the identity from the scope and never touch the global, which the scanner thread keeps for its own synchronous work. A successful publish credits progress to the scope's request unless that request deferred the retry. When a lane ends with its deferred error, LaneOutcome carries the error and identity together and LanePipeline installs the identity on the scanner thread only for the outcome actually returned to the supervisor. Concurrent publication touched unprotected hash maps. Dense publication remembers artifacts and every lane isolates failures against published_generated_artifacts, isolated_failed_indexes and isolated_failed_sources; those sets are now only reachable through helpers that hold a dedicated spin lock. * Discover standalone enrichments when Lite rebuilds its enrichment runtime liteMergedIndexesJsonAlloc built the merged catalog blob fed to createManagedDbEnrichments from listIndexes alone, so an extractor or chunker registered through antfly_db_add_enrichment_json with no index nesting the same declaration was accepted into the catalog but never got an asset producer or chunk provider: with no other producer the managed runtime was not created at all and the document's pending work was never serviced. The merged blob now also carries every standalone catalog entry under a reserved $enrichment:<kind>:<name> key in the shape the producer scanners already recognize; the managed-embedder scanners and the query-time semantic resolver only act on top-level embeddings entries, so the extra keys are inert to them. Covered by a merged-JSON unit test, an end-to-end run-until-idle drain of a standalone chunk enrichment (both added to the curated capi test list), and a Go binding test that drains a standalone extractor against the fake inference server. * Select the returned lane error before installing its retry identity in dispatch dispatchAsset and dispatchDense each drained their lane's failed previous quantum and installed that quantum's retry identity immediately. When both previous quanta had failed, the dense lane's identity was installed last while the asset error was the one returned, so the supervisor charged the wrong request's retry budget (reproduced: asset error returned with identity 42 instead of 41). Dispatch now returns a LaneOutcome without installing anything, and dispatchDeferredGeneratedWork selects the outcome the supervisor will see (asset before dense, as drainAll already did) before installing only that outcome's identity. * Bound the resolution-allocation test past the planning floor The test exercised a stub that over-allocated 8 KiB while resolving an invocation plan for a tiny request, which the new 1 MiB planning floor now admits. The bound it guards is unchanged; the stub over-allocates just past the floor instead. * Update the chunked dense unit tests for the explicit failure scope Two tests only compiled by the full storage test binary still called recordUniqueChunkedDenseRequestErrors and flushChunkedDenseItems with the pre-FailureScope signatures; the filtered local runs never analyzed them, CI's x86_64 job did. * Let an explicit Lite vacuum wait its turn instead of failing busy The online vacuum made eight opportunistic, non-blocking attempts to take the writer slot and gave up with FileBusy (ANTFLY_BUSY at the C ABI) when a writer was active at every one of them. Every Lite database now carries the default full-text index, whose maintenance keeps a background writer busy after a burst of writes, so on a slow CI host the capi open/export/import/vacuum test lost that race consistently on Linux. The final round now takes its fair, ticketed turn through reserveWriterSlotYielding, drains the residual captured changes while holding the slot (no mutation can start behind it), and publishes; earlier rounds stay opportunistic so the large copy never blocks foreground writers. Verified on Linux arm64 in a container. The blocked-dense-lane test also observed the gated producer's success before the asset lane had flushed and checkpointed, and asserted the cursor immediately; it now waits for the checkpoint itself. * Qualify the GLiNER2.5 fp16 encoder bundle The fp16-encoder bundle now has production qualification rows for native and Metal, single-window and long-document, and can be selected. The one single-window value that missed the 5e-4 tolerance is fp16 weight-rounding noise, not a defect: biases, layer norms, position tables and the whole extraction head stay fp32 in the artifact, both the native GEMM and the Metal kernel accumulate in fp32, and two independent kernels land on the identical value. Across the 62 pinned confidence values the max delta is 5.9e-4 with every decision identical, so 7.5e-4 is the reviewed single-window bound. Long documents are qualified with the same evidence the fp32 row used: geometry identical at both window sizes, the canonical-shape tests now parametrized over the bundle directory and run against the fp16 bundle on both backends, plus fp32-vs-fp16 parity through the long executor on the real VOPR.md, PDF.md-maximum and corpus-minimum documents: 445 matched decisions per backend, zero mismatches, max delta 1.7e-3 under a reviewed 2.5e-3 bound. The larger residual is the cross-window tie-break choosing different windows' near-tied estimates for the same span. Selection: exportBundle writes model_manifest.json at publish, gated on the reviewed qualification, and boundaryIdentityIsQualified recognizes converted bundles by their receipt precision; the fp16 bundle at ~/.antfly/inference/models/fastino/gliner2.5-base-v1-fp16 is selectable by name. fp32 stays the default: on the 40-section corpus through the in-process provider entry fp16 is a wash on native (0.47 vs 0.49 sections/s) and 2.9x slower on Metal (0.94 vs 2.72), because the managed session's resident-weight fast path is fp32-only. * Print the stable snapshot report when the tail assertion fails The C API vacuum test appends four junk bytes after the online vacuum and expects both stable-snapshot copies to report tail_bytes 4. The x86_64 CI runner fails the second copy's assertion while the same test passes on macOS and on Linux arm64; the assertion only reports that the substring was missing, so print the report to see what the runner produced. * Name the reason an online Lite vacuum gives up busy The online vacuum can return FileBusy from four places: another vacuum already holds the change capture, the capture overflowed before a catch-up, it overflowed while the writer slot was held, or every round found new changes after taking the slot. The C ABI collapses all of them to ANTFLY_BUSY, and the x86_64 CI runner reports that code where macOS and Linux arm64 complete the vacuum, so log which site fired and the capture size at that point. * Quiesce default-index maintenance before the C API vacuum and tail probes The test writes documents, then vacuums and appends four junk bytes to probe stable-snapshot tail accounting. Every fresh Lite database now provisions the default full-text index, so that maintenance can still be writing when the probes run on a slow host: the vacuum's writer-slot attempts collide with it, and a later checkpoint can extend the file past the junk so the tail is no longer four bytes. Drain to idle first; vacuum under concurrent maintenance has its own coverage in the Lite native suite. * Fence the final vacuum catch-up with the store mutex and bound the writer-slot wait Two defects in the online vacuum's final round, both surfaced by the x86_64 CI runner. The final round held the writer slot while catching the image up and publishing, but the slot only excludes an open transaction. Every mutation, transaction commits and the disk index's out-of-band catalog publications alike, is applied through the group-commit queue under the store mutex, and the residual copy ran outside that mutex, so on the runner one index publication landed after every slot-held catch-up and the vacuum gave up busy on all eight rounds. The final round now holds the store mutex across the catch-up and the publish, so the residual is exactly what was captured and nothing can be applied behind it. Waiting on the writer slot without bound deadlocked the caller that holds a writer and expects FileBusy (the 'reserves one writer' test), which hung the Lite native partition until CI's idle watchdog fired. Rounds that find the slot busy now back off briefly, up to 64 ms, so a background transaction mid-commit can finish while a held writer still ends in FileBusy. A new test keeps group-commit mutations landing for the whole vacuum and requires it to publish with every landed key intact. * Give the embedded modules the schema OpenAPI module schema/table_schema_impl.zig now takes the public table storage mode and the relational wire types from antfly_schema_openapi (#784), but the embedded module graph (embedded_support and the embedded db/api roots) and its wasm twin were never handed that module, so embedded-test no longer compiled. Add it to configureModule next to the other generated OpenAPI modules and to both callers' dependency tuples. CI does not run embedded-test, which is why main stayed green. The wasm bundle itself still does not build, for reasons that predate this change: 64-bit atomics in hbc_vector_directory, index_manager, core, search_exec, relational_columns, source_pin and hbc_adapter, and std.Io.Dir/std.Thread reached from the same graph, none of which the freestanding target implements. * Dump the full index observation when a restart hides searchable vectors pytest abbreviates a dict used as an assertion message, so the only CI failure of the progressive restart test left the freshness, publication and backfill fields unreadable. Fail through pytest.fail with the complete index read and server logs instead. * Format the release promotion condition test scripts/ci/check.sh format python fails on the file as merged from main (#834), which blocks the PR gate's sdks-ci job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the relational storage foundation, extracting relational indexes and their public APIs from the combined SQL work.
/indexesresource, including partial/covering indexes, durable build/repair/retirement lifecycle, and shared maintenance tooling.Latest review fixes
Validation
Latest local validation:
git diff --checkpassed.The two previously reported storage-suite failures are fixed: the owner fixture now includes
restore_bootstrap_json, and the bulk-query test completes the replacement text index's initial build before checking search visibility. Bulk writes were stored correctly; index initialization was incomplete in the test.After merging
origin/mainatb46f42c78:make fmtpassed; Go, TypeScript, and Python clients were regenerated from the merged specs.This validation does not claim the entire repository test suite or remote CI has passed.
Design and review guides
docs/relational-indexes-extraction.mddocs/relational-restore-architecture.mddocs/native-backup-seals.mdThis is a large cross-layer change; the public specs/generated clients, storage/index lifecycle, distributed integrity, and restore/online-merge paths should be reviewed as separate areas.