Repository navigation
feat!: v0.2.0 — observability, declared breaks and the codec fingerprint - #60
Merged
Merged
Conversation
BREAKING CHANGE: `BackendKind`, `Compression`, `SessionMode`,
`MemoryClass`, `MemoryTier`, `Tier`, `ModelSpec` and the TUI's `Panel`,
`KeyInput` and `Transition` are `#[non_exhaustive]`. A consumer matching
on any of them needs a `_` arm; in exchange, the next backend or codec
does not break them.
The interesting half is the enums that are *not* marked. `Predicate`,
`Comparison`, `Literal`, `AggregateFunction`, `DistanceMetric` and the
codec's `GpuNode` stay exhaustive on purpose: they are the plan
vocabulary, every kernel must handle every variant, and the CPU reference
is what the GPU paths are checked against. A `_` arm there is not forward
compatibility, it is a wrong answer that compiles. Marking them was the
first thing I tried and the compiler caught it — `evaluate_predicate` in
the CPU kernels demanded a catch-all, which is exactly the arm that must
not exist.
Two cross-crate matches needed a decision rather than a `_ => {}`. The
dashboard draws an unrecognised backend as `[?]` in yellow, because
silently drawing it as `[CPU]` would misreport where the work ran. The
detector returns a typed error naming the kind, never a quiet fallback to
the reference path.
RELEASING.md gains "What a version number means here": that a DataFusion,
Arrow or Ballista major bump is a minor bump before 1.0 and a major after
— breaking whether or not our own code changes, because ADR-0009 makes
DataFusion our public vocabulary and cargo-semver-checks cannot see a
break in a re-exported type. And the enum rule above, written down, so
the next person knows which list a new enum joins.
`datagen` moves behind a default-on feature so a direct consumer of
oxidelake-storage can leave the demo generator out. It does not disappear
from `oxidelake-api`'s tree: that arrives through oxidelake-runtime, whose
`oxide gen-data` needs it, and features are per crate rather than per
target. The issue's dependency-tree criterion turned out to rest on a
premise that does not hold — datagen imports only arrow, oxidelake-core
and two crate-internal modules, so there were never any generator
dependencies to remove.
`cargo semver-checks -p oxidelake-core` reports the enum change as a major
break, which is the declaration this release exists to make.
Closes #41
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
…ecode BREAKING CHANGE: the codec payload header grew four bytes and `VERSION` is 2. A 0.1.x executor and a 0.2.0 scheduler refuse each other's plans, which is the point. `postcard` is not self-describing. A field added to `GpuNode`, or two fields of the same width reordered, decodes without complaint into the wrong parameters — so a rolling upgrade did not fail, it computed a confident wrong answer from a plausible-looking plan. The version byte only helps if somebody remembers to bump it. The fingerprint is derived instead: FNV-1a over the crate version, the version byte, and `oxidelake_compute::FEATURE_MASK`. It therefore moves whether or not anybody remembered. The feature mask is the half that catches the quieter failure — a planner built with `predict` emits plans referencing a UDF an executor without it cannot resolve, which today fails at execution time and reads like a query bug rather than a deployment one. `check_fingerprint` is extracted rather than inlined so the refusal is testable without a `TaskContext`; a failure nobody can set up on purpose is a failure nobody tests. Its message names both fingerprints and both things that could differ, because the person reading it is looking at a cluster that half works. The insta snapshot is the review-time half: `tests/codec_wire.rs` pins the exact bytes of every `GpuNode` variant, so changing the wire form moves a committed file and has to be explained in the same pull request. I decoded all four encodings by hand before accepting it — variant tags, zigzag varints, little-endian f32s, string lengths — rather than accepting bytes because the test wrote them. Not done here: refusing at *registration* rather than at decode. That needs a Ballista registration hook, and decode-time refusal already stops the wrong answer; registration-time would only make the error arrive earlier and read better. Left on #42. Refs #42 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Closes #49 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
…uming it Closes #47 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
…he CPU Closes #32 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
…ics endpoint Closes #33 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
… is library-only Closes #25 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.2.0 — the observability release
A GPU deployment that silently ran everything on the CPU produced identical
rows, identical
EXPLAINtags and nothing abovedebug. This release makesthat visible, and declares the breaking changes 0.2.0 exists to carry.
It does not finish the production-readiness milestone. Ten Phase 8 lines
remain open; #28, #45 and #46 need a CUDA device to verify on, and #29's
streaming aggregate is what #25's real wiring waits for.
docs/roadmap.mdand the CHANGELOG say so in those words.
What lands
fallback_batchesonOperatorStats/OperatorSnapshot, a colour-codedcpu fallback N / Mline in the TUI Inspector, one
warn!per operator on first fallback withthe reason, and
placement notesprinted underoxide explainnamingevery node the rule left on the CPU and why.
oxide.operatorspan per operator and partition (operator,partition,target,backend);OxideSession::collectlogs oneINFOline perquery;
--metrics-porton the worker and scheduler behind the newmetricsfeature serves Prometheus text. The plan codec attaches aprocess-wide
TelemetryHubto every node it decodes — without it aworker's
/metricswould describe no work at all.--batch-size,--output table|json|csv,oxide-worker --backend, and one README table listing every flag andenvironment variable.
predictreads its activation from the model file (predict: require an explicit activation in the model header instead of assuming ReLU #47). Fixing thisfound that the loader applied no activation at all while the module
docs, the README and ADR-0015 all said "ReLU between them" — every
non-linear model was scored as a linear one, and the only test of the
numbers compared them against the same bare stack.
non_exhaustiveenums with thesemver policy written down (Mark public enums non_exhaustive and write the DataFusion-coupling semver policy #41), with the plan vocabulary left
exhaustive on purpose.
from the backend's
MemoryInfo, and the panel and docs say the spillmanager is a library no query path calls.
Verification
Full gate locally:
cargo fmt --check,clippy --workspace --all-targets --all-features -D warnings,cargo test --workspaceand--all-features(37 result lines, 0 failures),RUSTDOCFLAGS="-D warnings" cargo doc,cargo deny check, crate metadata, skill version, DCO,no-AI-attribution.
cargo semver-checks --workspaceran locally before tagging this time —skipping it is what failed the v0.1.4 tag. It passes for all nine crates at
0.1.4 → 0.2.0, and with
--release-type minorforced it lists exactly thebreaks the CHANGELOG declares:
ModelSpec's variant, thenon_exhaustiveenums in core/memory/runtime/storage/tui/compute, and the two new public
fields on
OperatorSnapshotandTelemetrySnapshot.End-to-end, on a real spawned cluster: a worker's
/metricswas empty beforea query and carried
oxide_operator_{rows_in,batches,fallback_batches}_totalafter it, with the
oxide.operatorspan wrapping the fallback warning.Closes #25
Closes #32
Closes #33
Closes #41
Closes #42
Closes #47
Closes #49