From 7486f9249ab38c5a82da5774799d23ebe1ce996e Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 13 Nov 2026 13:53:03 +1100 Subject: [PATCH 1/6] docs(readme): explain the supervisor, build and config The README opens with the supervisor's role around Vector, the build, the two transports and the config cascade in about 530 words. Endpoints, the metrics merge, reload and the chart stay in docs/internals.md. The README links there. The working guide moved to CONTRIBUTING.md. The crate description matches the GitHub description. --- CONTRIBUTING.md | 112 +++++++++++++++ Cargo.toml | 2 +- README.md | 375 ++++-------------------------------------------- 3 files changed, 142 insertions(+), 347 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b8f57cd..a8b797b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -161,3 +161,115 @@ This happens automatically - no manual intervention required. ## Questions If you have questions about contributing, please open an issue or contact us. + +## Context + +### What this is + +A Rust supervisor that runs Vector.dev as a child process so a Vector pipeline +behaves like every other DFE app -- one big-dial config, one ops port, an OTLP +push, and a chart assembled from its deployment contract. It is NOT +the transform engine and it does not build Vector: the image downloads the +upstream release binary at the version pinned by `VECTOR_VERSION` in +`src/deployment.rs`, and cargo compiles only the supervisor. It is also not a +singleton and not part of the default deploy -- `dfe-infra/apps.yaml` declares +it `multiplicity: per_config` with `scale_deployed: true` and an empty +`default_in`, so one deployment exists per source config and only when a +deployment asks for it. + +### Where things live + +| Path | What it holds | +|---|---| +| `src/main.rs` | CLI entry. Fills scalo's `CommonArgs` from the loaded config before `run_app`, which is what makes `metrics.address` and `logging.*` take effect | +| `src/config/` | `loader` -> `validate` -> `generate` -> `wiring` -> `assembler`, plus `reload`, `transforms` and `kafka_defaults` | +| `src/vector/` | `process` spawn and backoff, `lifecycle` states, and `binary` -- version selection kept pure and NOT on the run path, which its own header explains | +| `src/bridge.rs` | The direct-transport translation between scalo `Transport/Push` and Vector `PushEvents`. Does nothing on `bus` | +| `src/health.rs`, `src/metrics.rs`, `src/metrics/scrape.rs` | Readiness publishing, and the scrape that merges `vector_*` into scalo's registry | +| `src/deployment.rs`, `src/vector-layer.dockerfile` | `contract()`, the source the Dockerfile and the released chart are generated from, and the two-binary image override spliced onto scalo's generated Dockerfile | +| `templates/bus.yaml`, `templates/direct.yaml` | Whole runnable Vector topologies, one per transport, commented per field | +| `pipelines/filebeat/` | The shipped filebeat pipeline | +| `docs/` | `architecture.md` for the shape and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` | +| `tests/` | `integration`, `e2e`, `smoke`, and `TESTING.md` for how the broker and Vector binary are resolved | +| `scripts/fetch-vector.sh`, `scripts/pgo-workload.sh` | Downloads the pinned Vector for tests, and drives the PGO/BOLT workload | + +### Commands that prove a change + +```bash +make check # hyperi-ci check -- quality + test +cargo nextest run --lib # unit only, no infrastructure +cargo nextest run --test integration # config assembly, wiring, lifecycle, metrics +cargo nextest run --test smoke # CLI surface +cargo nextest run --test e2e # needs Docker or a live broker +cargo nextest run --run-ignored all # everything, opt-in cases included +``` + +What green does NOT mean: + +- **The default run skips the Vector validator.** Seven `#[ignore]` cases need a + real Vector binary -- six in `tests/integration/vector_validate.rs` and one in + `tests/e2e/kafka.rs`. Until you pass `--run-ignored all`, nothing has run + Vector's own validator over an assembled config. +- **Locally, a missing broker or Vector binary SKIPS rather than fails.** In CI + that same absence is an assertion failure instead, so CI is stricter than a + green laptop run. +- **`e2e::filebeat_kafka` runs by default but skips without a sibling repo.** The + corpus and elastic's golden events live in dfe-transform-vrl, so it needs that + repo checked out beside this one or `DFE_TRANSFORM_VRL_DIR` pointing at it. +- **A docs-only push runs no CI at all.** `paths-ignore` for `docs/**` and + `**.md` is on the `push` trigger but not on `pull_request`, so the PR is the + only place a docs change is checked. A silent push is not a pass. + +### What tends to bite + +| Don't | Do | Why | +|---|---|---| +| Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output and the release assembles the chart from the contract, so a hand edit is reverted or never ships | +| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes | +| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()` and scale on CPU plus scaling pressure | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The library's trigger also reads `config.kafka.*`, which this app does not have, and refuses to render | +| Set `vector.version` in a deployment's config and forget it | Leave it unset, so the default reads `VECTOR_VERSION` | A committed chart once said 0.48.0 while the image baked 0.57.0, and only `version_check: warn` kept pods starting. Under `strict` that pairing refuses to start at all | +| Mount a whole `templates/*.yaml` as a transform file | Copy only its `transforms:` block | The supervisor generates `sources:` and `sinks:` from the big dials and refuses a transform file declaring them | +| Change `password` to a `SensitiveString` to look safer | Leave it a `String` | `SensitiveString` serialises as `***REDACTED***` and the figment serialize-merge-deserialize round trip in `apply_figment_env()` destroys the value. Masking happens in logs and `Debug`, and Vector reads the credential from files through its `directory` secret backend, so it never lands in the Vector config | +| Put a `${VAR}` placeholder in a credential or a transform | Mount the secret and set `sasl.secret_dir`, or give the credential through `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`, and read the environment in VRL with `get_env_var` | Vector 0.57+ expands no `${VAR}` without `--dangerously-allow-env-var-interpolation`, which the supervisor never passes, so the placeholder reaches the broker as the password | +| Assume a config key you added is wired because it parses and validates | Prove it changes behaviour, and add it to the standing check that every committed config file loads | `metrics.address` and `logging.level` never left the struct. A config saying `metrics.address: 0.0.0.0:19099` still listened on 9090, so a deployment moving the metrics port lost every probe | +| Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on. Consumer lag sat at 72 across a quiet 90-second window while the app's own metrics said it had processed those same 72 events | +| Read `/readyz` as "carrying traffic" | Read it as "the child was alive 500ms after spawn" | It once answered an unconditional 200 because nothing published a readiness signal, so Vector could crash and restart hundreds of times with the pod still Ready and zero restarts | +| Swap the reload poller for inotify | Keep polling | S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events, so inotify works on every laptop and silently stops reloading in production | +| Add or rename a credential field without a schema marker | Schema it as `scalo::SensitiveString` with `#[schemars(with = ...)]`, as `SaslConfig::password` is | That puts `x-scalo-secret` and `writeOnly` on the field in the emitted config-schema, which is what tells a console to mask it. Without it, only dfe-engine's leaf-name fallback (`password`, `secret`, `token`, `api_key`, `private_key`, `passphrase`) redacts it, so a name outside that set returns an operator's Kafka password in the clear | + +### Where this sits + +Generated from `dfe-infra/suite.yaml` via +`python3 ../dfe-infra/scripts/dfe-stack suite --consumer dfe-transform-vector` +and `--producer dfe-transform-vector`. + +Inbound -- what this repo depends on: + +- **scalo-rs**, `cargo-dep`. `Cargo.toml` declares the `scalo` crate by range, + with a second range covering the dev dependency. A scalo release reaches this + repo, so widen the range if it does not admit the new version, then rebuild. +- **scalo-rs**, `generated-file`, lockstep. The committed `Dockerfile` is written + by `scalo::deployment::generate_dockerfile()` -- its own header names the + generator and the schema version -- and this repo splices the Vector layer onto + it. When the generator or its schema moves, regenerate with + `dfe-transform-vector emit-dockerfile > Dockerfile` and commit the diff. The + released chart is assembled on the scalo-service library chart at + `release.helm.library`, which moves with the scalo version in `Cargo.toml`. + +Outbound -- what depends on this repo: + +- **dfe-infra**, `image-pin`, lockstep. `dfe-infra/versions.yaml` pins this + repo's container image as a tag plus the digest that makes the tag + immutable, and the thin chart released here, which Argo deploys. Bump the + tag and re-resolve the digest. + +Two more relationships that are real but are NOT declared edges in the suite +graph, so no gate enforces them: + +- **dfe-infra `apps.yaml`** is the SSoT for this app's shape -- multiplicity, + scaling, transports, endpoints and which compiler derives its routing. + dfe-engine reads that manifest, so adding or changing the app is a manifest + edit, never an engine release. +- **dfe-transform-vrl** holds the filebeat corpus, the bundled pipeline and the + documented divergences that `e2e::filebeat_kafka` grades against. The + dependency is a test-time repo checkout only, and the case skips without it. diff --git a/Cargo.toml b/Cargo.toml index 8db156e..3ed9779 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,7 +11,7 @@ name = "dfe-transform-vector" version = "1.0.13" edition = "2024" authors = ["HyperI "] -description = "Kafka-to-Kafka transform pipelines, running Vector.dev as a managed subprocess. Runs standalone or as part of the HyperI Data Fusion Engine." +description = "Runs a Vector pipeline as a supervised transform stage in a DFE deployment." license = "BUSL-1.1" repository = "https://github.com/hyperi-io/dfe-transform-vector" rust-version = "1.95" diff --git a/README.md b/README.md index 7af87e3..62068ba 100644 --- a/README.md +++ b/README.md @@ -1,375 +1,58 @@ # dfe-transform-vector -[![Build Status](https://github.com/hyperi-io/dfe-transform-vector/actions/workflows/ci.yml/badge.svg)](https://github.com/hyperi-io/dfe-transform-vector/actions) -[![License](https://img.shields.io/badge/license-BUSL--1.1-blue)](https://github.com/hyperi-io/dfe-transform-vector/blob/main/LICENSE) +Runs a Vector pipeline as a supervised transform stage in a DFE deployment. -Runs a Vector pipeline as a transform stage in a DFE deployment. Vector does the transform, and this wrapper supplies the rest a stage needs: config, credentials, health, metrics and restarts. +[![Build Status](https://github.com/hyperi-io/dfe-transform-vector/actions/workflows/ci.yml/badge.svg)](https://github.com/hyperi-io/dfe-transform-vector/actions) +[![License](https://img.shields.io/badge/license-BUSL--1.1-blue)](LICENSE) -## What it gives you +## What it does -- Vector's own transforms beyond `remap`, where dfe-transform-vrl runs VRL only. -- One small YAML (pipeline name, brokers, topics, SASL and TLS) that generates Vector's source, sink and observability config, so you write only the `transforms:` and `enrichment_tables:` blocks. -- Kafka in and out, or a direct gRPC push with no broker, with the same transform on both. -- A crashed Vector restarted with backoff from 1 s to 60 s, and readiness that follows it. -- A transform change validated and sent to Vector with a SIGHUP, then read back from Vector's own counters as applied, refused (the running config is put back) or unconfirmed, each counted on its own label. -- Vector's `vector_*` metrics merged into the wrapper's, so one endpoint and one OTLP push carry both. -- Built on [scalo](https://github.com/hyperi-io/scalo-rs), so the wrapper's config, logs, health probes and deployment contract work the way they do in every DFE Rust service. +- Vector does the transform, and the supervisor supplies the rest a stage needs: config, credentials, health, metrics and restarts. Reach for it when you need one of Vector's own transforms beyond `remap`. For VRL alone, [dfe-transform-vrl](https://github.com/hyperi-io/dfe-transform-vrl) is smaller and in-process. +- One small YAML (pipeline name, brokers, topics, SASL and TLS) generates Vector's source, sink and observability config, so you write only the `transforms:` and `enrichment_tables:` blocks. +- Kafka in and out (`bus`), or a direct gRPC push with no broker (`direct`). The transform you author is the same on both. +- A crashed Vector is restarted with backoff from 1 s to 60 s, and readiness follows it. +- A transform change is validated, sent to Vector with a SIGHUP, and read back from Vector's own counters as applied, refused (the running config is put back) or unconfirmed. +- Vector's `vector_*` metrics are merged into the supervisor's, so one endpoint and one OTLP push carry both. -## Architecture +## Quick start -```mermaid -flowchart TB - SRC[("Previous stage
Kafka topic or gRPC push")] - SINK[("Next stage
Kafka topic or gRPC push")] - subgraph W["dfe-transform-vector (Rust, PID 1)"] - CE["Config engine
big-dial YAML -> Vector config dir"] - PM["Process manager
spawn / signal / crash recovery"] - BR["Bridge
direct transport only"] - VEC["Vector.dev
subprocess (--config-dir)"] - OPS["Ops :9090
/metrics /livez /readyz
wrapper + merged Vector"] - EXP["Vector exporter :9598
loopback only"] - VEC --> EXP --> OPS - CE --> VEC - PM --> VEC - end - SRC --> BR --> VEC --> BR --> SINK - SRC -.bus.-> VEC -.bus.-> SINK +```bash +cargo build --release +./target/release/dfe-transform-vector run --config config.example.yaml ``` -## Two transports, one template each - -One deployment runs one transport, and the transform you author is identical on -both. What changes is who hands the record over. - -`source.transport` and `sink.transport` also take `kafka` for `bus` and `grpc` for `direct`, the names the other DFE stages use. +[config.example.yaml](config.example.yaml) is the annotated reference. [templates/bus.yaml](templates/bus.yaml) and [templates/direct.yaml](templates/direct.yaml) are the whole Vector topology for each transport, commented per field and runnable as they stand: copy one, replace the `dfe_transform` step, and you have a working pipeline. In a deployment the supervisor generates `sources:` and `sinks:`, and refuses any other top-level key in a transform file. | | `bus` | `direct` | |---|---|---| -| In | Kafka `_land` | a scalo Push listener on `source.listen` | +| In | Kafka `_land` | a scalo Push listener on `source.listen` (port 6000) | | Out | Kafka `_load` | a scalo Push client to `sink.endpoint` | | Needs a broker | yes | no | -| Vector sees | `kafka` source and sink | `vector` source and sink on loopback | - -`templates/bus.yaml` and `templates/direct.yaml` are the whole Vector topology -for each, commented per field and runnable as they stand -- copy one, replace -the `dfe_transform` step, and you have a working pipeline. In a DFE deployment -the supervisor generates the `sources:` and `sinks:` blocks from the big dials -and you author only the `transforms:` block, plus `enrichment_tables:` loading -from `transforms.data_dir`. The supervisor refuses any other top-level key, so a -whole template mounted as a transform file does not start. - -On `direct` Vector does not speak scalo's `Transport/Push`, so the supervisor -translates. It accepts records on `source.listen`, hands them to Vector over -`bridge.to_vector`, takes them back on `bridge.from_vector`, and pushes them to -`sink.endpoint`. Both inner legs are loopback -- they exist inside one pod. -A push is answered only once the next hop has its records. A hop that cannot -move on is retried until the push's hold runs out, 18 s at `source.listen` and -16.5 s at `bridge.from_vector`, each send to the next stage giving up after -15 s, and then the push is answered `Unavailable` so its sender retries. A record -no retry would get through is dropped and counted in -`pipeline_dead_letters_dropped_total`: one over the next stage's message-size -ceiling, or one Vector refused for good because a sink it feeds rejected it -(`reason="rejected"`). Under memory pressure only -`source.listen` sheds pushes; the leg back from Vector keeps draining. - -## Behaviour - -- Transform files are loaded, validated and wired between `dfe_source` and - `dfe_sink`. -- The transform watcher polls. A source or sink change needs a pod restart. -- The Vector version is checked against the config (strict/warn/disabled). -- librdkafka defaults are baked in under a 4-layer cascade. -- A thin Helm chart is assembled at release from the deployment contract on the - scalo-service library chart. - -## Quick Start - -```bash -# Build -cargo build --release - -# Run with config file -./target/release/dfe-transform-vector run --config config.example.yaml - -# Generate deployment artefacts -./target/release/dfe-transform-vector emit-dockerfile -./target/release/dfe-transform-vector emit-compose -``` - -### Helm chart -No chart is committed here. At release, hyperi-ci runs the binary's `generate-artefacts` for the deployment contract and assembles a thin chart from it on the scalo-service library chart, at the version `release.helm.library` names in `.hyperi-ci.yaml`. Keep that version at the scalo version in `Cargo.toml`: a library renders only the contract version its scalo release writes. - -To see the chart a release would ship, build the binary and run `hyperi-ci chart assemble --binary target/debug/dfe-transform-vector --image ghcr.io/hyperi-io/dfe-transform-vector:@sha256: --version `. It prints the chart directory it wrote. - -The ScaledObject scales on CPU alone, because `contract()` sets `KafkaLagTrigger::disabled()`. Consumer-group lag rises when a downstream stage breaks, and more replicas cannot fix that. A deployment adds a trigger with the library's `keda.extraTriggers` value. - -scalo's runtime serves the scaling pressure at `/scaling/pressure` and writes it into the `scaling_pressure` gauge. The supervisor feeds it no components and no memory reading, so it reads 0. +One port, 9090, carries the ops surface: `/livez`, `/readyz` (200 only while Vector is up and its sink is delivering) and `/metrics`. Vector's own exporter on 9598 is loopback only. ## Configuration -See [config.example.yaml](https://github.com/hyperi-io/dfe-transform-vector/blob/main/config.example.yaml) -for a full annotated configuration. - -Key environment variable overrides (prefix `DFE_TRANSFORM_VECTOR_`, and the old `DFE_TRANSFORM_` names are no longer read): - -| Variable | Description | -|----------|-------------| -| `DFE_TRANSFORM_VECTOR_SOURCE_TRANSPORT` | `bus` (or `kafka`) or `direct` (or `grpc`) | -| `DFE_TRANSFORM_VECTOR_SOURCE_LISTEN` | Push listener bind address (direct) | -| `DFE_TRANSFORM_VECTOR_SOURCE__BROKERS` | Kafka source broker addresses | -| `DFE_TRANSFORM_VECTOR_SOURCE__TOPICS` | Source topic list | -| `DFE_TRANSFORM_VECTOR_SOURCE__GROUP_ID` | Consumer group ID | -| `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_{USERNAME,PASSWORD}` | Kafka SASL credentials per endpoint; the chart mounts the Kafka Secret's `username` and `password` into all four | -| `DFE_TRANSFORM_VECTOR_KAFKA_SECURITY_PROTOCOL` | Sets `source.tls.enabled` and `sink.tls.enabled` true when the value contains `SSL`; never sets them false | -| `DFE_TRANSFORM_VECTOR_SINK_TRANSPORT` | `bus` (or `kafka`) or `direct` (or `grpc`) | -| `DFE_TRANSFORM_VECTOR_SINK_ENDPOINT` | Next stage's Push listener (direct) | -| `DFE_TRANSFORM_VECTOR_SINK__BROKERS` | Kafka sink broker addresses | -| `DFE_TRANSFORM_VECTOR_SINK__TOPIC` | Sink output topic, and the routing key on direct | -| `DFE_TRANSFORM_VECTOR_BRIDGE_TO_VECTOR` | Where Vector accepts records (direct, loopback) | -| `DFE_TRANSFORM_VECTOR_BRIDGE_FROM_VECTOR` | Where the supervisor accepts them back (direct, loopback) | -| `DFE_TRANSFORM_VECTOR_PIPELINE__NAME` | Pipeline name | -| `DFE_TRANSFORM_VECTOR_TRANSFORMS__DIR` | Transform YAML directory | -| `DFE_TRANSFORM_VECTOR_LOGGING__LEVEL` | Log level (trace/debug/info/warn/error) | - -### Hot-reload - -**Hot-reloaded (takes effect via SIGHUP to Vector):** - -- Transform YAML file contents (modified/added/removed in watched directory) -- `transforms.dir` / `transforms.files` path changes, picked up from an edit to the main config file - -The watcher polls the main config file as well as the transform files, so a change listed below is logged as needing a restart when the file changes. - -**Requires pod restart:** - -- `source.*` -- the consumer or the Push listener is established at startup -- `sink.*` -- the producer or the Push client is established at startup -- `bridge.*` -- the two supervisor-to-Vector legs bind at startup -- `pipeline.name` -- consumer group_id and metrics labels set at startup -- `vector.*` -- binary path, data_dir, API address set at spawn -- `metrics.*` -- the ops HTTP server is bound at startup -- `logging.*` -- tracing subscriber configured at startup - -A transform change is re-assembled and validated first, including that every enrichment table file it names can be read: `vector validate --no-environment` never opens those files. Vector then applies the change, or refuses it and keeps the config it was running, and the SIGHUP succeeds either way. So the supervisor reads the outcome from Vector's `vector_reloaded_total` and `vector_component_errors_total{error_code="reload"}`, counts it in `transform_vector_config_reloads_total` as `success`, `error` or `unconfirmed`, and on a refusal puts the assembled config back to what Vector runs. A failed reload is tried again on the same files one poll interval later, the wait doubling to five minutes, until it succeeds or the files change. - -## Endpoints - -One port carries the whole ops surface, so there is a single answer to "is this -pod ready". - -| Endpoint | Port | Purpose | -|----------|------|---------| -| `GET /livez` | 9090 | K8s liveness and startup probes (the supervisor is up) | -| `GET /readyz` | 9090 | K8s readiness probe (200 only while the Vector subprocess is up and its sink is delivering -- see below) | -| `GET /metrics` | 9090 | Prometheus scrape (wrapper + merged Vector metrics) | -| `GET /metrics` | 9598 | Vector's own exporter, loopback only, for debugging | -| `Transport/Push` | 6000 | Records in, on the direct transport only | - -## Metrics - -9090 carries everything. The wrapper GETs Vector's `prometheus_exporter` on -`metrics.vector_metrics_address` (default `127.0.0.1:9598`) at scalo's metrics -interval, parses the exposition, and registers every sample on the same -registry scalo serves and pushes over OTLP. `vector_*` names and labels are -preserved, with the platform namespace and labels scalo applies to every other -metric. - -9598 is a loopback debug surface, not a scrape target. Nothing outside the pod -needs it: no second Prometheus target, no PodMonitor selector, no published -container port. Change the port with -`DFE_TRANSFORM_VECTOR_METRICS__VECTOR_METRICS_ADDRESS` (or `metrics.vector_metrics_address` -in the config file) and both the Vector sink and the wrapper's scrape follow it. - -The merge also feeds the app's own throughput counters, which otherwise read 0 -because Vector, not the wrapper, owns the Kafka client: +Settings load through a cascade: CLI, then `DFE_TRANSFORM_VECTOR_*` environment variables, then the config file, then defaults. The transport dials take a single underscore (`DFE_TRANSFORM_VECTOR_SOURCE_TRANSPORT`, `_SOURCE_LISTEN`, `_SINK_ENDPOINT`, the SASL pairs), and a key inside a section takes two (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS`, `_SINK__TOPIC`, `_TRANSFORMS__DIR`). [docs/config-schema.yaml](docs/config-schema.yaml) lists every key with its type and default. -| Wrapper metric | Vector source | -|----------------|---------------| -| `records_received_total` | `vector_component_received_events_total{component_id="dfe_source"}` | -| `records_processed_total`, `records_delivered_total` | `vector_component_sent_events_total{component_id="dfe_sink"}` | -| `records_error_total` | `vector_component_discarded_events_total` on `dfe_sink` (records it rejected) and on `dfe_size_cap` (records over the producer's `message.max.bytes`) | -| `transform_vector_sink_errors_total` | `vector_component_errors_total{component_id="dfe_sink"}` | -| `pipeline_dead_letters_dropped_total{reason="too_large"}` | what `vector_component_discarded_events_total{component_id="dfe_size_cap"}` grew by since the last scrape, so each drop counts once, as every other app counts a record over its ceiling | -| `pipeline_dead_letters_dropped_total{reason="rejected"}` | what `vector_component_discarded_events_total{component_id="dfe_sink"}` grew by since the last scrape: records the broker refused for good -- an unknown topic, a failed authorisation, an invalid record | +A transform file change hot-reloads. A change to `source.*`, `sink.*`, `bridge.*`, `pipeline.name`, `vector.*`, `metrics.*` or `logging.*` takes a pod restart, and the watcher logs it as such. The reload path, the readiness rules, the metrics merge and the Kafka tuning are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md). -An unreachable exporter is not fatal: the scrape warns at most once every five -minutes, counts `transform_vector_scrape_failures_total`, and leaves readiness -alone. So is an oversized one -- a response past 4 MiB is a failed scrape, not -a partial merge. - -A component Vector drops stops appearing in the exposition. Its gauges are -zeroed after `metrics.vector_metrics_expiry_ticks` scrapes without it (default -4), rather than reading their last value until the pod restarts. Its counters -are left flat, which already rates to zero. - -`/readyz` reports two things. First, whether the Vector subprocess EXISTS: the -child was still alive 500ms after spawn, which rules out the crash-on-start -cases (bad argv, unreadable config) and a pod sitting in crash-recovery -backoff. Vector's own startup routinely takes longer than that, so there is a -window where the pod reports ready and Vector is still coming up. - -Second, whether the sink is delivering. The sink never gives up on a record (`sink.message_timeout_ms: 0`), so a partition with no leader would otherwise hold records indefinitely under a live, Ready process. A record the broker refuses for good is dropped instead, so an ACL or missing-topic outage loses everything while the counters move. When the sink holds records, or has had records refused, and has delivered none for `metrics.sink_stall_secs` (default 60), `/readyz` answers 503 until it delivers again. An idle sink is never stalled. - -## Development - -```bash -# Run checks (fmt + clippy + test + deny) -make check - -# Run tests -cargo nextest run - -# Run the e2e suite, opt-in cases included (needs Docker + a Vector binary) -cargo nextest run --test e2e --run-ignored all - -# Run the Vector validate cases (needs a Vector binary) -cargo nextest run -E 'test(vector_validate)' --run-ignored all - -# Drive the release build's PGO workload by hand (needs no broker or Vector) -PGO_WORKLOAD_DURATION_SECS=60 scripts/pgo-workload.sh target/debug/dfe-transform-vector -``` +## Documentation -Release builds are PGO- and BOLT-optimised, off `scripts/pgo-workload.sh`: it -drives the direct transport, the subprocess watch and the Vector metrics merge -for five minutes on each arch. That covers the supervisor binary and nothing -else -- Vector's own per-record work happens in the binary the image downloads, -which this repo does not compile. +- [docs/architecture.md](docs/architecture.md): the problem, the ownership boundary and the invariants. +- [docs/internals.md](docs/internals.md): the config engine, the subprocess manager, health and metrics, the Vector version pin, the chart. +- [docs/migration.md](docs/migration.md): moving from the official Vector chart. +- [docs/librdkafka.md](docs/librdkafka.md): Kafka client tuning. +- [tests/TESTING.md](tests/TESTING.md): the test tiers, and the e2e cases that need Docker and a Vector binary. -The filebeat acceptance case (`e2e::filebeat_kafka`) runs by default -- it runs -on the test run's own broker container. It grades the filebeat corpus against elastic's -golden events, and both live in dfe-transform-vrl, so it skips with a message -unless that repo is checked out beside this one or `DFE_TRANSFORM_VRL_DIR` -points at it. +The suite's published docs are at https://dfe.docs.hyperi.io. -## Documentation +## Help and contributing -- [docs/architecture.md](https://github.com/hyperi-io/dfe-transform-vector/blob/main/docs/architecture.md) -- The problem, the ownership boundary and the invariants -- [docs/internals.md](https://github.com/hyperi-io/dfe-transform-vector/blob/main/docs/internals.md) -- Full architecture and design -- [docs/migration.md](https://github.com/hyperi-io/dfe-transform-vector/blob/main/docs/migration.md) -- Migration from official Vector chart -- [docs/librdkafka.md](https://github.com/hyperi-io/dfe-transform-vector/blob/main/docs/librdkafka.md) -- Kafka tuning reference - -## Issues +How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue. Open bugs: [this repo](https://github.com/hyperi-io/dfe-transform-vector/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs) ## License -This project is licensed under the Business Source License 1.1 -(BUSL-1.1). See [LICENSE](https://github.com/hyperi-io/dfe-transform-vector/blob/main/LICENSE) for details. - -Copyright (c) 2026 HYPERI PTY LIMITED - -For commercial licensing options, see [COMMERCIAL.md](https://github.com/hyperi-io/dfe-transform-vector/blob/main/COMMERCIAL.md). - -## Context - -### What this is - -A Rust supervisor that runs Vector.dev as a child process so a Vector pipeline -behaves like every other DFE app -- one big-dial config, one ops port, an OTLP -push, and a chart assembled from its deployment contract. It is NOT -the transform engine and it does not build Vector: the image downloads the -upstream release binary at the version pinned by `VECTOR_VERSION` in -`src/deployment.rs`, and cargo compiles only the supervisor. It is also not a -singleton and not part of the default deploy -- `dfe-infra/apps.yaml` declares -it `multiplicity: per_config` with `scale_deployed: true` and an empty -`default_in`, so one deployment exists per source config and only when a -deployment asks for it. - -### Where things live - -| Path | What it holds | -|---|---| -| `src/main.rs` | CLI entry. Fills scalo's `CommonArgs` from the loaded config before `run_app`, which is what makes `metrics.address` and `logging.*` take effect | -| `src/config/` | `loader` -> `validate` -> `generate` -> `wiring` -> `assembler`, plus `reload`, `transforms` and `kafka_defaults` | -| `src/vector/` | `process` spawn and backoff, `lifecycle` states, and `binary` -- version selection kept pure and NOT on the run path, which its own header explains | -| `src/bridge.rs` | The direct-transport translation between scalo `Transport/Push` and Vector `PushEvents`. Does nothing on `bus` | -| `src/health.rs`, `src/metrics.rs`, `src/metrics/scrape.rs` | Readiness publishing, and the scrape that merges `vector_*` into scalo's registry | -| `src/deployment.rs`, `src/vector-layer.dockerfile` | `contract()`, the source the Dockerfile and the released chart are generated from, and the two-binary image override spliced onto scalo's generated Dockerfile | -| `templates/bus.yaml`, `templates/direct.yaml` | Whole runnable Vector topologies, one per transport, commented per field | -| `pipelines/filebeat/` | The shipped filebeat pipeline | -| `docs/` | `architecture.md` for the shape and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` | -| `tests/` | `integration`, `e2e`, `smoke`, and `TESTING.md` for how the broker and Vector binary are resolved | -| `scripts/fetch-vector.sh`, `scripts/pgo-workload.sh` | Downloads the pinned Vector for tests, and drives the PGO/BOLT workload | - -### Commands that prove a change - -```bash -make check # hyperi-ci check -- quality + test -cargo nextest run --lib # unit only, no infrastructure -cargo nextest run --test integration # config assembly, wiring, lifecycle, metrics -cargo nextest run --test smoke # CLI surface -cargo nextest run --test e2e # needs Docker or a live broker -cargo nextest run --run-ignored all # everything, opt-in cases included -``` - -What green does NOT mean: - -- **The default run skips the Vector validator.** Seven `#[ignore]` cases need a - real Vector binary -- six in `tests/integration/vector_validate.rs` and one in - `tests/e2e/kafka.rs`. Until you pass `--run-ignored all`, nothing has run - Vector's own validator over an assembled config. -- **Locally, a missing broker or Vector binary SKIPS rather than fails.** In CI - that same absence is an assertion failure instead, so CI is stricter than a - green laptop run. -- **`e2e::filebeat_kafka` runs by default but skips without a sibling repo.** The - corpus and elastic's golden events live in dfe-transform-vrl, so it needs that - repo checked out beside this one or `DFE_TRANSFORM_VRL_DIR` pointing at it. -- **A docs-only push runs no CI at all.** `paths-ignore` for `docs/**` and - `**.md` is on the `push` trigger but not on `pull_request`, so the PR is the - only place a docs change is checked. A silent push is not a pass. - -### What tends to bite - -| Don't | Do | Why | -|---|---|---| -| Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output and the release assembles the chart from the contract, so a hand edit is reverted or never ships | -| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes | -| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()` and scale on CPU plus scaling pressure | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The library's trigger also reads `config.kafka.*`, which this app does not have, and refuses to render | -| Set `vector.version` in a deployment's config and forget it | Leave it unset, so the default reads `VECTOR_VERSION` | A committed chart once said 0.48.0 while the image baked 0.57.0, and only `version_check: warn` kept pods starting. Under `strict` that pairing refuses to start at all | -| Mount a whole `templates/*.yaml` as a transform file | Copy only its `transforms:` block | The supervisor generates `sources:` and `sinks:` from the big dials and refuses a transform file declaring them | -| Change `password` to a `SensitiveString` to look safer | Leave it a `String` | `SensitiveString` serialises as `***REDACTED***` and the figment serialize-merge-deserialize round trip in `apply_figment_env()` destroys the value. Masking happens in logs and `Debug`, and Vector reads the credential from files through its `directory` secret backend, so it never lands in the Vector config | -| Put a `${VAR}` placeholder in a credential or a transform | Mount the secret and set `sasl.secret_dir`, or give the credential through `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`; read the environment in VRL with `get_env_var` | Vector 0.57+ expands no `${VAR}` without `--dangerously-allow-env-var-interpolation`, which the supervisor never passes, so the placeholder reaches the broker as the password | -| Assume a config key you added is wired because it parses and validates | Prove it changes behaviour, and add it to the standing check that every committed config file loads | `metrics.address` and `logging.level` never left the struct. A config saying `metrics.address: 0.0.0.0:19099` still listened on 9090, so a deployment moving the metrics port lost every probe | -| Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on. Consumer lag sat at 72 across a quiet 90-second window while the app's own metrics said it had processed those same 72 events | -| Read `/readyz` as "carrying traffic" | Read it as "the child was alive 500ms after spawn" | It once answered an unconditional 200 because nothing published a readiness signal, so Vector could crash and restart hundreds of times with the pod still Ready and zero restarts | -| Swap the reload poller for inotify | Keep polling | S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events, so inotify works on every laptop and silently stops reloading in production | -| Add or rename a credential field without a schema marker | Schema it as `scalo::SensitiveString` with `#[schemars(with = ...)]`, as `SaslConfig::password` is | That puts `x-scalo-secret` and `writeOnly` on the field in the emitted config-schema, which is what tells a console to mask it. Without it, only dfe-engine's leaf-name fallback (`password`, `secret`, `token`, `api_key`, `private_key`, `passphrase`) redacts it, so a name outside that set returns an operator's Kafka password in the clear | - -### Where this sits - -Generated from `dfe-infra/suite.yaml` via -`python3 ../dfe-infra/scripts/dfe-stack suite --consumer dfe-transform-vector` -and `--producer dfe-transform-vector`. - -Inbound -- what this repo depends on: - -- **scalo-rs**, `cargo-dep`. `Cargo.toml` declares the `scalo` crate by range, - with a second range covering the dev dependency. A scalo release reaches this - repo, so widen the range if it does not admit the new version, then rebuild. -- **scalo-rs**, `generated-file`, lockstep. The committed `Dockerfile` is written - by `scalo::deployment::generate_dockerfile()` -- its own header names the - generator and the schema version -- and this repo splices the Vector layer onto - it. When the generator or its schema moves, regenerate with - `dfe-transform-vector emit-dockerfile > Dockerfile` and commit the diff. The - released chart is assembled on the scalo-service library chart at - `release.helm.library`, which moves with the scalo version in `Cargo.toml`. - -Outbound -- what depends on this repo: - -- **dfe-infra**, `image-pin`, lockstep. `dfe-infra/versions.yaml` pins this - repo's container image as a tag plus the digest that makes the tag - immutable, and the thin chart released here, which Argo deploys. Bump the - tag and re-resolve the digest. - -Two more relationships that are real but are NOT declared edges in the suite -graph, so no gate enforces them: - -- **dfe-infra `apps.yaml`** is the SSoT for this app's shape -- multiplicity, - scaling, transports, endpoints and which compiler derives its routing. - dfe-engine reads that manifest, so adding or changing the app is a manifest - edit, never an engine release. -- **dfe-transform-vrl** holds the filebeat corpus, the bundled pipeline and the - documented divergences that `e2e::filebeat_kafka` grades against. The - dependency is a test-time repo checkout only, and the case skips without it. +BUSL-1.1, see [LICENSE](LICENSE). From e237208db09177a581382020e5df9a2ec786f0ca Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 13 Nov 2026 13:59:21 +1100 Subject: [PATCH 2/6] docs(readme): name the dotenv layer and the readiness signals The cascade now lists the working directory's .env between the environment and the config file. The readiness row says /readyz proves the child is alive after spawn and the sink isn't stalled. It proves nothing about traffic. The docs site link is an autolink for markdownlint. --- CONTRIBUTING.md | 2 +- README.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a8b797b..3262e5f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -233,7 +233,7 @@ What green does NOT mean: | Put a `${VAR}` placeholder in a credential or a transform | Mount the secret and set `sasl.secret_dir`, or give the credential through `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`, and read the environment in VRL with `get_env_var` | Vector 0.57+ expands no `${VAR}` without `--dangerously-allow-env-var-interpolation`, which the supervisor never passes, so the placeholder reaches the broker as the password | | Assume a config key you added is wired because it parses and validates | Prove it changes behaviour, and add it to the standing check that every committed config file loads | `metrics.address` and `logging.level` never left the struct. A config saying `metrics.address: 0.0.0.0:19099` still listened on 9090, so a deployment moving the metrics port lost every probe | | Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on. Consumer lag sat at 72 across a quiet 90-second window while the app's own metrics said it had processed those same 72 events | -| Read `/readyz` as "carrying traffic" | Read it as "the child was alive 500ms after spawn" | It once answered an unconditional 200 because nothing published a readiness signal, so Vector could crash and restart hundreds of times with the pod still Ready and zero restarts | +| Read `/readyz` as "carrying traffic" | Read it as "the child was alive 500ms after spawn, and the sink is not stalled" | Readiness is those two signals and nothing more, so a pod can be Ready with no record moving | | Swap the reload poller for inotify | Keep polling | S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events, so inotify works on every laptop and silently stops reloading in production | | Add or rename a credential field without a schema marker | Schema it as `scalo::SensitiveString` with `#[schemars(with = ...)]`, as `SaslConfig::password` is | That puts `x-scalo-secret` and `writeOnly` on the field in the emitted config-schema, which is what tells a console to mask it. Without it, only dfe-engine's leaf-name fallback (`password`, `secret`, `token`, `api_key`, `private_key`, `passphrase`) redacts it, so a name outside that set returns an operator's Kafka password in the clear | diff --git a/README.md b/README.md index 62068ba..faf69d7 100644 --- a/README.md +++ b/README.md @@ -25,15 +25,15 @@ cargo build --release | | `bus` | `direct` | |---|---|---| -| In | Kafka `_land` | a scalo Push listener on `source.listen` (port 6000) | +| In | Kafka `_land` | a scalo Push listener on `source.listen` (port 6000 by default) | | Out | Kafka `_load` | a scalo Push client to `sink.endpoint` | | Needs a broker | yes | no | -One port, 9090, carries the ops surface: `/livez`, `/readyz` (200 only while Vector is up and its sink is delivering) and `/metrics`. Vector's own exporter on 9598 is loopback only. +One port, 9090 by default, carries the ops surface: `/livez`, `/readyz` (200 only while Vector is up and its sink is not stalled) and `/metrics`. Vector's own exporter, `127.0.0.1:9598` by default, is loopback only. ## Configuration -Settings load through a cascade: CLI, then `DFE_TRANSFORM_VECTOR_*` environment variables, then the config file, then defaults. The transport dials take a single underscore (`DFE_TRANSFORM_VECTOR_SOURCE_TRANSPORT`, `_SOURCE_LISTEN`, `_SINK_ENDPOINT`, the SASL pairs), and a key inside a section takes two (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS`, `_SINK__TOPIC`, `_TRANSFORMS__DIR`). [docs/config-schema.yaml](docs/config-schema.yaml) lists every key with its type and default. +Settings load through a cascade: CLI, then `DFE_TRANSFORM_VECTOR_*` environment variables, then the working directory's `.env`, then the config file, then defaults. The transport dials take a single underscore (`DFE_TRANSFORM_VECTOR_SOURCE_TRANSPORT`, `_SOURCE_LISTEN`, `_SINK_ENDPOINT`, the SASL pairs), and a key inside a section takes two (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS`, `_SINK__TOPIC`, `_TRANSFORMS__DIR`). [docs/config-schema.yaml](docs/config-schema.yaml) lists every key with its type and default. A transform file change hot-reloads. A change to `source.*`, `sink.*`, `bridge.*`, `pipeline.name`, `vector.*`, `metrics.*` or `logging.*` takes a pod restart, and the watcher logs it as such. The reload path, the readiness rules, the metrics merge and the Kafka tuning are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md). @@ -45,7 +45,7 @@ A transform file change hot-reloads. A change to `source.*`, `sink.*`, `bridge.* - [docs/librdkafka.md](docs/librdkafka.md): Kafka client tuning. - [tests/TESTING.md](tests/TESTING.md): the test tiers, and the e2e cases that need Docker and a Vector binary. -The suite's published docs are at https://dfe.docs.hyperi.io. +The suite's published docs are at . ## Help and contributing From f529964d5223db8decc18a1d372220fcb1ea7ffa Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 13 Nov 2026 14:04:01 +1100 Subject: [PATCH 3/6] docs(contributing): fix placeholder credential claim The bite table wrongly says a ${VAR} credential reaches the broker. The loader refuses any credential holding ${ before Vector starts. --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3262e5f..2315d27 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -230,7 +230,7 @@ What green does NOT mean: | Set `vector.version` in a deployment's config and forget it | Leave it unset, so the default reads `VECTOR_VERSION` | A committed chart once said 0.48.0 while the image baked 0.57.0, and only `version_check: warn` kept pods starting. Under `strict` that pairing refuses to start at all | | Mount a whole `templates/*.yaml` as a transform file | Copy only its `transforms:` block | The supervisor generates `sources:` and `sinks:` from the big dials and refuses a transform file declaring them | | Change `password` to a `SensitiveString` to look safer | Leave it a `String` | `SensitiveString` serialises as `***REDACTED***` and the figment serialize-merge-deserialize round trip in `apply_figment_env()` destroys the value. Masking happens in logs and `Debug`, and Vector reads the credential from files through its `directory` secret backend, so it never lands in the Vector config | -| Put a `${VAR}` placeholder in a credential or a transform | Mount the secret and set `sasl.secret_dir`, or give the credential through `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`, and read the environment in VRL with `get_env_var` | Vector 0.57+ expands no `${VAR}` without `--dangerously-allow-env-var-interpolation`, which the supervisor never passes, so the placeholder reaches the broker as the password | +| Put a `${VAR}` placeholder in a credential or a transform | Mount the secret and set `sasl.secret_dir`, or give the credential through `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`, and read the environment in VRL with `get_env_var` | Vector 0.57+ expands no `${VAR}` without `--dangerously-allow-env-var-interpolation`, which the supervisor never passes, so a credential holding `${` is refused at load rather than reaching the broker as the password | | Assume a config key you added is wired because it parses and validates | Prove it changes behaviour, and add it to the standing check that every committed config file loads | `metrics.address` and `logging.level` never left the struct. A config saying `metrics.address: 0.0.0.0:19099` still listened on 9090, so a deployment moving the metrics port lost every probe | | Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on. Consumer lag sat at 72 across a quiet 90-second window while the app's own metrics said it had processed those same 72 events | | Read `/readyz` as "carrying traffic" | Read it as "the child was alive 500ms after spawn, and the sink is not stalled" | Readiness is those two signals and nothing more, so a pod can be Ready with no record moving | From faa2180de48cfb148c448203afbf781730e91c17 Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 13 Nov 2026 14:10:37 +1100 Subject: [PATCH 4/6] docs(contributing): describe current behaviour in three rows The version pin, unwired config key and auto-commit rows describe current behaviour. Old measurements and dates are removed. --- CONTRIBUTING.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2315d27..35ea180 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -227,12 +227,12 @@ What green does NOT mean: | Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output and the release assembles the chart from the contract, so a hand edit is reverted or never ships | | Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes | | Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()` and scale on CPU plus scaling pressure | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The library's trigger also reads `config.kafka.*`, which this app does not have, and refuses to render | -| Set `vector.version` in a deployment's config and forget it | Leave it unset, so the default reads `VECTOR_VERSION` | A committed chart once said 0.48.0 while the image baked 0.57.0, and only `version_check: warn` kept pods starting. Under `strict` that pairing refuses to start at all | +| Set `vector.version` in a deployment's config and forget it | Leave it unset, so the default reads `VECTOR_VERSION` | A pinned version that differs from the binary the image bakes refuses to start under `strict`, and runs only because `version_check: warn` lets it | | Mount a whole `templates/*.yaml` as a transform file | Copy only its `transforms:` block | The supervisor generates `sources:` and `sinks:` from the big dials and refuses a transform file declaring them | | Change `password` to a `SensitiveString` to look safer | Leave it a `String` | `SensitiveString` serialises as `***REDACTED***` and the figment serialize-merge-deserialize round trip in `apply_figment_env()` destroys the value. Masking happens in logs and `Debug`, and Vector reads the credential from files through its `directory` secret backend, so it never lands in the Vector config | | Put a `${VAR}` placeholder in a credential or a transform | Mount the secret and set `sasl.secret_dir`, or give the credential through `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`, and read the environment in VRL with `get_env_var` | Vector 0.57+ expands no `${VAR}` without `--dangerously-allow-env-var-interpolation`, which the supervisor never passes, so a credential holding `${` is refused at load rather than reaching the broker as the password | -| Assume a config key you added is wired because it parses and validates | Prove it changes behaviour, and add it to the standing check that every committed config file loads | `metrics.address` and `logging.level` never left the struct. A config saying `metrics.address: 0.0.0.0:19099` still listened on 9090, so a deployment moving the metrics port lost every probe | -| Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on. Consumer lag sat at 72 across a quiet 90-second window while the app's own metrics said it had processed those same 72 events | +| Assume a config key you added is wired because it parses and validates | Prove it changes behaviour, and add it to the standing check that every committed config file loads | A key can parse, validate and reach nothing, so a deployment that moves a port on it keeps the old port and loses every probe | +| Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on, so with it off the stored offsets never reach the broker and consumer lag never clears | | Read `/readyz` as "carrying traffic" | Read it as "the child was alive 500ms after spawn, and the sink is not stalled" | Readiness is those two signals and nothing more, so a pod can be Ready with no record moving | | Swap the reload poller for inotify | Keep polling | S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events, so inotify works on every laptop and silently stops reloading in production | | Add or rename a credential field without a schema marker | Schema it as `scalo::SensitiveString` with `#[schemars(with = ...)]`, as `SaslConfig::password` is | That puts `x-scalo-secret` and `writeOnly` on the field in the emitted config-schema, which is what tells a console to mask it. Without it, only dfe-engine's leaf-name fallback (`password`, `secret`, `token`, `api_key`, `private_key`, `passphrase`) redacts it, so a name outside that set returns an operator's Kafka password in the clear | From 6a4f9e9bb808ba4a87267807b3ef42221243ee39 Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 13 Nov 2026 14:17:06 +1100 Subject: [PATCH 5/6] docs(readme): document reload and exporter settings The README says hot reload needs enabling. Changes that fail validation are never sent. It names metrics.vector_metrics_address as the exporter address setting. The address stays on loopback. --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index faf69d7..e0887ab 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Runs a Vector pipeline as a supervised transform stage in a DFE deployment. - One small YAML (pipeline name, brokers, topics, SASL and TLS) generates Vector's source, sink and observability config, so you write only the `transforms:` and `enrichment_tables:` blocks. - Kafka in and out (`bus`), or a direct gRPC push with no broker (`direct`). The transform you author is the same on both. - A crashed Vector is restarted with backoff from 1 s to 60 s, and readiness follows it. -- A transform change is validated, sent to Vector with a SIGHUP, and read back from Vector's own counters as applied, refused (the running config is put back) or unconfirmed. +- With reload enabled, a transform change is validated, sent to Vector with a SIGHUP, and read back from Vector's own counters as applied, refused (the running config is put back) or unconfirmed. One that fails validation is logged and never sent. - Vector's `vector_*` metrics are merged into the supervisor's, so one endpoint and one OTLP push carry both. ## Quick start @@ -29,13 +29,13 @@ cargo build --release | Out | Kafka `_load` | a scalo Push client to `sink.endpoint` | | Needs a broker | yes | no | -One port, 9090 by default, carries the ops surface: `/livez`, `/readyz` (200 only while Vector is up and its sink is not stalled) and `/metrics`. Vector's own exporter, `127.0.0.1:9598` by default, is loopback only. +One port, 9090 by default, carries the ops surface: `/livez`, `/readyz` (200 only while Vector is up and its sink is not stalled) and `/metrics`. Vector's own exporter defaults to `127.0.0.1:9598`, and `metrics.vector_metrics_address` stays on loopback. ## Configuration Settings load through a cascade: CLI, then `DFE_TRANSFORM_VECTOR_*` environment variables, then the working directory's `.env`, then the config file, then defaults. The transport dials take a single underscore (`DFE_TRANSFORM_VECTOR_SOURCE_TRANSPORT`, `_SOURCE_LISTEN`, `_SINK_ENDPOINT`, the SASL pairs), and a key inside a section takes two (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS`, `_SINK__TOPIC`, `_TRANSFORMS__DIR`). [docs/config-schema.yaml](docs/config-schema.yaml) lists every key with its type and default. -A transform file change hot-reloads. A change to `source.*`, `sink.*`, `bridge.*`, `pipeline.name`, `vector.*`, `metrics.*` or `logging.*` takes a pod restart, and the watcher logs it as such. The reload path, the readiness rules, the metrics merge and the Kafka tuning are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md). +A transform file change hot-reloads when reload is enabled, after it validates. A change to `source.*`, `sink.*`, `bridge.*`, `pipeline.name`, `vector.*`, `metrics.*` or `logging.*` takes a pod restart, and the watcher logs it as such. The reload path, the readiness rules, the metrics merge and the Kafka tuning are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md). ## Documentation From 3b9bd26c37bef2044ad4754d5c445b09c6192b0f Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 13 Nov 2026 14:21:50 +1100 Subject: [PATCH 6/6] docs(contributing): count ignored cases and state scaling default The validator note counts eight ignored cases across two files. The lag-trigger row says the default scales on CPU alone. A deployment can add a pressure trigger through keda.extraTriggers if needed. --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 35ea180..f3172f4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -206,7 +206,7 @@ cargo nextest run --run-ignored all # everything, opt-in cases included What green does NOT mean: -- **The default run skips the Vector validator.** Seven `#[ignore]` cases need a +- **The default run skips the Vector validator.** Eight `#[ignore]` cases (seven in `tests/integration/vector_validate.rs`, one in `tests/e2e/kafka.rs`) need a real Vector binary -- six in `tests/integration/vector_validate.rs` and one in `tests/e2e/kafka.rs`. Until you pass `--run-ignored all`, nothing has run Vector's own validator over an assembled config. @@ -226,7 +226,7 @@ What green does NOT mean: |---|---|---| | Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output and the release assembles the chart from the contract, so a hand edit is reverted or never ships | | Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes | -| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()` and scale on CPU plus scaling pressure | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The library's trigger also reads `config.kafka.*`, which this app does not have, and refuses to render | +| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()` and scale on CPU, adding a pressure trigger through the chart's `keda.extraTriggers` where a deployment wants one | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The library's trigger also reads `config.kafka.*`, which this app does not have, and refuses to render | | Set `vector.version` in a deployment's config and forget it | Leave it unset, so the default reads `VECTOR_VERSION` | A pinned version that differs from the binary the image bakes refuses to start under `strict`, and runs only because `version_check: warn` lets it | | Mount a whole `templates/*.yaml` as a transform file | Copy only its `transforms:` block | The supervisor generates `sources:` and `sinks:` from the big dials and refuses a transform file declaring them | | Change `password` to a `SensitiveString` to look safer | Leave it a `String` | `SensitiveString` serialises as `***REDACTED***` and the figment serialize-merge-deserialize round trip in `apply_figment_env()` destroys the value. Masking happens in logs and `Debug`, and Vector reads the credential from files through its `directory` secret backend, so it never lands in the Vector config |