dfe-transform-vrl

Runs your VRL transforms over a DFE deployment's event stream, in-process.

What it does

• Sits between the land topic and dfe-loader, and runs the VRL (Vector Remap Language) files in a directory, in filename order, against each event. The VRL engine is compiled in, so a transform is a file you change, not a release.
• Kafka in and out, or a direct gRPC push on a deployment with no broker. The transform you write is the same on both.
• At-least-once by default: the source is acknowledged only once the sink has the transformed records. source.acknowledgements.enabled: false acknowledges at receipt instead.
• Enrichment tables from CSV, JSON, YAML and STIX, read from VRL, plus MMDB and SQLite with the enrichment-mmdb and enrichment-sqlite build features. A table with a refresh setting reloads while the service runs. MMDB loads once at start.
• The host and network functions (get_env_var, get_hostname, http_request, dns_lookup, reverse_dns) do not compile, so a transform cannot reach outside the process.

JSON is the only payload format. A record that is not JSON is dropped, counted in records_error_total{stage="deserialise"}, and not redelivered. For Vector's own transforms beyond remap, use dfe-transform-vector.

Quick start

cargo build --release
cp config.example.yaml config.yaml      # set the brokers, topics and transforms.dir
./target/release/dfe-transform-vrl config-check --config config.yaml
./target/release/dfe-transform-vrl --config config.yaml

The smallest config names the source, a directory holding at least one .vrl file, and the sink:

pipeline:
  name: my-pipeline

source:
  brokers: ["kafka:9092"]
  topics: ["raw_events"]
  group_id: dfe-transform-vrl-my-pipeline

transforms:
  dir: /etc/dfe/transforms/

sink:
  brokers: ["kafka:9092"]
  topic: enriched_events

A transform file is plain VRL:

# transforms/01_parse.vrl
.parsed = parse_json!(.message)
del(.message)

Health and metrics answer on port 9090: /livez, /readyz (503 while memory is under pressure or a transport reports itself unhealthy, and 200 once the pipeline runs, an idle one included) and /metrics.

Configuration

config.example.yaml is the annotated reference, and docs/config-schema.yaml lists every key with its type and default. The big dials have flat environment overrides for Kubernetes, DFE_TRANSFORM_VRL_SOURCE_BROKERS, DFE_TRANSFORM_VRL_SINK_TOPIC, DFE_TRANSFORM_VRL_TRANSFORMS_DIR and the SASL pair among them. The whole list, Kafka over TLS with a private CA and the scalo sections that share the file on a double underscore are under Operating notes in docs/internals.md, and the settings that parse but reach nothing under Reload.

pipelines/filebeat/ ships a pure-VRL port of the DFE 2.1 filebeat templates (Cisco Meraki, IOS, Umbrella) with its timezones.csv table. Opt in by pointing transforms.dir and enrichment_tables at it. dfe-transform-elastic carries the same sources as compiled Rust.

Documentation

• docs/architecture.md: the shape, who owns the loop, the delivery contract and the two config authorities.
• docs/internals.md: data flow, VRL integration, memory budget, configuration, reload, operating notes.
• pipelines/filebeat/README.md: wiring and limits of the filebeat bundle.
• tests/TESTING.md: the test tiers.

The suite's published docs are at https://dfe.docs.hyperi.io, and this repository's pages at https://dfe.docs.hyperi.io/dfe-transform-vrl/.

Help and contributing

How to work in this repo, the commands that prove a change and what tends to bite are in CONTRIBUTING.md. A question that is not a bug goes in a new issue.

Open bugs: this repo | whole DFE suite

License

BUSL-1.1, see LICENSE.