The front door of a DFE deployment: takes events in the protocols agents already speak and hands them on as JSON.
- One service for HTTP(S) JSON, Vector's gRPC sink, OTLP, Beats, Splunk HEC, syslog, Fluent Forward, GELF, Prometheus Remote Write and webhooks, so an agent keeps the protocol it already speaks. Every handler normalises to JSON and hands the record to Kafka or straight to dfe-loader over gRPC.
- Ordered routing rules that set each record's source and org from fields it already carries, such as
data_stream.dataset, and pick its Kafka topic from the source. - A sender answered only once every destination has confirmed its records, on by default for each listener that can hold its answer.
POST /ingestanswers 202 when the batch is accepted, 503 withRetry-Afterwhen it cannot take it, and 413 over the body limit, 10,485,760 bytes by default. - Each record's org held to what its sender may use. A credential or listener bound in
ingest_bindingslands only in its own orgs, and an unbound sender only in the deployment's default org. - Header, bearer token and mTLS authentication.
- Built on scalo, so config, logs, metrics, health probes and memory backpressure work the way they do in every DFE Rust service.
| Protocol | Port | Source |
|---|---|---|
| HTTP(S) JSON | configurable | Vector, custom agents |
| gRPC Vector sink | configurable | Vector |
Webhook (POST /webhook/{caller}) |
shares HTTP, or 8090 | Product alert rules and notification hooks (runZero, ...) |
| OTLP gRPC | 4317 | OpenTelemetry collectors |
| OTLP HTTP | 4318 | OpenTelemetry collectors |
| Lumberjack/Beats | 5044 | Filebeat, Logstash Beats output |
| Splunk HEC | 8088 | Splunk forwarders, Fluentd HEC output |
| Syslog UDP/TCP/TLS | 514 / 6514 | syslogd, rsyslog, syslog-ng |
| Fluent Forward | 24224 | Fluentd, Fluent Bit |
| GELF TCP | 12201 | Graylog GELF output, Fluent Bit |
| Prometheus Remote Write | 9091 | Prometheus, VictoriaMetrics |
| Flow (NetFlow + sFlow), experimental | 2055 / 4739 / 6343 UDP | NetFlow v5/v9, IPFIX, sFlow v5 exporters (v7 not supported) |
HTTP always runs, because the probes ride its listener, and the other listeners are opt-in. A listener that answers at enqueue (acknowledgements off, or syslog, GELF and flow) buffers in memory behind a circuit breaker, and disk spillover is opt-in.
The build needs protoc, cmake and a C toolchain on PATH: the gRPC, OTLP and Prometheus Remote Write protocols compile vendored .proto files, and aws-lc-sys and rdkafka-sys build native code.
sudo apt-get install -y protobuf-compiler cmake build-essential # Debian or Ubuntu
brew install protobuf cmake # macOS
cargo build --release
./target/release/dfe-receiver --config config.yamlThe smallest config that moves data names the bind address, an auth mode, the brokers and the default source:
server:
bind_address: "0.0.0.0:8080"
auth:
mode: none
kafka:
brokers:
- "localhost:9092"
routing:
default_source: "main"
topic_suffix: "_land"Every setting is in config.example.yaml, and DFE_RECEIVER_SERVER__BIND_ADDRESS is how an environment override is written: the DFE_RECEIVER_ prefix, then the key path with double underscores. server.auth.mode is none, header, bearer or mtls. The routing rules, the webhook intake, the metrics and the hot-path design are in docs/internals.md.
- docs/internals.md: how it works, configuration, the webhook intake, metrics and the security model.
- docs/hardening.md: running it where it is reachable from outside.
- docs/sidecar-transports.md and docs/webhook-setup.md: the transport options and the webhook callers.
- docs/metrics.md and docs/performance.md: what it reports and what it costs.
- docs/architecture.md: where it sits in the suite and why it is built this way.
The suite's published docs are at https://dfe.docs.hyperi.io, and this repository's pages at https://dfe.docs.hyperi.io/dfe-receiver/.
How to work in this repo, the commands that prove a change and what tends to bite are in CONTRIBUTING.md. A question that is not a bug goes in a new issue.
Open bugs: this repo | whole DFE suite
BUSL-1.1, see LICENSE.