From 9c46aedad7f25488a8c9a5cd870873cd283dd02b Mon Sep 17 00:00:00 2001 From: Schnitz <12687466+CptSchnitz@users.noreply.github.com> Date: Tue, 21 Jul 2026 16:19:11 +0300 Subject: [PATCH 01/22] docs: spec --- docs/log-processing-sidecar-spec.md | 314 ++++++++++++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100644 docs/log-processing-sidecar-spec.md diff --git a/docs/log-processing-sidecar-spec.md b/docs/log-processing-sidecar-spec.md new file mode 100644 index 0000000..027c716 --- /dev/null +++ b/docs/log-processing-sidecar-spec.md @@ -0,0 +1,314 @@ +# Spec: Optional Fluent Bit log-processing sidecar + +## Problem Statement + +We run nginx as a shared proxy chart across many workloads. Today every access log +line nginx writes to stdout is collected centrally, but we **do not have permission to +run a DaemonSet**, so log collection happens by reading pod logs through the +**Kubernetes API**. That path is expensive and throttled, and its cost scales with the +*total* stdout volume — so we pay, at high volume, to ingest a firehose of access logs +that is overwhelmingly routine `2xx`/`3xx` traffic we never look at. + +We want to keep the small, valuable slice of logs (errors and specific status codes), +turn the log stream into useful metrics we can't get today, and stop paying to ship the +noise — and we need to cut that volume **as close to the source as possible**, because +the k8s-API collection path is the thing that costs us. + +Separately, our metrics today come only from nginx `stub_status` via the +`nginx-prometheus-exporter` sidecar (connection/throughput counters). We have no +per-status-code rates and no latency distributions, and we have a general metrics-infra +constraint: Prometheus discovers targets by a **single scrape annotation per pod**, and +we have **no remote_write endpoint** we want to stand up and babysit. + +## Solution + +Add an **optional, off-by-default in-pod Fluent Bit sidecar** that becomes the pod's +single point of log egress and metrics exposure when enabled: + +- nginx sends its full access/error log stream to Fluent Bit locally over **syslog UDP on + loopback** (best-effort). nginx still writes to stdout/stderr for `kubectl logs` + visibility, but the pod's central-collection scrape label is turned **off**, so the + k8s-API collector no longer ingests the firehose — this is where the cost is cut. +- Fluent Bit **filters** the access logs down to errors / selected status codes and + forwards only those to our **central Alloy over OTLP**; the rest are dropped. +- Fluent Bit **derives Prometheus metrics** from the same log stream (per-status-code + request counter, request-time latency histogram) and **merges** them with the existing + `stub_status` exporter's metrics onto a **single `/metrics` endpoint** it serves — so we + keep connection gauges, gain the new metrics, and still expose exactly one scrape target + with no Prometheus/remote_write change. + +When the feature is disabled, the chart behaves exactly as it does today. + +## User Stories + +1. As a platform operator, I want the log-processing sidecar to be **off by default**, so + that existing deployments of this chart are completely unaffected when they upgrade. +2. As a platform operator, I want a single `fluentbit.enabled` flag to turn the whole + feature on, so that adopting it is one deliberate switch. +3. As a cost owner, I want the pod's central-collection scrape label forced **off** when + the sidecar is enabled, so that we stop paying to ingest the full log firehose through + the k8s API. +4. As a platform operator, I want to keep controlling the scrape label myself when the + sidecar is **disabled**, so that today's behavior (logs collected as before) is + preserved and under my control. +5. As an operator debugging a pod, I want nginx to keep writing logs to **stdout/stderr** + even when the sidecar is on, so that `kubectl logs` still works for quick inspection. +6. As an operator reading `kubectl logs` on an enabled pod, I want stdout access logs in a + **human-readable** format (not raw JSON), so that they are easy to scan by eye. +7. As an operator, I want that human-readable stdout line to include the metadata we + already surface — notably the authenticated **client name** — plus request time and + upstream status, so that the readable log is actually useful and not a bare access line. +8. As a log consumer, I want the JSON-structured access log (our existing OTel-shaped + format) to flow to Fluent Bit over the syslog path, so that downstream tooling still + receives fully structured records. +9. As an SRE, I want Fluent Bit to forward only **server errors (5xx)** by default, so that + incident-relevant logs reach Loki without me configuring anything. +10. As an SRE, I want to also opt into forwarding **client errors (4xx)** with a boolean, + so that I can include them when they matter to me. +11. As an SRE, I want to specify an explicit **list of extra status codes** to forward + (e.g. `429`, `499`), so that I can capture specific conditions beyond the error classes. +12. As an SRE, I want everything not matched by my forwarding rules to be **dropped** by + the sidecar, so that only the valuable slice leaves the pod. +13. As an SRE, I want the forwarding rules expressed as **simple structured values**, so + that I don't have to learn Fluent Bit's config language for the common case. +14. As an observability engineer, I want Fluent Bit to derive a **per-status-code request + counter** from the access logs, so that I can chart request and error rates by status. +15. As an observability engineer, I want a **request-time latency histogram** derived from + the access logs, so that I can compute p50/p95/p99 latency that `stub_status` cannot + provide. +16. As an observability engineer, I want those metric definitions provided as an + **overridable default** set with safe, bounded cardinality, so that the feature is + useful the moment it's enabled and demonstrates the safe pattern. +17. As an advanced observability engineer, I want to supply my **own raw metric + definitions** (as Fluent Bit config code blocks) to derive additional metrics, so that + I'm not limited to the defaults. +18. As an advanced user, I want my raw metrics/config blocks rendered through Helm **`tpl`**, + so that I can reference chart values and release info inside them. +19. As an observability engineer, I want to be warned (in docs) never to label metrics by + high-cardinality fields (path, query, client IP, user-agent), so that I don't OOM the + sidecar or overload Prometheus. +20. As a Prometheus operator, I want Fluent Bit to also **scrape the existing + `nginx-prometheus-exporter`** and re-expose its metrics, so that `stub_status` + connection gauges are preserved. +21. As a Prometheus operator, I want Fluent Bit to serve **one merged `/metrics`** endpoint + (log-derived + scraped exporter), so that the single-scrape-annotation limit is + satisfied with no Prometheus change. +22. As a Prometheus operator, I want the pod's advertised scrape port (`mclabels.prometheus.port`) + to automatically point at Fluent Bit's merged endpoint when the feature is enabled, so + that discovery just works. +23. As a Prometheus operator, I want to make **no remote_write change** to Prometheus, so + that I don't have to stand up and monitor a new ingestion path. +24. As an SRE, I want nginx **error logs** also shipped through Fluent Bit on a separate + pipeline, so that error output is captured alongside access logs. +25. As an SRE, I want error logs parsed into **structured fields** (level, pid, message, + client, request) by Fluent Bit, so that they are queryable even though nginx OSS emits + them as plaintext. +26. As an SRE, I want **all** error logs at/above a configurable level forwarded (no + status-based filtering, since error logs have no status), so that I don't lose error + context. +27. As an SRE, I want error logs to reach the **same central OTLP logs endpoint** as access + logs, so that they land together and are sorted by severity downstream. +28. As a developer, I want a simple mechanism to **mount a custom Lua script** (rendered + through `tpl`) into Fluent Bit, so that I can implement advanced filtering (e.g. latency + thresholds) later without changing the chart. +29. As a reliability owner, I want the sidecar's failure to **never affect nginx** — nginx + must not block on the sidecar and its liveness must not depend on it — so that log + processing can never take down serving traffic. +30. As a reliability owner, I accept that logs are **best-effort** (datagrams may drop under + load or while the sidecar restarts), so that we get source-side volume reduction without + coupling nginx's fate to a log file or a reliable transport. +31. As a platform operator, I want the Fluent Bit image sourced and pinned the **same way as + the existing exporter image** (repository/tag/pullPolicy via `cloudProvider`), so that + it fits our registry and pull-secret conventions. +32. As a platform operator, I want the Fluent Bit configuration delivered via a + **ConfigMap** rendered through `tpl`, consistent with how the chart already ships + `nginx.conf`/`log_format.conf`. +33. As a platform operator, I want a dedicated, explicit **OTLP logs endpoint** value + (not reusing the traces `opentelemetry.exporterHost`), so that logs go to the correct + Alloy receiver. +34. As a chart maintainer, I want the existing `nginx-prometheus-exporter` to remain in + place (coexist), so that we don't lose connection-level signal and don't destabilize the + current metrics path. + +## Implementation Decisions + +**Tool selection — Fluent Bit.** Chosen over Grafana Alloy and the OpenTelemetry Collector. +Deciding factors, given locked constraints (scrape-only, single scrape annotation, keep +`stub_status` gauges, cut volume at source): +- Fluent Bit is the only candidate that can **merge** externally-scraped metrics + (`prometheus_scrape` input) with log-derived metrics (`log_to_metrics`) onto **one** + `prometheus_exporter` `/metrics` endpoint — Alloy's `/metrics` cannot re-expose scraped + series (it is push-only for those, requiring remote_write, which is out of scope), and the + OTel Collector has **no logs→histogram** capability at all. +- Fluent Bit's `log_to_metrics` supports **counter, gauge, and histogram** (including a + histogram over a numeric log field such as `request_time`). +- Lightest footprint of the three (C; ~20–30 MB) — relevant across a fleet of pods. +- Accepted trade-offs: it is not our existing org-standard agent (Alloy), and advanced + numeric filtering needs Lua — but the common status-based forwarding compiles to a plain + `grep` regex and needs no Lua. + +**Metrics egress model — scrape only, no remote_write.** Fluent Bit serves a single merged +`/metrics`. No change to Prometheus. Alloy-with-push was explicitly rejected because it is +the heaviest option and the only one requiring a new push endpoint we'd have to operate. + +**Coexistence with `nginx-prometheus-exporter`.** The exporter stays. Fluent Bit scrapes it +via `prometheus_scrape` and re-exposes its series alongside the log-derived metrics. When +the feature is enabled, `mclabels.prometheus.port` advertises **Fluent Bit's** merged port +instead of the exporter's port, so exactly one endpoint is scraped. + +**Transport — syslog over UDP loopback.** nginx emits its logs to Fluent Bit via +`access_log syslog:server=127.0.0.1:` and `error_log syslog:server=127.0.0.1:`. +UDP loopback was chosen over a unix domain socket because (a) it has no socket-file +startup-ordering dependency, and (b) it survives Fluent Bit restarts cleanly, whereas a +unix *datagram* socket suffers a stale-inode problem after the receiver recreates the socket +(persistent silent loss until an nginx reload). File-tailing was **rejected**: a shared log +file couples nginx's availability to disk pressure and to the sidecar keeping up — trading a +data-loss risk for an availability risk, which is unacceptable. + +**Loss model — best-effort, observable enough.** Datagram syslog is fire-and-forget; the +kernel drops when the receive buffer fills (e.g. during a burst). This is accepted. We will +size the receive buffer to reduce drops and rely on rough signals (UDP drop counters and/or +comparing nginx's request count against Fluent Bit's received count) to know when to scale +the sidecar. No elaborate drop-accounting is built. + +**Failure isolation.** Fluent Bit runs as a **regular sidecar container** (Kubernetes 1.24 +does not support native/`restartPolicy: Always` sidecars). Its health never gates nginx: +nginx never blocks on it (UDP is non-blocking), and Fluent Bit's liveness at most restarts +its own container. The existing exporter's liveness-probe comment ("the entire Pod … will +restart") is misleading — a liveness failure restarts only that container — and will be +revisited/cleaned up, not treated as a mandate. + +**Two nginx log destinations, format depends on feature state.** +- Feature **disabled** (today): access log → stdout as **JSON** (our existing + `log_format ... escape=json` template), collected as before. There is **no** native JSON + log format in open-source nginx; the manual `escape=json` template remains the approach. +- Feature **enabled**: access log → **stdout in a human-readable format** (combined-style, + augmented with the authenticated client name `$jwt_payload_sub`, `request_time`, and + upstream status) for `kubectl logs`, **and** → **syslog (JSON)** for Fluent Bit. Error log + → **stderr (plaintext)** for humans **and** → **syslog (plaintext)** for Fluent Bit. + +**Scrape-label wiring.** When `fluentbit.enabled=true`, the chart **forces** the +central-collection scrape label (`mclabels.logScraping` → `mapcolonies.io/alloy-api-logs`) +to `false`, with **no override**. When disabled, the label remains user-controlled with +today's default. This removes the double-ingest footgun. + +**Access-log forwarding — structured knobs → `grep`.** Chart-authored values compile to a +single Fluent Bit `grep` filter on the status field: +- `clientErrors: bool` (keep `4xx`), `serverErrors: bool` (keep `5xx`), `statusCodes: []` + (extra explicit codes). Because HTTP status is always three digits, these map to a clean + regex (e.g. `^[45]` for errors, alternation for explicit codes) — **no Lua** needed. +- Everything unmatched is dropped. Latency-threshold forwarding is **not** in scope (it would + require Lua) but is enabled later via the Lua escape hatch. + +**Metrics config — bounded defaults + raw passthrough.** The chart ships a default, +overridable `log_to_metrics` set with **safe, bounded cardinality**: a per-`status_code` +request counter and a `request_time` histogram (optionally split by method). Advanced users +supply **raw** Fluent Bit config blocks for additional metrics. Both the raw metrics config +and the Lua script are rendered through **`tpl`**. There is no hard cardinality enforcement +(impossible with raw config); docs warn against high-cardinality labels. + +**Error-log pipeline — separate, plaintext, structured in Fluent Bit.** A distinct syslog +input on its own port and Fluent Bit **tag** (`nginx.error` vs `nginx.access`), parsed by a +**regex parser** the chart ships for the standard nginx error format +(`time [level] pid#tid: *cid message, client:…, server:…, request:"…"`), yielding structured +fields. Forwarding policy is **forward-all at/above a configurable `minLevel`** (no +status-based filtering). Native JSON error logs (`error_log … json`, nginx 1.29.8) are +**NGINX Plus-only** and unavailable on the open-source `-otel` image, so plaintext + Fluent +Bit parsing is the approach. **No error-log metrics** are shipped by default; because metrics +config is tag-matched, a user can add error metrics later purely in raw config with no chart +change. + +**Log egress — OTLP to central Alloy.** Both pipelines forward via Fluent Bit's +`opentelemetry` output (OTLP/HTTP) to a **dedicated logs endpoint** (`fluentbit.output.logs.*`), +explicitly **not** the traces-only `opentelemetry.exporterHost`. Loki push is not used — the +central Alloy already accepts OTLP and owns the Loki backend. + +**Config delivery & image.** Fluent Bit config (inputs, filters, parsers, metrics, outputs, +optional Lua script) is delivered via a **ConfigMap rendered with `tpl`**, mirroring how +`nginx.conf`/`log_format.conf`/`default.conf` are shipped. The image follows the +`prometheusExporter.image` convention (`repository`/`tag`/`pullPolicy`, registry prefixed via +`cloudProvider`). + +**Proposed `values.yaml` shape (illustrative, from the design discussion):** + +```yaml +fluentbit: + enabled: false # off by default; when true → forces scrape label off + image: + repository: common/fluent-bit + tag: "3.x" + pullPolicy: IfNotPresent + resources: { enabled: true, value: { ... } } + output: + logs: # dedicated OTLP logs endpoint (NOT opentelemetry.exporterHost) + host: "" + port: 4318 + protocol: http + accessLog: + syslogPort: 5514 + stdoutReadable: true # human-readable stdout when enabled; JSON goes to syslog + forward: + clientErrors: false # keep 4xx + serverErrors: true # keep 5xx + statusCodes: [] # extra explicit codes, e.g. [429, 499] + metrics: + enabled: true + scrapeExporter: true # prometheus_scrape the nginx exporter → merge + port: 2021 # single merged /metrics → mclabels.prometheus.port when enabled + config: | # raw log_to_metrics blocks (tpl-rendered); safe defaults shipped + # per-status counter + request_time histogram + errorLog: + enabled: true + syslogPort: 5515 + minLevel: warn # forward all at/above this level + lua: + enabled: false + script: | # inline Lua, tpl-rendered, mounted for advanced filtering + # function keep(tag, ts, record) ... end +``` + +## Testing Decisions + +Automated tests are **out of scope by decision** — this repo has no test harness and the +team opted not to introduce one for this feature. Verification is manual, via `helm template` +against representative value sets (enabled/disabled, each forwarding knob, Lua on/off) to +confirm the rendered Deployment, ConfigMap, and `mclabels` labels/annotations match the +decisions above. Runtime behavior of nginx and Fluent Bit (actual datagram delivery, +filtering, drop behavior under load) is not asserted by any chart-level test and would only +be observed in a live environment. + +## Out of Scope + +- **Automated tests** (per decision above). +- **Remote_write / metrics push** and any Prometheus-side configuration change — explicitly + rejected; scrape-only is the model. +- **Grafana Alloy or OpenTelemetry Collector** as the sidecar — evaluated and not chosen. +- **File-tail / stdout-tail ingestion** — rejected as a production-availability risk. +- **Latency-threshold (numeric) forwarding rules** — not shipped as structured knobs; + achievable later via the Lua escape hatch. +- **Error-log-derived metrics** — not shipped by default (achievable via raw tag-matched + config). +- **Loki push protocol** — egress is OTLP only. +- **Native JSON error logs** — an NGINX Plus feature, unavailable on the open-source image. +- **Guaranteed / lossless log delivery** — the design is explicitly best-effort. +- **Native/`restartPolicy: Always` sidecars** — unavailable on Kubernetes 1.24. +- **Publishing this spec to the issue tracker** — created as a local file only. + +## Further Notes + +- **Primary success metric:** reduction in log volume ingested through the k8s-API + collection path once the scrape label is turned off and only filtered logs egress via + Fluent Bit. +- **Validation items flagged during design** (confirm during implementation, not blockers): + - **Datagram size:** access-log JSON can be large (long URLs/query strings); oversized + UDP datagrams may be truncated/dropped, causing a parse failure and line loss. Size the + receive buffer accordingly and confirm typical line sizes fit. + - Confirm the exact nginx directive/behavior for combining `error_log syslog:` with the + chosen output, and the standard error-format regex, on the pinned image. +- **Cluster/version context:** target is Kubernetes **1.24** (no native sidecars); nginx base + image is the open-source `nginxinc/nginx-unprivileged:1.29.8-alpine3.23-otel`. +- **Docs:** the chart's `values.md` is generated by helm-docs (CI); new values will need the + documentation-comment annotations the chart already uses. +- **Publishing blocked at time of writing:** `gh` auth token is invalid and the issue-tracker + triage-label vocabulary was not available; if this spec is later pushed to the tracker, + apply the `ready-for-agent` label. From 814180ce9197a8215b8c8f18e2d4ff7ae7f1dd8a Mon Sep 17 00:00:00 2001 From: CptSchnitz <12687466+CptSchnitz@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:14:21 +0300 Subject: [PATCH 02/22] feat: add Fluent Bit sidecar support with configurable metrics endpoint and resource limits - step 1 --- helm/templates/_fluentbit.tpl | 19 +++++++++++++++++++ helm/templates/_helpers.tpl | 31 ++++++++++++++++++++++++++++++- helm/templates/deployment.yaml | 9 ++++++--- helm/values.md | 10 ++++++++++ helm/values.yaml | 30 ++++++++++++++++++++++++++++++ 5 files changed, 95 insertions(+), 4 deletions(-) create mode 100644 helm/templates/_fluentbit.tpl diff --git a/helm/templates/_fluentbit.tpl b/helm/templates/_fluentbit.tpl new file mode 100644 index 0000000..acfb209 --- /dev/null +++ b/helm/templates/_fluentbit.tpl @@ -0,0 +1,19 @@ +{{/* +Fluent Bit sidecar container. Serves the merged Prometheus /metrics endpoint the pod +advertises when the feature is enabled. Its health never gates nginx. +*/}} +{{- define "nginx.fluentbitContainer" -}} +- name: fluent-bit + {{- with .Values.fluentbit.image }} + image: {{ include "nginx.cloudProviderDockerRegistryUrl" $ }}{{ .repository }}:{{ .tag }} + {{- end }} + imagePullPolicy: {{ .Values.fluentbit.image.pullPolicy }} + ports: + - name: metrics + containerPort: {{ .Values.fluentbit.accessLog.metrics.port }} + protocol: TCP + {{- if .Values.fluentbit.resources.enabled }} + resources: + {{- toYaml .Values.fluentbit.resources.value | nindent 4 }} + {{- end }} +{{- end -}} diff --git a/helm/templates/_helpers.tpl b/helm/templates/_helpers.tpl index cb74c7b..b351ece 100644 --- a/helm/templates/_helpers.tpl +++ b/helm/templates/_helpers.tpl @@ -30,6 +30,35 @@ Create chart name and version as used by the chart label. {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} {{- end }} +{{/* +mclabels labels. When Fluent Bit is enabled, force the log-scraping label off so the +central k8s-API collector stops ingesting the firehose. Override is set on a deep copy, +so the shared .Values is never mutated. (`merge` can't be used here: mergo treats the +`false` override as empty and keeps the original value.) +*/}} +{{- define "nginx.mclabels.labels" -}} +{{- $mclabels := .Values.mclabels -}} +{{- if .Values.fluentbit.enabled -}} +{{- $mclabels = set (deepCopy .Values.mclabels) "logScraping" false -}} +{{- end -}} +{{- include "mclabels.labels" (dict "Values" (dict "mclabels" $mclabels "global" .Values.global)) -}} +{{- end -}} + +{{/* +mclabels annotations. When Fluent Bit is enabled, point the advertised Prometheus port at +its merged /metrics endpoint. The override is set on a deep copy, so the exporter +container and Service keep using the exporter's own port. Unlike the labels helper, no +`global` is threaded through — mclabels.annotations reads only .Values.mclabels. +*/}} +{{- define "nginx.mclabels.annotations" -}} +{{- $mclabels := .Values.mclabels -}} +{{- if .Values.fluentbit.enabled -}} +{{- $prometheus := set (deepCopy .Values.mclabels.prometheus) "port" .Values.fluentbit.accessLog.metrics.port -}} +{{- $mclabels = set (deepCopy .Values.mclabels) "prometheus" $prometheus -}} +{{- end -}} +{{- include "mclabels.annotations" (dict "Values" (dict "mclabels" $mclabels)) -}} +{{- end -}} + {{/* Common labels */}} @@ -40,7 +69,7 @@ helm.sh/chart: {{ include "nginx.chart" . }} app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} -{{ include "mclabels.labels" . }} +{{ include "nginx.mclabels.labels" . }} {{- end }} {{/* diff --git a/helm/templates/deployment.yaml b/helm/templates/deployment.yaml index 7c77bbb..a1a995d 100644 --- a/helm/templates/deployment.yaml +++ b/helm/templates/deployment.yaml @@ -35,7 +35,7 @@ spec: {{- if .Values.additionalPodAnnotations }} {{- toYaml .Values.additionalPodAnnotations | nindent 8 }} {{- end }} - {{- include "mclabels.annotations" . | nindent 8 }} + {{- include "nginx.mclabels.annotations" . | nindent 8 }} spec: {{- if $cloudProviderImagePullSecretName }} imagePullSecrets: @@ -97,8 +97,8 @@ spec: - name: http containerPort: {{ $prometheusExporterPort }} protocol: TCP - # Note: if the Prometheus Exporter container isn't live the entire Pod - # (including the NGINX container) will restart + # Note: a livenessProbe failure restarts only this container, not the whole + # Pod — the NGINX container keeps serving traffic while the exporter restarts. livenessProbe: initialDelaySeconds: {{ .Values.initialDelaySeconds }} httpGet: @@ -109,6 +109,9 @@ spec: {{- toYaml .Values.prometheusExporter.resources.value | nindent 12 }} {{- end }} {{- end }} + {{- if .Values.fluentbit.enabled }} + {{- include "nginx.fluentbitContainer" . | nindent 8 }} + {{- end }} {{- if .Values.sidecars }} {{ tpl (.Values.sidecars) . | nindent 8 }} {{- end }} diff --git a/helm/values.md b/helm/values.md index c3f0e5b..d6e0cb3 100644 --- a/helm/values.md +++ b/helm/values.md @@ -32,6 +32,16 @@ A Helm chart for nginx | environment | string | `"development"` | Specify the environment for this deployment | | extraVolumeMounts | list | `[]` | List of extra volumeMounts that are added to the NGINX container | | extraVolumes | list | `[]` | List of extra volumes that are added to the Deployment | +| fluentbit.accessLog.metrics.port | int | `2021` | Port on which Fluent Bit serves the merged Prometheus /metrics endpoint. When the sidecar is enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). | +| fluentbit.enabled | bool | `false` | Enable or disable the optional Fluent Bit log-processing sidecar. When enabled, the central log-scraping label is forced off (no override) and the Prometheus scrape port is pointed at Fluent Bit's merged /metrics endpoint. When disabled (the default), the chart behaves exactly as before. | +| fluentbit.image.pullPolicy | string | `"IfNotPresent"` | Image pull policy for the Fluent Bit sidecar | +| fluentbit.image.repository | string | `"common/fluent-bit"` | Docker image name for the Fluent Bit sidecar | +| fluentbit.image.tag | string | `"5.0.7"` | Docker image tag for the Fluent Bit sidecar | +| fluentbit.resources.enabled | bool | `true` | Enable or disable resource limits and requests for the Fluent Bit sidecar | +| fluentbit.resources.value.limits.cpu | string | `"100m"` | CPU limit for the Fluent Bit sidecar | +| fluentbit.resources.value.limits.memory | string | `"128Mi"` | Memory limit for the Fluent Bit sidecar | +| fluentbit.resources.value.requests.cpu | string | `"100m"` | CPU request for the Fluent Bit sidecar | +| fluentbit.resources.value.requests.memory | string | `"128Mi"` | Memory request for the Fluent Bit sidecar | | fullnameOverride | string | `""` | String to fully override fullname template | | global.cloudProvider | object | `{}` | Global cloud provider configuration. | | global.environment | string | `""` | Global environment setting. | diff --git a/helm/values.yaml b/helm/values.yaml index c5e549e..2a40472 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -185,6 +185,36 @@ prometheusExporter: # -- Memory request for the main container memory: 128Mi + # @section -- Fluent Bit Log-Processing Sidecar +fluentbit: + # -- Enable or disable the optional Fluent Bit log-processing sidecar. When enabled, the central log-scraping label is forced off (no override) and the Prometheus scrape port is pointed at Fluent Bit's merged /metrics endpoint. When disabled (the default), the chart behaves exactly as before. + enabled: false + image: + # -- Docker image name for the Fluent Bit sidecar + repository: common/fluent-bit + # -- Docker image tag for the Fluent Bit sidecar + tag: "5.0.7" + # -- Image pull policy for the Fluent Bit sidecar + pullPolicy: IfNotPresent + resources: + # -- Enable or disable resource limits and requests for the Fluent Bit sidecar + enabled: true + value: + limits: + # -- CPU limit for the Fluent Bit sidecar + cpu: 100m + # -- Memory limit for the Fluent Bit sidecar + memory: 128Mi + requests: + # -- CPU request for the Fluent Bit sidecar + cpu: 100m + # -- Memory request for the Fluent Bit sidecar + memory: 128Mi + accessLog: + metrics: + # -- Port on which Fluent Bit serves the merged Prometheus /metrics endpoint. When the sidecar is enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). + port: 2021 + authorization: # -- Use authroization mechanism enabled: true From ebf96bbe910aa169e6a32082215e7e8322a7ab74 Mon Sep 17 00:00:00 2001 From: CptSchnitz <12687466+CptSchnitz@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:16:49 +0300 Subject: [PATCH 03/22] feat: implemented step 2 --- helm/config/fluent-bit.conf | 50 +++++++++++++++++++++++++ helm/config/log_format.conf | 12 ++++++ helm/config/nginx.conf | 3 +- helm/templates/_fluentbit.tpl | 4 ++ helm/templates/_helpers.tpl | 26 +++++++++++++ helm/templates/deployment.yaml | 5 +++ helm/templates/fluentbit-configmap.yaml | 10 +++++ helm/values.md | 8 ++++ helm/values.yaml | 19 ++++++++++ 9 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 helm/config/fluent-bit.conf create mode 100644 helm/templates/fluentbit-configmap.yaml diff --git a/helm/config/fluent-bit.conf b/helm/config/fluent-bit.conf new file mode 100644 index 0000000..14845b9 --- /dev/null +++ b/helm/config/fluent-bit.conf @@ -0,0 +1,50 @@ +# Fluent Bit configuration, rendered through Helm `tpl` from the chart ConfigMap +# (mirroring how nginx.conf / log_format.conf are shipped). +# +# Access-log forwarding pipeline: nginx ships its JSON access log here over syslog UDP on +# loopback (best-effort); Fluent Bit filters it down to the selected status codes and +# forwards only those to central Alloy over OTLP. Everything else is dropped at the source. + +[SERVICE] + flush 1 + daemon off + log_level info + parsers_file parsers.conf + +# nginx `access_log syslog:server=127.0.0.1:` delivers the JSON `main` format here. +[INPUT] + Name syslog + Tag nginx.access + Mode udp + Listen 127.0.0.1 + Port {{ .Values.fluentbit.accessLog.syslogPort }} + Buffer_Chunk_Size 32k + Buffer_Max_Size 64k + +# The syslog message body is the nginx JSON line; decode it so the OTel-shaped fields +# (including the nested http.response.status_code) become queryable by the grep filter. +[FILTER] + Name parser + Match nginx.access + Key_Name message + Parser json + Reserve_Data On + +# Keep only the status codes selected by fluentbit.accessLog.forward; drop everything else. +# When no forwarding rule is active the regex matches nothing, so all records are dropped. +[FILTER] + Name grep + Match nginx.access + Regex $Attributes['http.response.status_code'] {{ include "nginx.fluentbit.accessLogRegex" . }} + +# Forward the surviving records to the dedicated central Alloy OTLP/HTTP logs endpoint +# (fluentbit.output.logs.*, deliberately NOT the traces-only opentelemetry.exporterHost). +[OUTPUT] + Name opentelemetry + Match nginx.access + Host {{ .Values.fluentbit.output.logs.host }} + Port {{ .Values.fluentbit.output.logs.port }} + Logs_uri /v1/logs + {{- if eq .Values.fluentbit.output.logs.protocol "https" }} + Tls On + {{- end }} diff --git a/helm/config/log_format.conf b/helm/config/log_format.conf index 8dc82ee..595ba33 100644 --- a/helm/config/log_format.conf +++ b/helm/config/log_format.conf @@ -53,3 +53,15 @@ log_format main escape=json '"InstrumentationScope":"access.log",' '"Body":"$request"' '}'; +{{- if .Values.fluentbit.enabled }} + +# Human-readable (combined-style) access log for `kubectl logs`, augmented with the +# authenticated client name, request time and upstream status. Written to stdout when +# fluentbit.accessLog.stdoutReadable is true; the JSON `main` format always goes to +# Fluent Bit over syslog. +log_format readable +'$remote_addr {{ if .Values.authorization.enabled }}$jwt_payload_sub {{ end }}[$time_local] ' +'"$request" $status $body_bytes_sent ' +'"$http_referer" "$http_user_agent" ' +'rt=$request_time upstream_status=$upstream_status'; +{{- end }} diff --git a/helm/config/nginx.conf b/helm/config/nginx.conf index 08623b6..8c975dd 100644 --- a/helm/config/nginx.conf +++ b/helm/config/nginx.conf @@ -43,7 +43,8 @@ http { include /etc/nginx/log_format.conf; - access_log /var/log/nginx/access.log main; + access_log /var/log/nginx/access.log {{ if .Values.fluentbit.enabled }}{{ if .Values.fluentbit.accessLog.stdoutReadable }}readable{{ else }}main{{ end }}; + access_log syslog:server=127.0.0.1:{{ .Values.fluentbit.accessLog.syslogPort }} main{{ else }}main{{ end }}; sendfile on; #tcp_nopush on; diff --git a/helm/templates/_fluentbit.tpl b/helm/templates/_fluentbit.tpl index acfb209..34d6f2b 100644 --- a/helm/templates/_fluentbit.tpl +++ b/helm/templates/_fluentbit.tpl @@ -8,6 +8,10 @@ advertises when the feature is enabled. Its health never gates nginx. image: {{ include "nginx.cloudProviderDockerRegistryUrl" $ }}{{ .repository }}:{{ .tag }} {{- end }} imagePullPolicy: {{ .Values.fluentbit.image.pullPolicy }} + volumeMounts: + - name: fluentbit-config + mountPath: /fluent-bit/etc/fluent-bit.conf + subPath: fluent-bit.conf ports: - name: metrics containerPort: {{ .Values.fluentbit.accessLog.metrics.port }} diff --git a/helm/templates/_helpers.tpl b/helm/templates/_helpers.tpl index b351ece..d43a1bd 100644 --- a/helm/templates/_helpers.tpl +++ b/helm/templates/_helpers.tpl @@ -157,6 +157,32 @@ split_clients "$otel_trace_id" $ratio_sampler { } {{- end -}} +{{/* +Compile the access-log forwarding rules (fluentbit.accessLog.forward) into a single regex +for Fluent Bit's grep filter, matched against the HTTP status code. serverErrors adds 5xx +(`5`), clientErrors adds 4xx (`4`), and each explicit statusCodes entry is added verbatim; a +status matches when it starts with any alternative (e.g. `^(5|429)`). When no rule is active +the regex is `^$`, which matches only an empty string — so every real (non-empty) status is +dropped and nothing is forwarded. +*/}} +{{- define "nginx.fluentbit.accessLogRegex" -}} +{{- $parts := list -}} +{{- if .Values.fluentbit.accessLog.forward.serverErrors -}} +{{- $parts = append $parts "5" -}} +{{- end -}} +{{- if .Values.fluentbit.accessLog.forward.clientErrors -}} +{{- $parts = append $parts "4" -}} +{{- end -}} +{{- range .Values.fluentbit.accessLog.forward.statusCodes -}} +{{- $parts = append $parts (toString .) -}} +{{- end -}} +{{- if $parts -}} +^({{ join "|" $parts }}) +{{- else -}} +^$ +{{- end -}} +{{- end -}} + {{/* Generate OpenTelemetry trace configuration */}} diff --git a/helm/templates/deployment.yaml b/helm/templates/deployment.yaml index a1a995d..c1c00f9 100644 --- a/helm/templates/deployment.yaml +++ b/helm/templates/deployment.yaml @@ -48,6 +48,11 @@ spec: {{- if .Values.extraVolumes }} {{ tpl (toYaml .Values.extraVolumes) . | nindent 8 }} {{- end }} + {{- if .Values.fluentbit.enabled }} + - name: fluentbit-config + configMap: + name: {{ printf "%s-fluentbit-configmap" (include "nginx.fullname" .) }} + {{- end }} containers: - name: nginx {{- with .Values.image }} diff --git a/helm/templates/fluentbit-configmap.yaml b/helm/templates/fluentbit-configmap.yaml new file mode 100644 index 0000000..f8b3b4f --- /dev/null +++ b/helm/templates/fluentbit-configmap.yaml @@ -0,0 +1,10 @@ +{{- if .Values.fluentbit.enabled }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ printf "%s-fluentbit-configmap" (include "nginx.fullname" .) }} + labels: + {{- include "nginx.labels" . | nindent 4 }} +data: + fluent-bit.conf: {{ tpl (.Files.Get "config/fluent-bit.conf") . | quote }} +{{- end }} diff --git a/helm/values.md b/helm/values.md index d6e0cb3..4fb8703 100644 --- a/helm/values.md +++ b/helm/values.md @@ -32,11 +32,19 @@ A Helm chart for nginx | environment | string | `"development"` | Specify the environment for this deployment | | extraVolumeMounts | list | `[]` | List of extra volumeMounts that are added to the NGINX container | | extraVolumes | list | `[]` | List of extra volumes that are added to the Deployment | +| fluentbit.accessLog.forward.clientErrors | bool | `false` | Forward client errors (4xx) to central Alloy. | +| fluentbit.accessLog.forward.serverErrors | bool | `true` | Forward server errors (5xx) to central Alloy. On by default so incident-relevant logs reach Loki with no configuration. | +| fluentbit.accessLog.forward.statusCodes | list | `[]` | Extra explicit status codes to forward beyond the error classes, e.g. [429, 499]. Everything not matched by any rule is dropped. | | fluentbit.accessLog.metrics.port | int | `2021` | Port on which Fluent Bit serves the merged Prometheus /metrics endpoint. When the sidecar is enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). | +| fluentbit.accessLog.stdoutReadable | bool | `true` | When true, nginx writes a human-readable (combined-style) access log to stdout for kubectl logs; the JSON log always goes to Fluent Bit over syslog. When false, stdout also receives the JSON format. | +| fluentbit.accessLog.syslogPort | int | `5514` | UDP port on loopback where Fluent Bit's syslog input listens and to which nginx forwards its JSON-formatted access log. | | fluentbit.enabled | bool | `false` | Enable or disable the optional Fluent Bit log-processing sidecar. When enabled, the central log-scraping label is forced off (no override) and the Prometheus scrape port is pointed at Fluent Bit's merged /metrics endpoint. When disabled (the default), the chart behaves exactly as before. | | fluentbit.image.pullPolicy | string | `"IfNotPresent"` | Image pull policy for the Fluent Bit sidecar | | fluentbit.image.repository | string | `"common/fluent-bit"` | Docker image name for the Fluent Bit sidecar | | fluentbit.image.tag | string | `"5.0.7"` | Docker image tag for the Fluent Bit sidecar | +| fluentbit.output.logs.host | string | `""` | Host of the dedicated central Alloy OTLP logs endpoint Fluent Bit forwards to. Deliberately NOT reused from opentelemetry.exporterHost (that is the traces-only endpoint). Required when the sidecar is enabled. | +| fluentbit.output.logs.port | int | `4318` | Port of the central Alloy OTLP/HTTP logs endpoint. | +| fluentbit.output.logs.protocol | string | `"http"` | Protocol used to reach the OTLP logs endpoint (http or https). | | fluentbit.resources.enabled | bool | `true` | Enable or disable resource limits and requests for the Fluent Bit sidecar | | fluentbit.resources.value.limits.cpu | string | `"100m"` | CPU limit for the Fluent Bit sidecar | | fluentbit.resources.value.limits.memory | string | `"128Mi"` | Memory limit for the Fluent Bit sidecar | diff --git a/helm/values.yaml b/helm/values.yaml index 2a40472..2314ac2 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -210,7 +210,26 @@ fluentbit: cpu: 100m # -- Memory request for the Fluent Bit sidecar memory: 128Mi + output: + logs: + # -- Host of the dedicated central Alloy OTLP logs endpoint Fluent Bit forwards to. Deliberately NOT reused from opentelemetry.exporterHost (that is the traces-only endpoint). Required when the sidecar is enabled. + host: "" + # -- Port of the central Alloy OTLP/HTTP logs endpoint. + port: 4318 + # -- Protocol used to reach the OTLP logs endpoint (http or https). + protocol: http accessLog: + # -- UDP port on loopback where Fluent Bit's syslog input listens and to which nginx forwards its JSON-formatted access log. + syslogPort: 5514 + # -- When true, nginx writes a human-readable (combined-style) access log to stdout for kubectl logs; the JSON log always goes to Fluent Bit over syslog. When false, stdout also receives the JSON format. + stdoutReadable: true + forward: + # -- Forward server errors (5xx) to central Alloy. On by default so incident-relevant logs reach Loki with no configuration. + serverErrors: true + # -- Forward client errors (4xx) to central Alloy. + clientErrors: false + # -- Extra explicit status codes to forward beyond the error classes, e.g. [429, 499]. Everything not matched by any rule is dropped. + statusCodes: [] metrics: # -- Port on which Fluent Bit serves the merged Prometheus /metrics endpoint. When the sidecar is enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). port: 2021 From 225a263741ae4416a26134ed4c284ca7551e06cc Mon Sep 17 00:00:00 2001 From: CptSchnitz <12687466+CptSchnitz@users.noreply.github.com> Date: Wed, 22 Jul 2026 19:02:33 +0300 Subject: [PATCH 04/22] feat: implemented step 3 --- helm/config/fluent-bit.conf | 38 +++++++++++++++++++++++++++++++++++-- helm/values.md | 5 ++++- helm/values.yaml | 32 ++++++++++++++++++++++++++++++- 3 files changed, 71 insertions(+), 4 deletions(-) diff --git a/helm/config/fluent-bit.conf b/helm/config/fluent-bit.conf index 14845b9..a671c44 100644 --- a/helm/config/fluent-bit.conf +++ b/helm/config/fluent-bit.conf @@ -4,6 +4,9 @@ # Access-log forwarding pipeline: nginx ships its JSON access log here over syslog UDP on # loopback (best-effort); Fluent Bit filters it down to the selected status codes and # forwards only those to central Alloy over OTLP. Everything else is dropped at the source. +# +# Metrics pipeline (fluentbit.accessLog.metrics): derive Prometheus metrics from the same +# stream, merge with the scraped nginx exporter, and serve one /metrics endpoint. [SERVICE] flush 1 @@ -20,7 +23,24 @@ Port {{ .Values.fluentbit.accessLog.syslogPort }} Buffer_Chunk_Size 32k Buffer_Max_Size 64k - +{{ if .Values.fluentbit.accessLog.metrics.enabled }} +# Fluent Bit's own metrics (records/bytes in and out, retries, uptime) — a rough signal for +# sizing the sidecar (compare input records against nginx's request count). +[INPUT] + Name fluentbit_metrics + Tag nginx.metrics.fluentbit + Scrape_Interval 30 +{{ if .Values.fluentbit.accessLog.metrics.scrapeExporter }} +# Scrape the existing nginx-prometheus-exporter so its stub_status gauges merge into the +# single /metrics endpoint below (Port = the exporter's own mclabels.prometheus.port). +[INPUT] + Name prometheus_scrape + Tag nginx.metrics.exporter + Host 127.0.0.1 + Port {{ .Values.mclabels.prometheus.port }} + Metrics_Path /metrics +{{ end }} +{{- end }} # The syslog message body is the nginx JSON line; decode it so the OTel-shaped fields # (including the nested http.response.status_code) become queryable by the grep filter. [FILTER] @@ -29,7 +49,11 @@ Key_Name message Parser json Reserve_Data On - +{{ if .Values.fluentbit.accessLog.metrics.enabled }} +# Derive metrics from the full stream, before grep drops the non-forwarded records, so the +# counters cover every request. Raw, overridable fluentbit.accessLog.metrics.config (tpl). +{{ tpl .Values.fluentbit.accessLog.metrics.config . -}} +{{ end }} # Keep only the status codes selected by fluentbit.accessLog.forward; drop everything else. # When no forwarding rule is active the regex matches nothing, so all records are dropped. [FILTER] @@ -48,3 +72,13 @@ {{- if eq .Values.fluentbit.output.logs.protocol "https" }} Tls On {{- end }} +{{- if .Values.fluentbit.accessLog.metrics.enabled }} + +# Serve the merged /metrics (log-derived + scraped exporter + Fluent Bit's own) on +# metrics.port. Match * is safe — prometheus_exporter only handles metric events, not logs. +[OUTPUT] + Name prometheus_exporter + Match * + Host 0.0.0.0 + Port {{ .Values.fluentbit.accessLog.metrics.port }} +{{- end }} diff --git a/helm/values.md b/helm/values.md index 4fb8703..74e30ea 100644 --- a/helm/values.md +++ b/helm/values.md @@ -35,7 +35,10 @@ A Helm chart for nginx | fluentbit.accessLog.forward.clientErrors | bool | `false` | Forward client errors (4xx) to central Alloy. | | fluentbit.accessLog.forward.serverErrors | bool | `true` | Forward server errors (5xx) to central Alloy. On by default so incident-relevant logs reach Loki with no configuration. | | fluentbit.accessLog.forward.statusCodes | list | `[]` | Extra explicit status codes to forward beyond the error classes, e.g. [429, 499]. Everything not matched by any rule is dropped. | -| fluentbit.accessLog.metrics.port | int | `2021` | Port on which Fluent Bit serves the merged Prometheus /metrics endpoint. When the sidecar is enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). | +| fluentbit.accessLog.metrics.config | string | per-status-code counter + request_time histogram (see values.yaml) | Raw `log_to_metrics` blocks (tpl-rendered) that derive metrics from the access log. Default: a per-status-code counter and a request_time histogram. WARNING: never label by high-cardinality fields (path, query, client IP, user-agent) — it can OOM the sidecar. | +| fluentbit.accessLog.metrics.enabled | bool | `true` | Derive Prometheus metrics from the access log and serve them on the merged /metrics endpoint. When false, no metrics blocks are rendered. | +| fluentbit.accessLog.metrics.port | int | `2021` | Port on which Fluent Bit serves the merged /metrics endpoint. When enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). | +| fluentbit.accessLog.metrics.scrapeExporter | bool | `true` | Scrape the existing nginx-prometheus-exporter and merge its series into the endpoint, preserving the stub_status gauges. | | fluentbit.accessLog.stdoutReadable | bool | `true` | When true, nginx writes a human-readable (combined-style) access log to stdout for kubectl logs; the JSON log always goes to Fluent Bit over syslog. When false, stdout also receives the JSON format. | | fluentbit.accessLog.syslogPort | int | `5514` | UDP port on loopback where Fluent Bit's syslog input listens and to which nginx forwards its JSON-formatted access log. | | fluentbit.enabled | bool | `false` | Enable or disable the optional Fluent Bit log-processing sidecar. When enabled, the central log-scraping label is forced off (no override) and the Prometheus scrape port is pointed at Fluent Bit's merged /metrics endpoint. When disabled (the default), the chart behaves exactly as before. | diff --git a/helm/values.yaml b/helm/values.yaml index 2314ac2..7d5cd58 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -231,8 +231,38 @@ fluentbit: # -- Extra explicit status codes to forward beyond the error classes, e.g. [429, 499]. Everything not matched by any rule is dropped. statusCodes: [] metrics: - # -- Port on which Fluent Bit serves the merged Prometheus /metrics endpoint. When the sidecar is enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). + # -- Derive Prometheus metrics from the access log and serve them on the merged /metrics endpoint. When false, no metrics blocks are rendered. + enabled: true + # -- Scrape the existing nginx-prometheus-exporter and merge its series into the endpoint, preserving the stub_status gauges. + scrapeExporter: true + # -- Port on which Fluent Bit serves the merged /metrics endpoint. When enabled, this becomes the pod's advertised scrape port (mclabels.prometheus.port). port: 2021 + # -- Raw `log_to_metrics` blocks (tpl-rendered) that derive metrics from the access log. Default: a per-status-code counter and a request_time histogram. WARNING: never label by high-cardinality fields (path, query, client IP, user-agent) — it can OOM the sidecar. + # @default -- per-status-code counter + request_time histogram (see values.yaml) + config: | + # Per-status-code request counter (status_code is bounded, safe to label). + [FILTER] + Name log_to_metrics + Match nginx.access + Tag nginx.metrics + Metric_Mode counter + Metric_Namespace nginx + Metric_Subsystem http + Metric_Name requests_total + Metric_Description Total nginx HTTP requests by response status code + Add_Label status_code $Attributes['http.response.status_code'] + + # Request-time histogram in seconds. Buckets default to Prometheus's standard set. + [FILTER] + Name log_to_metrics + Match nginx.access + Tag nginx.metrics + Metric_Mode histogram + Metric_Namespace nginx + Metric_Subsystem http + Metric_Name request_duration_seconds + Metric_Description nginx request processing time in seconds + Value_Field $Attributes['mapcolonies.request_time'] authorization: # -- Use authroization mechanism From 4a70265ec936f9dfcee31ed74e15de6a5882389b Mon Sep 17 00:00:00 2001 From: CptSchnitz <12687466+CptSchnitz@users.noreply.github.com> Date: Wed, 22 Jul 2026 19:40:12 +0300 Subject: [PATCH 05/22] feat: implemented step 4 --- helm/config/fluent-bit-parsers.conf | 16 +++++++++++ helm/config/fluent-bit.conf | 38 ++++++++++++++++++++++++- helm/config/nginx.conf | 5 ++++ helm/templates/_fluentbit.tpl | 5 ++++ helm/templates/fluentbit-configmap.yaml | 3 ++ helm/values.md | 3 ++ helm/values.yaml | 7 +++++ 7 files changed, 76 insertions(+), 1 deletion(-) create mode 100644 helm/config/fluent-bit-parsers.conf diff --git a/helm/config/fluent-bit-parsers.conf b/helm/config/fluent-bit-parsers.conf new file mode 100644 index 0000000..a251eb2 --- /dev/null +++ b/helm/config/fluent-bit-parsers.conf @@ -0,0 +1,16 @@ +# Custom Fluent Bit parsers shipped by the chart (loaded via an extra `parsers_file` in the +# [SERVICE] section of fluent-bit.conf). The image's default parsers.conf (json, syslog, …) +# stays loaded alongside this file. + +# Parses the standard open-source nginx error-log line into structured fields. Format: +# 2024/01/15 10:23:45 [error] 1234#5678: *90 , client: …, server: …, request: "…" +# The leading timestamp is optional (some syslog paths strip it), the connection id (*cid) is +# optional (worker/startup messages omit it), and the client/server/request tail is optional +# and matched as a unit so a comma inside does not truncate it. +[PARSER] + Name nginx_error + Format regex + Regex ^(?:(?