[zanejohnson-azure]

Summary

On containerd, ContainerLogV2 multiline doesn't group Java (and .NET) stack traces whose exception header has no message, for example a bare java.lang.NullPointerException. This PR fixes that. Reported in ICM 21000002363585.

Root cause

• CRI lines have no trailing newline. On containerd, main.sh and main.ps1 switch the tail input to Multiline.Parser cri. CRI log values have no trailing \n; Docker json-file values do.
• The built-in start rule needs : or a newline after the class name. fluent-bit's built-in java start rule is /(.)(?:Exception|Error|Throwable|V8 errors stack trace)[:\r\n]/. On CRI, a header that ends with the exception class never matches. Examples:
  • a bare java.lang.NullPointerException
  • Exception in thread "main" java.lang.X
  • a bare ...Error
  • a .NET ---> System.X with no message
• Stale parser state explains the customer's shape. Upstream doesn't reset multiline rule state on a non-matching line. The \tat frames after the unmatched header can then group under state left by an earlier trace. The result is the header alone, then a record that starts with \tat ....
• Why most traces are fine. Headers with : message match, so most traces group correctly. Docker-runtime clusters aren't affected.

This isn't a documented ContainerLogV2 multiline limitation. The trace is a standard Java exception, and the same input groups correctly on Docker.

Fix

• New parser. azm-containers-parser-multiline.conf gets azmon_java, a copy of the built-in java parser (src/multiline/flb_ml_parser_java.c). Its start rule also matches at end of line, (?:[:\r\n]|$); every other rule is unchanged.
• Customizers. fluent-bit-conf-customizer.rb and fluent-bit-geneva-conf-customizer.rb render java as azmon_java in multiline.parser. For example, go,java,python becomes go,azmon_java,python.
• Why a new name. A custom [MULTILINE_PARSER] named java is shadowed by the built-in one.
• No ConfigMap change. AZMON_MULTILINE_LANGUAGES and telemetry still report java. dotnet already maps to java, so it also gets the fix. Linux and Windows share the parser file and customizers.

Validation

Production conf A/B on containerd.

• Setup: the customer's agent image (3.8.0, fluent-bit 5.0.4) with Linux fluent-bit.conf rendered by the customizer from ci_prod vs. this branch. The main.sh CRI sed, the real tail input, and the real filters are used; oms outputs are replaced with stdout.
• Table: number of records emitted.

Scenario	ci_prod	This PR	Docker runtime, built-in java*
bare NPE	6	3	3
bare NPE after an earlier trace (customer shape)	7	6	6
NPE with message	2	2	2
bare Exception in thread "main" X	4	2	2
Caused by chain	2	2	2
bare XError	4	2	2
.NET with message	2	2	2
.NET bare inner exception	4	3	3
false-positive guard (INFO ... Error + normal line)	2	2	2
nested exception is:	5	3	3

* Docker reference values come from the same inputs in a minimal multiline harness. On Docker, azmon_java output was identical to the built-in parser in every scenario, so there is no regression.

• Result: containerd now matches Docker in all 10 scenarios.
• Customer shape, before: the NPE header alone, then a 3-line record starting \tat ....
• Customer shape, after: the header and its 3 frames in one 4-line record.

Unit test. The new fluent-bit-conf-customizer_test.rb is picked up by test/unit-tests/test_driver.rb in the Ruby-Tests job. It checks that:

• both customizers, on the Linux and Windows confs, render multiline.parser go,azmon_java,python without logging config::error
• language lists without java are unchanged
• disabled multiline keeps the placeholders
• the azmon_java start rule matches message-less headers without a trailing newline (Ruby uses Onigmo, the same regex engine as fluent-bit)

With the ci_prod customizers and parser file, the test fails (2 failures); on this branch it passes. The full Ruby suite (test_driver.rb) also passes: 96 runs, 0 failures, 0 errors.

Remaining before merge

• Check the Windows image; fluent-bit 5.0.3 has the same built-in java parser.
• Run the multiline-validation A/B on a test cluster.

Follow-ups (not in this PR)

• File upstream fluent-bit issues:
  • the java start rule should accept end of line
  • multiline rule state should reset on a non-matching line
• Keep azmon_java in sync with the built-in parser on fluent-bit upgrades (noted in the conf comment).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[zanejohnson-azure]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

[zanejohnson-azure]

Cluster A/B validation: PASS

I ran the repo's multiline-validation A/B on an AKS test cluster: containerd, ContainerLogV2, MSI auth, one Ubuntu 24.04 node and one Windows Server 2022 node. Multiline was enabled for java, python, dotnet and go.

• OLD: prod ciprod:3.9.0-ci-prod-10-07-2026-cd634797 (plus win-).
• NEW: this PR. Images were built by a manual run of pipeline 444 on the branch, since PR builds don't push:
  • cidev:3.9.0-zane-multiline-java-cri-bare-header-10-21-2026-3447f118
  • cidev:win-3.9.0-zane-multiline-java-cri-bare-header-10-21-2026-3447f118-ltsc2022
• Controls:
  • Fluent-bit is the same in OLD and NEW: 5.0.4 on Linux, 5.0.3 on Windows.
  • NEW differs from OLD only by this PR's commit and [Generated by SRE Agent] docs: fill 3.9.0 image tag and Azure Linux version in release notes #1798, which touches ReleaseNotes.md only.
  • Both used the same jobs, redeployed after each image swap, and a 20-minute Log Analytics window starting at the same offset.
• Jobs:
  • The 8 standard jobs in test/scenario/multiline/. They only emit exception headers that carry a message.
  • An ad-hoc java-bare job on each OS. About once a second it prints a SEVERE line, then a bare java.lang.NullPointerException header, then 3 \tat frames.

Lang	OS	Stitched % OLD → NEW	MaxLen OLD → NEW	MaxLines OLD → NEW	Result
java-bare	Linux	0 → 50	91 → 225	1 → 4	Fixed
java-bare	Windows	0 → 50	91 → 225	1 → 4	Fixed
java	Linux / Windows	100 / 100, unchanged	167 / 168, unchanged	3, unchanged	PASS
dotnet	Linux / Windows	50 / 33.3, unchanged	538, unchanged	6, unchanged	PASS
golang	Linux / Windows	33.3 / 33.3, unchanged	238, unchanged	7, unchanged	PASS
python	Linux / Windows	33.3 / 50, unchanged	316, unchanged	6, unchanged	PASS

java-bare records per 20-minute window (Linux / Windows):

Record shape	OLD	NEW
SEVERE line	1199 / 1188	1199 / 1188
header and 3 frames as one 4-line record	0 / 0	1199 / 1188
orphan header	1199 / 1188	0 / 0
orphan \tat frame	3597 / 3564	0 / 0

• NEW reads 50% because each iteration produces one single-line SEVERE record plus one trace. Every trace is stitched.
• Iteration counts match OLD, and no other record shapes appeared, so nothing was lost or duplicated.

Also verified

• The rendered agent config on both OSes has multiline.parser azmon_java,python,go and the azmon_java parser block.
• Fluent-bit logged no errors on either OS.
  • Windows logged a few flush retries at startup while addon-token-adapter-win restarted once; it also restarted once under OLD.
  • The retries cleared before the jobs started.
• The PR build and the manual image build are green.

Not run on a node

• The Geneva conf path, which is covered by the unit test.
• Windows ltsc2019 and Linux arm64. The change is config-only, and every image variant is built from the same files.

This covers both "Remaining before merge" items: the Windows image check and the multiline-validation A/B.

[github-actions]

This PR is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.