Skip to content

Upgrade Windows Telegraf to 1.40.0 - #1785

Merged
suyadav1 merged 4 commits into
ci_prodfrom
zanejohnson-azure-windows-telegraf-correctness-fix
Sep 24, 2026
Merged

suyadav1 merged 4 commits into
ci_prodfrom
zanejohnson-azure-windows-telegraf-correctness-fix

Conversation

@zanejohnson-azure

@zanejohnson-azure zanejohnson-azure commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Upgrade the official Windows Telegraf package from 1.24.2 to 1.40.0 for discovery-token file rereading on a failed-poll retry. No Telegraf source patch or new production dependency.

Why the remaining changes are needed

File / area Reason and upstream reference
kubernetes/windows/setup.ps1 Download the official 1.40.0 archive; pin its SHA256 and fail on installation errors.
build/windows/installer/conf/telegraf.conf Preserve the overall 15-second scrape timeout. 1.24.2 assigned response_timeout to http.Client.Timeout; 1.40 documentation says it now covers headers only. Therefore use timeout = "15s".
build/common/installer/scripts/tomlparser-prom-customconfig.rb Remove the old Windows-only response_timeout override for generated namespace plugins. Linux already used timeout; Linux output, filter names/values and public ConfigMap keys are unchanged.
build/windows/installer/conf/telegraf-ama-logs-process-metrics.conf The optional service shares the upgraded executable. Replace removed pid_tag with the documented tag_with = ["pid"]. 1.40's option schema no longer has pid_tag. CPU/RSS filters, intervals and output stay unchanged.
main.ps1 + telegraf-windows-service.rb Stock 1.40 uses native service detection that requires a services.exe parent in session 0; the observed container uses session 5 and failed SCM startup. Reuse the bundled Ruby service dispatcher to run the unchanged executable with --console, handling both service roles, child exit and bounded logging/shutdown.
telegraf-windows-console.rb Provide native stop signaling and Windows job containment. Enroll the host before spawning so children cannot escape during startup. This is a helper library, not another service.
Tests, runners, docs and DevSkim comments Cover package/config preservation and service lifecycle; document these compatibility reasons and narrowly suppress public-checksum/test-loopback false positives.

Latest six-hour results

Commit df6e8d1e, build 126849: WS2022 observation on September 22, 00:03-06:03 UTC, with Prometheus and process-metrics collection both enabled.

  • Process metrics working: both services started normally. Application Insights received 360 CPU + 360 RSS measurements each for both Telegraf children and MonAgentCore, with 72/72 five-minute bins populated per series.
  • Data delivery working: ContainerLogV2: 21,413 rows; InsightsMetrics: 216,000 rows, each with 72/72 nonempty bins. No sustained delivery gap was observed.
  • No earlier rapid handle-growth pattern observed: Prometheus Telegraf 263 -> 282; process-metrics Telegraf 449 -> 403 (range 401-494). All recorded process identities were unchanged.

Consume the official upstream Windows package with a pinned SHA256 and migrate timeout, field-filter, and procstat PID-tag configuration. Preserve ConfigMap keys and Linux behavior; document the deferred discovery cleanup and deadline gaps.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Comment thread kubernetes/windows/setup.ps1 Fixed
Comment thread build/common/installer/scripts/tomlparser-prom-customconfig_test.rb Fixed
zanejohnson-azure and others added 3 commits September 17, 2026 23:53
Host the unchanged upstream executable with the existing win32-service dispatcher instead of relying on Telegraf's session-zero service detection. Preserve both service roles, bound shutdown and log growth, and contain the child in a kill-on-close job. Cover registration and lifecycle behavior without relaxing TLS, authentication, strict config parsing, or protected-memory defaults.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Enroll the service host in its kill-on-close job before creating the Telegraf child, retain the non-inheritable handle until host exit, and cover abrupt and normal startup-boundary exits with local Windows process tests. Wire Windows worker tests into the existing Ruby driver and narrowly suppress the public-checksum and test-loopback DevSkim false positives.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae
Remove the optional fieldinclude/fieldexclude migration and its OS-specific rendering/test helper. Preserve fieldpass/fielddrop and the original timeout variable while retaining the required overall-timeout and procstat PID-tag fixes. Add pinned upstream references for the compatibility rationale.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae
@zanejohnson-azure
zanejohnson-azure marked this pull request as ready for review September 22, 2026 20:09
@zanejohnson-azure
zanejohnson-azure requested a review from a team as a code owner September 22, 2026 20:09
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@suyadav1

Copy link
Copy Markdown
Contributor

Did we test the fix for WS2022 and WS2025 both? We should merge #1782 and verify there are no issues for ws2025?

@suyadav1

Copy link
Copy Markdown
Contributor

Can we add a check to kill Telegraf / restart the container if the number of handlers are increasing so it doesn't impact other processes?

@zanejohnson-azure

Copy link
Copy Markdown
Contributor Author

Can we add a check to kill Telegraf / restart the container if the number of handlers are increasing so it doesn't impact other processes?

i would not prefer to swallow it when it happens. if we hit handle increasing issue in the future, i would prefer to let it fail and then we need do the actual fix.

@zanejohnson-azure

Copy link
Copy Markdown
Contributor Author

Did we test the fix for WS2022 and WS2025 both? We should merge #1782 and verify there are no issues for ws2025?

synced offline, i will do the win25 fix after this PR is merged. will rebase win25 fix PR and validate then.

@suyadav1
suyadav1 merged commit b430069 into ci_prod Sep 24, 2026
20 checks passed
suyadav1 pushed a commit that referenced this pull request Sep 24, 2026
* Upgrade Windows Telegraf to 1.40.0

Consume the official upstream Windows package with a pinned SHA256 and migrate timeout, field-filter, and procstat PID-tag configuration. Preserve ConfigMap keys and Linux behavior; document the deferred discovery cleanup and deadline gaps.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix Telegraf service startup in Windows containers

Host the unchanged upstream executable with the existing win32-service dispatcher instead of relying on Telegraf's session-zero service detection. Preserve both service roles, bound shutdown and log growth, and contain the child in a kill-on-close job. Cover registration and lifecycle behavior without relaxing TLS, authentication, strict config parsing, or protected-memory defaults.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Close Telegraf service startup containment gap

Enroll the service host in its kill-on-close job before creating the Telegraf child, retain the non-inheritable handle until host exit, and cover abrupt and normal startup-boundary exits with local Windows process tests. Wire Windows worker tests into the existing Ruby driver and narrowly suppress the public-checksum and test-loopback DevSkim false positives.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae

* Keep existing Telegraf field filters in Windows upgrade

Remove the optional fieldinclude/fieldexclude migration and its OS-specific rendering/test helper. Preserve fieldpass/fielddrop and the original timeout variable while retaining the required overall-timeout and procstat PID-tag fixes. Add pinned upstream references for the compatibility rationale.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 08a78bbb-f751-41cb-87b6-96aaf4c1a1ae
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants