Upgrade Windows Telegraf to 1.40.0

Dev Guide.md

@@ -4,6 +4,58 @@ More advanced information needed to develop or build the docker provider will li
 
 <!-- TODO: eventually move dev info from README.md to here-->
 
+## Windows Telegraf dependency
+
+`kubernetes/windows/setup.ps1` installs the official Telegraf 1.40.0 Windows AMD64
+ZIP and verifies its pinned SHA256 before extraction. The package corresponds to
+upstream commit `e9017dc3266369d6fa185e0e130af1d1d4021ce9`. The existing Windows
+pipeline continues to sign `C:\opt\telegraf\telegraf.exe` as an OSS dependency.
+
+The official binary is built with Go 1.27.0 for `windows/amd64`, `GOAMD64=v1`.
+Go's [Windows OS floor](https://go.dev/wiki/MinimumRequirements#windows) is Windows
+10 or Windows Server 2016 and newer. Both repository image targets, LTSC2019 and
+LTSC2022, meet that floor; this does not replace validation inside those images
+or in installed-service mode.
+
+The Windows entrypoint registers both Telegraf services through
+`telegraf-windows-service.rb`, using the same already-installed `win32-service`
+dispatcher as Fluentd. Telegraf 1.40's native service detection requires its
+`services.exe` parent to be in session 0, which is not guaranteed in Windows containers.
+The host runs the unchanged executable with `--console`, monitors child exit, and
+forwards SCM stop through the child's private console/process group. Shutdown is
+bounded; the host joins a kill-on-close job before spawning, so the child inherits
+containment at creation, including if the host dies before `spawn` returns. The
+non-inheritable job handle stays open until host process exit so console cleanup
+does not kill the host before SCM shutdown completes. The service
+PID is the Ruby host; the Telegraf PID is its child. Role-only host arguments keep
+the existing procstat config-path filters selecting Telegraf rather than Ruby.
+Per-role logs under `C:\opt\telegraf\logs` rotate at 5 MiB with two backups.
+The native startup-boundary regression uses only local sleeping Ruby processes:
+`ruby build/windows/installer/scripts/telegraf-windows-console_test.rb`.
+It runs on Windows with the image's existing `ffi` dependency and skips on Linux.
+
+Upgrade the two configurations in `build/windows/installer/conf/` and
+`tomlparser-prom-customconfig.rb` together with the binary. Windows uses `timeout`
+to preserve the overall 15-second metric-scrape limit; upstream's
+[1.40 documentation](https://github.com/influxdata/telegraf/blob/e9017dc3266369d6fa185e0e130af1d1d4021ce9/plugins/inputs/prometheus/README.md#L152-L157)
+explains that `response_timeout` now covers headers only, unlike the
+[1.24.2 client timeout](https://github.com/influxdata/telegraf/blob/9550e7a533dd00632e14435e87ed3eb4b04832c6/plugins/inputs/prometheus/prometheus.go#L254-L261).
+Procstat uses `tag_with = ["pid"]` to retain PID tags. Existing `fieldpass`/`fielddrop`
+names are unchanged on both OSes because
+[1.40 still parses them](https://github.com/influxdata/telegraf/blob/e9017dc3266369d6fa185e0e130af1d1d4021ce9/config/config.go#L1649-L1687).
+Linux rendering remains unchanged. Run `ruby build/common/installer/scripts/tomlparser-prom-customconfig_test.rb`
+for rendering coverage with and without namespace filters. On Windows, set
+`TELEGRAF_WINDOWS_BINARY` to the extracted `telegraf.exe` to also load the generated
+configs with that binary in bounded `--test` mode, without Kubernetes access or
+running output plugins.
+
+This stock upgrade is a partial mitigation: node discovery rereads the token file
+on retries after a failed poll, without relying on file modification time. The
+[1.40.0 discovery code](https://github.com/influxdata/telegraf/blob/e9017dc3266369d6fa185e0e130af1d1d4021ce9/plugins/inputs/prometheus/kubernetes.go)
+still omits response cleanup on non-200 status codes and lacks an explicit
+discovery request timeout. The metric-scrape `timeout` does not bound that path.
+HTTP/2 negotiation is not a substitute for fixing those remaining issues.
+
 ## Testing
 Last updated 8/18/2021
 

build/common/installer/scripts/tomlparser-prom-customconfig.rb

@@ -92,8 +92,8 @@ def checkForType(variable, varType)
 
 # Telegraf parses interval values with Go's time.ParseDuration, which accepts one or more
 # decimal numbers each followed by a unit suffix (for example "30s", "1.5s" or "1h30m").
-# The unit set below is the intersection of what the Linux and Windows agents accept; "d"
-# is deliberately excluded because the older telegraf shipped on Windows rejects it.
+# Keep the existing config-map duration contract on both Linux and Windows; "d" remains
+# deliberately excluded even when newer telegraf versions accept days.
 # The anchors must be \A and \z (not ^ and $) so that a value such as "1m\n<injected toml>"
 # cannot pass validation by matching only its first line.
 TELEGRAF_DURATION_REGEX = /\A(?:(?:\d+(?:\.\d+)?|\.\d+)(?:ns|us|\u00B5s|\u03BCs|ms|s|m|h))+\z/
@@ -187,10 +187,6 @@ def createPrometheusPluginsWithNamespaceSetting(monitorKubernetesPods, monitorKu
     new_contents = new_contents.gsub("$AZMON_TELEGRAF_CUSTOM_PROM_SCRAPE_SCOPE", "# Commenting this out since new plugins will be created per namespace\n  # $AZMON_TELEGRAF_CUSTOM_PROM_SCRAPE_SCOPE")
 
     timeout_config_key = "timeout"
-    if is_windows?
-      # For windows, the timeout config key is different because of old version of telegraf
-      timeout_config_key = "response_timeout"
-    end
 
     pluginConfigsWithNamespaces = ""
     podScrapeScope = (@controller.casecmp(@replicaset) == 0) ? "cluster" : "node"

build/common/installer/scripts/tomlparser-prom-customconfig_test.rb

@@ -91,7 +91,8 @@ def run_parser(scenario, configmap)
         "SIDECAR_SCRAPING_ENABLED" => nil,
       }.merge(spec[:env])
 
-      stdout, stderr, = Open3.capture3(env, RbConfig.ruby, parser, chdir: sandbox)
+      stdout, stderr, status = Open3.capture3(env, RbConfig.ruby, parser, chdir: sandbox)
+      assert status.success?, "config parser failed: #{stdout}\n#{stderr}"
 
       telemetry_path = File.join(sandbox, "telemetry_prom_config_env_var")
       result = {
@@ -170,7 +171,7 @@ def test_invalid_telegraf_durations_are_rejected
       "1m;id",                  # shell metacharacters
       "$(id)",
       "`id`",
-      "1d",                     # rejected: the telegraf shipped on windows does not support days
+      "1d",                     # rejected by the existing config-map duration contract
       "1",                      # missing unit
       "m",                      # missing number
       "",
@@ -307,6 +308,98 @@ def test_namespace_plugins_are_generated_for_valid_namespaces
     assert_no_injected_plugins(:replicaset, result[:conf], "a benign namespace configuration")
   end
 
+  [:replicaset, :sidecar, :windows].each do |scenario|
+    { unfiltered: nil, empty_filter: [], namespace_filtered: ["default", "kube-system"] }.each do |name, namespaces|
+      define_method("test_prometheus_timeout_and_mapping_#{scenario}_#{name}") do
+        body = "monitor_kubernetes_pods = true\n" \
+               "interval = \"45s\"\n" \
+               "fieldpass = [\"requests_total\"]\n" \
+               "fielddrop = [\"debug_total\"]\n" \
+               "kubernetes_label_selector = \"app=metrics\"\n" \
+               "kubernetes_field_selector = \"spec.nodeName=test-node\""
+        body += "\nmonitor_kubernetes_pods_namespaces = #{namespaces.inspect}" unless namespaces.nil?
+
+        result = run_parser(scenario, configmap_for(scenario, body))
+        plugins = parse_generated_toml(result[:conf])["inputs"]["prometheus"]
+        monitored = plugins.select { |plugin| plugin["monitor_kubernetes_pods"] }
+        expected_namespaces = namespaces.nil? || namespaces.empty? ? [nil] : namespaces
+        assert_equal expected_namespaces, monitored.map { |plugin| plugin["monitor_kubernetes_pods_namespace"] }
+
+        monitored.each do |plugin|
+          assert_equal "15s", plugin["timeout"], "the overall scrape timeout must remain 15s"
+          refute plugin.key?("response_timeout"), "response_timeout only bounds response headers in current Telegraf"
+          assert_equal "45s", plugin["interval"]
+          assert_equal 2, plugin["metric_version"]
+          assert_equal "scrapeUrl", plugin["url_tag"]
+          assert_equal "pod_namespace", plugin["pod_namespace_label_name"]
+          assert_equal (scenario == :replicaset ? "cluster" : "node"), plugin["pod_scrape_scope"]
+          assert_equal ["requests_total"], plugin["fieldpass"]
+          assert_equal ["debug_total"], plugin["fielddrop"]
+          assert_equal "app=metrics", plugin["kubernetes_label_selector"]
+          assert_equal "spec.nodeName=test-node", plugin["kubernetes_field_selector"]
+        end
+
+        if scenario == :windows
+          plugins.each do |plugin|
+            assert_equal "15s", plugin["timeout"], "the base Windows plugin must also retain the overall timeout"
+            refute plugin.key?("response_timeout")
+          end
+        end
+      end
+    end
+  end
+
+  def test_windows_rendered_configs_load_in_packaged_telegraf
+    binary = ENV["TELEGRAF_WINDOWS_BINARY"]
+    skip "Set TELEGRAF_WINDOWS_BINARY to run the Windows binary config smoke test" if binary.nil? || binary.empty?
+    assert File.file?(binary), "TELEGRAF_WINDOWS_BINARY must point to telegraf.exe"
+
+    [nil, [], ["default", "kube-system"]].each do |namespaces|
+      body = "monitor_kubernetes_pods = true\nfieldpass = [\"requests_total\"]\nfielddrop = [\"debug_total\"]"
+      body += "\nmonitor_kubernetes_pods_namespaces = #{namespaces.inspect}" unless namespaces.nil?
+      conf = run_parser(:windows, configmap_for(:windows, body))[:conf]
+
+      # This smoke test loads the actual generated options without contacting Kubernetes
+      # or requiring a mounted service-account CA. --test never runs output plugins.
+      conf = conf.gsub(/^(\s*monitor_kubernetes_pods\s*=\s*)true[ \t]*$/) { "#{Regexp.last_match(1)}false" }
+      conf = conf.gsub(/^(\s*tls_ca\s*=\s*)"[^"]*"[ \t]*$/) { "#{Regexp.last_match(1)}\"\"" }
+
+      Dir.mktmpdir("windows-telegraf-config") do |dir|
+        path = File.join(dir, "telegraf.conf")
+        File.write(path, conf)
+        Open3.popen3({ "NODE_IP" => "127.0.0.1" }, binary, "--console", "--test", "--config", path) do |stdin, stdout, stderr, process| # DevSkim: ignore DS162092 -- Loopback-only config smoke test.
+          stdin.close
+          readers = [Thread.new { stdout.read }, Thread.new { stderr.read }]
+          completed = !process.join(20).nil?
+          Process.kill("KILL", process.pid) unless completed
+          output = readers.map(&:value).join("\n")
+          assert completed, "Telegraf config smoke test exceeded 20 seconds"
+          assert process.value.success?, "Telegraf rejected the rendered config for #{namespaces.inspect}: #{output}"
+        end
+      end
+    end
+  end
+
+  def test_windows_process_metrics_config_preserves_pid_tags_and_fields
+    path = File.join(REPO_ROOT, "build/windows/installer/conf/telegraf-ama-logs-process-metrics.conf")
+    config = Tomlrb.load_file(path)
+    assert_equal({ "telegraf_role" => "ama-logs-process-metrics" }, config["global_tags"])
+    assert_equal 11, config["inputs"]["procstat"].length
+    config["inputs"]["procstat"].each do |plugin|
+      assert_equal ["pid"], plugin["tag_with"]
+      refute plugin.key?("pid_tag")
+      assert_equal ["cpu_usage", "memory_rss"], plugin["fieldpass"]
+      assert_equal "native", plugin["pid_finder"]
+      assert_equal "agent_telemetry", plugin["name_override"]
+      assert_equal "t.azm.ms/", plugin["name_prefix"]
+      assert_equal "DaemonSet-Windows", plugin["tags"]["ControllerType"]
+    end
+    assert_equal(
+      { "ai.cloud.role" => "ControllerType", "ai.cloud.roleInstance" => "PodName" },
+      config["outputs"]["application_insights"].first["context_tag_sources"]
+    )
+  end
+
   def test_selectors_are_escaped_in_generated_namespace_plugins
     body = "monitor_kubernetes_pods = true\n" \
            "monitor_kubernetes_pods_namespaces = [\"default\"]\n" \

build/windows/installer/conf/telegraf-ama-logs-process-metrics.conf

@@ -20,7 +20,7 @@
   exe = "fluent-bit"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -36,7 +36,7 @@
   pattern = "telegraf.conf"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -52,7 +52,7 @@
   pattern = "telegraf-ama-logs-process-metrics.conf"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -68,7 +68,7 @@
   exe = "MonAgentLauncher"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -84,7 +84,7 @@
   exe = "MonAgentCore"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -100,7 +100,7 @@
   exe = "MonAgentHost"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -116,7 +116,7 @@
   exe = "MonAgentManager"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -132,7 +132,7 @@
   exe = "AzurePerfCollectorExtension"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -148,7 +148,7 @@
   exe = "AzureProfilerExtension"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -164,7 +164,7 @@
   exe = "AggregatorHost"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]
@@ -180,7 +180,7 @@
   exe = "powershell"
   interval = "60s"
   pid_finder = "native"
-  pid_tag = true
+  tag_with = ["pid"]
   name_override = "agent_telemetry"
   fieldpass = ["cpu_usage", "memory_rss"]
   [inputs.procstat.tags]

build/windows/installer/conf/telegraf.conf

@@ -149,8 +149,8 @@
   ## OR
   # bearer_token_string = "abc_123"
 
-  ## Specify timeout duration for slower prometheus clients (default is 3s)
-  response_timeout = "15s"
+  ## Overall HTTP timeout for metric scrapes, including the response body.
+  timeout = "15s"
 
   ## Optional TLS Config
   tls_ca = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

build/windows/installer/scripts/telegraf-windows-console.rb

@@ -0,0 +1,76 @@
+require "ffi"
+
+class TelegrafConsole
+  module Native
+    extend FFI::Library
+    ffi_lib "kernel32"
+    ffi_convention :stdcall
+    attach_function :GetConsoleProcessList, [:pointer, :ulong], :ulong
+    attach_function :AllocConsole, [], :bool
+    attach_function :FreeConsole, [], :bool
+    attach_function :GenerateConsoleCtrlEvent, [:ulong, :ulong], :bool
+    attach_function :GetLastError, [], :ulong
+    attach_function :GetCurrentProcess, [], :pointer
+    attach_function :CreateJobObjectW, [:pointer, :pointer], :pointer
+    attach_function :SetInformationJobObject, [:pointer, :int, :pointer, :ulong], :bool
+    attach_function :AssignProcessToJobObject, [:pointer, :pointer], :bool
+    attach_function :CloseHandle, [:pointer], :bool
+
+    class BasicLimits < FFI::Struct
+      layout :process_time, :int64, :job_time, :int64, :flags, :uint32,
+             :min_working_set, :size_t, :max_working_set, :size_t,
+             :active_processes, :uint32, :affinity, :size_t,
+             :priority, :uint32, :scheduling, :uint32
+    end
+
+    class ExtendedLimits < FFI::Struct
+      layout :basic, BasicLimits, :io_counters, [:uint64, 6],
+             :process_memory, :size_t, :job_memory, :size_t,
+             :peak_process_memory, :size_t, :peak_job_memory, :size_t
+    end
+  end
+
+  def prepare
+    buffer = FFI::MemoryPointer.new(:ulong, 1)
+    if Native.GetConsoleProcessList(buffer, 1) == 0
+      unless Native.AllocConsole
+        raise SystemCallError.new("AllocConsole", Native.GetLastError)
+      end
+      @allocated = true
+    end
+    @job = Native.CreateJobObjectW(nil, nil)
+    raise SystemCallError.new("CreateJobObject", Native.GetLastError) if @job.null?
+    limits = Native::ExtendedLimits.new
+    limits[:basic][:flags] = 0x2000 # JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE.
+    unless Native.SetInformationJobObject(@job, 9, limits.pointer, limits.size)
+      raise SystemCallError.new("SetInformationJobObject", Native.GetLastError)
+    end
+
+    # Enroll the host before spawning: children inherit membership atomically.
+    # The non-inheritable job handle remains owned only by this host.
+    unless Native.AssignProcessToJobObject(@job, Native.GetCurrentProcess)
+      raise SystemCallError.new("AssignProcessToJobObject(host)", Native.GetLastError)
+    end
+    @host_assigned = true
+  end
+
+  def interrupt(pid)
+    Native.GenerateConsoleCtrlEvent(1, pid) # CTRL_BREAK_EVENT for this process group.
+  end
+
+  def close
+    # Once enrolled, keep the handle until host exit. Closing it here would also
+    # kill the host before it can finish SCM stop notification and log cleanup.
+    if @job && !@job.null? && !@host_assigned
+      unless Native.CloseHandle(@job)
+        raise SystemCallError.new("CloseHandle(job)", Native.GetLastError)
+      end
+      @job = nil
+    end
+    return unless @allocated
+    unless Native.FreeConsole
+      raise SystemCallError.new("FreeConsole", Native.GetLastError)
+    end
+    @allocated = false
+  end
+end