optionally use watch events to track k8s pods

[pjfanning]

Motivation

kubernetes-api discovery performs a full pod list GET on every lookup. In a large cluster with frequent lookups that puts avoidable load on the API server, and the cost grows with the number of pods rather than with the number of changes.

The Kubernetes Watch API is the intended alternative: list once, then receive incremental ADDED / MODIFIED / DELETED events over a long-lived stream.

Modification

Add api-poll-mode, defaulting to "list" so existing behaviour is unchanged. With "watch", the first lookup performs an initial list to seed a cache and fix the point the watch resumes from, then opens a watch stream; later lookups read the cache without touching the API server.

Supporting changes:

• PodList gains ListMeta, so the watch can resume from the list's resourceVersion. An individual pod's version is not a valid resume point — in the existing pods.json fixture the list is at 16042 while the last item is at 7406832, and resuming from the latter would silently skip every event in between.
• Watch state (podCache, resourceVersion, kill switch) is held per label selector in a ConcurrentHashMap. One discovery instance can be asked about several service names, each mapping to its own selector, so sharing the cache would let one service's lookup return another service's pods.
• The stream is framed with Framing.delimiter over the raw bytes, bounded by watch-max-frame-length. Decoding each network chunk to a String before splitting corrupts any multi-byte UTF-8 character straddling a chunk boundary, and the resulting parse failure is easy to swallow as a dropped event.
• Reconnects rebuild from a full list whenever there is no valid resume point (410 Gone, or an ERROR event), so pods deleted while disconnected do not linger in the cache. Resuming the stream alone would leave them there indefinitely.
• lookup honours resolveTimeout in watch mode, as it already did in list mode.
• Pods without metadata.name are dropped rather than sharing a cache key.
• api-poll-mode and watch-max-frame-length are validated in Settings, so a typo such as "Watch" fails at startup instead of silently falling back to list mode.
• Watch streams are shut down in the service-unbind phase of Coordinated Shutdown, and reconnects stop once shutdown has begun.

Docs cover watch mode, the new settings, and the RBAC requirement (watch alongside list).

Result

Discovery can track pods from a single streaming connection instead of polling, with list mode untouched as the default.

Tests

• sbt "discovery-kubernetes-api/test" — 27 succeeded, 0 failed.

• KubernetesApiWatchSpec covers the cache transitions, resource-version tracking, ERROR handling, unnamed pods, per-selector isolation, and framing a multi-byte character split across two chunks.

• JsonFormatSpec asserts the list resourceVersion, which differs from every item's.

• SettingsSpec covers the new settings and their validation.

• Integration test against a real API server. test-watch.sh deploys the demo with api-poll-mode = watch and reuses the shared Kubernetes script, then asserts the cluster formed via the watch stream rather than by quietly falling back to list mode. It runs as a second job in the Kubernetes API workflow — a separate job rather than a matrix, so the existing job keeps its name for branch protection. Passing in CI:

  KubernetesApiServiceDiscovery - Watch stream started for label selector: [app=pekko-bootstrap-demo]
  ClusterWatcher - Cluster pekko://Appka@10.244.0.7:17355 >>> MemberUp(...)
  3 pod(s) started a Kubernetes watch stream
  

• sbt "discovery-kubernetes-api/mimaReportBinaryIssues" — success; PodList and Metadata gained fields, with excludes in watch-pod-changes.excludes.

• sbt "+headerCheckAll" and scalafmt checks — clean.

References

None - new feature