Conversation
A running ama-logs pod does not imply working telemetry: container logs reach the workspace over a local mdsd socket, so they keep flowing even when the agent's outbound telemetry path is entirely broken. Assert the ContainerLogDaemonSetHeartbeatEvent custom event arrives from every node running a DaemonSet pod. - Query the agent telemetry App Insights resource (heartbeats are agent self-telemetry and never land in the customer workspace), matching the cluster resource ID case-insensitively. - Only expect a heartbeat from pods older than the 300s publish interval so a rollout does not fail the assertion. - Generalize AssertContainerLogNodeCoverage into AssertNodeCoverage so both signals share the missing-node reporting. - Plumb AgentTelemetryResourceId through the testkube script and TestWorkflow as AGENT_TELEMETRY_RESOURCE_ID; the spec skips when unset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The results check compared .result.status against "failed", but that status was never what it read. `kubectl testkube get testworkflowexecution --output json` returns an empty file when the execution has just finished, and an empty file satisfies `jq empty`, so it passed validation and .result.status then yielded an empty string. When the file was not empty the status was often still "running", because it is not final the moment `watch` returns. Neither value equals "failed", so failing workflows were recorded as successful and the pipeline job went green: build 126041 reported success while ginkgo reported "FAIL! -- 0 Passed | 1 Failed". Poll for a terminal status rather than reading whichever one happens to be available, and let it decide the outcome only once it is terminal. When it never becomes terminal, fall back to the exit code of `kubectl testkube watch`, which is non-zero for a failed execution and zero for a passing one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6b1dd36a-91a3-4eeb-9ffa-0f89b90b76bd
…over customMetrics The heartbeat assertion accepted any heartbeat for the cluster inside the query window, so telemetry published by the previous image satisfied it and a rollout that never reported would still pass. Resolve the agent version each node is expected to report and assert per node against it, reporting a node that reported nothing separately from one that reported only another version, which is what a stale image or a half-finished rollout looks like. The agent reports that version from AGENT_VERSION, which the build bakes in from its telemetry tag rather than the image tag; the two are equal except on release builds, where TELEMETRY_TAG overrides it. The pipeline therefore passes the telemetry tag explicitly and the image tag on the pod is used only as a fallback. Also assert customEvents and customMetrics per node. The heartbeat only exercises the custom event path, so a break confined to metrics stayed invisible. traces is queried and reported but not required: it carries only agent log lines above Information level, so a healthy agent emits none -- in a sampled 30 minute window just 41,491 of the 107,982 clusters reporting a heartbeat emitted a single trace -- and any trace that does arrive still has to come from the deployed image. Fixes an incorrect format verb that took the address of a string, which blocked go test from running in the utils package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6b1dd36a-91a3-4eeb-9ffa-0f89b90b76bd
This reverts commit 5462df9 after live telemetry validation in pipeline 444. Normal build, deployment, and Testkube execution are restored; permanent telemetry checks remain.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
suyadav1
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AGENT_TELEMETRY_RESOURCE_IDand propagate Testkube failures.Validation
Known-version heartbeat and metrics passed on all five clusters; a nonexistent version failed as expected. Strict trace checks remain unchanged; missing traces on existing pods were accepted for this validation.