Init Spotter with discovery and pcap capabilities - #1433
Open
khyatimahendru wants to merge 64 commits into
Open
Conversation
…s in spotter CI job
…stener bindings for integration tests
…r and validate startup status
…cripts to eliminate deprecation warnings
…e CallbackAPIVersion.VERSION2
…N file or Site Model directory targets
…stration - Update edge/spotter/spotter_plan.md with Sub-phase 3.3 (Resource Contention Suite), Sub-phase 3.4 (Observability Framework), and technical notes on Streaming MQTT Protocol packet export. - Update edge/spotter/GEMINI.md with Target Execution Matrix, 3-Tier Production Release Pipeline, and metric telemetry standards. - Add --serial_no support to bin/spotter, bin/stop_spotter, and container/supervisor.sh for isolated multi-device management. - Update edge/spotter/src/agent.py and unit tests to support spotter_device_id and GCP JWT endpoint configuration. - Enhance Python clientlib MqttMessagingClient callback signatures for Paho MQTT v1/v2 compatibility and fix discovery node TLS CA handling for JWT authentication.
…e MQTT streaming - Add 'trace' value to discovery enumeration depth in common.json. - Add optional interface, filter, and max_bytes parameters to config_discovery_family.json for TRACE captures. - Define authoritative events_stream.json schema for sequential reliable data stream chunking over MQTT. - Regenerate Java and Python schema classes and documentation using bin/gencode. - Update Spotter README, testing standards, and development plan to remove GCS/HTTP streaming references and establish exclusive Streaming MQTT Protocol architecture.
…otter - Replace single-purpose blobset PCAP handler with TraceDiscoveryManager in agent.py. - Support selective routing in dual-process supervisor runtime: ingest only depth=trace discovery configs and ignore standard scans. - Emit base64-encoded StreamEvents chunks to events/stream with sequential event_no indexing and memory buffering. - Report capture status transitions under device state.discovery.families. - Expand unit test suite (test_agent.py) to cover non-trace config rejection and streaming payload format. - Refactor integration pipeline (test_pcap) to verify end-to-end Discovery TRACE trigger, stream chunk reassembly by event_no, and PCAP magic header integrity.
- Add pure unit tests in test_pcap.py verifying tcpdump packet capturing, duration cutoffs, volume limits, and SIGKILL timeout fallback. - Expand test_agent.py unit tests with negative paths and error handling verification for TraceDiscoveryManager (Level 500 status reports). - Introduce unified test runner bin/run_spotter_tests supporting unit, integration, and all test execution modes with automatic venv setup and PYTHONPATH binding. - Update testing standards documentation and development plan to reflect verified completion of Sub-phase 2.5.
…it test hygiene and Discovery TRACE standard
…d automated rollback - Register ota_package and discovery_rules blob handlers on SystemManager in agent.py. - Implement staging pipeline where wheel updates request supervisor restart via exit code 42 after final state publish. - Support dynamic hot-reloading of discovery signature rules without dropping broker TLS sockets. - Implement standalone micro self-test suite (self_test.py) validating imports, raw sockets, staging filesystem writeability, and event loop initialization. - Update supervisor.sh monitoring loop to intercept exit code 42, execute staged sandbox self-tests, promote active symlinks upon passing, and perform automated rollback upon failure. - Add comprehensive unit test coverage in test_agent.py and test_supervisor.py and mark Sub-phase 3.1 verified in development plan.
- Implement multi-provider metric exporter in metrics.py supporting Prometheus HTTP server (/metrics on port 9090) and native UDMI telemetry serialization (events/metrics). - Implement cgroup safety circuit breaker evaluation to throttle diagnostic captures before kernel OOM thresholds are exceeded. - Implement structured single-line JSON log formatter and W3C traceparent context generator in logger.py. - Introduce load stress test suite (bin/test_resource_contention) asserting zero file descriptor leaks under continuous packet sniffing workloads. - Introduce network resilience test suite (bin/test_fault_injection) verifying reconnect backoff recovery under simulated interruptions. - Expand unit test coverage (test_observability.py) and mark all Phase 3 tasks and Definition of Done criteria completed in spotter_plan.md.
…uggable providers - Refactor Spotter from dual-process supervisor model into unified single UDMI device. - Implement pluggable FamilyProvider interface with BacnetFamilyProvider, EtherFamilyProvider, and PassiveFamilyProvider. - Standardize on single device ID across config generator, agent runtime, and test orchestrator. - Add comprehensive test suites covering unit tests, container lifecycle, PCAP streaming over MQTT, resource contention, network fault resilience, and 100% differential discovery parity.
…ractive test execution
…remove config blob handlers
- Refactor DiscoveryManager background task from busy 10s polling to dynamic interval calculation (DiscoveryIntervalWatcher) - Default production metrics reporting rate to 300 seconds (5 minutes) while allowing test override - Normalize discovery depth matching in SpotterDiscoveryManager to exact enum/value check - Replace internal team references with generic host observability terminology - Add unit tests for metrics rate defaults and overrides
…hangs and deadlocks
… testing hygiene - Container & Runtime Config: Resolve container volume mount paths by generating isolated runtime configs in out/spotter_runtime_config.json, dynamically allocate pseudo-TTY only when attached to a TTY, and route logs to out/spotter/. - Key & Protocol Support: Automatically select ES256 for Elliptic Curve keys, and extract dynamic BACnet IP from device metadata and host network interfaces rather than hardcoding loopback. - Architecture & Circuit Breaker: Introduce SpotterSystemManager and wire host telemetry and safety circuit breaker checks across SystemManager, SpotterDiscoveryManager, and PCAP capture loops to throttle workloads under memory pressure. - Test Suite Hygiene: Replace shell-based FD checks with process-level /proc/PID/fd inspection, add negative leak verification, and add network resilience fault injection suites. - Identity & Site Parity: Rename DN-1 reference node identifiers to canonical DSN-1 across site models, test fixtures, scripts, and configurations. - CI/CD & Style Integrity: Lowercase repository owner in release workflows for ghcr.io compatibility, enforce Google Python Style (2-space indentation, <=80 char line length, 10.00/10 pylint score), and refine documentation terminology.
…rences - Create docs/tools/spotter.md detailing architecture, ephemeral PCAP streaming, host telemetry, safety circuit breaker, CLI orchestration, and verification testing suites. - Register spotter in docs/tools/readme.md and link from docs/specs/discovery.md. - Remove internal bug references (b/549909773 and b.corp links) across public documentation files.
grafnu
reviewed
Sep 3, 2026
grafnu
left a comment
grafnu
left a comment
Collaborator
There was a problem hiding this comment.
Mostly stuff about testing and then, of course, a naming decision!
| bin/udmi start sites/udmi_site_model //mqtt/localhost:46432 | ||
|
|
||
| # Launch Spotter targeting the local broker | ||
| ./bin/spotter sites/udmi_site_model //mqtt/localhost:46432 AHU-1 |
Collaborator
There was a problem hiding this comment.
nb, as mentioned elsewhere I think this could (also) be part of the tmux barbican session -- but maybe not as this is testing spotter proper. At least let's make sure we understand the plan for that.
Collaborator
Author
There was a problem hiding this comment.
I added spotter as an optional service in tmux bridgehead instead (similar to pubber).
| @@ -0,0 +1,184 @@ | |||
| # Spotter Testing & Verification Standards | |||
Collaborator
There was a problem hiding this comment.
spotter is going to be the most well-tested part of the system after this file!
| "discovery": { | ||
| "$ref": "file:events_discovery.json#" | ||
| }, | ||
| "stream": { |
Collaborator
There was a problem hiding this comment.
I'm just pondering if this should be stream, streams, or streamset -- it should definately support more than one stream at a time. At some point (doesn't need to be part of this PR) I think there would also be config/state messages associated with streams e.g. to indicate what has been received already, and at that point it would likely be a set of things like blobset or pointset which means streamset has parity.
| "description": "Version of the UDMI schema", | ||
| "type": "string" | ||
| }, | ||
| "session_id": { |
Collaborator
There was a problem hiding this comment.
With this structure, I think it might be streams since this is describing which one of the streams, which is different from pointset where the event itself has a set of points.
Collaborator
Author
There was a problem hiding this comment.
Done. I've renamed it to streams.
khyatimahendru
marked this pull request as ready for review
September 8, 2026 15:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.