Explorer of CAVE Health and Operations
echo implements a service to monitor the health of CAVE datastacks by periodically querying various endpoints and storing metadata about what succeeded and failed, as well as timing.
- User perspective: everything done from the "outside looking in" perspective of an average user.
- Low config: valid inputs ("fixtures") like root IDs are discovered automatically wherever possible and stashed in a JSON.
- Explicit typing of inputs: went with things like RootIDWithBounds to make very explicit intent for what a given probe input represents.
- Simple probes: probes hit one endpoint or at least one method.
- Configurable probes: Can ask for something to iterate over all fixtures of the correct type, just the first, sample one at random. Can also configure timeout per probe.
- Read-only probes: for the most part only testing endpoints that don't change data, with slight exceptions for things like level2 cache or skeleton service that may cache data on the backend.
- Tabular output: raw outputs are written to a Delta Lake. This will enable simple downstream queries/SQL. UI and other downstream summaries will be derived from this one source of truth.
- Condensed output for UI: recent outputs are packed into a JSON that is read by the UI. The UI uses conditional GETs to only grab new data when something changes.
- Stateful UI: selection in the UI website is reflected in the URL so that links can be passed around.