From 49fec03b5564284df46720684537f5fe4eab4a71 Mon Sep 17 00:00:00 2001 From: Siddharth Bhat <190953906+SiddharthBhat120@users.noreply.github.com> Date: Thu, 12 Nov 2026 06:06:17 +0000 Subject: [PATCH] Remove personal prep notes, add MIT licence CLAUDE.md and docs/DEMO_AND_INTERVIEW.md were working notes, not project documentation. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01X1gXuRfdpHqj9VsKi9qnmx --- CLAUDE.md | 21 ---------------- LICENSE | 21 ++++++++++++++++ docs/DEMO_AND_INTERVIEW.md | 49 -------------------------------------- 3 files changed, 21 insertions(+), 70 deletions(-) delete mode 100644 CLAUDE.md create mode 100644 LICENSE delete mode 100644 docs/DEMO_AND_INTERVIEW.md diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index c735da6..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,21 +0,0 @@ -# BlackBox - -Ingests logs, alerts, deploy events and chat messages from an incident, normalises them into events, -finds key moments, shows an interactive timeline, and drafts a postmortem where every claim links to -supporting event ids. - -## Stack -FastAPI, SQLite, Drain3, optional OpenAI-compatible LLM client (httpx), React 19 + Vite + vis-timeline, -deterministic chaos simulator in demo/ (Docker is optional). - -## Rules -- Type hints everywhere; pytest (`python -m pytest backend/tests -q`). -- All timestamps stored in UTC (`timeutil.to_utc`); queries always sort by time. -- LLM output validated with Pydantic; failures fall back to the template postmortem. -- Postmortem sentences must cite event ids; ids are verified against the store; invalid -> "unverified". -- Trigger candidates are correlations; never call one "the root cause". -- Blameless wording: no individual names in the postmortem. - -## Layout -backend/app (ingest/, store, patterns, metrics, correlate, pipeline, postmortem, main), backend/tests, -demo/simulate.py, eval/run_eval.py, frontend (dist committed), docs. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..765b545 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Siddharth Bhat + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/docs/DEMO_AND_INTERVIEW.md b/docs/DEMO_AND_INTERVIEW.md deleted file mode 100644 index 6c997b0..0000000 --- a/docs/DEMO_AND_INTERVIEW.md +++ /dev/null @@ -1,49 +0,0 @@ -# BlackBox: demo script, resume bullets, interview prep - -## 3-minute demo script - -1. **Problem (20 s).** "After an outage, someone spends hours scrolling logs and Slack to rebuild what happened. BlackBox does that first pass in seconds and shows its evidence." -2. **Break something (20 s).** Pick *Disk fills up*, press **Inject the fault and record**. "This runs a small shop for 70 minutes, injects a real fault, and adds two harmless decoy deploys so the answer is not obvious." -3. **Readouts (20 s).** "Started 12:48 in payments, detected two minutes later by an alert, mitigated after ten minutes, recovered after thirteen. And 37,000 log lines became 22 timeline items." -4. **Timeline (40 s).** Point at the error chart and the lanes. Click the *write failed: No space left on device* pattern: "463 lines, zero in the baseline, so it is new." Toggle *Show unchanged patterns* to show the noise that was hidden. -5. **Trigger (40 s).** "The top candidate is a payments deploy 20 minutes earlier that turned on DEBUG logging. It wins over a gateway deploy that was closer in time, because errors started in payments and payments logged new debug lines right after that deploy. The gateway change left no trace of its own." Open the answer key: correct. -6. **Postmortem (30 s).** Open the draft. Click an evidence chip: it jumps back to the event on the timeline. "Every sentence cites event ids that are checked. Names from chat are replaced: blameless by design." -7. **Numbers (10 s).** "Across 50 injected incidents it ranked the real trigger first 49 times, and the one miss is documented." - -## Resume bullets - -- Built **BlackBox**, an incident-timeline tool (FastAPI, SQLite, Drain3, React) that ingests logs, Alertmanager alerts, deploy/GitHub events and Slack exports into one UTC event model and drafts an evidence-linked, blameless postmortem. -- Reduced **~34,000 log lines to ~23 timeline items** per incident with Drain template mining and z-score anomaly detection; identified the triggering change first in **49 of 50 injected incidents** (29 of 30 held-out) with two decoy deploys per run. -- Built a chaos simulator (bad deploy, DB pool exhaustion, slow provider, bad config, disk full) with recorded ground truth and an evaluation harness measuring trigger rank, time-to-detect/recover error (under 1 min) and postmortem evidence coverage (100%). - -## 10 interview questions with answers - -**1. What is a postmortem, and why "blameless"?** -A written review after an incident: what happened, the impact, the timeline, why it happened, and what we will change. Blameless means we describe systems and decisions, not people's mistakes. If people fear blame, they hide details, and the organisation stops learning. BlackBox writes in that style and replaces names from chat with roles. - -**2. How does log template mining (Drain) work?** -Drain groups log lines by shape. It first splits by number of words, then walks a shallow tree keyed on the first words, then compares the line with existing groups word by word. If enough words match, the line joins the group and the differing words become wildcards. I also mask ids, numbers and hex values first, so `fetched order o-123 in 31ms` and `fetched order o-987 in 29ms` become `fetched order in ms`. - -**3. Correlation vs causation: how does your tool avoid claiming too much?** -It never says "root cause". It ranks candidates and shows why: minutes before the first error, whether the change touched the service that failed first, and whether that service behaved differently afterwards. An "outside cause" candidate is always present. The postmortem says the top candidate is confirmed only if reverting it stopped the errors. - -**4. Why did the disk-full deploy win even though a decoy was closer in time?** -Time is only one factor. The real trigger changed payments, where errors started, and payments logged new DEBUG lines right after it. The decoy changed gateway, whose only new behaviour was errors that are explained by payments failing below it. I give a change behaviour credit only for patterns that are not just echoes of a failing dependency. - -**5. What problems do timestamps and time zones cause across services?** -Different formats (ISO, epoch seconds, epoch milliseconds, `09:00:01,123`), missing zones, local times like +05:30, clock skew between hosts, and events arriving out of order. I parse everything to timezone-aware UTC at ingestion, read zone-less times in a configured zone, store epoch seconds so sorting is exact, and always query sorted by time. Tests cover +05:30, +0530, epoch ms and late-arriving events. - -**6. How do you find the first failing service?** -For each service I take the earliest event of a new or increased error pattern (exact timestamps), falling back to minute-level spikes. Callers often fail milliseconds after their callee, so if two services fail within 5 seconds and one calls the other, I pick the callee. A bug where latency spikes used the first log line of the minute made gateway look first; fixing it raised accuracy on that scenario. - -**7. How do you detect spikes without a trained model?** -Per service per minute I compute error rate and p95 latency. The baseline is the quiet period before the incident. A spike needs a z-score above 4, a minimum real size (2% errors, or 1.5x baseline latency), and two minutes in a row. It is simple, explainable, and every moment has a reason string such as "35% errors, z-score 82". - -**8. How do you stop the LLM from inventing facts in the postmortem?** -It only sees key moments and pinned events, each with its event ids, never raw logs. It must return strict JSON where each sentence lists its evidence ids. My code checks every id exists in the incident; sentences with none are marked unverified, and invalid ids are removed. If the JSON is broken, the template draft is used. - -**9. What are time-to-detect and time-to-recover, and why track them?** -TTD is from the first errors to the first alert or human report; TTR is from the first errors to normal service. They measure how good monitoring and response are. Teams try to push both down over time, and they are good follow-up targets: here, "errors ran 2 minutes before an alert" becomes a concrete action. - -**10. How would this scale to millions of log lines per minute?** -Mine templates at the edge or per shard so only template ids and counts travel; pre-aggregate per-minute metrics in a stream processor; store events in a column store like ClickHouse partitioned by time; keep raw lines in cheap object storage and fetch samples on demand. The analysis itself works on aggregates and key moments, so it stays fast.