Outage degradation alerting

.env.example

@@ -17,3 +17,21 @@ TG_CHAT_ID=123456789
 
 # Optional: HTTP timeout in seconds for Telegram/Discord requests (must be positive, default 10)
 # REQUEST_TIMEOUT=30
+
+# Optional: seconds between each speed test + device scan cycle (default 1800 = 30 min)
+# SLEEP_TIME=1800
+# Optional: how many cycles between detailed AI reports with graph (default 8, i.e. ~4 hours at the default SLEEP_TIME)
+# REPORT_CYCLE_COUNT=8
+# Optional: sets Ollama's num_ctx per-request via extra_body, to stop a local model's
+# default context window from silently truncating a long prompt + a day of history.
+# No effect on cloud OpenAI. Leave unset unless using a local/self-hosted AI backend.
+# AI_CONTEXT_SIZE=8192
+
+# Optional: instant outage/degradation alerting thresholds. A reading counts as
+# "bad" if it's below the download threshold OR above the ping threshold; a
+# speed test failing outright counts as bad too. An alert fires once per
+# episode (not every cycle) once N consecutive bad readings are seen, and
+# again once the connection recovers, with how long it lasted.
+# OUTAGE_DOWNLOAD_THRESHOLD_MBPS=20
+# OUTAGE_PING_THRESHOLD_MS=150
+# OUTAGE_CONSECUTIVE_READINGS=2

README.md

@@ -31,13 +31,14 @@ Every 4 hours, it delivers a **detailed report** complete with a 24-hour trend g
 
 ## Features & Workflow
 
-Every 30 minutes (`SLEEP_TIME` in `main.py`, default 1800 seconds):
+Every `SLEEP_TIME` seconds (default 1800 = 30 min, configurable):
 
 1. **Speed Test:** Measures download/upload speeds, ping latency, ISP, and test server details using `speedtest-cli` (see [the note on measurement mode](#a-note-on-measurement-mode)).
-2. **LAN Scan:** Scans the local subnet using `nmap` ARP scan to count active connected devices.
-3. **Local Storage:** Saves metrics & device tallies directly to a local `metrics.sql` SQLite database.
+2. **LAN Scan:** Scans the local subnet using `nmap` ARP scan to identify active devices, including MAC address, vendor, and hostname where resolvable (see [Device Watch](#device-watch)).
+3. **Local Storage:** Saves metrics & device details directly to a local `metrics.sql` SQLite database.
 4. **Status Alert:** Sends a concise status update to your chosen notifier (*"all good"* or *"line is dying"*).
-5. **24h AI Report:** Every 8th cycle (every 4h), generates a **24-hour trend graph** via `matplotlib` alongside a sarcastic LLM analysis of network load and speed fluctuations.
+5. **24h AI Report:** Every `REPORT_CYCLE_COUNT` cycles (default 8, i.e. ~4h), generates a **24-hour trend graph** via `matplotlib` alongside a sarcastic LLM analysis of network load, speed fluctuations, and any notable new devices on the network.
+6. **Instant Outage Alerting:** Watches every cycle for an outright failed speed test or a degraded reading, alerting immediately rather than waiting for the next scheduled report (see [Instant Outage & Degradation Alerting](#instant-outage--degradation-alerting)).
 
 ---
 
@@ -129,6 +130,9 @@ cp .env.example .env
 | `DISCORD_WEBHOOK_URL` | Discord channel webhook URL — required if `NOTIFIER=discord` |
 | `DB_PATH` | SQLite database file path (e.g. `metrics.sql`) |
 | `REQUEST_TIMEOUT` | *Optional.* HTTP timeout in seconds for Telegram/Discord requests (positive integer, default `30`) |
+| `SLEEP_TIME` | *Optional.* Seconds between each speed test + device scan cycle (positive integer, default `1800`) |
+| `REPORT_CYCLE_COUNT` | *Optional.* How many cycles between detailed AI reports with graph (positive integer, default `8`) |
+| `AI_CONTEXT_SIZE` | *Optional.* Sets Ollama's `num_ctx` per-request, to stop a local model's default context window from silently truncating a long prompt + a day of history. No effect on cloud OpenAI — leave unset unless self-hosting the AI backend. |
 
 > [!TIP]
 > **You're not locked into OpenAI.** `ai.py` talks to any OpenAI-compatible endpoint, so a local inference server (e.g. [Ollama](https://ollama.com), LM Studio) works too — just point `AI_BASE_URL` at it. For report quality that holds up, use a model with **at least ~7B parameters**; a solid local pick is **Gemma 4 12B at 4-bit (QAT) quantization** (`gemma4:12b-it-qat` via Ollama), which fits comfortably on 16GB of RAM.
@@ -144,6 +148,9 @@ uv run main.py
 > [!TIP]
 > Run the bot inside `tmux`/`screen` or set it up as a system service (`systemd`/`launchd`) to keep it running 24/7 in the background.
 
+> [!TIP]
+> Pass `--test-ai` (`uv run main.py --test-ai`) to force the very first cycle to run the full detailed report (AI commentary + graph + notifier delivery) immediately, then resume the normal `REPORT_CYCLE_COUNT` schedule automatically — no config to remember to revert afterward. Useful for verifying your AI backend and notifier work without waiting for the regular cadence.
+
 ---
 
 ## Notifications: Telegram or Discord
@@ -192,6 +199,18 @@ Two consequences worth knowing:
 
 Since netmon exists to track *trends*, consistency matters more than peak numbers: keep one measurement method for the lifetime of your database. Swapping the backend mid-history puts a step change in your 24-hour graph that the AI commentary will faithfully report as a real speed jump.
 
+### Jitter & Bufferbloat (Ookla backend only)
+
+Classic `speedtest-cli` has no jitter or packet-loss data. If you instead run netmon against an Ookla-compatible speed test backend that reports those fields, netmon picks them up automatically and surfaces them in both mini and detailed reports, alongside a note from the AI treating high jitter or nonzero packet loss as a sign of bufferbloat — a connection can have great raw Mbps numbers and still feel laggy under load if jitter is high. This is entirely additive: nothing changes in reports if your backend doesn't provide this data.
+
+---
+
+## Device Watch
+
+Every device scan records each device's MAC address, vendor (resolved from `nmap`'s built-in OUI database), and hostname where available. `nmap` can only resolve a MAC for hosts on the same local subnet it can ARP directly — off-subnet or otherwise hidden devices are still counted, just not identified.
+
+Using MAC address history, the detailed AI report includes a **Device Watch** section that flags any device whose MAC hasn't been seen on the network in the last 14 days, alongside a vendor-count breakdown of everything currently online. A device with no resolvable MAC is never flagged as new, since there's no reliable identity to compare against.
+
 ---
 
 ## Example Output
@@ -232,12 +251,16 @@ Server: <b>New York</b>
 Download: 178.5 Mbps
 Upload: 45.2 Mbps
 Ping: 23.1 ms
+Jitter: 4.2 ms | Packet Loss: 0.0%
 Devices Online: 9
 </pre>
 
 <b>24-Hour Dynamics Analysis</b>
 Over the last 24 hours, the download speed averaged <code>140 Mbps</code>, but we saw a massive drop to <code>20 Mbps</code> at 8:00 PM right as device count jumped from <code>4</code> to <code>11 devices</code>. Clearly, someone's hogging the bandwidth or the ISP's mice were busy chewing on the fiber line again. Latency remained stable except for a brief spike during peak hours.
 
+<b>Device Watch</b>
+One new gadget joined the party today: a device with no vendor or hostname info at all — worth a second glance. Everything else is the same suspects as always.
+
 <b>Data Transfer (Latest Test)</b>
 <pre>
 Downloaded: 160.0 MB
@@ -248,6 +271,37 @@ Uploaded: 70.0 MB
 Expect periodic speed drops whenever local freeloaders stream 4K movies or the ISP potato infrastructure struggles.
 ```
 
+> [!NOTE]
+> The AI is only ever asked for three short text fields (the dynamics analysis, the Device Watch line, and the conclusion) — the surrounding HTML structure above is assembled deterministically in code, not generated by the model. This keeps report formatting consistent regardless of which LLM is behind `AI_BASE_URL`, including smaller local models that would otherwise struggle to reproduce a long literal template reliably. The `Jitter` line only appears when your speed test backend reports it (see [Jitter & Bufferbloat](#jitter--bufferbloat-ookla-backend-only)) — it's silently omitted otherwise.
+
+---
+
+## Instant Outage & Degradation Alerting
+
+Waiting for the next scheduled detailed report to notice an outage could mean a multi-hour delay. netmon instead watches every cycle:
+
+* **Outage:** the speed test itself fails outright for `OUTAGE_CONSECUTIVE_READINGS` consecutive cycles.
+* **Degradation:** a successful reading falls below `OUTAGE_DOWNLOAD_THRESHOLD_MBPS` or above `OUTAGE_PING_THRESHOLD_MS` for the same number of consecutive cycles.
+
+Each fires an alert once per episode (not every cycle, to avoid spam), and again once the connection recovers, with how long the episode lasted.
+
+| Variable | Description |
+| :--- | :--- |
+| `OUTAGE_DOWNLOAD_THRESHOLD_MBPS` | *Optional.* Download speed below which a reading counts as degraded (positive number, default `20`) |
+| `OUTAGE_PING_THRESHOLD_MS` | *Optional.* Ping above which a reading counts as degraded (positive number, default `150`) |
+| `OUTAGE_CONSECUTIVE_READINGS` | *Optional.* Consecutive bad/failed readings before alerting (positive integer, default `2`) |
+
+> [!NOTE]
+> A failing or degraded speed test is treated as the exact condition this tool exists to detect, not a bug in netmon — it alerts and keeps retrying every cycle rather than crashing the process (see [Reliability](#reliability) below for the genuine-infra-failure case, which is handled differently on purpose).
+
+---
+
+## Reliability
+
+Speed test, device scan, database, or notifier-delivery failures are **never silently retried**. If one of these fails, netmon makes a best-effort attempt to post an alert to your configured notifier — so the failure is visible without checking server logs — then crashes rather than looping on a broken state. Check the service logs (`journalctl -u netmon` if running under `systemd`, or wherever your process manager sends output) for the full traceback, and your process manager's restart policy will bring it back up.
+
+This is deliberately different from how a *slow or unreachable AI backend* is handled: that degrades gracefully (the report still sends, just without AI commentary) rather than crashing, since a flaky LLM endpoint isn't the kind of infrastructure failure worth stopping the whole monitor over.
+
 ---
 
 ## Project Structure

ai.py

@@ -26,25 +26,43 @@ def init(cls, api_key: str, model: str, base_url: str) -> "Client":
         return cls(OpenAI(api_key=api_key, base_url=base_url), model)
 
 
-    def send_message(self, message:str, system_prompt: str) -> str:
+    def send_message(
+        self,
+        message: str,
+        system_prompt: str,
+        temperature: float = 0.9,
+        context_size: int | None = None,
+    ) -> str:
         self._validate_str(message, "message")
 
+        # Ollama's default context window (often 2048-4096 tokens depending
+        # on the model) is easy to exceed once the system prompt plus a
+        # day's worth of historical readings are combined — and unlike a
+        # clear error, exceeding it just silently truncates the prompt
+        # (typically from the start), which can quietly drop persona/format
+        # instructions while leaving the raw data intact. Passing num_ctx
+        # via extra_body raises this per-request for Ollama specifically.
+        # This is a no-op / harmless on real OpenAI's API since it's only
+        # added when context_size is explicitly set (e.g. for a local
+        # Ollama backend), not unconditionally on every request.
+        extra_body = {}
+        if context_size is not None:
+            extra_body["options"] = {"num_ctx": context_size}
+
         response = self.conn.chat.completions.create(
             model=self.model,
+            temperature=temperature,
             messages=[
                 {"role": "system", "content": system_prompt},
                 {"role": "user", "content": message}
-            ])
-
+            ],
+            extra_body=extra_body or None,
+        )
+
         if response.choices[0].message.content is not None:
             return response.choices[0].message.content
-        
+
         raise RuntimeError("AI response is empty")
 
     def close(self):
-        self.conn.close()
-
-
-
-
-
+        self.conn.close()

config.py

@@ -6,6 +6,11 @@
 logger = logging.getLogger(__name__)
 
 DEFAULT_REQUEST_TIMEOUT = 30
+DEFAULT_SLEEP_TIME = 1800
+DEFAULT_REPORT_CYCLE_COUNT = 8
+DEFAULT_OUTAGE_DOWNLOAD_THRESHOLD_MBPS = 20.0
+DEFAULT_OUTAGE_PING_THRESHOLD_MS = 150.0
+DEFAULT_OUTAGE_CONSECUTIVE_READINGS = 2
 
 class Config:
     def __init__(
@@ -18,7 +23,14 @@ def __init__(
         tg_bot_token: str = "",
         tg_chat_id: str = "",
         discord_webhook_url: str = "",
-        request_timeout: int = DEFAULT_REQUEST_TIMEOUT
+        request_timeout: int = DEFAULT_REQUEST_TIMEOUT,
+        sleep_time: int = DEFAULT_SLEEP_TIME,
+        report_cycle_count: int = DEFAULT_REPORT_CYCLE_COUNT,
+        test_ai: bool = False,
+        ai_context_size: int | None = None,
+        outage_download_threshold_mbps: float = DEFAULT_OUTAGE_DOWNLOAD_THRESHOLD_MBPS,
+        outage_ping_threshold_ms: float = DEFAULT_OUTAGE_PING_THRESHOLD_MS,
+        outage_consecutive_readings: int = DEFAULT_OUTAGE_CONSECUTIVE_READINGS,
     ):
         self.ai_api_key: str = ai_api_key
         self.db_path: str = db_path
@@ -29,6 +41,13 @@ def __init__(
         self.tg_chat_id: str = tg_chat_id
         self.discord_webhook_url: str = discord_webhook_url
         self.request_timeout: int = request_timeout
+        self.sleep_time: int = sleep_time
+        self.report_cycle_count: int = report_cycle_count
+        self.test_ai: bool = test_ai
+        self.ai_context_size: int | None = ai_context_size
+        self.outage_download_threshold_mbps: float = outage_download_threshold_mbps
+        self.outage_ping_threshold_ms: float = outage_ping_threshold_ms
+        self.outage_consecutive_readings: int = outage_consecutive_readings
 
     @staticmethod
     def _parse_args():
@@ -39,6 +58,15 @@ def _parse_args():
             default=".env",
             help="Path to the .env file (default: .env)"
         )
+        parser.add_argument(
+            "--test-ai",
+            action="store_true",
+            help="Force the very first cycle to run the full detailed report "
+                 "(AI commentary + graph + notifier delivery), then continue on "
+                 "the normal REPORT_CYCLE_COUNT schedule for every cycle after. "
+                 "Useful for verifying the AI backend and notifier work without "
+                 "waiting for the regular cadence or permanently changing config."
+        )
         return parser.parse_args()
 
     @classmethod
@@ -72,6 +100,56 @@ def init(cls):
         if request_timeout <= 0:
             raise RuntimeError(f"REQUEST_TIMEOUT must be positive, got: {request_timeout}")
 
+        try:
+            sleep_time = int(os.getenv("SLEEP_TIME", DEFAULT_SLEEP_TIME))
+        except ValueError:
+            raise RuntimeError(f"SLEEP_TIME must be an integer number of seconds, got: {os.getenv('SLEEP_TIME')!r}")
+        if sleep_time <= 0:
+            raise RuntimeError(f"SLEEP_TIME must be positive, got: {sleep_time}")
+
+        try:
+            report_cycle_count = int(os.getenv("REPORT_CYCLE_COUNT", DEFAULT_REPORT_CYCLE_COUNT))
+        except ValueError:
+            raise RuntimeError(f"REPORT_CYCLE_COUNT must be an integer, got: {os.getenv('REPORT_CYCLE_COUNT')!r}")
+        if report_cycle_count <= 0:
+            raise RuntimeError(f"REPORT_CYCLE_COUNT must be positive, got: {report_cycle_count}")
+
+        # Optional and unset by default — only meaningful for local
+        # OpenAI-compatible servers like Ollama, whose default context
+        # window can silently truncate a long system prompt + a day's
+        # worth of history once combined. Left as None, nothing extra is
+        # sent, so cloud OpenAI usage is unaffected.
+        ai_context_size_raw = os.getenv("AI_CONTEXT_SIZE")
+        ai_context_size: int | None = None
+        if ai_context_size_raw is not None and ai_context_size_raw.strip() != "":
+            try:
+                ai_context_size = int(ai_context_size_raw)
+            except ValueError:
+                raise RuntimeError(f"AI_CONTEXT_SIZE must be an integer, got: {ai_context_size_raw!r}")
+            if ai_context_size <= 0:
+                raise RuntimeError(f"AI_CONTEXT_SIZE must be positive, got: {ai_context_size}")
+
+        try:
+            outage_download_threshold_mbps = float(os.getenv("OUTAGE_DOWNLOAD_THRESHOLD_MBPS", DEFAULT_OUTAGE_DOWNLOAD_THRESHOLD_MBPS))
+        except ValueError:
+            raise RuntimeError(f"OUTAGE_DOWNLOAD_THRESHOLD_MBPS must be a number, got: {os.getenv('OUTAGE_DOWNLOAD_THRESHOLD_MBPS')!r}")
+        if outage_download_threshold_mbps <= 0:
+            raise RuntimeError(f"OUTAGE_DOWNLOAD_THRESHOLD_MBPS must be positive, got: {outage_download_threshold_mbps}")
+
+        try:
+            outage_ping_threshold_ms = float(os.getenv("OUTAGE_PING_THRESHOLD_MS", DEFAULT_OUTAGE_PING_THRESHOLD_MS))
+        except ValueError:
+            raise RuntimeError(f"OUTAGE_PING_THRESHOLD_MS must be a number, got: {os.getenv('OUTAGE_PING_THRESHOLD_MS')!r}")
+        if outage_ping_threshold_ms <= 0:
+            raise RuntimeError(f"OUTAGE_PING_THRESHOLD_MS must be positive, got: {outage_ping_threshold_ms}")
+
+        try:
+            outage_consecutive_readings = int(os.getenv("OUTAGE_CONSECUTIVE_READINGS", DEFAULT_OUTAGE_CONSECUTIVE_READINGS))
+        except ValueError:
+            raise RuntimeError(f"OUTAGE_CONSECUTIVE_READINGS must be an integer, got: {os.getenv('OUTAGE_CONSECUTIVE_READINGS')!r}")
+        if outage_consecutive_readings <= 0:
+            raise RuntimeError(f"OUTAGE_CONSECUTIVE_READINGS must be positive, got: {outage_consecutive_readings}")
+
         if notifier == "telegram":
             if tg_bot_token.strip() == "":
                 raise RuntimeError("TG_BOT_TOKEN not found or empty in environment")
@@ -84,5 +162,7 @@ def init(cls):
         return cls(
             ai_key, db_path, model, base_url, notifier,
             tg_bot_token, tg_chat_id, discord_webhook_url,
-            request_timeout,
+            request_timeout, sleep_time, report_cycle_count,
+            args.test_ai, ai_context_size,
+            outage_download_threshold_mbps, outage_ping_threshold_ms, outage_consecutive_readings,
         )