🕵️ NetGuard – Full-Stack NIDS & Security Observability Engine

A full end-to-end real-time Network Intrusion Detection System (NIDS).
Combines a Multi-threaded Python (Scapy) capture and analysis engine with DPI, Sliding-Window anomaly detection, Active Defense mechanisms, and a fully code-managed monitoring stack (Dashboard as Code) on Docker (Grafana + Loki + Promtail).

---

🔎 Overview & Architecture

NetGuard provides a complete solution for monitoring, analyzing, and responding to network security events across OSI layers 3, 4, and 7.
The architecture is built on a continuous data pipeline that separates packet capture, real-time processing, and feeding data into the visualization system:

📡 Network Traffic  ➔  🐍 Python Engine (Sniffer + Worker + GC)  ➔  📄 JSON Logs File

📊 Auto-Provisioned Grafana  ⬅️  🗄️ Loki DB  ⬅️  🔄 Promtail Shipper

---

📂 Project Structure

python_sniffer/
├── grafana/
│   └── dashboards/                 # Standard JSON Dashboards (Git Version-Controlled)
│       ├── dashboard-Live Security Log Stream.json
│       ├── dashboard-Security Events Distribution.json
│       ├── dashboard-Threat Timeline & Severity Levels.json
│       ├── dashboard-Top Suspicious Source IPs.json
│       └── dashboard-Total Security Alerts.json
├── provisioning/                   # Grafana Automated Provisioning Configs
│   ├── dashboards/
│   │   └── dashboards.yml
│   └── datasources/
│       └── datasources.yml
├── logs/                           # Runtime Log Directory (Ignored by Git)
├── .env.example
├── .gitignore
├── docker-compose.yml
├── main.py                         # NIDS Core Engine (Thread-Safe & GC Refactored)
├── promtail-config.yml
├── requirements.txt
└── test_attack.py                  # Traffic Simulator

---

🚀 Core Features

Domain	Feature	Status	Description
📡 Network	Real-time L2-L7 Sniffing	✅	Real-time capture and analysis of IP, TCP, UDP, and DNS traffic while preventing memory overflow (store=0).
🛡️ Cyber Security	Sliding-Window Detection	✅	Detection of DoS (SYN Flood) and port scans based on a precise moving time window.
⚡ Active Defense	Dynamic IP Isolation	✅	Active mitigation mechanism that isolates attacking addresses for a limited time (Blacklist with automatic expiry).
🔍 DPI Engine	Deep Packet Inspection	✅	Byte-level Raw Payload scanning to detect suspicious strings (SQLi, Credentials, Path Traversal).
⚙️ Architecture	Producer-Consumer & Thread-Safety	✅	Bounded Queue, threading.Lock locks, and a dedicated background Garbage Collector thread to prevent memory leaks.
📊 Observability & IaC	Dashboard as Code (Grafana + Loki)	✅	Five pre-defined dashboards in standard JSON format, automatically loaded on container startup via Provisioning files.
📝 Logging	Structured JSON Dual-Stream	✅	Colorized console output alongside structured JSON log writes, tailored for collection by Promtail.
🧪 Testing	Traffic Attack Simulator	✅	Simulation script (test_attack.py) that generates synthetic attack traffic to validate detection mechanisms.

---

🛠️ Technologies & Architectural Highlights

• Python & Scapy: Raw-socket-level packet capture, protocol parsing, and deep payload-level inspection (DPI).
• Producer-Consumer Architecture: Full separation between packet capture and analysis via queue.Queue(maxsize=10000), preventing packet loss under load.
• Thread-Safety & Active Defense: Whitelist/Blacklist state management and anomaly detection guarded by threading.Lock to prevent data races, alongside dynamic, time-limited blocking of attacking IP addresses.
• Background Garbage Collector: A dedicated background thread that cleans up stale data structures (Sliding Window History & Blacklist) from memory every 30 seconds, synchronously and thread-safely, ensuring zero memory leaks from dormant IP addresses.
• Promtail & Grafana Loki: Shipping of structured JSON logs from the local logs directory and indexing them in Loki.
• Dashboards as Code (IaC): Full version control of 5 dashboards in Git under grafana/dashboards/, automatically loaded into Grafana on container startup.
• Docker Compose Stack: One-click deployment of the entire observability infrastructure.

---

📝 JSON Log Structure (Structured Logging)

{
  "timestamp": "2026-08-06T10:30:15.123456",
  "level": "WARNING",
  "message": "[PORT SCAN DETECTED] Host 10.0.0.4 scanned 18 unique ports",
  "logger": "NetworkGuardian",
  "src_ip": "10.0.0.4",
  "event_type": "PORT_SCAN",
  "details": "18 ports scanned"
}

---

⚙️ Installation & Quick Start

## 1. Clone the repository
git clone https://github.com/Raz-Eini/python_sniffer.git
cd python_sniffer
2. Environment Setup

cp .env.example .env # Set your Grafana password in .env
3. Start Observability Stack (Grafana, Loki, Promtail)

Grafana will automatically provision all dashboards from grafana/dashboards/

docker compose up -d
4. Setup Python Environment

python -m venv .venv
..venv\Scripts\activate     # On Windows
source .venv/bin/activate    # On Linux/Mac
pip install -r requirements.txt
5. Run NIDS Engine (Requires Administrator / Root)

On Linux / Mac:

sudo .venv/bin/python main.py
On Windows (Run PowerShell / CMD as Administrator):

python main.py
6. (Optional) Run Attack Simulator in a separate terminal

python test_attack.py

📊 Accessing Grafana: Open your browser to http://localhost:3000 (username: admin, password set in .env). All dashboards will already be loaded and ready to use!

---

📄 License

This project is distributed under the MIT license – free to use and modify for educational and research purposes.

---

👨‍💻 Raz Eini (2026)