Crowd Surveillance Anomaly Detection System

Real-time violence detection from video feeds using rule-based behavioral analysis — no model training required.

Overview

This system processes video footage frame by frame, detects and tracks every person in the scene, and applies three independent behavioral rules to identify violent or aggressive activity in real time. When an alert fires, the screen shows a red bounding box and banner, and the system automatically saves a pre-incident clip for review — including 2 seconds of footage before the alert triggered.

Built as a 6th semester college project with no custom model training. All components use pre-trained open-source models.

Results

Evaluated on 50 labelled clips (25 violence, 25 non-violence) from the Real-Life Violence Situations Dataset.

Metric	Value
Detection Rate (Recall)	80%
False Positive Rate	4%
Overall Accuracy	88%
True Positives	20 / 25
False Negatives	5 / 25
True Negatives	24 / 25
False Positives	1 / 25
Processing Speed (GPU)	15 – 75 FPS
Processing Speed (CPU)	5 – 10 FPS

How It Works

Every video frame passes through a 4-stage pipeline:

Video Input
    │
    ▼
YOLOv8 Nano ──────── detects all persons → [x1,y1,x2,y2,conf]
    │
    ▼
DeepSORT Tracker ─── assigns persistent IDs, stores 10-frame position history
    │
    ▼
PoseAnalyzer ─────── evaluates 3 rules per person per frame
    │         ├── Rule 1: Velocity   > 15 px/frame (rolling 5-frame avg)
    │         ├── Rule 2: IoU Overlap > 0.20 with any other person
    │         └── Rule 3: Wrist above shoulder (MediaPipe landmarks)
    │
    ▼
Alert Logic ──────── fires if: velocity AND (overlap OR pose)
                     requires 3 consecutive frames before alerting
    │
    ▼
Output: annotated video + alert banner + auto-saved incident clips

The 3 Detection Rules

Rule 1 — Velocity Measures how fast each tracked person is moving. A 5-frame rolling average eliminates tracker jitter. Crossing 15 px/frame indicates aggressive motion.

Rule 2 — Bounding Box Overlap (IoU) Measures whether two people's bounding boxes share more than 20% of their combined area — consistent with grappling or direct contact.

Rule 3 — Arm Elevation (MediaPipe Pose) Checks whether a wrist keypoint is positioned above the corresponding shoulder in image coordinates. Requires landmark visibility > 0.5 to fire.

Alert Fusion

alert = velocity_flag AND (overlap_flag OR pose_flag)

Requiring velocity as a mandatory condition eliminates the majority of false positives from people who are simply close or raising an arm for non-violent reasons.

Temporal Smoothing An alert only fires when the same person triggers the condition for 3 or more consecutive frames. Single-frame detection artifacts are silently ignored.

Features

Real-time processing — works on video files, webcam, or any RTSP stream
Yellow skeleton overlay — body joints and limb connections drawn per person
Color-coded bounding boxes — green (normal) / red (alert)
Pre-incident clip saving — 2 seconds before + 3 seconds after each alert, automatically saved to alert_clips/
Structured logging — every alert logged with timestamp, person ID, exact rule values, and consecutive frame count
GPU acceleration — YOLO runs on CUDA; MediaPipe runs on CPU
Batch evaluation — separate evaluate.py script for measuring TP/FP/TN/FN across a labelled clip set
Modular architecture — each component is independently replaceable

Project Structure

fight_detection/
│
├── main.py                     # Main pipeline, clip saver, alert logger
├── detector.py                 # YOLOv8 wrapper — person detection
├── tracker.py                  # DeepSORT wrapper — persistent ID tracking
├── pose_analyzer.py            # 3 behavioral rules + MediaPipe pose
├── visualizer.py               # Skeleton drawing, bounding boxes, alert banner
├── config.py                   # All thresholds in one place
├── evaluate.py                 # Batch evaluation on labelled clip sets
│
├── pose_landmarker_lite.task   # MediaPipe model file (download separately)
├── requirements.txt
│
├── output_annotated.mp4        # Annotated output video (auto-created)
├── detection_log.txt           # Per-alert log (auto-created)
├── evaluation_results.txt      # Evaluation metrics (auto-created)
└── alert_clips/                # Auto-saved incident clips
    ├── alert_1.mp4
    ├── alert_2.mp4
    └── ...

Requirements

Hardware

CPU: Intel i5 / Ryzen 5 or better
RAM: 8 GB recommended
GPU: NVIDIA GPU with CUDA support (optional but recommended — RTX 3050 or better)
Storage: ~3 GB for models and libraries

Software

Windows 10 / 11 (tested on Windows 11)
Python 3.10 or 3.11
NVIDIA drivers with CUDA 12.x support (for GPU acceleration)

Installation

1. Clone the repository

git clone https://github.com/yourusername/fight-detection.git
cd fight-detection

2. Create and activate a virtual environment

# Windows
python -m venv venv
venv\Scripts\activate

# Mac / Linux
python -m venv venv
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

Or install manually:

pip install ultralytics mediapipe==0.10.32 opencv-python numpy deep-sort-realtime

4. Install PyTorch with GPU support (recommended)

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Skip this step if you have no NVIDIA GPU. The system will run on CPU only.

Verify GPU is available:

python -c "import torch; print('CUDA:', torch.cuda.is_available())"

5. Download the MediaPipe pose model

Open this URL in your browser and save the file to the project root:

https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task

The file must be named pose_landmarker_lite.task and placed in the project root directory.

Usage

Run on a video file

Open main.py and set the video path:

VIDEO_PATH = r"C:\path\to\your\video.mp4"
cap = cv2.VideoCapture(VIDEO_PATH)

Then run:

python main.py

Run on a webcam

Comment out VIDEO_PATH and switch to webcam input:

# VIDEO_PATH = r"C:\path\to\video.mp4"
cap = cv2.VideoCapture(0)   # 0 = default webcam, 1 = external

Run on an IP camera / RTSP stream

cap = cv2.VideoCapture("rtsp://username:password@192.168.1.x:554/stream")

# Or using DroidCam (phone as webcam):
cap = cv2.VideoCapture("http://192.168.1.x:4747/video")

Press Q at any time to stop.

Output

File	Contents
`output_annotated.mp4`	Full annotated video of the session
`alert_clips/alert_N.mp4`	Auto-saved incident clips (2s before + 3s after)
`detection_log.txt`	Per-alert log with timestamps and rule values

Evaluation

To measure system accuracy on a labelled clip set:

1. Configure clip lists in `evaluate.py`

TEST_DIR = r"C:\path\to\test_videos"

VIOLENCE_CLIPS    = ["V_1.mp4", "V_2.mp4", ...]
NONVIOLENCE_CLIPS = ["NV_1.mp4", "NV_2.mp4", ...]

2. Run evaluation

python evaluate.py

Results are printed to the terminal and saved to evaluation_results.txt.

evaluate.py uses analyze_person_fast which skips MediaPipe pose estimation for batch speed. Results reflect velocity + overlap rules only. This gives conservative estimates aligned with the two strongest detection signals.

Configuration

All thresholds are centralised in config.py:

VELOCITY_THRESHOLD  = 15     # px/frame — minimum speed to flag Rule 1
OVERLAP_THRESHOLD   = 0.20   # IoU — minimum bounding box overlap for Rule 2
CONFIDENCE_THRESHOLD = 0.5   # YOLO detection confidence minimum
SMOOTHING_WINDOW    = 3      # consecutive frames before alert fires (main.py)
BUFFER_SECONDS      = 2      # seconds of pre-alert footage to buffer
POST_ALERT_SECS     = 3      # seconds of post-alert footage to save

Tuning guide

Symptom	Fix
Too many false alerts	Increase `VELOCITY_THRESHOLD` or `OVERLAP_THRESHOLD`
Missing real fights	Decrease `VELOCITY_THRESHOLD` or `OVERLAP_THRESHOLD`
Alerts fire too slowly	Decrease `SMOOTHING_WINDOW` (minimum 2)
Single-frame false positives	Increase `SMOOTHING_WINDOW`

Detection Log Format

Every alert produces a log entry like:

17:33:56  ALERT | t=12.73s frame=382 | Person #5 | rules=['velocity', 'overlap'] | vel=20.1px iou=0.30 pose=False | consecutive=3 frames

Field	Description
`t=12.73s`	Timestamp in the video
`frame=382`	Frame number
`Person #5`	DeepSORT track ID
`rules=[...]`	Which rules triggered
`vel=20.1px`	Rolling average velocity
`iou=0.30`	Maximum IoU with any other person
`pose=False`	Whether arm raise was detected
`consecutive=3`	Frames the condition held before alerting

Known Limitations

Overhead cameras — Pose estimation (Rule 3) fails from top-down angles. Bounding box shapes and overlap patterns change significantly.
Slow grappling — Sustained low-movement violence (chokes, holds) may not cross the velocity threshold.
Pixel-based velocity — Not normalized for camera distance. Persons far from the camera generate lower pixel displacement for the same physical movement.
Single-camera scope — No multi-camera tracking or cross-feed coordination.
Threshold generalization — Values were calibrated on the test dataset. Different environments (different lighting, crowd density, camera angles) may require recalibration.
MediaPipe on CPU — Pose estimation runs on CPU regardless of GPU availability. This is the primary throughput bottleneck.
No face recognition or identification — Track IDs are session-scoped integers only.

Stack

Component	Library	Role
Person Detection	YOLOv8 Nano (Ultralytics)	Locates all persons per frame
Multi-Object Tracking	DeepSORT (deep-sort-realtime)	Assigns persistent IDs
Pose Estimation	MediaPipe PoseLandmarker 0.10.32	Body joint detection
Video I/O	OpenCV	Frame reading, annotation, output
GPU Inference	PyTorch 2.6.0 + CUDA 12.4	YOLO acceleration
Numerical Operations	NumPy	Array math
Visualization	Matplotlib	Evaluation graphs

Dataset

Real-Life Violence Situations Dataset

Source: Kaggle — Mohamed Mustafa
2,000 short clips (5–10 seconds each): 1,000 violence / 1,000 non-violence
Real-world footage — not staged or synthetic
Used for evaluation only — no model training performed

Future Improvements

Replace MediaPipe with YOLOv8-pose for GPU-accelerated joint detection
Add velocity normalization by bounding box height to correct for camera distance
Implement loitering detection as a fourth behavioral rule
Build a web dashboard (Flask/Streamlit) for live feed monitoring and alert review
Add multi-camera coordination via shared event log
Implement operator feedback loop — dismissed alerts inform threshold calibration
Add audio integration for additional behavioral signals

References

Ultralytics YOLOv8 — https://github.com/ultralytics/ultralytics
MediaPipe Pose Landmarker — https://developers.google.com/mediapipe/solutions/vision/pose_landmarker
DeepSORT — https://github.com/levan92/deep_sort_realtime
Bewley et al. (2016) — Simple Online and Realtime Tracking (SORT). ICIP 2016.
Wojke et al. (2017) — Deep SORT with Deep Association Metric. ICIP 2017.
Jocher et al. (2023) — Ultralytics YOLOv8. doi:10.5281/zenodo.7347926

Built as part of Project 1 — 6th Semester, 2025–2026

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
README.md		README.md
config.py		config.py
detector.py		detector.py
evaluate.py		evaluate.py
main.py		main.py
pose_analyzer.py		pose_analyzer.py
requirements.txt		requirements.txt
tracker.py		tracker.py
visualizer.py		visualizer.py

Latest commit

History

Folders and files

Repository files navigation