Stereo-Object-Detection

🚀 Smart Vision System using ESP32-CAM + YOLO

📌 Problem Statement

Visually impaired individuals and general users often lack an affordable, portable system for real-time environmental awareness. Existing solutions are expensive, bulky, or require cloud dependency.

---

💡 Solution

We built a low-cost, real-time smart vision system using an ESP32-CAM and an Android application powered by YOLO object detection.

The system captures live images, transmits them wirelessly, processes them on-device, and provides visual + audio feedback to the user.

---

🧠 Key Features

• 📷 Real-time image capture using ESP32-CAM
• 📡 Wireless transmission over WiFi (HTTP)
• 🤖 On-device object detection using YOLOv8
• 📏 Distance estimation (based on bounding box scaling)
• 🔊 Voice feedback using Text-to-Speech (TTS)
• 📱 Clean Android UI for portable,offline and light monitoring
• 💻 WEB-APP for more accurate detections, hosted locally
• ⚡ Low-cost and portable edge solution

---

🏗️ System Architecture

       ┌──────────────────────┐
       │      ESP32-CAM (L)   │
       └─────────┬────────────┘
                 │
                 │ WiFi (Image Stream / Capture)
                 ▼
       ┌──────────────────────┐
       │      ESP32-CAM (R)   │
       └─────────┬────────────┘
                 │
                 ▼
        ┌──────────────────────┐
        │   Web-App Server     │
        │     (app.py)         │
        ├──────────────────────┤
        │ • Stereo Matching    │
        │ • WLS Filtering      │
        │ • Depth Estimation   │
        │ • YOLOv8 Detection   │
        └─────────┬────────────┘
                  │
                  ▼
        ┌──────────────────────┐
        │   Browser UI (User)  │
        ├──────────────────────┤
        │ • Camera Feeds       │
        │ • Depth Map          │
        │ • Bounding Boxes     │
        │ • Distance (meters)  │
        └──────────────────────┘


    (Alternative Mobile Pipeline)

       ┌──────────────────────┐
       │     ESP32-CAM        │
       └─────────┬────────────┘
                 │ WiFi (Image POST)
                 ▼
       ┌──────────────────────┐
       │   Android Device     │
       ├──────────────────────┤
       │ • YOLOv8 Detection   │
       │ • Distance Estimation│
       │ • Text-to-Speech     │
       └─────────┬────────────┘
                 ▼
       ┌──────────────────────┐
       │      User Output     │
       ├──────────────────────┤
       │ • Bounding Boxes     │
       │ • Labels             │
       │ • Audio Alerts       │
       └──────────────────────┘

---

🔄 Workflow Diagram

🌐 WEB-APP Pipeline

ESP32-CAM (L/R) -> WEB-APP: Send Stereo Frames
WEB-APP -> Stereo Engine: Compute Disparity (WLS)
WEB-APP -> YOLOv8: Detect Objects
YOLOv8 -> WEB-APP: Detection Results
WEB-APP -> WEB-APP: Distance Calculation
WEB-APP -> User (Browser): Display + Depth Map + Distances

📱 ANDROID Pipeline

ESP32-CAM -> APP: Send Image Frame
APP -> YOLOv8: Process Image
YOLOv8 -> APP: Detection Results
APP -> APP: Distance Calculation
APP -> User: Display + Voice Output

🔌 Hardware Requirements

Component	Description
ESP32-CAM (AI Thinker)	Camera + microcontroller
5V Power Supply	Stable external power
Jumper Wires	Connections

---

🔧 Hardware Setup

Connections:

• 5V → ESP32 5V
• GND → ESP32 GND

⚠️ Important:

• Use stable 5V supply (ESP32-CAM is sensitive to voltage drops)
• During programming: connect GPIO0 → GND
• After upload: remove GPIO0 connection

---

📡 Software Stack

🔹 Embedded (ESP32)

• Arduino IDE
• ESP32 Camera Library
• WiFi + HTTP Streaming

🔹 WEB-APP (Main System)

• Python (Flask / FastAPI)
• OpenCV (Stereo Matching + WLS Filtering)
• YOLOv8 (PyTorch / ONNX)
• NumPy (Depth Computation)
• HTML + CSS + JavaScript (Browser UI)

🔹 Mobile App (Alternative)

• Android (Kotlin / Java)
• YOLOv8 (TFLite)
• TextToSpeech API

---

⚙️ Installation Guide

1️⃣ ESP32 Setup

1. Install Arduino IDE

2. Add ESP32 board:

   https://dl.espressif.com/dl/package_esp32_index.json
   

3. Select:

   Board: AI Thinker ESP32-CAM
   

4. Upload code to both cameras (Left & Right)

5. Open Serial Monitor → note both IP addresses

---

2️⃣ WEB-APP Setup (Main System)

1. Clone the repository:

   git clone https://github.com/your-username/your-repo.git
   cd your-repo
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Update camera URLs in app.py:

   LEFT_CAM_URL = "http://<left_cam_ip>"
   RIGHT_CAM_URL = "http://<right_cam_ip>"
   

4. Run the server:

   python app.py
   

5. Open in browser:

   http://<your-device-ip>:5000
   

---

3️⃣ Android App Setup (Alternative)

1. Open project in Android Studio

2. Connect phone

3. Grant permissions:

   • Camera
   • Microphone (for TTS output)
   • Internet

4. Run the app

---

🚀 Usage Instructions

1. Turn on mobile hotspot
2. Power ESP32-CAM
3. Ensure both devices are on same network
4. Enter ESP32 IP in app (if required)
5. Start detection

---

📏 Distance Estimation (Stereo Triangulation)

This project estimates distance using stereo triangulation, similar to how human vision perceives depth.

🔢 Core Formula

Distance= f*B/d​

f (Focal Length): Camera parameter in pixels (FOCAL_LENGTH_PX = 1700)

B (Baseline): Distance between the two cameras (BASELINE_CM = 10 cm)

d (Disparity): Horizontal pixel difference of the object between left and right images

🎯 How Disparity is Calculated:

Object Matching Detects the same object (e.g., person) in both images using label, position, and size. Refined Matching Extracts a small patch from the object in the left image Uses OpenCV matchTemplate to find the best match in the right image Computes disparity as the difference in X-coordinates

🔄 Smoothing for Stability:

To reduce noise and flickering, the system applies a rolling average (last 10 values) to stabilize disparity and produce smoother distance estimates.

📌 Key Insight Large disparity → Object is close Small disparity → Object is far

---

⚠️ Limitations

• Dependent on lighting conditions
• Distance estimation is approximate
• Network latency may affect performance
• ESP32 has limited processing power

---

🔮 Future Improvements

• 📍 GPS tracking
• 🎯 Better distance estimation using depth models
• 🔋 Battery-powered portable design
• 🧠 Custom-trained YOLO model

---

🧪 Demo Instructions (for Judges)

1. Power the ESP32-CAM
2. Open the Android app
3. Show live detection
4. Demonstrate voice output
5. Move objects closer/farther to show distance effect

---

🏆 Innovation Highlights

• Combines IoT + Computer Vision + Mobile AI
• Fully portable and low-cost
• Works in real-time without cloud dependency
• Scalable for smart assistive systems

---

👨‍💻 Team

• Ryan Mittal
• Abdul
• Aryan Khare

---