RemindMe - AR Glasses to help dementia patients stay connected with their loved ones.

A comprehensive real-time face recognition system designed to assist individuals with dementia by identifying people in their environment and providing contextual information through visual and audio feedback.

Project Overview

RemindMe is an assistive technology solution that helps individuals with dementia recognize and remember people in their daily lives. The system uses computer vision to identify faces in real-time, displays information on an LCD screen, and provides audio announcements when requested. It also allows users to add new people to the system through a simple capture process.

Demo Video

A short demo showing RemindMe recognizing faces in real time, displaying names and relationships on the LCD, and providing audio feedback through text-to-speech.

Key Capabilities

• Real-time Face Recognition: Continuously monitors webcam feed and identifies known individuals
• Visual Display: Shows recognized person's name and relationship on LCD screen
• Audio Feedback: Text-to-speech announcements of recognized individuals
• Database Storage: MongoDB stores face embeddings and relationship information
• Easy Person Addition: Button-triggered workflow to add new people to the system
• Relationship Tracking: Maintains contextual information about each person (e.g., "mother", "caregiver", "doctor")

---

Features

✅ Real-time face recognition using webcam
✅ MongoDB database for persistent face storage
✅ LCD display showing name and relationship
✅ Text-to-speech announcements (macOS native or pyttsx3)
✅ Two-button interface for TTS and person capture
✅ Relationship tracking for contextual information
✅ Performance optimized for real-time processing
✅ Multi-platform support (macOS, Linux, Windows)
✅ Automatic face embedding averaging for better accuracy

---

Architecture Overview

The system consists of three main components:

1. Python Application (main.py)

   • Face detection and recognition engine
   • Database management
   • Serial communication handler
   • Text-to-speech controller

2. ESP32/Arduino Firmware (LCD_Dementia/src/main.cpp)

   • LCD display controller
   • Button input handler
   • Serial communication bridge

3. MongoDB Database

   • Stores face embeddings
   • Maintains person names and relationships
   • Enables persistent data across sessions

Data Flow

Webcam → Python (Face Detection) → Face Recognition → MongoDB Query
                                                      ↓
ESP32 ← Serial Communication ← Person Info + Relationship
  ↓
LCD Display + Button Input → Serial → Python (TTS/Capture)

Component Interaction

1. Face Recognition Loop: Continuously processes webcam frames, detects faces, and matches against database
2. Serial Communication: Bidirectional communication between Python and ESP32
   • Python → ESP32: Person name and relationship updates
   • ESP32 → Python: Button press events (TTS_REQUEST, CAPTURE_REQUEST)
3. Database Operations: Face embeddings stored as numpy arrays (converted to lists), queried for matching
4. TTS System: Thread-safe text-to-speech using platform-native solutions

---

Hardware Requirements

Required Components

• Computer (macOS, Linux, or Windows)

  • Webcam (built-in or USB)
  • USB port for serial communication

• ESP32 Development Board

  • Any ESP32 variant (ESP32-WROOM, ESP32-DevKit, etc.)

• 16x2 I2C LCD Display

  • Compatible with LiquidCrystal_I2C library
  • Common I2C address: 0x27 or 0x3F

• Two Push Buttons

  • Momentary push buttons
  • Internal pull-up resistors used (no external resistors needed)

Wiring Diagram

ESP32 Connections:
├── LCD Display (I2C)
│   ├── SDA → GPIO 21
│   ├── SCL → GPIO 22
│   ├── VCC → 5V
│   └── GND → GND
│
└── Buttons
    ├── TTS Button → GPIO 18 → GND (when pressed)
    └── Capture Button → GPIO 19 → GND (when pressed)

Note: Buttons use internal pull-up resistors. When pressed, they connect to GND (LOW state).

---

Software Requirements

Python Version

• Python 3.8, 3.9, 3.10, or 3.11 (Python 3.9 or 3.10 recommended)

Required Python Packages

See requirements.txt for complete list:

• opencv-python>=4.8.0 - Computer vision and webcam handling
• face_recognition>=1.3.0 - Face detection and recognition
• numpy>=1.24.0 - Numerical operations
• pymongo==4.4.1 - MongoDB database driver
• python-dotenv==1.0.0 - Environment variable management
• pyserial>=3.5 - Serial communication
• pyttsx3>=2.90 - Text-to-speech (fallback for non-macOS)

System Dependencies

macOS:

• CMake (for dlib compilation): brew install cmake
• Native say command (built-in, no installation needed)

Linux:

• CMake: sudo apt-get install cmake
• espeak or festival for TTS

Windows:

• CMake: Download from cmake.org
• SAPI5 TTS (built into Windows)

Database

• MongoDB (local or cloud instance)
  • MongoDB Atlas (cloud) recommended for easy setup
  • Or local MongoDB installation

Development Tools

• PlatformIO or Arduino IDE for ESP32 firmware
• Git for version control

---

Installation

Step 1: Clone Repository

git clone <repository-url>
cd Dementia_Glasses

Step 2: Create Virtual Environment

# Check Python version (should be 3.8-3.11)
python3 --version

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
source venv/bin/activate  # macOS/Linux
# or
venv\Scripts\activate     # Windows

Step 3: Install System Dependencies

macOS:

brew install cmake

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install cmake

Windows: Download and install CMake from cmake.org

Step 4: Install Python Dependencies

pip install -r requirements.txt

Note for macOS users: If face_recognition installation fails due to dlib:

# Install cmake first (see Step 3)
pip install dlib
pip install -r requirements.txt

Step 5: Install ESP32 Firmware

Using PlatformIO (Recommended):

cd LCD_Dementia
platformio run --target upload

Using Arduino IDE:

1. Install ESP32 board support in Arduino IDE
2. Install LiquidCrystal_I2C library
3. Open LCD_Dementia/src/main.cpp
4. Select ESP32 board and upload

---

Configuration & Setup

MongoDB Setup

Option A: MongoDB Atlas (Cloud - Recommended)

1. Sign up at MongoDB Atlas
2. Create a free cluster
3. Create a database user
4. Whitelist your IP address (or use 0.0.0.0/0 for development)
5. Get connection string (format: mongodb+srv://username:password@cluster.mongodb.net/)

Option B: Local MongoDB

1. Install MongoDB locally
2. Start MongoDB service
3. Connection string: mongodb://localhost:27017/

Connect ESP32

1. Connect ESP32 to computer via USB

2. Note the serial port:

   • macOS: /dev/cu.usbserial-* or /dev/cu.SLAB_USBtoUART
   • Linux: /dev/ttyUSB0 or /dev/ttyACM0
   • Windows: COM3, COM4, etc.

3. The Python script will auto-detect the port, but you can verify:

python3 -m serial.tools.list_ports

Initial Person Database Setup

Before running the main application, add at least one person to the database:

Option A: Using capture_face.py script:

python capture_face.py
# Enter person name when prompted
# Enter relationship when prompted
# Follow on-screen instructions to capture photos

Option B: Programmatically (for testing):

from config_db import persons
import face_recognition
import numpy as np
from PIL import Image

# Load an image and create embedding
image = face_recognition.load_image_file("path/to/person.jpg")
encoding = face_recognition.face_encodings(image)[0]

# Save to database
persons.insert_one({
    "name": "Person Name",
    "relationship": "mother",
    "embeddings": encoding.tolist()
})

---

Usage

Starting the Application

1. Activate virtual environment:

source venv/bin/activate  # macOS/Linux

2. Ensure ESP32 is connected and firmware is uploaded

3. Start the application:

python main.py

Application Workflow

1. Initialization:

   • Loads face embeddings from MongoDB
   • Connects to ESP32 via serial
   • Initializes webcam
   • Displays "Unknown" on LCD

2. Face Recognition:

   • Continuously processes webcam frames
   • Detects and recognizes faces
   • Updates LCD every 2 seconds with name and relationship
   • Draws bounding boxes on video feed

3. Button Interactions:

   TTS Button (GPIO 18):

   • Press to hear: "This is [Name] and they are your [Relationship]"
   • Works for recognized faces only
   • Non-blocking (doesn't interrupt face recognition)

   Capture Button (GPIO 19):

   • Press to add a new person
   • Follows interactive prompts:
     1. Confirmation: "Would you like to add a new person? (Yes/No)"
     2. Name input: "Enter name of person you want to add"
     3. Relationship input: "Enter their relationship to you"
     4. Automatic capture of 5 frames
     5. Saves to MongoDB and reloads database

4. Exiting:

   • Press q key in the video window to quit
   • Serial connection closes automatically

Performance Tuning

Edit main.py to adjust performance settings:

# Face recognition settings
SIMILARITY_THRESHOLD = 0.5  # Lower = stricter (0.3-0.6 recommended)
PROCESS_EVERY_N_FRAMES = 2  # Process every Nth frame (1=every frame)
PROCESSING_SCALE = 0.25      # Resize factor (0.25=quarter size, faster)
USE_HOG_MODEL = True         # True=HOG (fast), False=CNN (accurate but slow)
CAMERA_WIDTH = 640           # Camera resolution
CAMERA_HEIGHT = 480

# Update frequency
LCD_UPDATE_INTERVAL = 2      # Seconds between LCD updates

---

File Structure

Dementia_Glasses/
├── main.py                 # Main application (face recognition, serial comm)
├── capture_face.py         # Standalone script to add people to database
├── config_db.py            # MongoDB connection configuration
├── tts.py                  # Text-to-speech utilities (legacy, not used)
├── requirements.txt        # Python dependencies
├── .env                    # Environment variables (MongoDB credentials)
├── .gitignore              # Git ignore rules
├── README.md               # This file
│
├── known_faces/            # Legacy folder (not used, data in MongoDB)
│   ├── Person1/
│   └── Person2/
│
└── LCD_Dementia/           # ESP32 firmware
    ├── platformio.ini      # PlatformIO configuration
    └── src/
        └── main.cpp        # ESP32 Arduino code

Database Schema

Collection: persons

{
  "_id": ObjectId("..."),
  "name": "John Doe",           // Person's name
  "relationship": "caregiver",  // Relationship to user
  "embeddings": [0.123, ...]    // 128-dimensional face embedding (list)
}

Serial Communication Protocol

Python → ESP32:

Format: "name|relationship\n"
Example: "John Doe|caregiver\n"

ESP32 → Python:

"TTS_REQUEST\n"      // TTS button pressed (GPIO 18)
"CAPTURE_REQUEST\n"  // Capture button pressed (GPIO 19)

Threading Model

• Main Thread: Face recognition loop, video processing
• TTS Thread: Text-to-speech execution (non-blocking)
• Serial Thread: Handled in main loop (non-blocking reads)

---

API Documentation

Main Application Functions

speak_text(text)

Non-blocking text-to-speech. Runs in separate thread.

Parameters:

• text (str): Text to speak

Platform Behavior:

• macOS: Uses native say command
• Other platforms: Uses pyttsx3

speak_text_blocking(text)

Blocking text-to-speech. Waits for completion.

Parameters:

• text (str): Text to speak

Use Case: Interactive prompts during capture flow

send_face_name_to_lcd(face_name)

Sends person information to ESP32 LCD.

Parameters:

• face_name (str): Recognized person's name

Format: "name|relationship\n"

process_serial_requests()

Handles incoming serial messages from ESP32.

Actions:

• TTS_REQUEST: Triggers TTS for current face
• CAPTURE_REQUEST: Initiates person capture flow

reload_known_faces()

Reloads face database from MongoDB.

Use Case: After adding new person via capture flow

Database Functions

capture_face.py - Standalone Script

Function: process_and_save_person(name, relation)

Captures photos and saves to database.

Parameters:

• name (str): Person's name
• relation (str): Relationship to user

Process:

1. Captures 10 frames with countdown
2. Extracts face encodings from each frame
3. Averages encodings
4. Saves to MongoDB

ESP32 Functions

displayFaceName(faceName, relationship)

Updates LCD display with person information.

Parameters:

• faceName (str): Person's name (line 1)
• relationship (str): Relationship (line 2)

Button Handlers

• GPIO 18: Sends "TTS_REQUEST" on press
• GPIO 19: Sends "CAPTURE_REQUEST" on press
• Both use 200ms debounce delay

---

Troubleshooting

Common Issues

"TTS unavailable" or TTS not working

macOS:

• Verify say command works: say "test"
• Check system permissions for microphone/audio

Other platforms:

• Install TTS engine: sudo apt-get install espeak (Linux)
• Check pyttsx3 installation: pip install pyttsx3

"No serial port found"

Solutions:

• Verify ESP32 is connected via USB
• Check USB cable (data cable, not charge-only)
• Install USB drivers:
  • CH340: CH340 Driver
  • CP2102: CP2102 Driver
• List available ports: python3 -m serial.tools.list_ports

"No embeddings found in database"

Solution:

• Add at least one person using capture_face.py
• Verify MongoDB connection in .env file
• Check database name matches DB_NAME in .env

Face recognition not working / low accuracy

Solutions:

• Adjust SIMILARITY_THRESHOLD (try 0.4-0.6)
• Ensure good lighting
• Face should be front-facing and clearly visible
• Add more training images per person (use capture_face.py multiple times)

MongoDB connection errors

Solutions:

• Verify MONGO_URI in .env is correct
• Check MongoDB Atlas IP whitelist (if using cloud)
• Test connection: mongosh "<MONGO_URI>"
• Ensure database name exists

ESP32 not responding

Solutions:

• Verify firmware is uploaded correctly
• Check wiring (SDA/SCL, button connections)
• Monitor serial output: screen /dev/cu.usbserial-* 115200 (macOS)
• Verify I2C LCD address (common: 0x27 or 0x3F)

🔗 Devpost

Check out our Devpost submission for RemindMe:
https://devpost.com/software/remindme-dvlhj2