Lens

Local visual telemetry, Visual Question Answering (VQA), and OCR gateway microservice for Citadel. Lens acts as the "eyes" of the local AI agent system, allowing it to inspect environment state, read screen elements, verify UI layouts, and analyze user emotion, focus, and gestures in real time.

---

Demo

Here is a typical interaction pattern with the /v1/vision/telemetry endpoint returning object, facial, and body telemetry:

# Submit a video frame for high-speed local telemetry extraction
curl -X POST http://localhost:8004/v1/vision/telemetry \
  -F "file=@tests/fixtures/sample_text.png"

Response payload:

{
  "face": {
    "pose": {"pitch": -5.2, "yaw": 12.8, "roll": 1.4},
    "gaze": {"horizontal": -0.15, "vertical": 0.05},
    "blendshapes": {
      "mouthSmileLeft": 0.82,
      "mouthSmileRight": 0.85,
      "browDownLeft": 0.02,
      "browDownRight": 0.01,
      "jawOpen": 0.0
    },
    "emotions": {
      "happy": 0.83,
      "neutral": 0.12,
      "sad": 0.01,
      "angry": 0.01,
      "surprise": 0.03
    }
  },
  "body": {
    "posture": {
      "shoulder_alignment_angle": 1.2,
      "slouch_metric": 0.05
    },
    "hands": [
      {
        "side": "left",
        "gesture": "open_palm",
        "confidence": 0.94
      }
    ]
  },
  "objects": [
    {
      "label": "cell phone",
      "confidence": 0.88,
      "bbox": {
        "x": 0.12,
        "y": 0.65,
        "width": 0.15,
        "height": 0.22
      }
    }
  ]
}

---

Features

• Visual Question Answering & VQA: Analyze visual interface state and answer complex natural language queries using a local VLM.
• Biometric Telemetry: Calculate head pose (pitch, yaw, roll), 2D pupil gaze coordinates, and 52 native ARKit blendshapes (to classify user emotions).
• Body & Hand Posture: Analyze spine slouching (nose vs shoulder midpoint height), shoulder alignment, and classify 5 discrete hand gestures (open palm, closed fist, thumbs up, thumbs down, pointing).
• CPU-Accelerated Object Tracking: Detect 80 classes of physical items (phones, laptops, cups) locally using OpenCV's DNN engine bypassing heavy PyTorch libraries.
• Built-in Deskew & CLAHE Preprocessing: Automate bounded resizing, upscaling, deskewing, Gaussian denoising, and contrast normalization before processing.

---

Tech Stack

• Python 3.12: Standard scripting runtime.
• FastAPI & Uvicorn: Lightweight asynchronous web framework with standard request/response validation.
• OpenCV (cv2): Used for rapid image decoding, PnP geometry calculation, and loading the ONNX object detector.
• MediaPipe Tasks API: Machine learning pipeline for face landmarker, pose tracker, and hand models running locally under 150ms.
• Ollama REST Client: Proxies heavy multimodal cognitive VQA tasks to a local Metal GPU-accelerated Ollama process, conserving microservice memory space.

---

Getting Started

Prerequisites

• macOS 14+ (recommended to leverage the native Neural Engine & Apple Vision OCR APIs).
• Python 3.12+
• Local Ollama server running (http://localhost:11434) with the model pulled (e.g. ollama pull qwen2.5vl:3b).

Local Installation

1. Clone the repository:

   git clone git@github.com:rounakbajpayee/lens.git
   cd lens

2. Setup virtual environment:

   python -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install packages:

   pip install -r requirements.txt
   pip install -r requirements-dev.txt

   Note: If running on macOS, install the Apple Vision bindings manually:

   pip install pyobjc-framework-Vision pyobjc-framework-Quartz

4. Setup local environment configuration:

   cp .env.example .env

5. Install pre-commit hooks:

   pre-commit install

Running Locally

To launch the development server:

uvicorn src.main:app --reload --port 8004

---

Architecture

Lens is structured as a decoupled web gateway routing requests across lightweight in-process engines and the local Ollama daemon:

                  ┌───────────────────────┐
                  │      Client App       │
                  └───────────┬───────────┘
                              │ HTTP Requests
                              ▼
                  ┌───────────────────────┐
                  │    FastAPI Router     │
                  └─────┬───────────┬─────┘
                        │           │
          ┌─────────────┘           └─────────────┐
          ▼ in-process                            ▼ REST call
┌───────────────────────┐               ┌───────────────────┐
│   Telemetry Engines   │               │ Local Ollama VLM  │
│ (MediaPipe/OpenCV-DNN)│               │  (qwen2.5vl:3b)   │
└───────────────────────┘               └───────────────────┘

---

API Reference

GET /health

Returns status of the LENS service.

GET /metrics

Returns in-memory operational metrics (total request count, duration summaries, errors).

POST /v1/ocr/extract

Extract text from an image.

• Content-Type: multipart/form-data
• Fields:
  • file: Image binary file (Required).
  • engine: vision (macOS default) or surya (optional).
  • structured: true or false (optional, returns detailed bounding boxes).
  • lang: OCR language code for engines that support language routing; defaults to en.
  • cache: true or false; when enabled, identical image/engine/structured/lang requests are served from a bounded in-memory LRU cache.

Structured OCR bounding boxes use normalized coordinates (x, y, width, height) in the range 0.0 to 1.0.

POST /v1/vision/analyze

Execute VQA on an image.

• Content-Type: multipart/form-data
• Fields:
  • file: Image binary file (Required).
  • prompt: Query prompt (optional, defaults to "Describe this image in detail.").
  • model: Ollama model name (optional, defaults to qwen2.5vl:3b).

POST /v1/vision/telemetry

Extract facial, body, hand, and object telemetry.

• Content-Type: multipart/form-data
• Query Params:
  • face: true or false (default: true).
  • body: true or false (default: true).
  • objects: true or false (default: true).

---

Roadmap / Known Issues

• MediaPipe Headless Run: When executing tests on headless Linux systems (like GitHub Actions runners), MediaPipe can crash due to missing OpenGL libraries. We resolve this by ensuring OpenGL packages are installed in the host system.
• WebSockets Stream: Introduce a streaming WebSocket port for continuous low-latency video feed processing in a future phase.

---

License

This project is licensed under the AGPLv3. For commercial use without open-sourcing your application, please contact the author to purchase a commercial license.