Skip to content

PRD: Gemma 4 brain integration into Vector AI system #1

Description

@chatde

Overview

The Vector AI Brain at /Volumes/AI-Models/vector/ already has a production-grade autonomous system:

  • Priority-based BehaviorArbiter (SAFETY > ONBOARD > AI > IDLE)
  • Persistent MemoryBank with Jaccard scene similarity
  • LearningEngine for autonomous environment discovery
  • MissionController (SecurityPatrol, ObjectCensus, RoomCheck, PresenceWatch)
  • SpatialMemory for room mapping
  • VoiceHandler for voice commands

Current AI stack: LLaVA (vision) + Llama 3.1 (personality) — two separate models, no memory context injection.

Goal

Replace both models with Gemma 4 (single call) that:

  1. Sees through Vector's camera (multimodal vision)
  2. Gets injected with full MemoryBank context — every prior observation, known objects, room history
  3. Returns unified nav decision + personality reaction + memory note in one JSON response
  4. Plugs transparently into the existing BehaviorArbiter + MemoryBank + MissionController

Architectural decisions

  • Swap mechanism: GEMMA4=1 env var in rover_brain.py — zero changes to existing code paths
  • Model: gemma4:e4b via Ollama REST at http://localhost:11434
  • Context injection: MemoryBank.get_recent(5) + known_objects + SpatialMemory summary prepended to every prompt
  • Output contract: JSON with keys: scene, objects, nav_decision, danger_level, reaction, memory_note
  • Patcher: patch_all_models() monkey-patches mission_controller._ask_vision and learning_engine LLaVA calls
  • Fallback: if Gemma 4 unavailable, falls back to original AIBrain (LLaVA)

Delivered so far

  • gemma4_brain.py — Gemma4Brain class + patch_all_models() + standalone CLI
  • rover_brain.py — GEMMA4=1 swap point added
  • gemmi_processor.py — Gemma 4 replaces LLaVA as Ollama fallback
  • gemma4_companion.py — lightweight companion loop using existing SDK config

What remains

See child issues for implementation slices.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions