diff --git a/README.md b/README.md index f483e6b..d353d65 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,8 @@ SAM 3.1 finds your things in it, and Muse Voice Transcribe + Muse Spark Open hea

…then tap Take me there, and the camera checks it's really there when you arrive.

+

Watch the 3-minute demo · Try the sample home

+

Built during the Meta Global AI Developer Hackathon, Oct 26 to Nov 4, 2026. Pre-existing work: none.
Indie track · Theme: Build for Your World

@@ -78,7 +80,7 @@ Indie track · Theme: Build for Your World

### For judges, in one minute -**Start with the demo.** Nothing to install: open +**Start with the [3-minute video](https://youtu.be/vRAR3FHNSks): a blindfolded walk to a watch, one take.** Then try it, with nothing to install: open **[recall-ten-khaki.vercel.app/demo](https://recall-ten-khaki.vercel.app/demo)**, pick "Where are my keys?", and follow the walk to them in the author's real scanned living room. To run the same thing from this repo, with no keys, no GPU and no account: @@ -93,11 +95,20 @@ come from the planner the phone uses, on the scan's floor. | Criterion | Where to look | |---|---| -| **Impact** (30 %) | [Why I built it](#why-i-built-it): a true story · [What it is](#what-it-is): built first for blind and low-vision people, for anyone who misplaces things, voice first · [Privacy](docs/privacy.md): the home stays on your own computer · the [tester session plan](evals/usability/) | +| **Impact** (30 %) | [Why I built it](#why-i-built-it): a true story · [What it is](#what-it-is): built first for blind and low-vision people, for anyone who misplaces things, voice first · [Privacy](docs/privacy.md): the home stays on your own computer · a [first real tester](#results-so-far) with low vision, and the [fuller session plan](evals/usability/) | | **Use of Meta AI tooling** (25 %) | [Only Meta models, each with one job](#how-it-works-the-meta-models): MapAnything, SAM 3.1, Muse Spark Open (Muse Glimmer on your own Mac), Muse Voice Transcribe, Muse Spark 1.1 · [Recall's three agents](#recalls-three-agents) · [Sharing with robots](#sharing-with-robots) over A2A | -| **Demo** (25 %) | [The demo page](https://recall-ten-khaki.vercel.app/demo) and `make demo`: a real run, no keys · the screenshots below are real output · [How to run it](#how-to-run-it) | +| **Demo** (25 %) | [The 3-minute video](https://youtu.be/vRAR3FHNSks) · [The demo page](https://recall-ten-khaki.vercel.app/demo) and `make demo`: a real run, recorded and labelled, no keys · the screenshots below are real output · [How to run it](#how-to-run-it) | | **Technical depth** (20 %) | [Architecture](docs/architecture.md) · [28 decisions](docs/decisions.md) · [Guardrails](#guardrails): geometry decides the facts, the model only phrases them · [Algorithms](#algorithms) · tests on every part ([Results so far](#results-so-far)) | +**Real use, from the app's own logs** (31 Oct to 4 Nov 2026, one flat, one iPhone; not a study, [details](evals/results.md)): + +| Measure | Result | +|---|---| +| Answers grounded in what Recall really saw | **49 of 49** | +| Gap between camera and memory, confirmed arrivals | **2 cm** median (n = 22) | +| Arrival checks where the camera saw the thing | **26 of 31** | +| Guide, question to answer | **2.3 s** median (n = 46) | + --- ## Contents @@ -129,43 +140,44 @@ come from the planner the phone uses, on the scan's floor. **Real output, from the author's own living room:** one walk with an iPhone, rebuilt on a Mac. The room map shows floor, furniture and walls as cells, and the things as dots. The scan holds no photos of the home; the only footage in this repo is one short screen -recording and one frame of it, which the author reviewed and chose to publish (`apps/web/public/demo-video/`). +recording and one frame of it, on the demo page, which the author reviewed and chose to publish (`apps/web/public/demo-video/`). - - - - - - - - - - - - - - - - - - + + + - - - + + + - - - + + +
Find: the question 'Where are my keys?' and the answer 'Your keys are near the desk in the living room at waist height, seen on Friday at 04:21', with Take me there and Why? buttons, and the room map belowFind: the Where is box with Speak and Ask, then Your rooms: the Living room card with its map, 20 things as blue dots, and See it in AR and Share for robotsWhy Recall thinks so: the answer, 'Phrased by Muse Spark Open from 1 sighting, in 1.8 seconds', and the sighting: seen in the Living room scan on 30 Oct 2026, 70% sure
Ask. Muse Spark Open picks the tools; the place comes from the 3D memory. Take me there walks you to it.Your rooms. The map MapAnything and SAM 3.1 built from one walk. Swipe between rooms; tap one to see only its things, or see them through the camera in AR.Why? The sighting behind the answer, and which model worded it, in how long.
Your things: the Living room, 20 things, on or near the cabinet: keys, wallet, spectacle case, laptop, bottle, watch and book, each with its height and a Go buttonFound: search, All, To check and Checked filters, group by place, height or kind; books listed by kind, and a brown book marked Seen onceRobot Access: the switch to share the home map with robots, off; the Living room map for robots; what robots receive and what is never shared
Your things. By room, then the furniture they're on or near. Go on every row.Found. Check what the scan found; fix a name, colour or height; the unsure ones are marked.Robot Access. Off by default. A robot gets the map and where things are, never photos or anyone's position.Find: the question 'Where are my keys?' and the answer 'Your keys are near the desk in the living room at waist height, seen on Friday at 04:21', with Take me there and Why? buttons, and the room map below

Ask. Muse Spark Open picks the tools; the place comes from the 3D memory. Take me there walks you to it.

Find: the Where is box with Speak and Ask, then Your rooms: the Living room card with its map, 20 things as blue dots, and See it in AR and Share for robots

Your rooms. The map MapAnything and SAM 3.1 built from one walk. Swipe between rooms; tap one to see only its things, or see them through the camera in AR.

Why Recall thinks so: the answer, 'Phrased by Muse Spark Open from 1 sighting, in 1.8 seconds', and the sighting: seen in the Living room scan on 30 Oct 2026, 70% sure

Why? The sighting behind the answer, and which model worded it, in how long.
Scans: Scan a room with a Start a scan button, then Your scans: Living room, Usable, 173 frames, processed, and a note that this walk didn't line up with the last scan, so Recall can't say what movedDevices: Where Recall runs, with tabs for iPhone, Watch, Glasses and Robots, and the iPhone card marked Available nowYour things: the Living room, 20 things, on or near the cabinet: keys, wallet, spectacle case, laptop, bottle, watch and book, each with its height and a Go button

Your things. By room, then the furniture they're on or near. Go on every row.

Found: search, All, To check and Checked filters, group by place, height or kind; books listed by kind, and a brown book marked Seen once

Found. Check what the scan found; fix a name, colour or height; the unsure ones are marked.

Robot Access: the switch to share the home map with robots, off; the Living room map for robots; what robots receive and what is never shared

Robot Access. Off by default. A robot gets the map and where things are, never photos or anyone's position.
Scans. Every walk gets a quality report. When a walk doesn't line up with the last one, Recall says so rather than guessing what moved.Devices. iPhone is available now; Apple Watch and home robots are previews; Meta AI glasses are planned.Scans: Scan a room with a Start a scan button, then Your scans: Living room, Usable, 173 frames, processed, and a note that this walk didn't line up with the last scan, so Recall can't say what moved

Scans. Every walk gets a quality report. When a walk doesn't line up with the last one, Recall says so rather than guessing what moved.

Devices: Where Recall runs, with tabs for iPhone, Watch, Glasses and Robots, and the iPhone card marked Available now

Devices. iPhone is available now; Apple Watch and home robots are previews; Meta AI glasses are planned.

Illustration of the See it in AR screen in Edges mode: furniture drawn as pale outlines on black, labelled tags and rings on things such as grey watch and brown keys, a Speak button, and the same things as a list below

See it in AR. What Recall remembers, labelled through the camera, each with how sure it is, and the same things as a list. Illustration of the app's AR screen in its Edges look, drawn without the home; the tags are real ones from a walk.
+### The website + +**[The landing page](https://recall-ten-khaki.vercel.app).** What Recall is, and every answer spoken out loud. + ![The landing page: "Find your keys", RECALL in Braille, built first for blind and low-vision people and useful to anyone who forgets where they put things, the Ask Recall card, and Explore a real room · Actual scan](docs/media/hero.png) +
+ +**[The demo page](https://recall-ten-khaki.vercel.app/demo).** Pick a question and follow the walk to the thing, in the real scan: every measured point on the left, the architect's view on the right. The answers are real, recorded once, and labelled as recorded. + +![The demo page, Ask, then walk: a row of questions with Where is my watch? selected and marked Real recording; below it the scan's points and an architect's view of the room from above, each with a green route to the thing](docs/media/site-demo.png) + +
+ +**[The viewer](https://recall-ten-khaki.vercel.app/homes/flat/viewer).** Which Meta model made each part of this scan, and what each of Recall's agents may do. The same page has the full measurements and everything SAM 3.1 found. + +![The viewer: a table of what each Meta model made for this scan and where it ran, then a table of Recall's agents with each one's job, model and limits](docs/media/site-viewer.png) + ## What it is **What** @@ -204,7 +216,7 @@ recording and one frame of it, which the author reviewed and chose to publish (` | Architecture, decisions, privacy note, measurements | [`docs/`](docs) | [architecture](docs/architecture.md) · [decisions](docs/decisions.md) · [privacy](docs/privacy.md) · [verification](docs/verification.md) | | Tests and CI: API, pipeline, website, iPhone (unit, integration, accessibility, usability) | `services/*/tests`, `apps/web/e2e`, `apps/ios/RecallUITests`, `.github/workflows` | Green | | A usability session plan for a blind or low-vision tester | `evals/usability/` | Ready | -| The 3-minute demo video | | Coming | +| The 3-minute demo video | [youtu.be/vRAR3FHNSks](https://youtu.be/vRAR3FHNSks) | Ready | ## How the code is laid out @@ -624,6 +636,7 @@ erDiagram Measured, dated, and recorded in `docs/verification.md`. Nothing below is estimated. +- **First real tester:** a 65-year-old with low vision used the app on the author's iPhone and said that after a couple of runs he could guide himself through the rooms and find his wallet. One person's account, not a measured study. - **Reconstruction speed (M5 Pro, Apple GPU):** one room, 63 frames from a 21-second phone video, MapAnything with the Apache-2.0 weights: **19.6 s** of inference, **14.8 GB** peak memory, 11.3 s to load the model. - **Scale, images only:** two tape-measured desks (78 cm and 61 cm) came out about **23 % too large**, both by the same factor, so the shape is right and the absolute scale is not. This is why the pipeline feeds the phone's depth and poses into MapAnything. The tape-measured 20 cm check with the phone's inputs is still open. - **Room processing:** **60 s** per room with the warm model server (143 s before it), on the M5 Pro. diff --git a/THIRD_PARTY.md b/THIRD_PARTY.md index 2ad1259..0d0b702 100644 --- a/THIRD_PARTY.md +++ b/THIRD_PARTY.md @@ -105,7 +105,7 @@ Speech input is Muse Voice Transcribe (above). | What | Where | Terms | |---|---|---| | Sound of our own screen recording of a walk | `apps/web/public/demo-video/watch.mp4` | Ours, Apache-2.0 | -| Music: [Soft Technology Music](https://pixabay.com/music/ambient-soft-technology-music-629712/) by MiroMaxMusic | Background of the demo video only; the file is not in this repo | [Pixabay Content License](https://pixabay.com/service/license-summary/): free, commercial use allowed, no attribution required. | +| Music: original track made for this film with Claude | Background of the demo video only; the file is not in this repo | Ours: Anthropic's terms assign the output to the user, and no attribution is required | | Sound effects in the demo video | The demo video only; not in this repo | Ours: generated from noise and sine waves by our own script, no samples | ## Images diff --git a/apps/web/public/INSTALL.md b/apps/web/public/INSTALL.md index 525cc43..38620ae 100644 --- a/apps/web/public/INSTALL.md +++ b/apps/web/public/INSTALL.md @@ -109,8 +109,8 @@ pnpm dev ### A3. Open it - `http://localhost:3000/`: the landing page. -- `http://localhost:3000/homes/flat/viewer`: the **real scan** of the author's living room. It is geometry only, and - no photo of the room is published. It shows: +- `http://localhost:3000/homes/flat/viewer`: the **real scan** of the author's living room. The scan is geometry only + and holds no photos of the room. It shows: - the 3D room; - which Meta model made each part; - the measured numbers; @@ -181,7 +181,8 @@ Open `http://localhost:3000/demo`. **Check:** - the badge says "Recorded demo · real scan, real answers"; -- Recall's answer to "Where are my keys?" is shown, with the walk to them as clock-face steps; +- the page opens on "Where is my watch?": Recall's answer, the walk as clock-face steps, and a real recording of that + walk, which the author chose to publish; - choosing "Where is my bag?" draws a green route on the floor of the 3D room, one step at a time, while "you" move along it and the phone beside it shows the step being spoken (a simulated screen, labelled as such); - choosing "Where is my laptop?" offers the two laptops Recall found, each with its own walk; @@ -200,7 +201,7 @@ curl -s -X POST http://127.0.0.1:8000/ask \ -d '{"home_id": "flat", "question": "Where are my keys?"}' ``` -**Check:** a JSON answer like `"Your brown keys are near the desk in the living room at knee height, seen on Sunday at +**Check:** a JSON answer like `"Your keys were near the desk in the living room at knee height, seen on Sunday at 19:11."`, with `"model": "replay:muse-spark-open (recorded)"`. Every demo answer is labelled as recorded. The API documentation (Swagger UI) is at `http://127.0.0.1:8000/docs`. diff --git a/docs/install/README.md b/docs/install/README.md index 525cc43..38620ae 100644 --- a/docs/install/README.md +++ b/docs/install/README.md @@ -109,8 +109,8 @@ pnpm dev ### A3. Open it - `http://localhost:3000/`: the landing page. -- `http://localhost:3000/homes/flat/viewer`: the **real scan** of the author's living room. It is geometry only, and - no photo of the room is published. It shows: +- `http://localhost:3000/homes/flat/viewer`: the **real scan** of the author's living room. The scan is geometry only + and holds no photos of the room. It shows: - the 3D room; - which Meta model made each part; - the measured numbers; @@ -181,7 +181,8 @@ Open `http://localhost:3000/demo`. **Check:** - the badge says "Recorded demo · real scan, real answers"; -- Recall's answer to "Where are my keys?" is shown, with the walk to them as clock-face steps; +- the page opens on "Where is my watch?": Recall's answer, the walk as clock-face steps, and a real recording of that + walk, which the author chose to publish; - choosing "Where is my bag?" draws a green route on the floor of the 3D room, one step at a time, while "you" move along it and the phone beside it shows the step being spoken (a simulated screen, labelled as such); - choosing "Where is my laptop?" offers the two laptops Recall found, each with its own walk; @@ -200,7 +201,7 @@ curl -s -X POST http://127.0.0.1:8000/ask \ -d '{"home_id": "flat", "question": "Where are my keys?"}' ``` -**Check:** a JSON answer like `"Your brown keys are near the desk in the living room at knee height, seen on Sunday at +**Check:** a JSON answer like `"Your keys were near the desk in the living room at knee height, seen on Sunday at 19:11."`, with `"model": "replay:muse-spark-open (recorded)"`. Every demo answer is labelled as recorded. The API documentation (Swagger UI) is at `http://127.0.0.1:8000/docs`. diff --git a/docs/media/app-ar.png b/docs/media/app-ar.png new file mode 100644 index 0000000..0040ed6 Binary files /dev/null and b/docs/media/app-ar.png differ diff --git a/docs/media/app-devices.png b/docs/media/app-devices.png index 9f9d72c..d0cf7aa 100644 Binary files a/docs/media/app-devices.png and b/docs/media/app-devices.png differ diff --git a/docs/media/app-found.png b/docs/media/app-found.png index 6eceec3..fbad598 100644 Binary files a/docs/media/app-found.png and b/docs/media/app-found.png differ diff --git a/docs/media/app-rooms.png b/docs/media/app-rooms.png index 3f71c29..d4c6fd1 100644 Binary files a/docs/media/app-rooms.png and b/docs/media/app-rooms.png differ diff --git a/docs/media/app-scans.png b/docs/media/app-scans.png index 3ac8806..d9463b3 100644 Binary files a/docs/media/app-scans.png and b/docs/media/app-scans.png differ diff --git a/docs/media/app-things.png b/docs/media/app-things.png index 8dfc74a..3738da8 100644 Binary files a/docs/media/app-things.png and b/docs/media/app-things.png differ diff --git a/docs/media/site-demo.png b/docs/media/site-demo.png new file mode 100644 index 0000000..3579ac6 Binary files /dev/null and b/docs/media/site-demo.png differ diff --git a/docs/media/site-viewer.png b/docs/media/site-viewer.png new file mode 100644 index 0000000..480c4b6 Binary files /dev/null and b/docs/media/site-viewer.png differ