Repository navigation
Conversation
…them The real scan's walls run at about 42 degrees, but each furniture box was a world-axis min/max box, so anything against a wall cut through it (up to 29 cm). Boxes are now fitted in the room's own frame from points inside the room, clipped to it, and drawn turned (centre, size, yaw). A wall that RANSAC found twice (its face and a plane in front of it) becomes one wall on its outer line: the published scan goes from 6 wall sheets to 3. Re-exported real-scan/flat from the same episode; the point and mesh files are byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…side Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PnP, no model)
The pipeline now writes a place index per scan (ORB features of every
keyframe lifted to 3D with the phone's LiDAR depth and ARKit pose), and
POST /homes/{id}/locate matches one camera photo against it: the room it
was taken in, the camera's pose in the room map's world, and what to point
at. It runs beside ARKit's relocalisation, never instead of it.
On the real living-room scans: held-out keyframes 1.2 cm median error;
photos from other sessions fix 59% of the time and 89% of fixes agree
(the phone applies a pose only when two fixes agree); 22-35 ms per match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, a growing map, named coaching - Guide warms up and uses the map of the thing's own room (it used the newest scan of any room). - While ARKit hasn't matched, PhotoAssist sends about one photo a second to the Mac's matcher: it says when the phone is in another room, names furniture to point at, and applies a pose only when two fixes agree (25 cm, 10 degrees). ARKit's own match always replaces it. - ARLocator keeps labels, arrows and the route under one map node, so a photo correction moves them together. - Once ARKit recognises the room, the map it grew is saved beside the scan's own (worldmap.grown.arworldmap), so the next visit matches from there too. - See it in AR shows the list at 25 s but keeps looking for 2 minutes and switches the labels on when the room is recognised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sults; today's test counts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to each thing samples/replay/flat is re-recorded from scan bb449f4c (real Muse Spark Open calls) and now holds the floor geometry routes need: the point cloud thinned to 5 cm, positions only, and the walk's camera poses only. make demo-walks runs the replayed Guide and the phone's route planner from where the scan began and writes the website's walks.json; make demo starts the API and the website together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the real scan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A small launcher (scripts/demo.py, standard library) checks the tools and ports, starts the API and the website with their output in data/demo/*.log, waits until both really answer, then prints the page to open. Each recorded question the API answers gets one line; typed text is never logged. Ctrl+C, a closed terminal or a kill stops both servers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with its own walk Four more recorded questions (watch, mug, charger, passport; re-recorded with real Muse Spark Open calls). When the Guide finds several, each match from its own find_object tool becomes a choice with its route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and the phone beside the walk The phone shows the step being spoken (a labelled simulated screen), or plays a real screen recording when this computer has one, with the route following the video. Every question is tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r only The clips show the home, so they go to apps/web/public/local-video/ (gitignored) and never reach the hosted site; make demo says whether any were found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… computer has one A "Real recording" badge on the question it belongs to, and a shortcut beside the phone that plays it. Neither shows without local clips, so the hosted page is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…closed, cheering New exports only (SAM_MORE_FRAMES and a phone colour); the landing page's frames are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The real points gather into the room, then the walls, furniture labels, the keys' pin and the planner's route with its clock-face steps, then the measured 3 cm. Time comes from window.__film.seek(t), never the clock, so it can be captured frame by frame. Not linked from the site, not indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the camera stays open A double tap on a tag or a list row, Speak, or VoiceOver's two-finger double tap on what the camera points at starts the walk in the same view and ARKit session; its panel takes the list's place, only that thing's label stays, and the labels come back at the end. Several alike here: Recall asks which, decided from the labels and the words said, not from how the model phrased it. A thing in another room is answered, not walked to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ou said One camera frame moved a watch 82 cm, onto the TV, so the next walk's arrow pointed there. SAM alone no longer moves a thing: Muse Spark 1.1 may downgrade a move to uncertain, which leaves memory as it was. The privacy note, the API's privacy flows and the docs now say so. In AR the words Muse Voice Transcribe heard show lightly over the camera, and a label seen only once says why it can't be walked to instead of failing with an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as that frame make demo-clips can now cut a still from a recording as well as clips, from any private folder; the watch clip is the AR find by voice. The frame, like the clips, stays on this computer: the hosted page is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SAM runs on a sample of each scan, and no real rescan has lined up with an earlier one yet, so the compare step is unit-tested only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opening AR now says it is scanning the room, then asks what you're looking for and listens without a tap. After a find it asks whether to go back: yes retraces the route on the phone, no thank you ends the walk, and anything else is a new question. The recorder ends a turn on silence from the microphone's level alone, and waits out VoiceOver by the line's length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /reply sends the words as JSON data to the Guide's own model, so private mode stays on the Mac, and takes one strict JSON intent: go back, done or a new question. The phone falls back to its word rules when the Mac can't tell, can't be reached or takes over 6 s. Audited without the words, rate-limited, and listed in the privacy tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Point the phone at the shelf" asks a blind person to find what they can't see. Recognising a room now says to turn slowly at chest height, and the other sight-based lines say what to do with your body. Furniture isn't named until the room is recognised: memory alone named the wrong thing on 2 of 7 real frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The nav's two buttons are short so the bar stays one line, and the Accessibility menu is an icon with a tooltip; the hero keeps the badges and screen readers hear them. Every width from 640 to 1920 px is tested. Keyboard budget to the Ask card rises by these two stops, from 18 to 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The demo opens on the watch, and its simulated phone looks like the AR view, drawn with no photo of the home. The demo and the viewer say how the iPhone works in AR, and that it listens only on your turn: speech becomes text after the turn, it isn't a live stream. The viewer adds numbers from the app's own logs, dated and marked as not a study, and reads the room aloud with the browser's voice. Long pages get Back to top. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.