Repository navigation
Actions: the layer between motion and meaning - #377
Merged
Merged
Conversation
First part of #373. The toolbox measures motion and says nothing about actions, so this adds the middle term and leaves room for the third. An `Action` is a span with provenance, a `features` dict and a `labels` dict. `segment_actions` cuts spans out of a motion envelope; `action_type` describes the shape of one; `MgVideo.actions_from_motion()` does both and builds its envelope from POSE rather than pixels, which is what makes the result follow the person instead of the camera --- a pan moves every pixel and moves no landmark relative to the others. The separation of `features` from `labels` is the design rather than tidiness. Measuring fills features; naming fills labels. A shape is something the signal shows and a gesture is something a recogniser claims, and the two fail differently: a missed boundary loses an action outright, while a wrong name leaves the action there to be relabelled. It also lets the skeleton-based, video-model and dense-trajectory recognisers ARJ asked for coexist, since each writes under its own key and they are allowed to disagree. TWO ERRORS OF MINE, both caught by the known-answer tests rather than by reading. Half height was measured from the span's own minimum. A motion envelope has a meaningful zero, so for a rectangular action the minimum IS the peak, and the steadiest movement possible came out impulsive. Fixing that, a decaying impulse began reading as sustained, because segmentation cuts an action at a threshold and therefore hands this function an impulse already truncated to its loud third. Rather than tune the threshold until the three test signals passed, I measured all three after segmentation: time above half height reads 0.385 for a decay, 0.510 for a tremolo and 1.000 for a plateau, which does not separate them, while the energy centroid reads 0.332, 0.481 and 0.500, which does. The call rests on the centroid now, that being also the standard descriptor in the sound-object lineage this typology comes from, and the numbers are in the docstring so the threshold can be argued with. 706 tests pass, 4 skipped. mypy clean --- which caught me naming `musicalgestures` in an annotation without importing it, the same bug fixed in `_videoadjust.py` hours earlier, now that the step blocks rather than prints. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written at the end of a session that ran across four repositories, so the state is not visible from any one of them. Leads with what is waiting rather than what was done, because one item is easy to lose: micromotion's `heading-persistence` branch is committed and never pushed, it has no upstream, and `git status -sb` therefore prints no ahead marker for it --- which reads like "in sync". It is three commits ahead of origin/main and two of those predate today, so pushing it publishes work that was already sitting there. Records the release-process correction as a procedure rather than a note: the version lives in three places, not the two RELEASING.md named, and the check to run before tagging is one command. Also records the argument for keeping the mypy gate closed, which is not the count. It is the table of what the count was hiding --- a crop window that crashed on its own instruction, an audio axis that crashed past an hour, a Flow you could not hold, and a public method that had never worked. Ends on the manuscript, which is where the session started and where it did not go. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First part of #373. 706 tests pass, mypy clean.
The toolbox measures motion and says nothing about actions. This adds the middle term and leaves room for the third.
Action— a span with provenance, afeaturesdict and alabelsdict.segment_actions— cuts spans from a motion envelope.action_type— impulsive / sustained / iterative, following Sound Actions.MgVideo.actions_from_motion()— does both, building its envelope from pose rather than pixels. That is what makes the result follow the person instead of the camera, as Add action and gesture recognition #373 requires: a pan moves every pixel and moves no landmark relative to the others.Why
featuresandlabelsare separateMeasuring fills
features; naming fillslabels. A shape is something the signal shows; a gesture is something a recogniser claims. They also fail differently — a missed boundary loses an action outright, a wrong name leaves the action there to be relabelled.It is also what lets the four approaches coexist: the skeleton-based, video-model and dense-trajectory recognisers each write under their own key, and are allowed to disagree.
Two errors of mine, both caught by the tests
Time-above-half does not separate them; the centroid does. The call rests on the centroid — also the standard descriptor in the sound-object lineage this typology comes from — and the numbers are in the docstring so the threshold can be argued with.
Next: the recognisers, which now have somewhere to write.
🤖 Generated with Claude Code