Repository navigation
Conversation
dedup: only suppressed a row when it was similar to the immediately previously kept row, so two near-duplicates separated by an unrelated photo both survived — contradicting the documented guarantee to keep only photos less than the threshold similar to each other. Track the embeddings of every kept row and suppress any row similar to any of them. Covered by dedup_search_test.go (non-adjacent near-duplicates, synthetic CLIP embeddings).
Owner
Author
|
I don't think this is the best way to go with O^2, so might reopen later |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dedup:only suppressed a row when it was similar to the immediately previously kept row, so two near-duplicates separated by an unrelated photo both survived — contradicting the documented guarantee to keep only photos less than the threshold similar to each other. Track the embeddings of every kept row and suppress any row similar to any of them.Split out of #201 (search date matching / limit-after-scoring), which this commit is cherry-picked from — that PR now reverts this change. Merge #201 first to keep the diff clean; the two are otherwise independent.
Cost
O(n²) cosine comparisons in Go over kept rows — acceptable at current collection sizes; noted for future optimization.
Test plan
dedup_search_test.go: non-adjacent near-duplicates with synthetic CLIP embeddings — the duplicate after an unrelated photo is now suppressed.main(this branch):TestListDeduplicateSuppressesNonAdjacentNearDuplicatespasses.Related
created:local-date matching, limit-after-scoring for embedding queries)dedup:+ limit, thegoScoresgate needs aDeduplicate.Presentterm (that fix lands in fix: search date matching and embedding-score limits #201's code; this branch has nogoScoresyet, so dedup: (without embeddings) + limit: SQL LIMIT pre-clips the row stream before Go dedup runs #199 stays open for whichever lands the SQL-limit change)