Skip to content

v1.8.1: fix Chase re-import duplicating rows in IndexedDB - #2

Merged
gitluis merged 3 commits into
masterfrom
fix-chase-reimport-duplicates
Oct 25, 2026
Merged

gitluis merged 3 commits into
masterfrom
fix-chase-reimport-duplicates

Conversation

@gitluis

@gitluis gitluis commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Stacked on #1. This PR's base is security-fixes-restore-ai. Merge #1 first. If GitHub doesn't switch this PR's base to master automatically, change it by hand.

Summary

Re-importing a Chase CSV no longer duplicates every Chase row in IndexedDB. Each import now also removes the extra copies the bug already created. Released as v1.8.1.

The bug

  • Where: DB.importTransactions() assigned the _2/_3 occurrence suffix whenever an ID already existed in the database.
  • Effect:
    • On a re-upload, every Chase row collided with its earlier self and was inserted again as …_2. In testing, 165 Chase rows became 330 after a single re-import.
    • Ally files were unaffected, because they skip collisions.
  • When it showed up: only when the app booted from IndexedDB rather than finlens-state.json. The in-memory list is rebuilt from the CSVs on import.
  • Second problem: in memory, two identical rows shared the same base ID, while the database stored them as base and _2. Overrides couldn't target the second copy reliably.

The fix (068a2c6)

  • parseCsvFile() assigns suffixes, counting identical rows within the file: the first keeps the base hash, the second gets _2. They pass through normalizeRow(..., occurrence) to mkTx(), so IDs are final and identical in memory, in IndexedDB and in txOverrides.
  • DB.importTransactions() skips IDs it already holds. It reads the stored keys once with getAllKeys(), and no longer suffixes on collision or logs a console error per skipped row.
  • Cleanup of existing damage: each import deletes suffixed rows whose base transaction is in the file but whose exact ID is not.
    • This is safe because same-day identical rows always appear together in a bank export, so the file is authoritative for how many copies exist.
    • Rows whose base isn't in the file are never touched.
    • The import results card shows "N leftover duplicate copies removed".
  • Identical Ally rows (same date, bank time, description and amount) are now dropped in memory too, matching the database and CLAUDE.md §10.
  • Custom schemas without a time column now get suffixes by default instead of losing same-day identical rows.

Docs (a4d123a)

  • CLAUDE.md §10 rewritten.
  • Release: CHANGELOG entry and version 1.8.1 in all four places.

How it was tested

A headless Chromium run with the sample CSVs. 18/18 new checks pass, and the v1.8.0 suite still passes 27/27.

  • First import: 257 rows in memory and in IndexedDB. Memory IDs are unique and match the database exactly, and the identical Chase pair is base + _2.
  • Re-import: IndexedDB stays at 257, where it was 422 before. For Chase: 0 inserted, 165 skipped, 0 removed.
  • Seeded legacy damage (bogus _2 copies of 20 rows, a bogus _3 of the identical pair, and an unrelated suffixed row from "another file"):
    • Re-import removes exactly the 21 bogus copies.
    • It keeps the legitimate pair and the unrelated row.
    • The results card reports the removal.
  • Overrides: an override on the _2 row survives a re-import.
  • Reload: after a reload from IndexedDB, 257 unique transactions.
  • Appended export: a first file with one parking charge, then a newer file with both. The second import inserts only the new _2 row (1 inserted, 164 skipped).
  • Ally: a duplicated Ally line is kept once.

After merging

If your real database was affected, re-import your Chase CSVs once. The results card will show how many leftover copies were removed.

Noticed, not changed

Console error on reload: "Canvas is already in use" can appear for the Overview charts after a page reload. This already happens on the base branch: two overview renders run back to back during boot, and their deferred chart setups collide. The charts still render. It belongs in its own small fix.


Generated by Claude Code

DB.importTransactions() assigned the _2/_3 occurrence suffix whenever an ID
already existed, so every re-uploaded Chase row was inserted again as _2
(165 -> 330 in testing). Suffixes are now assigned once in parseCsvFile(),
counted within the file, so IDs are final and identical in memory and in
IndexedDB; the import skips IDs already stored (one getAllKeys read).

Each import also removes leftover copies from the bug: suffixed rows whose
base is in the file but whose exact ID is not. The results card reports them.
Identical Ally rows are dropped in memory too; custom schemas without a time
column get suffixes by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AT1Bh6Sscb8ogzzuf8U3Jc
…import-duplicates

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AT1Bh6Sscb8ogzzuf8U3Jc
@gitluis

gitluis commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Review: manual test of fix-chase-reimport-duplicates (de21c3e)

Date: 2026-10-24
Environment: Windows, Edge InPrivate (empty IndexedDB), index.html opened via file://
Test data: samples/chase_card_sample.csv (165 rows, including one identical same-day pair)

Results

# Test Expected Result
1 First import (Settings → Import → Pick CSV Files, assigned to Chase) 165 new, 0 skipped ✅ Pass
2 Reload (F5) and boot from IndexedDB Still 165 transactions ✅ Pass (Settings → General → Danger Zone shows 165)
3 Re-import the same file, then reload 0 new, 165 skipped; still 165 after reload (master doubles to 330) ✅ Pass
4 Identical same-day rows are kept SJ PARKING METER, Aug 14, −$2.00 appears exactly twice ✅ Pass
5 Console errors during imports and reload No red errors ✅ Pass

Not tested yet

  • Real data: I haven't re-imported my real Chase CSVs to confirm the cleanup of leftover duplicates ("N leftover duplicate copies removed").

Note for future testing

The Transactions view only shows the selected date range, and it defaults to This Month. Set it to All Time before checking counts or searching for older rows like the Aug 14 pair. Settings → General → Danger Zone shows the total count regardless of the range.

Verdict

Fix works as described on the sample data. Approval pending the real-data re-import.

@gitluis
gitluis merged commit 5fdca99 into master Oct 25, 2026
@gitluis
gitluis deleted the fix-chase-reimport-duplicates branch October 25, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants