Skip to content

v1.16.19: Fix overlapping CSV exports of one account counting the shared rows twice - #42

Open
gitluis wants to merge 2 commits into
fix-json-import-edits-not-appliedfrom
fix-overlapping-exports-duplicates
Open

gitluis wants to merge 2 commits into
fix-json-import-edits-not-appliedfrom
fix-overlapping-exports-duplicates

Conversation

@gitluis

@gitluis gitluis commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #41 (fix-json-import-edits-not-applied). Merge order: #16, #17, #19, #20, #21, #24–#38, #39–#50, then #10, then #23.
Changes earlier fixes' code: mergeImported() (#26) and importParsedCsv() (#25). It doesn't need #40's or #41's code.

Summary

Two CSV exports of one account that share dates, imported together, left the shared rows twice in memory and in finlens-state.json. IndexedDB held them once. Example: a May – mid-September and an August – October checking export in the folder.

On the samples split that way (23 shared rows), FinLens showed 280 transactions instead of 257. Every total counted the shared rows twice. A page reload kept it, because boot loads the state file first.

Now an import keeps one transaction per id, in memory, IndexedDB and the state file, however many files overlap. A state file already saved with duplicates is cleaned up when it loads.

Cause

  • Both files give the shared rows the same id. The id hashes the row and the account (CLAUDE.md §10). Checking and savings always land in one account. A card does only when both files have the same name: the fallback folder picker (browsers without the folder API) reads subfolders, so two card.csv files from monthly folders arrive together.
  • The in-memory merge kept every copy. loadFiles() collected every file's transactions, and mergeImported() returned kept.concat(fresh). DB.importTransactions() skips an id it already holds, so IndexedDB had one copy. The report's guess was right.
  • What the duplicates did:
    • Net worth, cash flow, the Transactions KPIs and its count all counted the shared rows twice.
    • The second copy of a transfer stayed a plain withdrawal, deposit or card payment. detectTransfers() tracks legs by id, so it skips the copy once the first is marked. That copy counted as spending or income.
    • An edited row showed twice. A bucket listed a linked transaction twice.
  • A second gap, in IndexedDB. Its leftover-copy check runs once per file. A later export can hold one more identical same-day card row (_3) than an earlier one. With the later file read first, the earlier file then deleted _3 from the store as a "leftover copy". Memory had it; a boot from IndexedDB didn't. It only showed once memory was fixed, and the same fix point (one import, several files) covers it.
  • Occurrence suffixes are fine. _2, _3 are counted within each file, so both files give a same-day pair the same two ids, and they collapse to one pair.
  • Master has the bug. Its loadFiles() appends each file's rows to State.transactions. Checked on origin/master (e8e18e7, v1.10.5) by calling loadFiles() with the same two checking slices plus the other samples: 280 in memory with 23 duplicate ids, 257 in IndexedDB. Its autosave writes State.transactions, so its state file would hold them too (read from the code, not run).

What changed

  • mergeImported() (core.js) keeps one transaction per id. The first file's copy stays, because that's the row IndexedDB stores (later files skip the id). A loaded id held twice also keeps one copy.
  • DB.importTransactions(txs, accountId, schemaId, importIds) (db.js) takes a new optional 4th argument: every id the import's files give. A row with one of those ids is never removed as a leftover copy.
  • loadFiles() builds importIds and passes it through importParsedCsv(p, importIds).
  • loadStateFromFile() keeps one copy of an id the file holds twice, logs how many it dropped, and schedules a save so the file is rewritten.
  • Unchanged:
    • Importing the exports one after the other already worked: the second replaces the shared rows.
    • Last import still lists a later file's shared rows as "already imported".
    • Ids, suffix counting, parseCsvFile() and the AI Eval's buildDataset() are untouched.
  • Version: v1.16.19 in CHANGELOG (dated 2026-11-02), footer, README (both places), Settings → General and CLAUDE.md §1. CLAUDE.md also updated: §4.5, §8, §10, §19 and "Last updated".

Not verified

  • The fallback folder picker with real subfolders. The card cases choose two same-named files from two directories through #filePick; #folderPick (webkitdirectory) goes through the same handleFilePick(), but I didn't drive it.
  • Firefox and Safari, where that picker is the one used.
  • Two card exports with different names. They're two accounts (§18), so their shared rows have different ids and both stay. That's how FinLens tells cards apart, and it isn't changed here.
  • Which copy wins when two files disagree on a field the id doesn't cover (a card row's bank Category). The first file's copy stays, matching IndexedDB; file order is the folder's or the picker's. I didn't find such rows in the samples.
  • Master's state file. I checked master's memory and IndexedDB, not its finlens-state.json.
  • Phone layouts (desktop only, by decision).
Automated checks: earlier run, not re-run · 6fc8bda

Moved unchanged from this PR's "Test results" comment on 2026-11-04; not re-run, so not in the new table format. Originally a description section titled "How it was tested".

Light checks, by Luis's call to save usage: on this branch's final commit 6fc8bda (chained on #41) only boot.js, race.js and this PR's own tests ran. Headless Chromium at 1280×900 against samples/, with the pinned CDN libraries served from npm; each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>.

Script Result on 6fc8bda
boot.js 257 transactions, 24 views, 0 errors
race.js 24 views, 0 errors
test-overlapping-exports.js (new in this PR) 46/46

Full set, once, on the top of the stack (#23, 819e809: every PR below it merged in): every base suite and every fix test from #24 to #50, plus #10's and #23's, each in its final variant. All pass: test-clarity.js 55/55, test-insights.js 137/137, test-finances.js 126/126, test-plan.js 127/127, boot.js 257 transactions and 0 errors, race.js 0 errors; #24–#38's tests (clear-edits 48/48, rebuild-folder 57/57, partial-import 47/47, json-import 28/28, new-browser-copy 10/10, json-export 31/31, plan-links 60/60, rule-flags 67/67, drawer-flags 36/36, start-balance 193/193, ai-summary-period 78/78, account-names 40/40, account-order 32/32, drawer-focus 167/167, cleanup 21/21); #39–#50's tests (drawer-fields 45/45, read-as 52/52, json-import-edits 29/29, overlapping-exports 46/46, override-balance 160/160, card-override 277/277, ai-summary-trend 91/91, transfer-user-type 39/39, untransfer-category 27/27, recategorize 55/55, retype-transfer 135/135, minor-fixes 34/34); test-dbtab.js 73/73, test-save-status.js 64/64, test-master-port.js 16/16. Three scripts needed a test-only change on that run, listed in #23 and in the PRs they belong to; the app code wasn't changed.

  • No phone checks (desktop only).

Measured before the chain

The numbers below are the PR author's, on the commit before it was chained (built on the old #38 tip ecfcab2, v1.16.15, or on the fix below it there; version numbers renumbered to the chain's).

Headless Chromium, desktop 1280×900, with the harness's fake folder. Suites ran one at a time.

  • test-overlapping-exports.js (new). It builds the fixtures by slicing samples/:

    • checking: Aug 1 – Oct 20 and May 1 – Sep 15 (23 shared rows), named checking_aug_oct.csv and checking_may_sep.csv;
    • card: Jul 1 – Oct 20 and May 1 – Sep 30 (87 shared rows, including the samples' one true same-day pair), both named credit_card_sample.csv, from two directories;
    • a later card export with a 3rd identical copy of that pair.

    The reference is a full import of the unsliced samples (257), or of the samples with the 3rd copy (258). Each case compares every transaction with it (id, account, type, categories, description), and counts memory, IndexedDB and the state file, with duplicate ids.

    1. Two overlapping checking exports in the folder: boot import, Reload CSVs, autosave, page reload, boot from IndexedDB, Overview and Transactions totals.
    2. The two card exports chosen together: the same-day pair stays base + _2. Then edits and a bucket link on shared rows, both files chosen again, autosave and reload.
    3. The same exports one after the other.
    4. The later export with 3 copies and an earlier one with 2, in both orders: base + _2 + _3 in memory and IndexedDB, nothing removed as a leftover copy, then a boot from IndexedDB.
    5. Edits and a link on a checking row, then the folder's checking export replaced by the two overlapping ones.
    6. A state file with 12 repeated ids: loaded with one copy of each, and rewritten.

    Totals are read from the screens: the Overview's net worth, latest month, cash flow and account cards, and the Transactions KPIs and count line.

    • Parent (b4a87e6): 15/46 passed, in a temporary worktree (since removed). Case 1: 280 in memory with 23 duplicate ids, 257 in IndexedDB, 280 in the state file and after a reload. Case 2: 344 with 87 duplicate ids; the pair shows as base, base, _2, _2; the bucket lists the linked row twice. Case 4, later file first: IndexedDB 257 against 258 unique ids in memory (_3 deleted). Case 6: 269 loaded and saved back. The passes: the references, case 3 (one after the other already worked), the IndexedDB boots of cases 1 and 5, and the no-console-error checks.
    • This branch (b648694): 46/46 passed.
  • test-read-as.js: 52/52 on the parent, 52/52 on this branch.

  • test-json-import-edits.js: 29/29 on the parent, 29/29 on this branch.

  • run-tip.sh 15, the same counts on the parent and on this branch:

    • test-clarity 55/55, test-insights 137/137, test-finances 126/126, test-plan 127/127.
    • boot: 257 transactions, 0 errors.
    • race: only renderImportLog errors (expected on this base until v1.10.5: fix Import Logs throwing when the view is left before it loads #18's fix is in the stack). Parent 7; this branch 10 on the first run and 6 on the rerun. None from this change.
    • test-clear-edits 47/47, test-rebuild-folder 57/57, test-partial-import 47/47, test-json-import 28/28, test-new-browser-copy 10/10, test-json-export 31/31.
    • test-plan-links 60/60, test-rule-flags 67/67, test-drawer-flags 36/36, test-start-balance 193/193, test-ai-summary-period 78/78.
    • test-account-names 40/40, test-account-order 32/32, test-drawer-focus 167/167, test-cleanup 21/21.
  • On this branch, the first run-tip.sh run was stopped by the tool's time limit before test-cleanup. I ran test-cleanup and race again on their own (ONLY=…). The parent ran in three groups the same way.

  • No existing suite changed.

  • Master: a one-off probe (probe-master-overlap.js, scratch only) on origin/master, described under Cause.

How to reproduce the bug

Run this on the base branch, fix-json-import-edits-not-applied. Master has the bug too.

  1. Do the Setup from the testing steps below, but check out fix-json-import-edits-not-applied instead of fix-overlapping-exports-duplicates. Use the scratch folder with the two overlapping checking files.
  2. On the welcome screen, click Pick your CSV folder.
  3. Choose the scratch folder and allow both prompts Edge shows.
  4. Click the calendar icon (Date range) and choose All time.
  5. On Overview, read the net worth and the Bank Checking card.
  6. Click Transactions and read the Transactions tile, Money in and Money out.
  7. Wait about 10 seconds, then run the PowerShell count from Setup step 5.

What goes wrong:

  • Net worth reads $16,361.06, not $15,499.81. Bank Checking shows "104 transactions" and $14,182.68, not 81 and $13,321.43.
  • The Transactions tile reads 280, not 257. Money in is +$39,874.88 and Money out is −$22,268.82.
  • The state file holds 280 transactions, while IndexedDB holds 257.
Manual QA steps

Setup

  1. git checkout fix-overlapping-exports-duplicates.
  2. Make a scratch folder, for example C:\temp\finlens-overlap. Copy the three files from samples/ into it. Never use your real data folder for this PR: it writes finlens-state.json.
  3. In that scratch folder, split the checking export into two overlapping exports. Open PowerShell in the folder and run:
    $h = Get-Content checking_sample.csv -TotalCount 1
    $r = Get-Content checking_sample.csv | Select-Object -Skip 1
    @($h) + ($r | Where-Object { $_.Substring(0,10) -le '2026-09-15' }) | Set-Content checking_may_sep.csv
    @($h) + ($r | Where-Object { $_.Substring(0,10) -ge '2026-08-01' }) | Set-Content checking_aug_oct.csv
    Remove-Item checking_sample.csv
    The folder now holds checking_may_sep.csv (66 rows), checking_aug_oct.csv (38 rows), savings_sample.csv and credit_card_sample.csv. The two checking files share 23 rows (Aug 1 – Sep 15).
  4. Open an Edge InPrivate window (Ctrl+Shift+N), so your real browser data isn't touched. Open index.html from the repo in it.
  5. To count the transactions in the state file at any point, run this in PowerShell in the scratch folder:
    (Get-Content .\finlens-state.json -Raw | ConvertFrom-Json).transactions.Count

1. Two overlapping exports in the folder count each row once

  1. On the welcome screen, click Pick your CSV folder.
  2. Choose the scratch folder and allow both prompts Edge shows (view files, then save changes).
  3. Click the calendar icon (Date range) in the app bar and choose All time.
  4. On Overview, read the net worth and the Bank Checking card.
  5. Click Transactions and read the Transactions tile and the count line under the list.
  6. Click the gear icon (Settings), then the Import tab, and read Last import.

Expected:

  • Net worth $15,499.81.
  • Bank Checking: Checking · 81 transactions, $13,321.43.
  • Transactions tile 257; count line "Showing 50 of 257 · 50 per page".
  • Money in +$32,524.88, Money out −$16,795.07.
  • Last import lists both checking files. One shows "23 already imported" (which one depends on the order the folder gives the files).

Before this fix: net worth $16,361.06, Bank Checking "104 transactions" $14,182.68, Transactions 280, Money in +$39,874.88, Money out −$22,268.82.

2. The state file and IndexedDB hold one copy

  1. Wait about 10 seconds (auto-save runs 5 seconds after a change). Hovering the Synced badge in the app bar shows "last saved ".
  2. Run the PowerShell count from Setup step 5.
  3. Press F12 → Application tab → IndexedDB → finlens_db → transactions. Read "Total entries" at the bottom of the table (click the refresh icon if it's empty).

Expected:

  • PowerShell prints 257.
  • IndexedDB transactions holds 257 entries.

Before this fix: the state file held 280; IndexedDB held 257.

3. Reload keeps 257

  1. Press Ctrl+F5 in the same InPrivate window.
  2. Set Date range to All time again (boot always starts on This month).
  3. Click Transactions.

Expected:

  • Transactions 257, net worth on Overview $15,499.81.
  • F12 → Console shows [FinLens] Boot: Loaded state from finlens-state.json — 257 transactions.

Before this fix: 280 after the reload, because boot reads the state file first.

4. An edit on a shared row shows once, and survives Reload CSVs

  1. On Transactions (All time), type RENT PAYMENT in Search description or notes.
  2. Click the Sep 1 row's description (RENT PAYMENT PALM COURT APTS, −$1,350.00). It's in both checking files.
  3. In Transaction detail, type test note in Notes and click Save changes.
  4. Go to Settings → Import and click Reload CSVs.
  5. Go back to Transactions and search RENT PAYMENT again.

Expected:

  • Count line "Showing 6 of 6 (257 before filters)".
  • The Sep 1 row appears once, with "test note" under its description.
  • Last import (Settings → Import) shows 0 new for every file.

Before this fix: "Showing 8 of 8 (280 before filters)": Sep 1 and Aug 1 each listed twice.

5. A state file saved with duplicates is cleaned on load

  1. Close every InPrivate window (this wipes its storage). Delete finlens-state.json from the scratch folder.
  2. git checkout fix-json-import-edits-not-applied (the base branch).
  3. Open a new InPrivate window, open index.html, click Pick your CSV folder, choose the scratch folder and allow both prompts.
  4. Wait about 10 seconds for the auto-save, then run the PowerShell count. It should print 280 (the bug, saved).
  5. git checkout fix-overlapping-exports-duplicates.
  6. Press Ctrl+F5 in the same InPrivate window. Keep F12 → Console open.
  7. Set Date range to All time and open Transactions.
  8. Wait about 5 seconds and run the PowerShell count again.

Expected:

  • Console: [FinLens] finlens-state.json held 23 transactions twice (the same id); one copy of each is kept.
  • Transactions 257.
  • PowerShell prints 257 (the file was saved again).

6. Regression: importing the exports one after the other

  1. Close every InPrivate window. Open a new one and open index.html.
  2. Click choose the files on the welcome screen (or Settings → Import → Choose CSV files).
  3. Select checking_may_sep.csv, savings_sample.csv and credit_card_sample.csv, then click Import 3 files.
  4. Set Date range to All time and note the Transactions count.
  5. Go to Settings → Import, click Choose CSV files, select only checking_aug_oct.csv, and click Import 1 file.
  6. Read Last import, then the Transactions count.

Expected:

  • After step 4: 242 transactions.
  • Last import: checking_aug_oct.csv "15 new · 23 already imported".
  • After step 6: 257 transactions.
  • Same on the base branch (this path already worked).

7. Choosing both overlapping files again changes nothing

  1. Continue from test 6.
  2. Settings → Import → Choose CSV files, select checking_may_sep.csv and checking_aug_oct.csv together, click Import 2 files.
  3. Open Transactions (All time).

Expected:

  • Last import: "0 new · 66 already imported" and "0 new · 38 already imported".
  • Transactions stays 257.

Before this fix: 280.

Please check before merging

  • Put two overlapping checking exports in the folder (or choose them together), Reload CSVs, and check the Transactions count and the Overview's net worth match a single export of the whole range.
  • If your current finlens-state.json has duplicates (an overlapping folder before this fix), check the console says how many were held twice on the next boot, and that the count drops.
  • Agree that the first file's copy wins when two files give the same id, to match what IndexedDB stores.
  • Agree to the clean-up in loadStateFromFile(). It's part of this fix so totals are right for data already saved with the bug, without waiting for a Reload.
  • Manual QA passed in Edge/Chrome with a real folder

https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft


Generated by Claude Code

…wice (v1.16.19)

Two exports of one account that share dates, imported together (a May to
mid-September and an August to October checking export in the folder, say),
left the shared rows twice in memory and, through the autosave, in
finlens-state.json, while IndexedDB held them once. On the samples split that
way (23 shared rows): 280 transactions instead of 257. Every total counted
the shared rows twice, the second copy of a transfer stayed a plain
withdrawal or deposit (detection tracks legs by id), an edited row showed
twice and a bucket listed a linked transaction twice. A reload kept it,
since boot loads the state file first. A card is affected when both files
have the same name (the fallback folder picker reads subfolders).

Cause: both files give the shared rows the same ids, and loadFiles() /
mergeImported() added every file's transactions, while
DB.importTransactions() skips an id it already holds. master has it too: its
import appends each file's rows (280 in memory, 257 in IndexedDB).

- mergeImported() keeps one transaction per id, the first file's, which is
  the row IndexedDB stores. Same-day identical card rows keep their _2, _3:
  they're counted per file, so both files give them the same ids.
- DB.importTransactions() takes every id of the import (importIds, passed by
  loadFiles() through importParsedCsv()) and never removes one as a leftover
  copy. Its check ran per file, so an earlier export read after a later one
  with one more same-day copy deleted that copy from the store.
- loadStateFromFile() keeps one copy of an id a state file holds twice, and
  saves the file again.
- Version bumped to v1.16.19 (CHANGELOG, footer, README, Settings,
  CLAUDE.md); CLAUDE.md §4.5, §8, §10, §19 updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
…3.0) into fix-overlapping-exports-duplicates

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBUZo6LRYpLthSp2PRi

# Conflicts:
#	js/views-settings.js
@gitluis gitluis changed the title v1.16.19: fix overlapping CSV exports of one account counting the shared rows twice v1.16.19: Fix overlapping CSV exports of one account counting the shared rows twice Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants