Repository navigation
Conversation
…wice (v1.16.19) Two exports of one account that share dates, imported together (a May to mid-September and an August to October checking export in the folder, say), left the shared rows twice in memory and, through the autosave, in finlens-state.json, while IndexedDB held them once. On the samples split that way (23 shared rows): 280 transactions instead of 257. Every total counted the shared rows twice, the second copy of a transfer stayed a plain withdrawal or deposit (detection tracks legs by id), an edited row showed twice and a bucket listed a linked transaction twice. A reload kept it, since boot loads the state file first. A card is affected when both files have the same name (the fallback folder picker reads subfolders). Cause: both files give the shared rows the same ids, and loadFiles() / mergeImported() added every file's transactions, while DB.importTransactions() skips an id it already holds. master has it too: its import appends each file's rows (280 in memory, 257 in IndexedDB). - mergeImported() keeps one transaction per id, the first file's, which is the row IndexedDB stores. Same-day identical card rows keep their _2, _3: they're counted per file, so both files give them the same ids. - DB.importTransactions() takes every id of the import (importIds, passed by loadFiles() through importParsedCsv()) and never removes one as a leftover copy. Its check ran per file, so an earlier export read after a later one with one more same-day copy deleted that copy from the store. - loadStateFromFile() keeps one copy of an id a state file holds twice, and saves the file again. - Version bumped to v1.16.19 (CHANGELOG, footer, README, Settings, CLAUDE.md); CLAUDE.md §4.5, §8, §10, §19 updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
Open
17 tasks
…3.0) into fix-overlapping-exports-duplicates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Z8iBUZo6LRYpLthSp2PRi # Conflicts: # js/views-settings.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two CSV exports of one account that share dates, imported together, left the shared rows twice in memory and in
finlens-state.json. IndexedDB held them once. Example: a May – mid-September and an August – October checking export in the folder.On the samples split that way (23 shared rows), FinLens showed 280 transactions instead of 257. Every total counted the shared rows twice. A page reload kept it, because boot loads the state file first.
Now an import keeps one transaction per id, in memory, IndexedDB and the state file, however many files overlap. A state file already saved with duplicates is cleaned up when it loads.
Cause
card.csvfiles from monthly folders arrive together.loadFiles()collected every file's transactions, andmergeImported()returnedkept.concat(fresh).DB.importTransactions()skips an id it already holds, so IndexedDB had one copy. The report's guess was right.detectTransfers()tracks legs by id, so it skips the copy once the first is marked. That copy counted as spending or income._3) than an earlier one. With the later file read first, the earlier file then deleted_3from the store as a "leftover copy". Memory had it; a boot from IndexedDB didn't. It only showed once memory was fixed, and the same fix point (one import, several files) covers it._2,_3are counted within each file, so both files give a same-day pair the same two ids, and they collapse to one pair.loadFiles()appends each file's rows toState.transactions. Checked onorigin/master(e8e18e7, v1.10.5) by callingloadFiles()with the same two checking slices plus the other samples: 280 in memory with 23 duplicate ids, 257 in IndexedDB. Its autosave writesState.transactions, so its state file would hold them too (read from the code, not run).What changed
mergeImported()(core.js) keeps one transaction per id. The first file's copy stays, because that's the row IndexedDB stores (later files skip the id). A loaded id held twice also keeps one copy.DB.importTransactions(txs, accountId, schemaId, importIds)(db.js) takes a new optional 4th argument: every id the import's files give. A row with one of those ids is never removed as a leftover copy.loadFiles()buildsimportIdsand passes it throughimportParsedCsv(p, importIds).loadStateFromFile()keeps one copy of an id the file holds twice, logs how many it dropped, and schedules a save so the file is rewritten.parseCsvFile()and the AI Eval'sbuildDataset()are untouched.Not verified
#filePick;#folderPick(webkitdirectory) goes through the samehandleFilePick(), but I didn't drive it.finlens-state.json.Automated checks: earlier run, not re-run · 6fc8bda
Moved unchanged from this PR's "Test results" comment on 2026-11-04; not re-run, so not in the new table format. Originally a description section titled "How it was tested".
Light checks, by Luis's call to save usage: on this branch's final commit
6fc8bda(chained on #41) onlyboot.js,race.jsand this PR's own tests ran. Headless Chromium at 1280×900 againstsamples/, with the pinned CDN libraries served from npm; each script ran asNODE_PATH=$(npm root -g) node <script> <repo> <outdir>.6fc8bdaboot.jsrace.jstest-overlapping-exports.js(new in this PR)Full set, once, on the top of the stack (#23,
819e809: every PR below it merged in): every base suite and every fix test from #24 to #50, plus #10's and #23's, each in its final variant. All pass:test-clarity.js55/55,test-insights.js137/137,test-finances.js126/126,test-plan.js127/127,boot.js257 transactions and 0 errors,race.js0 errors; #24–#38's tests (clear-edits 48/48, rebuild-folder 57/57, partial-import 47/47, json-import 28/28, new-browser-copy 10/10, json-export 31/31, plan-links 60/60, rule-flags 67/67, drawer-flags 36/36, start-balance 193/193, ai-summary-period 78/78, account-names 40/40, account-order 32/32, drawer-focus 167/167, cleanup 21/21); #39–#50's tests (drawer-fields 45/45, read-as 52/52, json-import-edits 29/29, overlapping-exports 46/46, override-balance 160/160, card-override 277/277, ai-summary-trend 91/91, transfer-user-type 39/39, untransfer-category 27/27, recategorize 55/55, retype-transfer 135/135, minor-fixes 34/34);test-dbtab.js73/73,test-save-status.js64/64,test-master-port.js16/16. Three scripts needed a test-only change on that run, listed in #23 and in the PRs they belong to; the app code wasn't changed.Measured before the chain
The numbers below are the PR author's, on the commit before it was chained (built on the old #38 tip
ecfcab2, v1.16.15, or on the fix below it there; version numbers renumbered to the chain's).Headless Chromium, desktop 1280×900, with the harness's fake folder. Suites ran one at a time.
test-overlapping-exports.js(new). It builds the fixtures by slicingsamples/:checking_aug_oct.csvandchecking_may_sep.csv;credit_card_sample.csv, from two directories;The reference is a full import of the unsliced samples (257), or of the samples with the 3rd copy (258). Each case compares every transaction with it (id, account, type, categories, description), and counts memory, IndexedDB and the state file, with duplicate ids.
_2. Then edits and a bucket link on shared rows, both files chosen again, autosave and reload._2+_3in memory and IndexedDB, nothing removed as a leftover copy, then a boot from IndexedDB.Totals are read from the screens: the Overview's net worth, latest month, cash flow and account cards, and the Transactions KPIs and count line.
b4a87e6): 15/46 passed, in a temporary worktree (since removed). Case 1: 280 in memory with 23 duplicate ids, 257 in IndexedDB, 280 in the state file and after a reload. Case 2: 344 with 87 duplicate ids; the pair shows as base, base,_2,_2; the bucket lists the linked row twice. Case 4, later file first: IndexedDB 257 against 258 unique ids in memory (_3deleted). Case 6: 269 loaded and saved back. The passes: the references, case 3 (one after the other already worked), the IndexedDB boots of cases 1 and 5, and the no-console-error checks.b648694): 46/46 passed.test-read-as.js: 52/52 on the parent, 52/52 on this branch.test-json-import-edits.js: 29/29 on the parent, 29/29 on this branch.run-tip.sh 15, the same counts on the parent and on this branch:renderImportLogerrors (expected on this base until v1.10.5: fix Import Logs throwing when the view is left before it loads #18's fix is in the stack). Parent 7; this branch 10 on the first run and 6 on the rerun. None from this change.On this branch, the first
run-tip.shrun was stopped by the tool's time limit beforetest-cleanup. I rantest-cleanupandraceagain on their own (ONLY=…). The parent ran in three groups the same way.No existing suite changed.
Master: a one-off probe (
probe-master-overlap.js, scratch only) onorigin/master, described under Cause.How to reproduce the bug
Run this on the base branch,
fix-json-import-edits-not-applied. Master has the bug too.fix-json-import-edits-not-appliedinstead offix-overlapping-exports-duplicates. Use the scratch folder with the two overlapping checking files.What goes wrong:
Manual QA steps
Setup
git checkout fix-overlapping-exports-duplicates.C:\temp\finlens-overlap. Copy the three files fromsamples/into it. Never use your real data folder for this PR: it writesfinlens-state.json.checking_may_sep.csv(66 rows),checking_aug_oct.csv(38 rows),savings_sample.csvandcredit_card_sample.csv. The two checking files share 23 rows (Aug 1 – Sep 15).index.htmlfrom the repo in it.1. Two overlapping exports in the folder count each row once
Expected:
Before this fix: net worth $16,361.06, Bank Checking "104 transactions" $14,182.68, Transactions 280, Money in +$39,874.88, Money out −$22,268.82.
2. The state file and IndexedDB hold one copy
Expected:
transactionsholds 257 entries.Before this fix: the state file held 280; IndexedDB held 257.
3. Reload keeps 257
Expected:
[FinLens] Boot: Loaded state from finlens-state.json — 257 transactions.Before this fix: 280 after the reload, because boot reads the state file first.
4. An edit on a shared row shows once, and survives Reload CSVs
RENT PAYMENTin Search description or notes.test notein Notes and click Save changes.RENT PAYMENTagain.Expected:
Before this fix: "Showing 8 of 8 (280 before filters)": Sep 1 and Aug 1 each listed twice.
5. A state file saved with duplicates is cleaned on load
finlens-state.jsonfrom the scratch folder.git checkout fix-json-import-edits-not-applied(the base branch).index.html, click Pick your CSV folder, choose the scratch folder and allow both prompts.git checkout fix-overlapping-exports-duplicates.Expected:
[FinLens] finlens-state.json held 23 transactions twice (the same id); one copy of each is kept.6. Regression: importing the exports one after the other
index.html.checking_may_sep.csv,savings_sample.csvandcredit_card_sample.csv, then click Import 3 files.checking_aug_oct.csv, and click Import 1 file.Expected:
7. Choosing both overlapping files again changes nothing
checking_may_sep.csvandchecking_aug_oct.csvtogether, click Import 2 files.Expected:
Before this fix: 280.
Please check before merging
finlens-state.jsonhas duplicates (an overlapping folder before this fix), check the console says how many were held twice on the next boot, and that the count drops.loadStateFromFile(). It's part of this fix so totals are right for data already saved with the bug, without waiting for a Reload.https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
Generated by Claude Code