Skip to content

v1.16.3: Fix importing some CSV files losing the other loaded transactions - #26

Open
gitluis wants to merge 3 commits into
fix-rebuild-empty-folderfrom
fix-partial-file-import
Open

gitluis wants to merge 3 commits into
fix-rebuild-empty-folderfrom
fix-partial-file-import

Conversation

@gitluis

@gitluis gitluis commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #25 (fix-rebuild-empty-folder), in a chain of fixes on top of the Clarity redesign (#16 → #17 → #19 → #20 → #21). Merge in order: #16, #17, #19, #20, #21, then the fixes #24–#38 and #39–#50 in order, then #10, then #23. #18 is a separate patch on master (already merged).
Depends on earlier fixes' code: #24's recorded CSV type and category (_originalType, _originalCategories) and #25's read-first import (importParsedCsv(), loadFiles()).

Summary

Importing only some of your CSV files replaced every loaded transaction with those files' ones. For example, choose just the card file on the Welcome or in Settings → Import. On the samples, 257 became 165. IndexedDB kept all 257, but the autosave wrote 165 to finlens-state.json, and boot loads that file first. After a reload, the other accounts were gone.

Now an import adds the files' new transactions and replaces the ones it already has. Every other loaded transaction stays, as IndexedDB already does.

Cause

  • Where: loadFiles() in js/core.js. The import step started with State.transactions = [] and then pushed only the imported files' transactions. DB.importTransactions() never removes a row it holds (§10), so after any partial import, memory and IndexedDB disagreed. The state file followed memory, and it wins at boot (§4.5).

  • What else it broke:

    • Card payments lost their pair with the checking autopay ("Payment Thank You-Mobile": transfer → payment).
    • Edited transactions in the other accounts disappeared from every screen.
  • Folder reloads had it too. Reload CSVs replaced memory with whatever the folder held. These cases lost rows the same way:

    • a newer export that starts where the last one ended. The Import tab itself suggests "start the next export on …";
    • an old CSV moved out of the folder.

    On the samples, an October-only card export left 117 of 257. So folder reloads now keep rows too; see "Please check".

  • master has the bug. origin/master (v1.10.4) has the same State.transactions = [] in loadFiles(), and the same file-picker path.

What changed

  • mergeImported() (new, js/core.js) builds the loaded transactions after an import, the way DB.importTransactions() builds the store:
    • a file's transaction replaces a loaded one with the same id;
    • new ids are added;
    • a leftover suffixed copy (left by ≤v1.8.0) is removed, by the same rule as the store;
    • every other loaded transaction stays.
  • undoTransferDetection() (new) puts a kept transfer back to its CSV type and category, with its edit, before detectTransfers() runs again.
    • Why it's needed: detection only pairs transactions that aren't transfers yet. Without this, a kept leg's partner, just re-read from a file, stayed unpaired.
    • A type the user set to transfer stays.
    • A transfer from a ≤v1.16.0 state file has no _originalType/_originalCategories. Its IndexedDB row gives them. Without a row, the type goes back to debit or credit by sign and the category comes from autoCategory(), as v1.16.1's Clear transaction edits does. Then detection pairs it again.
  • Sample data: importing your own files still replaces the samples. That branch of loadFiles() now empties State.transactions itself.
  • Rebuild from your folder still replaces everything: it drops the loaded data before the import (_dropImported()).
  • Version v1.16.3: CHANGELOG, footer, README (two places), Settings → General, CLAUDE.md §1 and Last updated.
  • CLAUDE.md: §4.5 (an import never drops loaded transactions), §8 (the new functions), §10 (memory follows the same re-upload rules) and §11 (kept transfers are detected again).

Not verified

  • Real folders. The folder is a fake handle. Nothing was run in Chrome or Edge against a real folder or a real bank export.
  • Large data. Only the 257 sample rows were used. mergeImported() uses sets, and reads IndexedDB once, only when a kept transfer lacks its CSV values. Its speed on thousands of rows wasn't measured.
  • Samples loaded on top of your own data now merge in memory instead of replacing it. The UI doesn't offer this (the Welcome shows only with no transactions); not tested.
  • Safari and Firefox weren't run.
  • Phone layout wasn't checked (desktop only).
Automated checks: earlier run, not re-run · 9137aa4

Moved unchanged from this PR's "Test results" comment on 2026-11-04; not re-run, so not in the new table format. Originally a description section titled "How it was tested".

Headless Chromium (Playwright) at 1280×900 against samples/ (257 transactions), with the pinned CDN libraries served from npm, on this branch's final commit 9137aa4, chained on #25. Each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>, from one harness shared by every branch in the stack: the base suites, plus every test added by this PR or by a PR below it (cumulative regression).

Script Result on 9137aa4
test-clarity.js 55/55
test-insights.js 137/137
test-finances.js 126/126
test-plan.js 127/127
boot.js 257 transactions, 24 views, 0 errors
race.js 7 errors, all renderImportLog (the race #18 fixes on master; not in this stack)
test-clear-edits.js (from #24) 47/47
test-rebuild-folder.js (from #25) 57/57
test-partial-import.js (new in this PR) 47/47

Measured before the chain

By the agent that wrote the fix, on 9137aa4 (this branch's commit too: it was already built on #25's 277ebe2, so it didn't change when the stack was built).

Headless Chromium against samples/, with CDNs served from npm. Each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>.

New: test-partial-import.js (47 checks). The fake folder is the same one test-rebuild-folder.js uses. Every step checks:

  • memory, IndexedDB and the state file (count, accounts, duplicate ids);
  • each transaction against a full import of the three samples: account, type, categories and description, so transfers must pair as in a full import;
  • a page reload.

Cases:

  1. One file, then two: the card file into an empty browser, then checking and savings, then the card file again. Checks no duplicates and that IndexedDB skipped all 165.
  2. One file over everything loaded: the card file, then only savings. This is the reported bug.
  3. A newer card export that starts on Oct 1: with Choose CSV files, then through the folder with Reload CSVs.
  4. A full folder reload: still replaces every loaded row with the file's new copy, as before.
  5. Your own file over the sample data: the samples go. A file that can't be read leaves them.
  6. A leftover duplicate copy: removed from memory as from IndexedDB.
  7. A v1.16.0 state file with an empty IndexedDB: kept transfers with no recorded CSV type still pair as in a full import.
  8. Edits on kept transactions stay.
  9. No IndexedDB.
Script Before (277ebe2) After (9137aa4)
test-partial-import.js (new) 22/47 47/47
test-rebuild-folder.js (previous PR) 57/57 57/57
test-clarity.js 55/55 55/55
test-finances.js 126/126 126/126
test-plan.js 127/127 127/127
test-insights.js 137/137 137/137
test-clear-edits.js 47/47 47/47
boot.js 0 errors 0 errors
race.js 6 renderImportLog errors 6 renderImportLog errors
  • Before, in numbers:
    • Case 2: 165 in memory and the state file, 257 in IndexedDB. The reload showed 165. Then importing only savings left 11.
    • Case 1: checking and savings after the card left 92.
    • Case 3: 25, then 117.
    • Case 8: both edited transactions were gone.
    • Cases 4 and 5 passed before too: a full folder reload and the sample-data rule behave as they did.
  • race.js: these are the known errors PR v1.10.5: fix Import Logs throwing when the view is left before it loads #18 fixes; none come from this change.
  • Existing suites: none was edited.
How to reproduce the bug

Run this on the base branch, fix-rebuild-empty-folder. master has the same bug.

  1. Do the Setup from the testing steps below, but check out fix-rebuild-empty-folder instead of fix-partial-file-import. Use the scratch folder with the three sample CSVs, never your real data folder. You can skip step 3 (the October-only file).
  2. Go to Settings → Import. Scroll to Using Safari or Firefox? and click Choose CSV files.
  3. Select only credit_card_sample.csv from the scratch folder.
  4. In Choose how to read this file, leave Read as on Credit card. Click Import 1 file.
  5. Go to Transactions (All time, All accounts). Search PAYROLL.
  6. Wait 10 seconds, press F5, set All time, and go to Transactions again.
  7. Go to Settings → Data & reset and read This browser's database.

What goes wrong:

  • The Transactions tile reads 165, not 257. PAYROLL finds nothing: checking and savings are gone from every screen.
  • After the reload it still reads 165, because finlens-state.json was written with 165, while This browser's database still shows 257 transactions.
Manual QA steps

Setup

  1. git checkout fix-partial-file-import.
  2. Make a scratch folder, for example C:\temp\finlens-26. Copy the three CSVs from samples/ into it. Never use your real data folder for this PR.
  3. Make an October-only card export for test 4: copy samples/credit_card_sample.csv to C:\temp\card_oct.csv (outside the scratch folder). Open it in Notepad. Keep the header and the first 25 data rows (through 2026-10-01,10/02/2026,WHOLE FOODS MKT,…). Delete every line after that, starting at 2026-09-26,09/27/2026,CHIPOTLE 2410. Save. Rename it to credit_card_sample.csv inside a new folder C:\temp\card-oct\ (same name as the original, so it maps to the same card account).
  4. Open index.html in Edge. Click Pick your CSV folder and choose the scratch folder. Allow access.
  5. Click the calendar icon (Date range) and choose All time. Do this again after every reload.
  6. Go to Transactions. The Transactions tile reads 257 (81 checking, 11 savings, 165 card).
  7. Go to Settings → Import → Save now.

To count one account: click the bank icon (Accounts) in the top bar and pick that account; pick All accounts to go back.

1. Importing only the card file keeps the other accounts (main fix)

  1. Go to Settings → Import. Scroll to Using Safari or Firefox? and click Choose CSV files.
  2. Select only credit_card_sample.csv from the scratch folder.
  3. In Choose how to read this file, leave Read as on Credit card. Click Import 1 file.
  4. Look at Last import.
  5. Go to Transactions (All time, All accounts).

Expected:

  • Last import shows the card file with 0 new · 165 already imported.
  • The Transactions tile reads 257, not 165.
  • Searching PAYROLL still finds the checking deposits, and Interest Paid still finds the savings rows.

Before this fix: the tile read 165, and checking and savings were gone from every screen.

2. Still 257 after the autosave and a reload (persistence)

  1. Wait 10 seconds (the autosave runs 5 seconds after a change).
  2. Press F5 and set All time.
  3. Go to Transactions.
  4. Go to Settings → Data & reset and read This browser's database.

Expected:

  • The tile reads 257 after the reload.
  • This browser's database shows 257 transactions.

Before this fix: finlens-state.json was written with 165, and the reload showed 165 while IndexedDB still had 257.

3. Card payments stay paired with the checking autopay

  1. Go to Transfers (All time).
  2. Note the Transfers tile (for example "N paired · M one-sided").
  3. Repeat test 1 (import only credit_card_sample.csv with Choose CSV files).
  4. Go back to Transfers.
  5. Go to Transactions, search Payment Thank You.

Expected:

  • The Transfers tile shows the same paired and one-sided counts as in step 2.
  • The Oct 6 $836.91 checking → card pair is still listed.
  • The Payment Thank You-Mobile rows show Transfer in, not a card payment category.

Before this fix: the payments became unpaired (type "payment"), because the checking autopays had been dropped.

4. A newer export that starts where the last one ended keeps the older rows

  1. Go to Settings → Import → Choose CSV files.
  2. Select C:\temp\card-oct\credit_card_sample.csv (the October-only file). Click Import 1 file.
  3. Look at Last import, then go to Transactions.

Expected:

  • Last import shows 0 new · 25 already imported.
  • The tile still reads 257.
  • With Accounts set to the card only, the card still has rows from May 2026 (for example search CHIPOTLE and see dates back to May).

Before this fix: the tile read 117 (only the 25 October card rows plus checking and savings).

5. Edits on other accounts survive a partial import

  1. Go to Transactions, search LUMA ELECTRIC, open the Oct 12 row (−$108.81).
  2. Change Description to POWER BILL TEST and add kept? in Notes. Click Save changes.
  3. Go to Settings → Import → Choose CSV files, select only savings_sample.csv from the scratch folder, click Import 1 file.
  4. Go to Transactions and search POWER BILL TEST.

Expected:

  • The row is there with the note kept?.
  • The tile reads 257.

Before this fix: only the 11 savings rows were left, and the edited row was gone.

6. Reload CSVs and Rebuild still work (regression)

  1. Go to Settings → Import → Reload CSVs.
  2. Check Last import: each sample shows 0 new · N already imported (81, 11, 165). The tile on Transactions reads 257.
  3. Go to Settings → Data & reset → Rebuild from your folder → Rebuild → Click again to confirm.

Expected:

  • Green callout: Rebuilt from "finlens-26": 257 transactions re-imported. Backup file updated.
  • This browser's database shows 257 transactions · 3 accounts.
  • The POWER BILL TEST edit from test 5 is still there (Rebuild keeps edits).

7. Your own files still replace the sample data

  1. Open an InPrivate window (Ctrl+Shift+N) and open index.html.
  2. On the Welcome, click Try it with sample data. Opened from disk, FinLens asks you to choose the files: select the three CSVs in samples/.
  3. The yellow banner says You're trying FinLens with sample data. Transactions reads 257 (the date range is already All time).
  4. Go to Settings → Import → Choose CSV files. Select only C:\temp\card-oct\credit_card_sample.csv. Click Import 1 file.
  5. Go to Transactions.

Expected:

  • The sample-data banner is gone.
  • Set Date range to All time if needed: the tile reads 25 (only the October card rows). The samples were replaced, not merged.
  • F12 → Console shows no red errors.

Please check before merging

  • Folder reloads now keep rows the folder no longer has, matching IndexedDB. Before, memory dropped them, the state file followed, and a boot from IndexedDB brought them back. Is this the behaviour you want?
  • Read as, then Reload CSVs. Change a file's Read as, then Reload CSVs, and its rows now show under both accounts until you Rebuild. On the samples, savings read as checking gives 268, the same as IndexedDB. Before: 257 in memory, 268 in IndexedDB. The Read-as message already says to use Rebuild for rows already imported. Reword it to say a plain reload adds them again? test-insights.js checks the "from the next reload" wording.
  • A changed row from the bank shows twice until Rebuild, e.g. a pending row that posted with a new description or amount. IndexedDB already kept both.
  • Older state files: for a kept transfer from a ≤v1.16.0 state file with no IndexedDB row, a card's own Category and Payment type can't be recovered. This is the same limit as v1.16.1.
  • Stack: this PR is based on fix-rebuild-empty-folder (v1.16.2).
  • Manual QA passed in Edge/Chrome with a real folder

https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft


Generated by Claude Code

…the next boot (v1.16.3)

loadFiles() replaced every loaded transaction with the imported files' ones. On the
samples, importing only the card file (Choose CSV files) left 165 of 257 in memory.
IndexedDB kept all 257, since DB.importTransactions() only adds rows, but the autosave
wrote 165 to finlens-state.json, which boot loads first. The same happened on Reload
CSVs when the folder no longer held all of an account's rows (a newer export that
starts where the last one ended, an old CSV moved out). master has the same code.

- loadFiles() merges (mergeImported()): the files' transactions replace loaded ones
  with the same id, new ones are added, every other loaded transaction stays. A
  leftover suffixed copy is removed from memory as importTransactions() removes it
  from the store.
- Kept transfers go back to their CSV type and category, with their edit, before
  detection (undoTransferDetection()), so they pair as in a full import. A transfer
  from a <=v1.16.0 state file uses its IndexedDB row, else debit/credit by sign and
  autoCategory().
- Importing your own files still replaces the sample data (it now empties
  State.transactions there explicitly). Rebuild from your folder still replaces
  everything: it drops the loaded data before the import.
- Version bumped to v1.16.3 (CHANGELOG, footer, README, Settings, CLAUDE.md);
  CLAUDE.md §4.5, §8, §10 and §11 updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
…al-file-import

Carries master's v1.10.3-v1.10.5 up the stack: declined answers (#14),
literal single "~" (#15) and the Import Logs race fix (#18), with
clarity-insights' port of #14 to the rebuilt AI tab.

One conflict: CLAUDE.md's "Last updated" line. Kept this branch's
line and added the v1.10.3-v1.10.5 history. The CHANGELOG merged cleanly.
Versions stay 1.16.3. renderImportLog() keeps #18's guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
… fix-partial-file-import

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBUZo6LRYpLthSp2PRi

# Conflicts:
#	js/views-settings.js
@gitluis gitluis changed the title v1.16.3: fix importing some CSV files losing the other loaded transactions v1.16.3: Fix importing some CSV files losing the other loaded transactions Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data loss Fixes a bug that loses or corrupts saved data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants