Skip to content

v1.16.2: Fix Rebuild from your folder clearing everything when the folder has no CSV it can read - #25

Open
gitluis wants to merge 3 commits into
fix-clear-edits-wipes-transactionsfrom
fix-rebuild-empty-folder
Open

gitluis wants to merge 3 commits into
fix-clear-edits-wipes-transactionsfrom
fix-rebuild-empty-folder

Conversation

@gitluis

@gitluis gitluis commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #24 (fix-clear-edits-wipes-transactions), in a chain of fixes on top of the Clarity redesign (#16 → #17 → #19 → #20 → #21). Merge in order: #16, #17, #19, #20, #21, then the fixes #24–#38 and #39–#50 in order, then #10, then #23. #18 is a separate patch on master (already merged).
Written on top of #24, but its code doesn't need #24's: it is stacked so versions and the CHANGELOG merge cleanly.

Summary

Settings → Data & reset → Rebuild from your folder cleared the transactions, accounts and import log before it read the folder. Two kinds of folder ended with 0 transactions, and that was saved to finlens-state.json:

  • a folder with no CSV, only finlens-state.json;
  • a folder with only CSVs FinLens can't read.

Now the rebuild reads and checks every CSV first. It clears only when the files can be read. Otherwise nothing changes, and the tab says why.

The other imports had the same problem one level down: Reload CSVs, picking or reconnecting a folder, and Choose CSV files. loadFiles() emptied the loaded transactions before parsing. Now an import that reads no transaction changes nothing.

Cause

  • Rebuild: DataCleanup.rebuildFromFolder() called _dropImported() and DB.clearStores(['transactions', 'accounts', 'import_log']), then reloadFromDirHandle(). With no CSV, reloadFromDirHandle() showed an alert and returned. With unreadable CSVs, loadFiles() imported nothing. Either way _finish() then wrote 0 transactions to finlens-state.json, which boot loads first. On the samples, a reload afterwards shows 0.
  • Partial failure in Rebuild:
    • One CSV unreadable among several: the rebuild went ahead without it, and that account's transactions were gone. In the test, checking: 257 → 176.
    • A CSV the browser couldn't open: entry.getFile() threw after the clear. The tab said "Something went wrong", and memory and IndexedDB were left empty.
    • A failing IndexedDB write: parseCsvFile()'s complete: async callback never resolved, so the rebuild hung on "Working…".
  • Reload, pick, reconnect, choose files: loadFiles() set State.transactions = [] before parsing. When no file gave a transaction, none were left in memory. The autosave then wrote 0 to the state file. IndexedDB still held 257, but the state file wins at boot, so a reload showed 0.
    • Reconnect only imports into a browser with no transactions, so it had nothing to lose.
    • A folder with no CSV at all was already safe on Reload: it stops at the alert.
  • master has the bug. origin/master (v1.10.4) has the same rebuildFromFolder() and the same State.transactions = [] in loadFiles().

What changed

  • loadFiles() reads every file before changing anything (js/core.js).

    • If no transaction was read, it stops and changes nothing: memory, browser storage, IndexedDB, the sample data and the import pause all stay. An alert names each file and why, unless quiet.
    • It returns { imported, reason, transactions, files }.
    • New options: quiet, unopened (files the caller couldn't open), and beforeImport(result). beforeImport runs after the read; returning false stops the import.
  • parseCsvFile() now only reads. It returns { accountId, rows, txs, problem, … } and writes nothing. problem is one of 'unknown-format', 'no-rows' or 'unreadable'.

    • Schema detection, account ids and occurrence suffixes are unchanged, so the AI Eval's mirror (§21) needs no change.
  • importParsedCsv() (new) does what parseCsvFile() used to do after parsing: the account in State and IndexedDB, DB.importTransactions(), the import log, State.fileMeta.

    • A failing IndexedDB write is logged and the import carries on.
    • A file with no readable row no longer creates an empty account.
  • rebuildFromFolder() (js/views-cleanup.js) clears inside beforeImport, after the read, and only when _rebuildBlockers() finds nothing. A blocker is a file FinLens would import but couldn't read:

    • the browser couldn't open or parse it;
    • it is in a known format (by columns or name) but has no readable row;
    • it was imported before, and its columns are no longer recognised.

    With a blocker, nothing is cleared and the tab says why, in the callout the Data & reset actions use (_done('bad', …)). The tab has no per-row status line.

    • A CSV in another format that was never imported is skipped, as every import skips it. The success message names it.
  • reloadFromDirHandle() (js/views-data.js):

    • reports a CSV it can't open with the others, instead of throwing;
    • takes opts and returns loadFiles()'s result;
    • renders after it fills Last import. Before, Last import showed the previous import until the next render.
  • importAssignedFiles(): when nothing was imported, the chosen files stay listed, so Read as can be changed and tried again.

  • Last import lists files that weren't imported, with the reason (captureImportResults(), shared by both import paths).

  • New helpers: importProblemText() and importFileProblem() put a file's problem in words.

  • Version v1.16.2: CHANGELOG, footer, README (two places), Settings → General, CLAUDE.md §1 and Last updated.

  • CLAUDE.md: §8 (the new functions and options) and §20 (the Rebuild row, plus a "Read before clearing" rule).

  • Review tip: most of the core.js diff is parseCsvFile()'s body re-indented inside a try. View it with whitespace hidden.

Not verified

  • Real folders. The folder is a fake FileSystemDirectoryHandle. Nothing was run in Chrome or Edge against a real folder.
  • A locked file. The NotReadableError is simulated. A real file locked by another program wasn't tried.
  • Picking a folder (the Welcome's or Settings' Pick a folder) wasn't run: it needs showDirectoryPicker. It uses reloadFromDirHandle(), which the Reload cases cover.
  • Reconnect wasn't run with an unreadable folder. By reading the code, it only imports into a browser with no transactions.
  • Boot with an unreadable folder and no transactions now shows the new alert, like the existing "No CSV files found" alert at boot. Not run.
  • Reload and the folder picker: before, a CSV that couldn't be opened made Reload CSVs open the folder picker (reloadDataFiles()'s catch). That is from reading the code, not run.
  • Safari and Firefox weren't run.
  • Phone layout wasn't checked (desktop only).
Automated checks: earlier run, not re-run · 277ebe2

Moved unchanged from this PR's "Test results" comment on 2026-11-04; not re-run, so not in the new table format. Originally a description section titled "How it was tested".

Headless Chromium (Playwright) at 1280×900 against samples/ (257 transactions), with the pinned CDN libraries served from npm, on this branch's final commit 277ebe2, chained on #24. Each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>, from one harness shared by every branch in the stack: the base suites, plus every test added by this PR or by a PR below it (cumulative regression).

Script Result on 277ebe2
test-clarity.js 55/55
test-insights.js 137/137
test-finances.js 126/126
test-plan.js 127/127
boot.js 257 transactions, 24 views, 0 errors
race.js 7 errors, all renderImportLog (the race #18 fixes on master; not in this stack)
test-clear-edits.js (from #24) 47/47
test-rebuild-folder.js (new in this PR) 57/57

Measured before the chain

By the agent that wrote the fix, on 277ebe2 (this branch's commit too: it was already built on #24's 93a373a, so it didn't change when the stack was built).

Headless Chromium against samples/, with CDNs served from npm. Each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>.

New: test-rebuild-folder.js (57 checks). Every case starts in a fresh browser whose connected folder holds the three samples: boot imports 257, then Save now. The folder is a fake handle, as in test-clear-edits.js; it survives reloads and can be changed per case. Each case checks:

  • memory;
  • IndexedDB transactions, accounts and import log;
  • the state file, right away and after the 5-second autosave;
  • the message;
  • a page reload.

Cases:

  • A: a folder with no CSV.
  • B: only unreadable CSVs.
  • C1: one sample broken.
  • C2: a CSV in another format next to the samples.
  • C3: a CSV the browser can't open.
  • C4: a file imported before whose columns changed.
  • D: a normal rebuild, with reloads from the file and from IndexedDB.
  • E1, E2, E3: Reload CSVs with an empty, unreadable or partly unreadable folder.
  • F: Choose CSV files with one broken file.
  • G: IndexedDB fails on the 2nd file during a rebuild.
Script Before (93a373a) After (277ebe2)
test-rebuild-folder.js (new) 25/57 57/57
test-clarity.js 55/55 55/55
test-finances.js 126/126 126/126
test-plan.js 127/127 127/127
test-insights.js 137/137 137/137
test-clear-edits.js 47/47 47/47
boot.js 0 errors 0 errors
race.js 8 renderImportLog errors 6 renderImportLog errors
  • Before, in numbers:
    • A: memory, IndexedDB and the state file all went to 0, and the reload showed 0.
    • C1: 176 kept, checking lost.
    • E2: memory 0, state file 0, IndexedDB 257. The reload showed 0.
    • G: stuck on "Working…".
  • race.js: these are the known errors PR v1.10.5: fix Import Logs throwing when the view is left before it loads #18 fixes. The count varies by run; none come from this change. The line number moved by one because of the Last import change.
  • Existing suites: none was edited.
How to reproduce the bug

Run this on the base branch, fix-clear-edits-wipes-transactions. master has the same bug.

  1. Do the Setup from the testing steps below, but check out fix-clear-edits-wipes-transactions instead of fix-rebuild-empty-folder. Use the scratch folder with the three sample CSVs, never your real data folder. (real browser: the tests used a fake folder)
  2. Move the three sample CSVs from the scratch folder into the parked folder. The scratch folder now holds only finlens-state.json.
  3. Go to Settings → Data & reset. This browser's database shows 257 transactions and 3 accounts.
  4. On Rebuild from your folder, click Rebuild, then Click again to confirm.
  5. Click OK on the alert.
  6. Go to Transactions (All time).
  7. Press F5, set All time again, and open finlens-state.json from the scratch folder.

What goes wrong:

  • An alert says "No CSV files found in the selected folder.", then the success message says 0 transactions re-imported.
  • Transactions reads 0, not 257, and This browser's database shows 0.
  • After the reload it still shows 0, and finlens-state.json has "transactions": [].
Manual QA steps

Setup

  1. git checkout fix-rebuild-empty-folder.
  2. Make a scratch folder, for example C:\temp\finlens-25. Copy the three CSVs from samples/ into it. Never use your real data folder for this PR. Keep samples/ itself untouched so you can copy fresh files back.
  3. Make a second folder next to it, C:\temp\finlens-25-parked, to move files into and out of.
  4. In the scratch folder, create notes.csv in Notepad with these two lines, and save it:
    Foo,Bar
    1,2
    
    Move notes.csv into the parked folder for now.
  5. Open index.html in Edge. Click Pick your CSV folder and choose the scratch folder. Allow access.
  6. Click the calendar icon (Date range) and choose All time. Do this again after every reload.
  7. Go to Transactions. The Transactions tile reads 257.
  8. Go to Settings → Import → Save now.

Every button in Data & reset → Clean up needs two clicks: the button, then Click again to confirm.

1. Rebuild with no CSV in the folder changes nothing (main fix)

  1. Move the three sample CSVs from the scratch folder into the parked folder. The scratch folder now holds only finlens-state.json.
  2. Go to Settings → Data & reset. Note the counts under This browser's database (257 transactions · 3 accounts · …).
  3. On Rebuild from your folder, click Rebuild, then Click again to confirm.

Expected:

  • A red callout at the top: Nothing was cleared: "finlens-25" has no CSV files.
  • No alert box pops up.
  • This browser's database still shows 257 transactions and 3 accounts.
  • Transactions (All time) still reads 257.

Before this fix: an alert "No CSV files found in the selected folder.", then the success message with 0 transactions re-imported, and 0 everywhere.

2. Nothing is lost after a reload (persistence)

  1. Right after test 1, press F5. Set All time.
  2. Go to Transactions.
  3. Open finlens-state.json in the scratch folder in Notepad and search for "transactions".

Expected:

  • The tile reads 257.
  • The file's transactions list is not empty (it still lists the sample rows, e.g. UBER TRIP).

Before this fix: the reload shows 0, and the file has "transactions": [].

3. Rebuild with only an unreadable CSV changes nothing

  1. Move notes.csv from the parked folder into the scratch folder (the samples stay parked).
  2. Go to Settings → Data & reset → Rebuild from your folder → Rebuild → Click again to confirm.

Expected:

  • Red callout: Nothing was cleared: FinLens couldn't read any transactions from the CSV files in "finlens-25". notes.csv: its columns don't match a format FinLens reads.
  • Transactions still reads 257.

4. Reload CSVs with only an unreadable CSV changes nothing

  1. With the scratch folder still holding only notes.csv and finlens-state.json, go to Settings → Import.
  2. Click Reload CSVs.
  3. Read the alert, then click OK.
  4. Wait 10 seconds (the autosave runs 5 seconds after a change). Press F5 and set All time.

Expected:

  • The alert says FinLens couldn't read any transactions from this file, so nothing was changed. followed by notes.csv: its columns don't match a format FinLens reads.
  • Transactions reads 257 before and after the reload.

Before this fix: memory and finlens-state.json went to 0 (IndexedDB kept 257), and the reload showed 0.

5. One broken sample blocks the rebuild

  1. Copy the three sample CSVs from samples/ into the scratch folder (so it holds the three samples, notes.csv and finlens-state.json).
  2. Open the scratch folder's checking_sample.csv in Notepad. Use Replace (Ctrl+H): find 2026-, replace with bad-, Replace all. Save and close.
  3. Go to Settings → Data & reset → Rebuild from your folder → Rebuild → Click again to confirm.

Expected:

  • Red callout: Nothing was cleared: FinLens couldn't read checking_sample.csv (none of its 81 rows has a date and an amount FinLens can read). Rebuild re-reads every CSV from scratch, so that file's transactions would be left out. Fix or replace it, or move it out of the folder, then rebuild again.
  • notes.csv is not named as a blocker.
  • Transactions still reads 257. Searching PAYROLL still finds the checking deposits.

Before this fix: the rebuild went ahead without checking: 176 transactions, and every checking row gone.

6. Choose CSV files with a broken file keeps the file listed

  1. Go to Settings → Import. Scroll to Using Safari or Firefox? and click Choose CSV files.
  2. Pick only the broken checking_sample.csv from the scratch folder.
  3. In Choose how to read this file, leave Read as on Checking. Click Import 1 file.
  4. Read the alert and click OK.

Expected:

  • The alert says nothing was changed and names checking_sample.csv: none of its 81 rows has a date and an amount FinLens can read.
  • The Choose how to read this file panel is still shown with the file, so you can change Read as or click Cancel.
  • Transactions still reads 257.
  • Click Cancel to close the panel.

7. A normal rebuild still works, and names a skipped CSV (regression)

  1. Copy a fresh checking_sample.csv from samples/ over the broken one in the scratch folder. Leave notes.csv there.
  2. Go to Settings → Data & reset → Rebuild from your folder → Rebuild → Click again to confirm.
  3. Go to Settings → Import and look at Last import.
  4. Press F5, set All time, and go to Transactions.

Expected:

  • Green callout: Rebuilt from "finlens-25": 257 transactions re-imported. Not read: notes.csv (its columns don't match a format FinLens reads). Backup file updated.
  • Last import lists the three samples with their counts, plus a notes.csv row: Not imported: its columns don't match a format FinLens reads.
  • After the reload the tile reads 257.
  • F12 → Console: a yellow parseCsvFile: no schema matched for notes.csv warning is fine; no red errors.

8. Reload CSVs with a good folder updates Last import right away

  1. Delete notes.csv from the scratch folder.
  2. Go to Settings → Import and click Reload CSVs.

Expected:

  • No alert.
  • Last import updates at once to the three samples, each with 0 new · N already imported (81 for checking, 11 for savings, 165 for the card), and no notes.csv row. Before this fix, Last import kept showing the previous import until you left and came back to the tab.
  • Transactions reads 257.

Please check before merging

  • The Rebuild rule. One unreadable CSV that FinLens would import blocks the whole rebuild. That includes a file matched only by its name: a CSV called checklist.csv or credit_score.csv that isn't an export. Is blocking right, or should those be skipped?
  • Alerts. An import that reads nothing says why in an alert(), as the existing "No CSV files found" alert does, at boot too. Is that fine, or should it be a status line?
  • Chosen files stay listed after an import that read nothing. Is that the behaviour you want?
  • Unchanged, but worth knowing: Rebuild still drops accounts whose CSV isn't in the folder, e.g. a card imported with Choose CSV files from elsewhere. That is the existing "from scratch" behaviour.
  • Found, not fixed: after a successful rebuild the button reads "Rebuild", not "Rebuilt". loadFiles() renders mid-rebuild and uses up DataCleanup.lastDone. The parent commit does the same.
  • Merge conflicts, resolved in the stack: fix-account-name-reset (v1.16.12: Fix clearing an account name not bringing the default name back #35) and fix-account-order-after-import (v1.16.13: Fix account display order lost when importing files one at a time #36) change the account set-up that was in parseCsvFile() and the account order in loadFiles(). The set-up block now lives in importParsedCsv(); those PRs re-apply their changes there, on top of this one.
  • Manual QA passed in Edge/Chrome with a real folder

https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft


Generated by Claude Code

…s no CSV it can read (v1.16.2)

DataCleanup.rebuildFromFolder() cleared the transactions, accounts and import log
(memory and IndexedDB) before reading the folder. With only finlens-state.json in it,
or only CSVs FinLens can't parse, 0 transactions were left, and _finish() wrote that
to finlens-state.json, which boot loads first. Reload CSVs, picking or reconnecting a
folder and Choose CSV files had the same problem one level down: loadFiles() emptied
State.transactions before parsing, and the autosave then wrote 0 to the file.
master has the same code.

- loadFiles() reads every file first and changes nothing (memory, storage, IndexedDB,
  sample data, the import pause) unless at least one transaction was read. An alert
  names each file and why; chosen files stay listed so Read as can be changed.
- parseCsvFile() now only reads a file: { accountId, rows, txs, problem, ... }, no
  writes. importParsedCsv() does the import; a failing IndexedDB write is logged and
  the import carries on (before, that file's import never finished).
- rebuildFromFolder() clears inside loadFiles()'s new beforeImport hook, after the
  read, and only when _rebuildBlockers() finds no file it would import but couldn't
  read (unopenable, no readable row, or imported before with columns it no longer
  knows). Otherwise nothing is cleared and the tab says why. CSVs in other formats
  that were never imported are skipped and named.
- reloadFromDirHandle() reports a CSV it can't open instead of stopping, returns
  loadFiles()'s result, and renders after filling Last import, which lists files that
  weren't imported.
- A file with no readable row no longer creates an empty account.
- Version bumped to v1.16.2 (CHANGELOG, footer, README, Settings, CLAUDE.md);
  CLAUDE.md §8 and §20 updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
… fix-rebuild-empty-folder

Carries master's v1.10.3-v1.10.5 up the stack: declined answers (#14),
literal single "~" (#15) and the Import Logs race fix (#18), with
clarity-insights' port of #14 to the rebuilt AI tab.

One conflict: CLAUDE.md's "Last updated" line. Kept this branch's
line and added the v1.10.3-v1.10.5 history. The CHANGELOG merged cleanly.
Versions stay 1.16.2. renderImportLog() keeps #18's guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
…13.0) into fix-rebuild-empty-folder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBUZo6LRYpLthSp2PRi

# Conflicts:
#	js/views-settings.js
@gitluis gitluis changed the title v1.16.2: fix Rebuild from your folder clearing everything when the folder has no CSV it can read v1.16.2: Fix Rebuild from your folder clearing everything when the folder has no CSV it can read Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data loss Fixes a bug that loses or corrupts saved data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants