Skip to content

v1.16.4: Fix JSON import losing spotted-charge choices and category rules - #27

Open
gitluis wants to merge 3 commits into
fix-partial-file-importfrom
fix-json-import-keys
Open

gitluis wants to merge 3 commits into
fix-partial-file-importfrom
fix-json-import-keys

Conversation

@gitluis

@gitluis gitluis commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #26 (fix-partial-file-import), in a chain of fixes on top of the Clarity redesign (#16 → #17 → #19 → #20 → #21). Merge in order: #16, #17, #19, #20, #21, then the fixes #24–#38 and #39–#50 in order, then #10, then #23. #18 is a separate patch on master (already merged).
This fix doesn't use any earlier fix's code; it is stacked only so versions and the CHANGELOG merge cleanly.

Summary

After Settings → Data & reset → Import JSON, the spotted-charge choices (Bills → Spotted recurring charges: Confirm / Dismiss) and the category rules (Spending → Category rules) showed at first, then were gone after a reload. This fixes the import, and moves the places that save keys by State name in a loop onto one shared mapping. It also fixes a smaller problem found during the audit: a null value in the file broke categorization. A third finding, a new database dropping three keys, was part of this commit before the chain; it is now its own PR, #28, stacked on this one.

Cause

  • Two keys have older storage names. Browser storage (localStorage and IndexedDB app_state) keeps State.recurringConfirmed as recurring and State.categoryRules as catRules. Every other key is stored under its State name.
  • The import used State names. importJsonFile() called lsSet(k, d[k]) with the State name, so these two went to finlens_recurringConfirmed / finlens_categoryRules and app_state recurringConfirmed / categoryRules. Nothing reads those keys. At boot, lsGet('recurring'), lsGet('catRules') and DB.loadAll() loaded the old values.
  • The state file was right. saveStateToFile() and loadStateFromFile() use State names, and the debounced save writes from memory. So with a connected folder, a reload more than 5 seconds after the import loaded the file and kept the import. Without a folder, or once folder access had lapsed (the browser was closed), the import was lost.

What changed

The mapping

  • core.js: STORAGE_KEYS = { recurringConfirmed: 'recurring', categoryRules: 'catRules' } and storageKey(stateKey). The storage names don't change, so existing data loads as before. State's initializer reads both keys through it.

Every key, State name → storage key

State key Storage key Notes
recurringConfirmed recurring mismatch, fixed in the import
categoryRules catRules mismatch, fixed in the import
budgets, buckets, debts, bills, txOverrides, manualBalances, categories, apiKey, aiHistory, scenarios, investAlloc, debtPlan, cardDetails, sampleData, theme, lastFolderName, csvAutoImportPaused, fileAssignments, accountOrder, accountLabels, txRules, transferKeywords, customSchemas same as the State name no mismatch

Places audited

  • importJsonFile() (ui.js): the bug. It now saves through storageKey(). A key set to null in the file now counts as missing and keeps the current value; before, a null categoryRules made autoCategory() throw.
  • DB.loadAll() (db.js): was right (its own list of storage names). Not changed here; v1.16.5: Fix a new browser database dropping the card details, the sample-data flag and the import pause #28 moves it onto storageKey() and a shared list.
  • DB._migrateSync() (db.js, a new database copies localStorage into app_state): names were right, but its hand-written list missed cardDetails, sampleData and csvAutoImportPaused, which DB.loadAll() then reset to their defaults. Real data loss, fixed separately in v1.16.5: Fix a new browser database dropping the card details, the sample-data flag and the import pause #28.
  • Data & reset (views-cleanup.js): was right (PLAN_ALSO listed 'recurring' by hand). The plan clears now save through storageKey(), and PLAN_ALSO no longer carries a storage key of its own. No change in behavior.
  • migrate.js: all five of its KEYS are stored under their State names, so it was right. migrateStores() now reads and writes browser storage through storageKey(). No change in behavior.
  • AI Eval sandbox (views-eval.js): swaps State keys in memory only and never writes storage. No change.
  • saveStateToFile(), loadStateFromFile(), boot, _restoreStateIfBrowserEmpty(): use State names in the file and don't write browser storage. No change.
  • The direct call sites (lsSet('recurring', …), lsSet('catRules', …) in ui.js and views-data.js) were right. Unchanged.

Version and docs

Not verified

  • A real folder. The state-file check uses a fake FileSystemDirectoryHandle. Real Chrome or Edge with a real folder wasn't run.
  • Firefox and Safari weren't run (Chromium only).
  • A real 1.0 export. No 1.0 writer exists in the repo's history: the export has said 1.1 since the first commit. The test uses a hand-made 1.0-shaped file with fewer keys.
  • Leftover keys. An import before this fix left recurringConfirmed and categoryRules in browser storage. Nothing reads them and Start over removes them, but this PR doesn't clean them up. Anyone who imported before the fix should import the file again.
Automated checks: earlier run, not re-run · 3a21a04

Moved unchanged from this PR's "Test results" comment on 2026-11-04; not re-run, so not in the new table format. Originally a description section titled "How it was tested".

Headless Chromium (Playwright) at 1280×900 against samples/ (257 transactions), with the pinned CDN libraries served from npm, on this branch's final commit 3a21a04, chained on #26. Each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>, from one harness shared by every branch in the stack: the base suites, plus every test added by this PR or by a PR below it (cumulative regression).

Script Result on 3a21a04
test-clarity.js 55/55
test-insights.js 137/137
test-finances.js 126/126
test-plan.js 127/127
boot.js 257 transactions, 24 views, 0 errors
race.js 8 errors, all renderImportLog (the race #18 fixes on master; not in this stack)
test-clear-edits.js (from #24) 47/47
test-rebuild-folder.js (from #25) 57/57
test-partial-import.js (from #26) 47/47
test-json-import.js (new in this PR) 28/28

Measured before the chain

By the agent that wrote the fix, on d47dd8d (built on ddc8a5f, then numbered v1.16.1), which also held the #28 fix. Version numbers below are as the agent wrote them, except the PR's own version.

Headless Chromium (Playwright) against samples/, on commit d47dd8d. The CDN libraries are served from npm (same versions; the SRI hashes in index.html verify).

  • test-clarity.js: 55/55.
  • test-finances.js: 126/126.
  • test-plan.js: 127/127.
  • test-insights.js: 137/137. One check edited: "General: version line" matched v1.16.0 literally. It now reads the version from the index.html footer, so it follows a version bump and also checks that the two places agree.
  • boot.js: 257 transactions, 24 views, 0 errors.
  • race.js: 7 errors, all renderImportLog / importLogsBody, which v1.10.5: fix Import Logs throwing when the view is left before it loads #18 fixes. v1.16.0 shows 7 too. None come from this change.
  • test-json-import.js (new): 31/31. On v1.16.0 (ddc8a5f) it fails 11 of 31, all from this bug and the two side issues. It covers:
    • Round trip: dismiss one spotted charge and confirm another, add two category rules (all through the UI), export, Start over, re-import samples/, import the file, reload. After the reload, Bills shows the charge as Dismissed and Spending lists both rules.
    • Storage: after the import, recurring and catRules hold the imported values in localStorage and app_state. Nothing is written under recurringConfirmed or categoryRules.
    • State file: with a fake folder handle, the debounced save writes the imported values and no API key. Loading that file as boot does restores them.
    • Older exports: a 1.0-shaped file, a 1.1 file from before v1.15.0 (no debtPlan / cardDetails), a file with no version, and a file with null values. Each leaves the keys it lacks unchanged in State and storage, also after a reload.
    • Clear bills clears recurring in storage.
    • New database: with IndexedDB deleted and localStorage kept, every value survives the reload, cardDetails included. On v1.16.0 cardDetails was lost. (This section moved to v1.16.5: Fix a new browser database dropping the card details, the sample-data flag and the import pause #28's test-new-browser-copy.js with the fix it covers; the 3 checks it held are not in this branch's 28.)
    • Migration: old account codes in localStorage and app_state are still renamed at boot. Passes on v1.16.0 too, as a check that behavior didn't change.
How to reproduce the bug

Run this on the base branch, fix-partial-file-import.

  1. Do the Setup from the testing steps below (InPrivate window, sample data, no folder connected), but check out fix-partial-file-import instead of fix-json-import-keys. You end with a fresh sample-data session and the export file from setup step 8.
  2. Go to Settings → Data & reset → Import JSON…. Choose the export.
  3. Click OK on Imported successfully.
  4. Go to Transactions → Spending pill → Category rules. UBER TRIP → Personal is listed.
  5. Go to the Bills pill → Spotted recurring charges. SPOTIFY USA shows Dismissed.
  6. Press F5.
  7. Check Category rules and Spotted recurring charges again.
  8. Press F12 → Application → Local storage → the file:// entry.

What goes wrong:

  • After F5, Category rules says No rules yet., and SPOTIFY USA is back to Dismiss / Confirm as bill. (NETFLIX still shows ✓ Added to bills, because the bill itself was saved.)
  • Local storage has stray finlens_categoryRules and finlens_recurringConfirmed keys holding the imported values, which nothing reads.
Manual QA steps

Setup

  1. git checkout fix-json-import-keys.
  2. These tests use the sample data with no folder connected: that is where the bug showed. Do not connect your real folder. Use an InPrivate window (Ctrl+Shift+N) so nothing touches your normal browser data. InPrivate keeps its data across F5 until you close every InPrivate window.
  3. In the InPrivate window, open index.html.
  4. On the Welcome, click Try it with sample data. Opened from disk, FinLens asks you to choose the files: select the three CSVs in samples/. The yellow banner says You're trying FinLens with sample data. The date range is set to All time for you.

Make the backup file:

  1. Go to Transactions → Spending pill. Scroll to Category rules. In Description contains type UBER TRIP, set Category to Personal, click Add rule. The rule row UBER TRIP → Personal appears.
  2. Go to the Bills pill. Under Spotted recurring charges, click Dismiss on SPOTIFY USA. It shows Dismissed with an Undo link.
  3. On the NETFLIX row, click Confirm as bill. It shows ✓ Added to bills.
  4. Go to the gear icon → Settings → Data & reset → Export and import → Export JSON. Save the file (finlens-export-YYYYMMDD.json) somewhere like C:\temp.
  5. Close every InPrivate window. Open a new InPrivate window, open index.html, and load the sample data again (steps 3–4). The Category rules list says No rules yet., and SPOTIFY and NETFLIX show Dismiss / Confirm as bill.

1. Imported spotted-charge choices and category rules survive a reload (main fix)

  1. Go to Settings → Data & reset → Import JSON…. Choose the export from step 8.
  2. Click OK on Imported successfully.
  3. Go to Transactions → Spending pill → Category rules. Check the list.
  4. Go to the Bills pill → Spotted recurring charges. Check SPOTIFY and NETFLIX.
  5. Press F5.
  6. Repeat steps 3 and 4.

Expected:

  • Before and after the reload, Category rules lists UBER TRIP → Personal.
  • Before and after the reload, SPOTIFY USA shows Dismissed with Undo, and NETFLIX shows ✓ Added to bills.
  • F12 → Application → Local storage → the file:// entry has finlens_catRules and finlens_recurring holding the imported values. There is no finlens_categoryRules or finlens_recurringConfirmed key.

Before this fix: after F5 the rules list said No rules yet. and SPOTIFY went back to Dismiss / Confirm as bill. NETFLIX still showed ✓ Added to bills (that comes from the bill itself, which was saved). Local storage had the stray finlens_categoryRules and finlens_recurringConfirmed keys.

2. A null value in the file keeps the current value

  1. Copy the export file to export-null.json. Open it in Notepad.
  2. Change the "categoryRules": [ … ] value to "categoryRules": null (replace the whole array, brackets included). Save.
  3. Go to Settings → Data & reset → Import JSON… and choose export-null.json. Click OK.
  4. Go to Transactions → Spending pill.
  5. Press F5 and go to the Spending pill again.

Expected:

  • Imported successfully. (not Import failed).
  • The Spending screen renders, and Category rules still lists UBER TRIP → Personal.
  • F12 → Console shows no red error such as State.categoryRules is not iterable.

Before this fix: categoryRules became null, and categorizing threw a red error in the Console (for example on Re-categorize all or the next import).

3. Rules and choices made in the app still save (regression)

  1. Go to Transactions → Spending pill → Category rules. Add TARGET → Housing. Click Add rule.
  2. Delete the UBER TRIP rule with its × button.
  3. Go to the Bills pill. On SPOTIFY USA click Undo. On another spotted charge (for example T-MOBILE AUTOPAY) click Dismiss.
  4. Press F5.

Expected:

  • Category rules lists only TARGET → Housing.
  • SPOTIFY USA shows Dismiss / Confirm as bill; T-MOBILE AUTOPAY shows Dismissed.

4. Clear bills still forgets the spotted-charge choices

  1. Go to Settings → Data & reset → Clean up. The Clear bills row reads 1 bill. Also forgets which spotted recurring charges you confirmed or dismissed.
  2. Click Clear, then Click again to confirm.
  3. Press F5. Go to Transactions → Bills pill.

Expected:

  • Green callout Bills cleared.
  • After the reload, NETFLIX and T-MOBILE AUTOPAY both show Dismiss / Confirm as bill again, and no bill is listed.
  • Category rules on the Spending pill still lists TARGET → Housing (only bills were cleared).

5. Import with a folder connected still works (regression)

  1. Close every InPrivate window. Make a scratch folder, for example C:\temp\finlens-27, and copy the three samples/ CSVs into it.
  2. Open a new InPrivate window, open index.html, click Pick your CSV folder, choose the scratch folder and allow access.
  3. Go to Settings → Data & reset → Import JSON… and choose the export from setup step 8. Click OK.
  4. Wait 10 seconds, then press F5.
  5. Check Category rules (Spending pill) and Spotted recurring charges (Bills pill). Set Date range to All time first.

Expected:

  • UBER TRIP → Personal is listed, SPOTIFY USA is Dismissed, NETFLIX is ✓ Added to bills.
  • finlens-state.json in the scratch folder has "categoryRules" with the UBER TRIP rule and "recurringConfirmed" with SPOTIFY.

Please check before merging

  • Import a JSON export in your own browser, reload, and check Bills → Spotted recurring charges and Spending → Category rules.
  • You're fine with a null value in an import keeping the current value, rather than replacing it.
  • The DB._migrateSync() fix (it also copies cardDetails, sampleData and csvAutoImportPaused) is split out into v1.16.5: Fix a new browser database dropping the card details, the sample-data flag and the import pause #28, with its CHANGELOG entry, docs and test.
  • The migrate.js and views-cleanup.js changes are refactors with no change in behavior.
  • Version v1.16.4 everywhere: footer, README badge and footer, Settings → General, CHANGELOG, CLAUDE.md §1.
  • Manual QA passed in Edge/Chrome with a real folder

https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft


Generated by Claude Code

…16.4)

Browser storage keeps State.recurringConfirmed as 'recurring' and
State.categoryRules as 'catRules'. importJsonFile() saved every key
under its State name, so these two went to keys nothing reads and the
old values came back on the next reload.

- core.js: STORAGE_KEYS / storageKey(stateKey), the one State key ->
  storage key mapping. The storage names don't change.
- Import saves through storageKey(). A null value counts as missing
  and keeps the current value (a null categoryRules broke
  categorization).
- Data & reset plan clears and migrate.js go through storageKey(); no
  change in behavior.
- CHANGELOG, version in the four places, CLAUDE.md §1, §4.5, §8, §12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
…mport-keys

Carries master's v1.10.3-v1.10.5 up the stack: declined answers (#14),
literal single "~" (#15) and the Import Logs race fix (#18), with
clarity-insights' port of #14 to the rebuilt AI tab.

One conflict: CLAUDE.md's "Last updated" line. Kept this branch's
line and added the v1.10.3-v1.10.5 history. The CHANGELOG merged cleanly.
Versions stay 1.16.4. renderImportLog() keeps #18's guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
@gitluis gitluis added bug Something isn't working data loss Fixes a bug that loses or corrupts saved data labels Nov 3, 2026 — with Claude
@gitluis gitluis added this to the Clarity stack (v1.12.0 → v1.18.1) milestone Nov 3, 2026 — with Claude
…fix-json-import-keys

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBUZo6LRYpLthSp2PRi

# Conflicts:
#	js/views-settings.js
@gitluis gitluis changed the title v1.16.4: fix JSON import losing spotted-charge choices and category rules v1.16.4: Fix JSON import losing spotted-charge choices and category rules Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data loss Fixes a bug that loses or corrupts saved data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants