Skip to content

v1.16.25: Fix "Re-categorize all" changes being lost after a reload - #48

Open
gitluis wants to merge 2 commits into
fix-untransfer-categoryfrom
fix-recategorize-all-not-saved
Open

gitluis wants to merge 2 commits into
fix-untransfer-categoryfrom
fix-recategorize-all-not-saved

Conversation

@gitluis

@gitluis gitluis commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #47 (fix-untransfer-category). Merge order: #16, #17, #19, #20, #21, #24–#38, #39–#50, then #10, then #23.
Uses #24's recorded CSV category (_originalCategories). It doesn't need the fixes between #38 and this one.

Summary

Re-categorize all (Spending → Category rules) changed categories in memory only. After a reload, the imported categories came back. Now its result is saved and applied whenever transactions are built or loaded, so it survives a reload from IndexedDB, a boot from finlens-state.json and a CSV re-import.

The same function also overwrote two things it shouldn't have. With the result now saved, those would have become permanent, so they're fixed here too:

  • Card rows with no matching rule lost their CSV Category to the built-in rules (UBER TRIP: Travel → Transport).
  • Transfers lost Transfer in / Transfer out.

Cause

  • Nothing saved it. recategorizeAll() (ui.js) set t.categories on the loaded transactions and called renderActive(). No lsSet(), no state-file save, no IndexedDB write.
  • Category rules don't re-apply on load. Unlike flag rules since v1.16.8: Fix flags set by transaction rules disappearing after a reload #31 (txFlags()), they're applied only on import (normalizeRow()) and by this button. So a reload had nothing to rebuild the result from:
    • IndexedDB boot: DB.loadAll() → reconstituteTx() takes the category from the stored row, which holds the import's.
    • State-file boot: loadStateFromFile() takes the categories the file was saved with. The file got the new ones only by accident: when another change (adding a rule, a drawer save) had scheduled the 5-second debounced save and it ran after the click. Wait more than 5 seconds after the last change, click Re-categorize all, and nothing is written.
    • Re-import: mergeImported() replaces each loaded transaction with the file's fresh one, so card rows got their CSV Category back.
  • Card rows and transfers: it gave every transaction without an edit autoCategory(description). For a card row with no matching rule that's the built-in category, not its CSV Category. For a transfer leg it's Transfer (or a rule's category), not Transfer in / Transfer out.
  • master has the bug (v1.10.5): the same recategorizeAll(), also unsaved, also skipping only edits.

What changed

  • New State key ruleCategories ({ [txId]: category }): what Re-categorize all gave each transaction, only where it differs from the import's category (_originalCategories) or from its IndexedDB row's. The two can differ: after a rule is added and the CSVs imported again, memory re-reads a checking row with the rule, while the store keeps the row from its first import.
    • Saved like any setting: lsGet/lsSet (localStorage, IndexedDB app_state via DB.APP_STATE_KEYS) and saveStateToFile().
    • The click calls lsSet('ruleCategories', …), which also schedules the state-file save.
  • Applied when a transaction is built or loaded:
    • mkTx() (import): edit, then ruleCategories[id], then the import's category.
    • reconstituteTx(raw, ov, rules, ruleCats) (IndexedDB boot, state-file boot, Clear transaction edits, queryTransactions()): the same order. A record whose type is already transfer keeps its own category, because a state-file entry holds the detection's Transfer in / out and that boot doesn't run detectTransfers(). IndexedDB rows hold the CSV type, and DB.loadAll() detects transfers afterwards.
    • loadStateFromFile() passes the file's own ruleCategories (State only gets it after the function returns), as it already does for txRules.
  • recategorizeAll() (now async):
    • Each transaction gets categoryFromRules() (new, core.js): a category rule matching _originalDesc; else, for a format with its own Category column (the card export), the import's category; else the built-in rules.
    • Shown on every transaction except one with a category edit (the edit wins) and a detected transfer (keeps its transfer category). Saved for those too, so it shows once the edit is cleared or the transaction stops being a transfer.
    • Removing a rule, then Re-categorize all, puts its rows back to the import's category and removes their entries.
    • The import's category is _originalCategories. A transaction from a finlens-state.json written before v1.16.1 (every file master writes) doesn't record it: then its IndexedDB row's; with no row, the category it shows, when nothing but the import can have set it. It reads the rows once per click (DB._getAllTransactions()).
    • Entries for transactions not loaded are kept, as edits are.
  • AI Eval sandbox (views-eval.js) sets ruleCategories: {}, since mkTx() reads it (CLAUDE.md §21).
  • DB Schema view (views-db.js): the app_state note lists ruleCategories.
  • Not in the JSON export. It's worked out from the category rules, which are exported. Left out on purpose and listed with the other left-out keys.
  • Version v1.16.25 in CHANGELOG.md, index.html, README.md, js/views-settings.js and CLAUDE.md §1.
  • Docs: CLAUDE.md §4.1, §4.2 (new "Categories" paragraph), §8, §9, §20, "Last updated"; README Spending.

Why not write the result into the IndexedDB transactions rows, as one option was:

  • After a boot or restore from finlens-state.json, the store can be empty or partial (§4.5). Then there's no row to write.
  • Rebuild from your folder and Clear transactions & accounts empty the store.
  • A re-import skips rows already stored and puts the file's fresh transaction in memory, so the two would disagree.
  • A row is documented to hold the CSV's values (§10), and _originalCategories is read from it.

A map keyed by transaction id avoids all four, the same way txOverrides does for edits.

Not verified

  • A real browser with a real folder. The state-file paths use a fake folder handle (as the other suites do), not Chrome/Edge's File System Access API.
  • Data written by master. Section G removes _originalCategories from a payload to imitate it; I didn't load a real v1.10.5 database or state file.
  • Large data sets. Re-categorize all saves one map entry per changed transaction. Not timed beyond the samples (~260 rows).
  • Firefox and Safari. Not run.
  • Combined with the parallel transfer-detection changes (detectTransfers(), reapplyTransferKeywords()). I didn't touch either. A transaction that stops being a transfer during a session (a transfer keyword removed) shows whatever reapplyTransferKeywords() / undoTransferDetection() give it until the next reload, where its saved category applies.

Known limits (in the CHANGELOG):

  • A card row from a pre-v1.16.1 state file, with no IndexedDB row, that a rule re-categorized gets the built-in category if the rule is later removed: FinLens doesn't know its CSV Category. Re-importing its CSV and clicking Re-categorize all brings it back.
  • ruleCategories isn't in the JSON export (see above).

Found, not fixed (separate issues):

  • Card rows ignore category rules on import. normalizeRow() keeps a card row's CSV Category even when a rule matches, though the panel says "Rules apply to every import". Card rows imported after a Re-categorize all show their CSV Category until it's clicked again.
  • A rule added between two imports reaches re-read checking rows in memory only. Add WALMART → Shopping without clicking Re-categorize all, then import the same CSVs again: the 16 WALMART rows show Shopping, because mkTx() re-reads them with the rule. A reload from IndexedDB shows Groceries, because DB.importTransactions() skips rows it already has, so they keep the first import's category. Same on the base and on this branch (a scratch probe). Clicking Re-categorize all after the import now makes it stick (section H), but without the click the two still disagree.
Automated checks: earlier run, not re-run · bdd7386

Moved unchanged from this PR's "Test results" comment on 2026-11-04; not re-run, so not in the new table format. Originally a description section titled "How it was tested".

Light checks, by Luis's call to save usage: on this branch's final commit bdd7386 (chained on #47) only boot.js, race.js and this PR's own tests ran. Headless Chromium at 1280×900 against samples/, with the pinned CDN libraries served from npm; each script ran as NODE_PATH=$(npm root -g) node <script> <repo> <outdir>.

Script Result on bdd7386
boot.js 257 transactions, 24 views, 0 errors
race.js 24 views, 0 errors
test-recategorize.js (new in this PR) 55/55

Full set, once, on the top of the stack (#23, 819e809: every PR below it merged in): every base suite and every fix test from #24 to #50, plus #10's and #23's, each in its final variant. All pass: test-clarity.js 55/55, test-insights.js 137/137, test-finances.js 126/126, test-plan.js 127/127, boot.js 257 transactions and 0 errors, race.js 0 errors; #24–#38's tests (clear-edits 48/48, rebuild-folder 57/57, partial-import 47/47, json-import 28/28, new-browser-copy 10/10, json-export 31/31, plan-links 60/60, rule-flags 67/67, drawer-flags 36/36, start-balance 193/193, ai-summary-period 78/78, account-names 40/40, account-order 32/32, drawer-focus 167/167, cleanup 21/21); #39–#50's tests (drawer-fields 45/45, read-as 52/52, json-import-edits 29/29, overlapping-exports 46/46, override-balance 160/160, card-override 277/277, ai-summary-trend 91/91, transfer-user-type 39/39, untransfer-category 27/27, recategorize 55/55, retype-transfer 135/135, minor-fixes 34/34); test-dbtab.js 73/73, test-save-status.js 64/64, test-master-port.js 16/16. Three scripts needed a test-only change on that run, listed in #23 and in the PRs they belong to; the app code wasn't changed.

  • No phone checks (desktop only).

Measured before the chain

The numbers below are the PR author's, on the commit before it was chained (built on the old #38 tip ecfcab2, v1.16.15, or on the fix below it there; version numbers renumbered to the chain's).

Headless Chromium (Playwright) against samples/, desktop viewport, one suite at a time.

New: test-recategorize.js (sections A–H, 55 checks):

  • A. Add 3 rules in the panel (WALMART → Shopping on checking, STARBUCKS → Personal on the card, ONLINE TRANSFER → Housing on the 12 transfer legs). Edit one WALMART and one STARBUCKS row in the drawer. Click Re-categorize all. Then check: only the rule-matched rows change; card rows with no rule keep their CSV Category; transfers keep Transfer in / out; the edits win; the Transactions table matches State. Then reload from IndexedDB, re-import the same CSVs, and reload again.
  • B. Boot a new browser from the finlens-state.json A writes after its reload, reload, Re-categorize all again: no change.
  • C. Folder connected, booted from the file. Rules and edits saved, wait 6 s, then Re-categorize all and wait 6 s. Checks that a new file is written with the new categories, and that a reload booting from that file keeps them.
  • D. Remove the WALMART and STARBUCKS rules, then Re-categorize all: back to Groceries and Food & Drink, entries removed, edits kept, transfers unchanged; reload. Remove the last rule: nothing saved, every row as imported except the 2 edits.
  • E. No rules: Re-categorize all changes no category and saves nothing; reload.
  • F. Clear transaction edits after Re-categorize all: the edited row takes the rule's category; reload.
  • G. A state file without _originalCategories (the shape master writes): in a new browser (no rows), and with IndexedDB rows present (rule removed → back to the CSV Category from the rows).
  • H. Add rules, import the same CSVs again (memory re-reads WALMART as Shopping, the stored rows still say Groceries), Re-categorize all, reload from IndexedDB: Shopping and Personal kept. Then remove the rules, Re-categorize all, reload: every row as imported. A first version of this fix compared only with _originalCategories and failed this section's reload check (54/55); the row comparison was added for it.
Run ecfcab2 (base) this branch
test-recategorize.js 29/55 55/55

On the base, the failures are the bug itself: reloads, the state-file boot, the re-import and the debounced save lose the categories. Re-categorize all with no rules changes 136 sample rows. The ONLINE TRANSFER legs become Housing / Transfer.

Existing suites: run-tip.sh 15 <repo> <outdir> 1 (base suites plus every fix test of #24–#38):

Suite ecfcab2 (base) 1710fdd (this branch)
test-clarity.js 55/55 55/55
test-insights.js 137/137 137/137
test-finances.js 126/126 126/126
test-plan.js 127/127 127/127
test-clear-edits.js 47/47 47/47
test-rebuild-folder.js 57/57 57/57
test-partial-import.js 47/47 47/47
test-json-import.js 28/28 28/28
test-new-browser-copy.js 10/10 10/10
test-json-export.js 31/31 31/31
test-plan-links.js 60/60 60/60
test-rule-flags.js 67/67 67/67
test-drawer-flags.js 36/36 36/36
test-start-balance.js 193/193 193/193
test-ai-summary-period.js 78/78 78/78
test-account-names.js 40/40 40/40
test-account-order.js 32/32 32/32
test-drawer-focus.js 167/167 167/167
test-cleanup.js 21/21 21/21
boot.js 257 transactions, 24 views, 0 errors 257 transactions, 24 views, 0 errors
race.js 24 views, 7 errors 24 views, 6 errors
  • race.js: every error on both is the known renderImportLog one (views-data.js:263, "Cannot set properties of null"), from v1.10.5: fix Import Logs throwing when the view is left before it loads #18's fix not being in the stack yet. None comes from this change; the count varies run to run (6–8).
  • No existing suite was edited.
  • Command: NODE_PATH=$(npm root -g) node test-recategorize.js <repo> <outdir>; suites: run-tip.sh 15 <repo> <outdir> 1.
How to reproduce the bug

Run this on the base branch, fix-untransfer-category. Master has the bug too.

  1. Do the Setup from the testing steps below, but check out fix-untransfer-category instead of fix-recategorize-all-not-saved. Use the scratch folder with the samples only.
  2. On Spending → Category rules, with no rules, click Re-categorize all.
  3. On Transactions, search UBER TRIP, then STARBUCKS, then ONLINE TRANSFER.
  4. Back on Category rules, add WALMART → Shopping and STARBUCKS → Personal.
  5. Click Re-categorize all.
  6. On Transactions, search WALMART and STARBUCKS. They show Shopping and Personal.
  7. Press F5, wait for the data to load, and search WALMART and STARBUCKS again.

What goes wrong:

  • After step 2, with no rules at all, UBER TRIP shows Transport instead of Travel, STARBUCKS shows Food & Dining instead of Food & Drink, and the transfer legs show Transfer instead of Transfer out / Transfer in.
  • After the reload in step 7, WALMART is back to Groceries and STARBUCKS to Food & Drink. The Re-categorize all result was never saved.
Manual QA steps

Setup

  1. git checkout fix-recategorize-all-not-saved.
  2. Copy checking_sample.csv, credit_card_sample.csv and savings_sample.csv from samples/ into a new scratch folder (e.g. C:\temp\finlens-48). This PR changes what is saved, so never use your real folder.
  3. Open index.html in Edge.
  4. Go to Settings (gear) → Import → Pick a folder, choose the scratch folder, and allow read and write access.
  5. In the top bar, open the date range and pick All time.
  6. Go to Transactions. Click the Spending pill. Scroll to the Category rules panel at the bottom. It should say No rules yet.

Useful searches on Transactions (pill All, box Search description or notes): WALMART (16 checking rows, imported as Groceries), STARBUCKS (11 card rows, CSV Category Food & Drink), UBER TRIP (17 card rows, CSV Category Travel), ONLINE TRANSFER (12 rows: Transfer out on checking, Transfer in on savings).

1. Re-categorize all with no rules changes nothing

  1. On Spending → Category rules, with no rules, click Re-categorize all.
  2. Go to Transactions, search UBER TRIP.
  3. Search STARBUCKS.
  4. Search ONLINE TRANSFER.

Expected:

  • UBER TRIP rows still show Travel.
  • STARBUCKS rows still show Food & Drink.
  • Transfer legs still show Transfer out / Transfer in.

Before this fix: UBER TRIP turned into Transport, STARBUCKS into Food & Dining, and the transfer legs into Transfer.

2. Rules apply, and hold after a reload

  1. On Spending → Category rules, type WALMART in Description contains, choose Shopping in Category, click Add rule.
  2. Add a second rule: STARBUCKS → Personal.
  3. Click Re-categorize all.
  4. Go to Transactions, search WALMART, then STARBUCKS.
  5. Press F5. Wait for the data to load.
  6. Search WALMART and STARBUCKS again.

Expected:

  • Before and after F5: all 16 WALMART rows show Shopping, all 11 STARBUCKS rows show Personal.
  • UBER TRIP rows still show Travel.

Before this fix: after F5, WALMART went back to Groceries and STARBUCKS to Food & Drink.

3. Holds when booting from finlens-state.json (wait out the auto-save first)

  1. After test 2, wait at least 10 seconds so the pending auto-save has run.
  2. Remove the WALMART rule (the × next to it), add it again (WALMART → Shopping), then wait 10 seconds.
  3. Click Re-categorize all. Wait 10 seconds. Do nothing else.
  4. Open an InPrivate window in Edge and open index.html there (empty browser storage).
  5. Settings → Import → Pick a folder, choose the same scratch folder, allow access.
  6. Set the date range to All time. Search WALMART and STARBUCKS on Transactions.

Expected:

  • WALMART shows Shopping and STARBUCKS shows Personal in the InPrivate window, which only has the state file to go on.
  • Optional: open finlens-state.json in a text editor and find a "ruleCategories" object listing transaction ids.

Before this fix: a click with no other change pending wrote nothing, so the new window showed Groceries / Food & Drink.

Close the InPrivate window before continuing.

4. Holds after re-importing the CSVs

  1. Back in the normal window, go to Settings → Import and click Reload CSVs.
  2. Go to Transactions, search WALMART and STARBUCKS.

Expected:

  • WALMART still Shopping, STARBUCKS still Personal.

Before this fix: STARBUCKS (a card row) got its CSV Category Food & Drink back.

5. Transfers keep Transfer in / out even when a rule matches them

  1. On Spending → Category rules, add ONLINE TRANSFER → Housing.
  2. Click Re-categorize all.
  3. Go to Transactions, search ONLINE TRANSFER.
  4. Click the Transfers pill.
  5. Press F5 and check both again.

Expected:

  • All 12 legs still show Transfer out (checking) / Transfer in (savings), before and after F5.
  • They still appear on Transfers, and the Spending totals did not jump by the transfer amounts.

Before this fix: they became Housing.

6. A category set in the drawer still wins

  1. On Transactions, search WALMART and click one row to open the drawer.
  2. Set Category to Entertainment and click Save changes.
  3. Go to Spending → Category rules, click Re-categorize all.
  4. Back on Transactions, search WALMART.
  5. Press F5 and look again.

Expected:

  • The edited row shows Entertainment; the other 15 show Shopping. Same after F5.

7. Clear transaction edits: the edited row takes the rule's category

  1. Go to Settings → Data & reset.
  2. In Clean up, click Clear next to Clear transaction edits, then click again to confirm.
  3. Go to Transactions, search WALMART.
  4. Press F5 and look again.

Expected:

  • The row edited in test 6 now shows Shopping (the rule), not Groceries. Same after F5.

8. Removing rules puts rows back

  1. On Spending → Category rules, click × on the WALMART and STARBUCKS rules.
  2. Click Re-categorize all.
  3. Search WALMART, then STARBUCKS on Transactions.
  4. Press F5 and look again.
  5. Remove the ONLINE TRANSFER rule, click Re-categorize all, press F5.

Expected:

  • WALMART back to Groceries, STARBUCKS back to Food & Drink, before and after F5.
  • Transfer legs still Transfer out / Transfer in.
  • Optional, F12 → Application → IndexedDB → finlens_db → app_state → key ruleCategories: after the last step its value is {} (empty).

9. Regression: adding a rule alone still doesn't touch existing rows

  1. Add UBER → Transport. Do not click Re-categorize all.
  2. Search UBER TRIP on Transactions.
  3. Now click Re-categorize all and search again.

Expected:

  • Before the click: still Travel. After the click: Transport.
  • Remove the rule and click Re-categorize all: back to Travel.

Please check before merging

  • Re-categorize all after adding a rule: the categories hold after a reload, with and without a connected folder.
  • A card transaction no rule matches keeps its CSV Category (e.g. UBER TRIP stays Travel with no rules).
  • Transfers keep Transfer in / Transfer out after Re-categorize all, even with a rule that matches them.
  • A category you set in the drawer still wins.
  • Removing a rule, then Re-categorize all, puts its transactions back.
  • You're fine with a new State key (ruleCategories) rather than writing into the IndexedDB rows (reasons above).
  • You're fine with Clear transaction edits keeping the rules' categories (a re-categorized, edited row shows the rule's category after the clear), as flags follow their rules.
  • You're fine with leaving ruleCategories out of the JSON export.
  • Manual QA passed in Edge/Chrome with a real folder

https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft


Generated by Claude Code

recategorizeAll() changed categories in memory only and saved nothing.
Category rules don't re-apply on load, so a reload from IndexedDB took
the import's category from each stored row, and a boot from
finlens-state.json had the new categories only if another change had
scheduled the debounced save. A CSV re-import also brought card rows'
CSV categories back.

What Re-categorize all gives is now saved in State.ruleCategories
({ [txId]: category }), where it differs from the import's category or
from the stored row's: localStorage, IndexedDB app_state and the state
file. mkTx() and reconstituteTx() apply it after an edit, so it holds
after a reload from either place and after a re-import.

Saved, two things it did wrong would have become permanent, so they are
fixed with it: card rows with no matching rule now keep their CSV
Category (categoryFromRules()), and detected transfers keep
Transfer in / Transfer out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft
…fix-recategorize-all-not-saved

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Z8iBUZo6LRYpLthSp2PRi

# Conflicts:
#	js/views-settings.js
@gitluis gitluis changed the title v1.16.25: fix "Re-categorize all" changes being lost after a reload v1.16.25: Fix "Re-categorize all" changes being lost after a reload Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data loss Fixes a bug that loses or corrupts saved data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants