From 6fc8bda61e5d372eb68c914035df407903082183 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 2 Nov 2026 15:20:20 +0000 Subject: [PATCH] Fix overlapping CSV exports of one account counting the shared rows twice (v1.16.19) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two exports of one account that share dates, imported together (a May to mid-September and an August to October checking export in the folder, say), left the shared rows twice in memory and, through the autosave, in finlens-state.json, while IndexedDB held them once. On the samples split that way (23 shared rows): 280 transactions instead of 257. Every total counted the shared rows twice, the second copy of a transfer stayed a plain withdrawal or deposit (detection tracks legs by id), an edited row showed twice and a bucket listed a linked transaction twice. A reload kept it, since boot loads the state file first. A card is affected when both files have the same name (the fallback folder picker reads subfolders). Cause: both files give the shared rows the same ids, and loadFiles() / mergeImported() added every file's transactions, while DB.importTransactions() skips an id it already holds. master has it too: its import appends each file's rows (280 in memory, 257 in IndexedDB). - mergeImported() keeps one transaction per id, the first file's, which is the row IndexedDB stores. Same-day identical card rows keep their _2, _3: they're counted per file, so both files give them the same ids. - DB.importTransactions() takes every id of the import (importIds, passed by loadFiles() through importParsedCsv()) and never removes one as a leftover copy. Its check ran per file, so an earlier export read after a later one with one more same-day copy deleted that copy from the store. - loadStateFromFile() keeps one copy of an id a state file holds twice, and saves the file again. - Version bumped to v1.16.19 (CHANGELOG, footer, README, Settings, CLAUDE.md); CLAUDE.md §4.5, §8, §10, §19 updated. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Z8iBXzUjiQbYziDZ8CZft --- CHANGELOG.md | 18 ++++++++++++++++++ CLAUDE.md | 20 +++++++++++++++----- README.md | 4 ++-- index.html | 2 +- js/core.js | 34 ++++++++++++++++++++++++++++++---- js/db.js | 11 +++++++++-- js/views-settings.js | 2 +- 7 files changed, 76 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87a392b..72a8c9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,24 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). --- +## [1.16.19] — 2026-11-02 + +### Fixed +- **Two overlapping exports of one account, imported together, counted the rows they share twice.** + - **When:** the folder, or Choose CSV files, holds two exports of the same account that share dates, such as a May – mid-September and an August – October checking export. It happened on every import that reads them together: boot, Reload CSVs, Reconnect, Rebuild from your folder, Choose CSV files. Checking and savings always go into one account. A card does when both files have the same name: the folder picker browsers without the folder API use reads subfolders, so two `card.csv` files from monthly folders arrive together. + - **What it did:** on the samples, checking split into May 1 – Sep 15 and Aug 1 – Oct 20 (23 shared rows) gave 280 transactions instead of 257 in memory and `finlens-state.json`, while IndexedDB held 257. Every total counted the shared rows twice: net worth, cash flow, the Transactions KPIs and count. The second copy of a transfer stayed a plain withdrawal, deposit or card payment (detection skips an id it has already marked), so it counted as spending or income too. An edited row showed twice, and a bucket listed a linked transaction twice. A page reload kept it all, since boot loads `finlens-state.json` first; only a boot from IndexedDB didn't. + - **Cause:** both files give the shared rows the same ids (CLAUDE.md §10). `loadFiles()` collected every file's transactions, and `mergeImported()` added them all, while `DB.importTransactions()` skips an id it already holds. `master` has it too: its import appends each file's rows (280 in memory, 257 in IndexedDB on the same files). + - **Fix:** `mergeImported()` keeps one transaction per id, the first file's, which is the row IndexedDB stores. Same-day identical card rows keep their `_2`, `_3`: they're counted within each file, so both files give them the same ids, and an export with one more copy (a later export, made after another identical charge posted that day) adds only that one. + - **IndexedDB too:** its leftover-copy check ran file by file. With that later export read first, the earlier one then deleted the extra copy (`_3`) from the store, so memory had it and a boot from IndexedDB didn't. `DB.importTransactions()` now takes every id of the import (`importIds`, from `loadFiles()` through `importParsedCsv()`) and never removes one of them. + - **Already saved:** a `finlens-state.json` that holds a transaction twice is loaded with one copy of each (`loadStateFromFile()`), and saved again. + - **Unchanged:** importing the exports one after the other already worked, since the second replaces the shared rows. Last import still lists a later file's shared rows as "already imported". + - **Not covered:** two card exports with different names are two accounts (§18), so the rows they share have different ids and both stay. That's how FinLens tells cards apart; it isn't changed here. + +### Docs +- **CLAUDE.md:** §1 version; §4.5 one transaction per id; §8 `mergeImported()`, `importParsedCsv()`; §10 overlapping exports of one account; §19 `DB.importTransactions()` keeps the import's ids. + +--- + ## [1.16.18] — 2026-11-02 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 96feb5f..fc6ef83 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,7 +59,7 @@ Until then, this section is the rule. FinLens is a **single-origin, zero-build, vanilla JavaScript personal finance dashboard** that runs entirely in the browser. There is no server, no npm, no bundler, no framework, and no TypeScript. The entry point is `index.html`; everything else is a plain `.js` file in the `js/` folder. -**Current version:** v1.16.18 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release). +**Current version:** v1.16.19 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release). **What it does:** Imports CSV exports from a bank (checking + savings) and credit cards, parses and normalizes them into a unified transaction model, and renders a multi-tab financial dashboard with budgeting, debt tracking, savings forecasting, investment planning, and Claude-powered AI analysis. @@ -358,6 +358,7 @@ loadStateFromFile() → found? - **IndexedDB may not hold the loaded transactions.** After a boot or restore from `finlens-state.json` (a new browser, cleared site data, `_restoreStateIfBrowserEmpty()`), the `transactions` store can be empty or hold only some of them. Code that changes the loaded transactions works from `State.transactions`, never by replacing them with the store's rows: a store row may only fill in a value a transaction lacks. Up to v1.16.0, Clear transaction edits replaced them and wiped every transaction (§20). - **An import never drops loaded transactions** (v1.16.3). `loadFiles()` adds the files' transactions to the loaded ones (`mergeImported()`): the same id is replaced by the file's, every other loaded transaction stays, as `DB.importTransactions()` keeps every row (§10). Up to v1.16.2 it replaced all of them with the files' ones, so importing only the card file left only the card's in memory, and the autosave wrote that to the file that boot loads first. Removing rows is Rebuild from your folder's job (§20), with one exception, next. - **A file read into another account takes its rows with it** (v1.16.17). Changing a file's **Read as** changes the account, so its rows come back with new ids (§10). `loadFiles()` then removes the file's earlier rows from the old account: memory, IndexedDB and, through the autosave, the state file. Their edits and plan links move to the new ids. Up to v1.16.16 both copies stayed, until Rebuild. Rules in §10. +- **One transaction per id** (v1.16.19). Two exports of one account that share dates give the shared rows the same ids (§10). `mergeImported()` keeps one, the first file's, as IndexedDB keeps the first row it stores. Up to v1.16.18 memory kept both, and the autosave wrote both to the state file. `loadStateFromFile()` keeps one copy of an id the file holds twice. Anything that adds to `State.transactions` must keep ids unique: totals, transfer detection (it tracks legs by id) and links all assume it. - **Secrets never reach the file.** `saveStateToFile()` deliberately omits `apiKey`, and `loadStateFromFile()` strips it from files written by ≤v1.7.0 (moving it to browser storage if needed, then scheduling a re-save). Keep both behaviors when changing the serialized field list. --- @@ -612,10 +613,12 @@ loadFiles(files, assignments, { sample, quiet, beforeImport, unopened }?) // the // The files' transactions join the loaded ones, none dropped (mergeImported(), v1.16.3). // quiet: no alert, the caller reports; beforeImport(result): runs after the read, false stops it; // unopened: [{ name }] files the caller couldn't open. Returns { imported, reason, transactions, files } -parseCsvFile(file, kind) // reads one CSV → { accountId, rows, txs, problem, … } and writes nothing; importParsedCsv(p) imports it +parseCsvFile(file, kind) // reads one CSV → { accountId, rows, txs, problem, … } and writes nothing; importParsedCsv(p, importIds) imports it + // (importIds: every id of the import, which IndexedDB's leftover-copy check keeps, v1.16.19) importProblemText(res) // why a loadFiles() result's files gave no transactions, in words (importFileProblem(f) for one) reloadFromDirHandle(handle, opts?) // (views-data.js) the folder's CSVs → loadFiles(); opts as loadFiles()'s; returns its result -mergeImported(loaded, fresh) // loaded + the files' transactions by id, leftover copies out, kept transfers undetected (v1.16.3) +mergeImported(loaded, fresh) // loaded + the files' transactions by id, leftover copies out, kept transfers undetected (v1.16.3); + // one transaction per id, the first file's when two files give it (v1.16.19) undoTransferDetection(t, row?) // a detected transfer back to its CSV type and category, with its edit, before detectTransfers() accountFiles(a) // the CSV names an account came from: a.files, else [a.sourceFile] (saved before v1.16.17) earlierCopies(read) // (v1.16.17) loaded rows a re-read file gave under another account → Map(old id → { id, from, file }); reads only @@ -768,10 +771,16 @@ The account id comes last on purpose: `migrate.js` relies on it to rename accoun - Suffixes are allowed when `schema.allowOccurrenceSuffix` is true, or when it's unset and the schema has no `time` column (custom schemas). The checking and savings schemas set it to `false`: with a bank-assigned `Time`, an identical row is a true duplicate and only the first copy is kept. - **Never assign suffixes on a database collision.** Up to v1.8.0, `DB.importTransactions()` did, which re-inserted every row of a re-uploaded credit card file as an `_2` copy. -**Re-upload safety:** IDs are final when they reach `DB.importTransactions()`, which reads the stored keys once and skips any ID it already holds. Re-uploading a file — or a newer export with rows appended — inserts only genuinely new rows. On each import it also deletes leftover copies from the old bug: a suffixed row whose base transaction is in the file but whose exact ID is not. Same-day identical rows always appear together in a bank export, so the file is authoritative for how many copies exist. The import results card reports these as "leftover duplicate copies removed". +**Re-upload safety:** IDs are final when they reach `DB.importTransactions()`, which reads the stored keys once and skips any ID it already holds. Re-uploading a file — or a newer export with rows appended — inserts only genuinely new rows. On each import it also deletes leftover copies from the old bug: a suffixed row whose base transaction is in the file but whose exact ID is not (and that no other file of the import gives, v1.16.19, below). Same-day identical rows always appear together in a bank export, so the file is authoritative for how many copies exist. The import results card reports these as "leftover duplicate copies removed". **Memory follows the same rules** (v1.16.3). `loadFiles()` puts the files' transactions into `State.transactions` the way `DB.importTransactions()` puts them into the store (`mergeImported()`): an id it already has is replaced by the file's, a new one is added, a leftover copy as above is removed, and every other loaded transaction stays. So importing one file, or a newer export that starts later, keeps the other accounts and the older rows, and `finlens-state.json` gets them all. +**Overlapping exports of one account** (v1.16.19). Two exports that share dates (May – mid-September and August – October, imported together) give the shared rows the same ids, `_2`, `_3` included, since copies are counted within each file. Checking and savings always share an account; a card only when both files have the same name (§18), as two `card.csv` files from subfolders through the fallback folder picker. +- **Memory:** `mergeImported()` keeps one transaction per id, the first file's. IndexedDB matches: the later files skip the ids the first one stored. +- **Leftover copies:** `DB.importTransactions(txs, accountId, schemaId, importIds)` never removes an id another file of the import gives (`importIds`, every id of the import, from `loadFiles()`). An export made after one more identical charge posted that day has one more copy; it stays, whichever file is read first. +- **Up to v1.16.18** memory kept both copies, the autosave wrote them to `finlens-state.json`, and every total counted the shared rows twice. `loadStateFromFile()` keeps one copy of an id the file holds twice, and saves the file again. +- **Card exports with different names** are different accounts, so the rows they share have different ids and both stay. + **"Read as" changes the ids** (v1.16.17). A file read as another kind than before (Settings → Import → Files in the folder, or the Choose CSV files card) goes into another account, so every row gets a new id. Before importing, `earlierCopies()` finds the file's earlier rows; after, `moveEarlierCopies()` takes them out. - **Where it looks:** only accounts the file went into before: `accountFiles(a)` lists it, or, for an account saved before v1.16.17 (no `files`), its `sourceFile` is the file or the import log says the file went into it. Without this, re-importing one file would take another account's row that happens to match exactly (checking and savings can both post the same fee at the same second). - **What it finds:** for each of the file's rows, the id it had in that account, from its new id (`IdMigration._rekey()`, §22). That keeps the time or post date, so it's exact; a match on date, description and amount would mix up two same-day rows of the same amount. A loaded row with that id goes, unless a file in this import gives that id too. @@ -1070,7 +1079,7 @@ Multiple credit card files can be loaded simultaneously. Each gets a unique `acc --- -*Last updated: v1.16.18 — 2026-11-02 (a JSON import's transaction edits show at once: §9 `applyImportedEdits()`; §20 what takes effect without a reload; v1.16.17: a file read into another account takes its rows with it: §4.3 the account's `files`, `accountFiles()`; §4.5 the exception to an import never dropping rows; §8 `accountFiles()`, `earlierCopies()`, `moveEarlierCopies()`; §10 Read as changes the ids; §19 `DB.removeTransactions()`, `DB.patchAccount()`; §22 `_rekey()` is also used by imports; v1.16.16: the transaction and transfer drawers store only the fields you change: §4.1 TxOverride, §11 transfer keywords, §12 state mutations; v1.16.15: §1 desktop only; §3 `views-database.js` removed, the Changelog view reads CHANGELOG.md; §6, §18, §19 point to the current screens; v1.16.14: §9 the drawer is a modal dialog: inert page, Tab wraps, first field focused, focus returns after a re-render; v1.16.13: account order: §4.1 `accountOrder` keeps ids not loaded; §8 `orderedAccounts()`; §18 display order; v1.16.12: account names: §4.1 `accountLabels`; §4.3 an account's `name` is its default; §8 `defaultAccountName()`; §9 `acctName()`; §13 the summary lists accounts by `acctName()`; §18 names you type, and clearing them; v1.16.11: AI summary balances at the end of the selected period: §13 `summaryBalanceEnd()` and the date notes; §21 the parser reads the balance lines by how they start; v1.16.10: starting balance counted in every range: §8 `startingBalance()`, `currentBalances()`; §13 summary balances; §18 one balance rule; v1.16.9: the transaction drawer stores only the flags you change: §4.2 flags, §12 state mutations; v1.16.8: flags from transaction rules are worked out on load, never read from storage: §4.2 flags, §8 `txRuleFlag()` / `txFlags()` / `applyTxRulesToAll()`, §19 flags in the transactions store; v1.16.7: transaction links to budgets, buckets and debts by id: §1 version; §3 migrate.js role; §4.1 the `id` fields, TxOverride and its `link` shape; §8 `PLAN_LINK_TYPES`, `newPlanId()`; §9 `planItem()`, `linkedTransactions()`, `unlinkPlanItems()`, `ensurePlanIds()`; §20 clearing budgets, buckets or debts removes their links; §22 the plan-link migration; v1.16.6: JSON export carries account names and order, Read as, flag rules, transfer keywords and theme: §9 `JSON_EXPORT_KEYS`; §12 adding a persisted field; §20 what the export holds; v1.16.5: a new database copies every saved setting: §4.5 the keys that go through `storageKey()`; §12 adding a persisted field; §19 `DB.APP_STATE_KEYS`; v1.16.4: JSON import saves under the storage keys: §4.5 `STORAGE_KEYS` / `storageKey()`; §8 `storageKey()`; §12 adding a persisted field uses the State name; v1.16.3: An import keeps the loaded transactions it doesn't replace: §4.5 an import never drops loaded transactions; §8 `mergeImported()`, `undoTransferDetection()`; §10 memory follows the same re-upload rules; §11 kept transfers are detected again; v1.16.2: Imports read every file before changing anything: §8 `loadFiles()` options and result, `parseCsvFile()` reads only, `importParsedCsv()`, `importProblemText()`, `reloadFromDirHandle()`; §20 Rebuild from your folder reads before it clears, read before clearing; v1.16.1: Clear transaction edits works from memory: §4.2 `_originalCategories`, `_originalType` and what a transaction keeps of its row; §4.5 IndexedDB may not hold the loaded transactions; §10 a stored row holds the CSV's values; §20 Clear transaction edits; v1.16.0: Trends, AI Analysis, Settings and the Welcome rebuilt: §3 file roles; §4.1 `cardDetails`, `sampleData`, AI history `model`; §6 tile tokens and Insights/Settings/Welcome components; §7 rule 7 covers Trends; §8 `loadFiles()` sample option; §9 `shellHero()`, `NO_RANGE_VIEWS`, ☰ → ×, JSON export/import buttons; §12 Settings tabs renamed, the Welcome, range label hidden on Trends and Settings; §13 quick analyses, tab state, history; §20 new fields, Use my own files, Export/Import JSON; §21 quick analyses send `AI_PROMPTS`; v1.15.0: Plan screens rebuilt: §3 file roles; §4.1 `debtPlan` and the Budget / Bucket / Debt / Scenario shapes; §6 `--teal-text` and Plan components; §7 rule 7 covers the Plan charts; §8 plan helpers (`payoffPlan()`, `growthByYear()` …); §9 plan form helpers, `simulatePayoff()` kept unused; §12 Plan titles, Forecast tabs, range label hidden on Plan screens; §13 debt balances in the summary; §20 Clear forecast scenarios / investment mix, Clear debts resets `debtPlan`; v1.14.0: Finances screens rebuilt: §4.1 Bill shape and `recurringConfirmed`; §6 new tokens and Finances components; §7 rule 7 inline SVG charts; §8 new aggregation helpers; §9 drawer, `TX_VIEWS`, SVG/heatmap/format helpers, bill helpers, removed renderers; §11 keywords re-run detection, `transferPairs()`, card payments; §12 page actions and Transactions pills; §20 Clear bills also clears spotted-charge choices; v1.13.0: Clarity redesign — §6 tokens and components; §9 theme, chart colors and app bar helpers; §12 app bar navigation replaces the sidebar and the emoji rule; §17 footer; §20 Data & Reset; v1.12.0: bank-neutral account codes, names, samples and docs; §22 `migrate.js` added; §3, §4, §5, §10, §18, §21 updated; v1.10.5: no section changes: Import Logs race fix; v1.10.4: §8, §13 single `~` is literal; §21 excerpts escape `~`; v1.10.3: §13 refusals; §21 declined runs; v1.10.2: §13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)* +*Last updated: v1.16.19 — 2026-11-02 (overlapping exports of one account count each shared row once: §4.5 one transaction per id; §8 `mergeImported()`, `importParsedCsv()`; §10 overlapping exports of one account; §19 the leftover-copy check keeps the import's ids; v1.16.18: a JSON import's transaction edits show at once: §9 `applyImportedEdits()`; §20 what takes effect without a reload; v1.16.17: a file read into another account takes its rows with it: §4.3 the account's `files`, `accountFiles()`; §4.5 the exception to an import never dropping rows; §8 `accountFiles()`, `earlierCopies()`, `moveEarlierCopies()`; §10 Read as changes the ids; §19 `DB.removeTransactions()`, `DB.patchAccount()`; §22 `_rekey()` is also used by imports; v1.16.16: the transaction and transfer drawers store only the fields you change: §4.1 TxOverride, §11 transfer keywords, §12 state mutations; v1.16.15: §1 desktop only; §3 `views-database.js` removed, the Changelog view reads CHANGELOG.md; §6, §18, §19 point to the current screens; v1.16.14: §9 the drawer is a modal dialog: inert page, Tab wraps, first field focused, focus returns after a re-render; v1.16.13: account order: §4.1 `accountOrder` keeps ids not loaded; §8 `orderedAccounts()`; §18 display order; v1.16.12: account names: §4.1 `accountLabels`; §4.3 an account's `name` is its default; §8 `defaultAccountName()`; §9 `acctName()`; §13 the summary lists accounts by `acctName()`; §18 names you type, and clearing them; v1.16.11: AI summary balances at the end of the selected period: §13 `summaryBalanceEnd()` and the date notes; §21 the parser reads the balance lines by how they start; v1.16.10: starting balance counted in every range: §8 `startingBalance()`, `currentBalances()`; §13 summary balances; §18 one balance rule; v1.16.9: the transaction drawer stores only the flags you change: §4.2 flags, §12 state mutations; v1.16.8: flags from transaction rules are worked out on load, never read from storage: §4.2 flags, §8 `txRuleFlag()` / `txFlags()` / `applyTxRulesToAll()`, §19 flags in the transactions store; v1.16.7: transaction links to budgets, buckets and debts by id: §1 version; §3 migrate.js role; §4.1 the `id` fields, TxOverride and its `link` shape; §8 `PLAN_LINK_TYPES`, `newPlanId()`; §9 `planItem()`, `linkedTransactions()`, `unlinkPlanItems()`, `ensurePlanIds()`; §20 clearing budgets, buckets or debts removes their links; §22 the plan-link migration; v1.16.6: JSON export carries account names and order, Read as, flag rules, transfer keywords and theme: §9 `JSON_EXPORT_KEYS`; §12 adding a persisted field; §20 what the export holds; v1.16.5: a new database copies every saved setting: §4.5 the keys that go through `storageKey()`; §12 adding a persisted field; §19 `DB.APP_STATE_KEYS`; v1.16.4: JSON import saves under the storage keys: §4.5 `STORAGE_KEYS` / `storageKey()`; §8 `storageKey()`; §12 adding a persisted field uses the State name; v1.16.3: An import keeps the loaded transactions it doesn't replace: §4.5 an import never drops loaded transactions; §8 `mergeImported()`, `undoTransferDetection()`; §10 memory follows the same re-upload rules; §11 kept transfers are detected again; v1.16.2: Imports read every file before changing anything: §8 `loadFiles()` options and result, `parseCsvFile()` reads only, `importParsedCsv()`, `importProblemText()`, `reloadFromDirHandle()`; §20 Rebuild from your folder reads before it clears, read before clearing; v1.16.1: Clear transaction edits works from memory: §4.2 `_originalCategories`, `_originalType` and what a transaction keeps of its row; §4.5 IndexedDB may not hold the loaded transactions; §10 a stored row holds the CSV's values; §20 Clear transaction edits; v1.16.0: Trends, AI Analysis, Settings and the Welcome rebuilt: §3 file roles; §4.1 `cardDetails`, `sampleData`, AI history `model`; §6 tile tokens and Insights/Settings/Welcome components; §7 rule 7 covers Trends; §8 `loadFiles()` sample option; §9 `shellHero()`, `NO_RANGE_VIEWS`, ☰ → ×, JSON export/import buttons; §12 Settings tabs renamed, the Welcome, range label hidden on Trends and Settings; §13 quick analyses, tab state, history; §20 new fields, Use my own files, Export/Import JSON; §21 quick analyses send `AI_PROMPTS`; v1.15.0: Plan screens rebuilt: §3 file roles; §4.1 `debtPlan` and the Budget / Bucket / Debt / Scenario shapes; §6 `--teal-text` and Plan components; §7 rule 7 covers the Plan charts; §8 plan helpers (`payoffPlan()`, `growthByYear()` …); §9 plan form helpers, `simulatePayoff()` kept unused; §12 Plan titles, Forecast tabs, range label hidden on Plan screens; §13 debt balances in the summary; §20 Clear forecast scenarios / investment mix, Clear debts resets `debtPlan`; v1.14.0: Finances screens rebuilt: §4.1 Bill shape and `recurringConfirmed`; §6 new tokens and Finances components; §7 rule 7 inline SVG charts; §8 new aggregation helpers; §9 drawer, `TX_VIEWS`, SVG/heatmap/format helpers, bill helpers, removed renderers; §11 keywords re-run detection, `transferPairs()`, card payments; §12 page actions and Transactions pills; §20 Clear bills also clears spotted-charge choices; v1.13.0: Clarity redesign — §6 tokens and components; §9 theme, chart colors and app bar helpers; §12 app bar navigation replaces the sidebar and the emoji rule; §17 footer; §20 Data & Reset; v1.12.0: bank-neutral account codes, names, samples and docs; §22 `migrate.js` added; §3, §4, §5, §10, §18, §21 updated; v1.10.5: no section changes: Import Logs race fix; v1.10.4: §8, §13 single `~` is literal; §21 excerpts escape `~`; v1.10.3: §13 refusals; §21 declined runs; v1.10.2: §13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)* --- @@ -1088,6 +1097,7 @@ The full IndexedDB implementation is live. See **`docs/archive/INDEXEDDB_PLAN.md - **Transfer Keywords** — `State.transferKeywords` is edited on the Transfers screen (§11). `detectTransfers()` builds its regex from this list. - **Flags in the `transactions` store (v1.16.8)** — `DB.importTransactions()` still writes `isRecurring`, `isBill` and `isSubscription` into each new row, as they are at import, but FinLens never reads them back (only the developer DB Inspector shows them). Rows already stored are skipped on re-import, so they are never updated. `reconstituteTx()` works the flags out from `txOverrides` and the current rules (§4.2). - **A file read into another account (v1.16.17, §10)** is one of the few times rows leave the store (with Data & reset's clears, §20; an import's leftover copies, §10; the account-code migration, §22). `DB.removeTransactions([{ id, account }])` deletes the old rows and lowers each account's `txCount` (its date range stays). `DB.patchAccount(id, patch | null)` updates or deletes an account record, and never creates one. +- **The leftover-copy check keeps the import's ids (v1.16.19, §10).** `DB.importTransactions()` runs once per file; its 4th argument, `importIds`, lists every id the import's files give, and no row with one of them is deleted as a leftover copy. Up to v1.16.18 a file with fewer same-day copies than another file of the same import, read after it, deleted the extra copy. ### Adding a new bank Add a new object to `BANK_SCHEMAS` in `core.js`. No other code changes needed if the bank follows standard column conventions (`date`, `amount`, `description`). For non-standard conventions use: `amountSign: 'inverted'`, custom `columns` map, `dateFormat` (`'YYYY-MM-DD'` | `'MM/DD/YYYY'` | `'DD/MM/YYYY'`), and `postDateFormat` if the post-date column uses a different format than the primary date column (see the `credit_card` schema). diff --git a/README.md b/README.md index 9445559..b3e0e77 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > A zero-install, browser-native finance dashboard. Drop in your bank CSVs, get instant clarity on your money. -**Version:** v1.16.18  ·  **Stack:** Vanilla JS, HTML, CSS  ·  **Install:** None +**Version:** v1.16.19  ·  **Stack:** Vanilla JS, HTML, CSS  ·  **Install:** None --- @@ -263,4 +263,4 @@ See [CHANGELOG.md](CHANGELOG.md) for the full release history. --- -*FinLens v1.16.18 — Built for one person's finances, designed to be useful for anyone's.* +*FinLens v1.16.19 — Built for one person's finances, designed to be useful for anyone's.* diff --git a/index.html b/index.html index 199723b..a957b94 100644 --- a/index.html +++ b/index.html @@ -1202,7 +1202,7 @@
-
v1.16.18 · FinLens
+
v1.16.19 · FinLens
diff --git a/js/core.js b/js/core.js index 24474e8..8ec7d0c 100644 --- a/js/core.js +++ b/js/core.js @@ -170,6 +170,16 @@ async function loadStateFromFile() { // getter into a plain value. Rebuild them exactly as DB.loadAll() does, or // monthKey()/date comparisons throw or misbehave (Income, Trends, AI summary). if (Array.isArray(loaded.transactions)) { + // One transaction per id (v1.16.19). Up to v1.16.18 two overlapping exports of one + // account imported together saved the rows they share twice (mergeImported()). One id + // is one CSV row in one account (CLAUDE.md §10), so the first copy is kept, and the file + // is saved again without the others. + const seen = new Set(), all = loaded.transactions.length; + loaded.transactions = loaded.transactions.filter(raw => { if (seen.has(raw.id)) return false; seen.add(raw.id); return true; }); + if (loaded.transactions.length < all) { + console.log(`[FinLens] finlens-state.json held ${all - loaded.transactions.length} transaction${all - loaded.transactions.length === 1 ? '' : 's'} twice (the same id); one copy of each is kept`); + _saveStateDebounced(State); + } const ovs = loaded.txOverrides || State.txOverrides || {}; // Rule flags come from the file's own rules: State only gets them after this returns. const rules = Array.isArray(loaded.txRules) ? loaded.txRules : State.txRules; @@ -787,7 +797,9 @@ function parseCsvFile(file, forcedKind) { // logs the import and adds it to State.fileMeta. Returns the file's transactions; // loadFiles() puts them in State.transactions. A failing database write is logged and the // import carries on, so the transactions still reach memory and finlens-state.json. -async function importParsedCsv(p) { +// importIds: every id the files of this import give (loadFiles()); IndexedDB keeps them +// when it removes leftover copies (DB.importTransactions(), v1.16.19). +async function importParsedCsv(p, importIds) { const { file, schema, kind, accountId, accountName, txs } = p; // Initialise account in State (in-memory). Its name is always the default: a name // you typed stays in State.accountLabels, and acctName() shows that first. (Older @@ -822,7 +834,7 @@ async function importParsedCsv(p) { initialBalance: State.manualBalances[accountId+'_initial'] != null ? Number(State.manualBalances[accountId+'_initial']) : null, }); - importResult = await DB.importTransactions(txs, accountId, schema.id); + importResult = await DB.importTransactions(txs, accountId, schema.id, importIds); await DB.logImport({ filename: file.name, @@ -927,10 +939,14 @@ async function loadFiles(fileList, assignments, opts = {}) { // account, so with other ids. Its earlier rows are found before anything is written // (v1.16.17), then leave their old account below. const moves = await earlierCopies(read); + // Every id the files of this import give. Two overlapping exports of one account give the + // rows they share with the same id (§10): IndexedDB's leftover-copy check for one file must + // not delete a row another file of this import gives (v1.16.19). + const importIds = new Set(read.filter(p => p.schema && !p.problem).flatMap(p => p.txs.map(t => t.id))); const fresh = []; for (const p of read) { if (!p.schema || p.problem) continue; // a file that gave no transactions creates no account - fresh.push(...await importParsedCsv(p)); + fresh.push(...await importParsedCsv(p, importIds)); } if (moves.size) await moveEarlierCopies(moves, read, fresh); // The files' transactions join the loaded ones; none of the others is dropped (v1.16.3). @@ -973,11 +989,21 @@ async function loadFiles(fileList, assignments, opts = {}) { // id whose base is in the files but whose exact id isn't. // - A kept transfer leg goes back to its values before detection (undoTransferDetection()), // so loadFiles()'s detectTransfers() can pair it again, with a row just re-read too. +// - One transaction per id (v1.16.19). Two overlapping exports of one account (a September +// export and a September–October one, read together) give the rows they share with the +// same id: the id hashes the row and the account (CLAUDE.md §10), and a file's _2, _3 … +// are counted in that file alone, so its same-day copies get the same ids in both files. +// The first file's copy stays, as in IndexedDB, where the later files skip that id. Up to +// v1.16.18 both copies were kept, so the shared rows counted twice in every total, and the +// autosave wrote them twice to finlens-state.json. A loaded id held twice (a state file +// saved then) keeps one copy too. async function mergeImported(loaded, fresh) { const baseOf = (id) => id.replace(/_\d+$/, ''); + const once = (list) => { const seen = new Set(); return list.filter(t => !seen.has(t.id) && !!seen.add(t.id)); }; + fresh = once(fresh); const ids = new Set(fresh.map(t => t.id)); const bases = new Set(fresh.map(t => baseOf(t.id))); - const kept = loaded.filter(t => !ids.has(t.id) && !(t.id !== baseOf(t.id) && bases.has(baseOf(t.id)))); + const kept = once(loaded.filter(t => !ids.has(t.id) && !(t.id !== baseOf(t.id) && bases.has(baseOf(t.id))))); // A transfer from a ≤v1.16.0 finlens-state.json doesn't record its CSV type and // category; its IndexedDB row does, when this browser has one. const known = (t) => t._originalType && Array.isArray(t._originalCategories) && t._originalCategories.length; diff --git a/js/db.js b/js/db.js index 87bb556..cc53bc1 100644 --- a/js/db.js +++ b/js/db.js @@ -192,9 +192,16 @@ const DB = { * such row). Same-day identical rows always appear together in a bank export, so the * file is authoritative for how many copies of each of its transactions exist. * + * importIds (optional, v1.16.19): every ID the files of this import give (loadFiles()). + * A row with one of them is never a leftover copy, so with two overlapping exports of one + * account read together, the one with fewer copies of a same-day row (exported before the + * last one posted) doesn't delete the copy the other file just added. Up to v1.16.18 it + * did when that file came second, and the browser database then held one row fewer than + * memory and finlens-state.json. + * * Returns { inserted, skipped, removed, dateRangeMin, dateRangeMax, gapWarning }. */ - async importTransactions(txArray, accountId, schemaId) { + async importTransactions(txArray, accountId, schemaId, importIds) { if (!this._db || !txArray.length) { return { inserted: 0, skipped: 0, removed: 0, dateRangeMin: null, dateRangeMax: null, gapWarning: null }; } @@ -283,7 +290,7 @@ const DB = { const batchIds = new Set(sorted.map(t => t.id)); const batchBases = new Set(sorted.map(t => baseOf(t.id))); const stale = [...existingIds].filter(id => - id !== baseOf(id) && !batchIds.has(id) && batchBases.has(baseOf(id))); + id !== baseOf(id) && !batchIds.has(id) && !(importIds && importIds.has(id)) && batchBases.has(baseOf(id))); if (stale.length) { removed = await new Promise((res) => { const t = this._db.transaction('transactions', 'readwrite'); diff --git a/js/views-settings.js b/js/views-settings.js index fbd7a40..ba4e7ad 100644 --- a/js/views-settings.js +++ b/js/views-settings.js @@ -67,7 +67,7 @@ function _renderGeneralSettings(root) { ``).join('')} -
VersionFinLens v1.16.18
+
VersionFinLens v1.16.19
`); const opts=$$('[data-theme-opt]',root); opts.forEach((b,i)=>{