diff --git a/CHANGELOG.md b/CHANGELOG.md index adde633..0dae7fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,27 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). --- +## [1.16.2] — 2026-11-02 + +### Fixed +- **Rebuild from your folder cleared everything when the folder had no CSV FinLens could read.** + - **When:** Settings → Data & reset → Rebuild from your folder, with a connected folder that holds no CSV (only `finlens-state.json`), only CSVs FinLens can't read, or a CSV the browser can't open. + - **Cause:** `DataCleanup.rebuildFromFolder()` cleared the transactions, accounts and import log (memory and IndexedDB) before reading the folder. With nothing to import, 0 transactions were left, and `_finish()` wrote that to `finlens-state.json`, which boot loads first. Two related cases: with one CSV unreadable among several, the rebuild went ahead without it, so that account's transactions were gone; a CSV the browser couldn't open stopped the rebuild after the clear ("Something went wrong"). The bug was already on `master`. + - **Fix:** the rebuild reads and checks every CSV first. It clears and re-imports only when at least one transaction was read and every file it would import could be read (`_rebuildBlockers()`). Otherwise nothing changes, and the tab says why: the folder has no CSV, or which file couldn't be read and why. A CSV in a format FinLens doesn't read, and never imported, is skipped as every import skips it, and the message names it. +- **Reload CSVs, picking or reconnecting a folder, and Choose CSV files emptied the loaded transactions when no file could be read.** + - **Cause:** `loadFiles()` emptied `State.transactions` before parsing. When no file gave a transaction, none were left in memory, and the autosave wrote that to `finlens-state.json`. IndexedDB still held them, but the state file wins at boot. A folder with no CSV at all was already safe: Reload stopped with an alert. The bug was already on `master`. + - **Fix:** `loadFiles()` reads every file first and changes nothing unless at least one transaction was read: not memory, browser storage, IndexedDB, the sample data or the import pause. `parseCsvFile()` now only reads; `importParsedCsv()` imports. An alert names each file and why. Chosen files stay listed, so their Read as can be changed and tried again. +- **Smaller fixes in the same code:** + - **A CSV the browser can't open** (another program has it locked) is reported with the others. Before, it stopped the whole reload. + - **A file with no readable row** no longer creates an empty account. + - **A failing IndexedDB write during an import** is logged and the import carries on, so the transactions still reach memory and the state file. Before, that file's import never finished, and Rebuild stayed on "Working…". + - **Settings → Import → Last import** lists the files that weren't imported, with the reason. After Reload CSVs it now shows that reload right away; before, it showed the previous import until the next render. + +### Docs +- **CLAUDE.md:** §8 `loadFiles()` options and result, `parseCsvFile()`, `importParsedCsv()`, `importProblemText()`; §20 Rebuild from your folder reads before it clears, and an import that reads nothing changes nothing. + +--- + ## [1.16.1] — 2026-11-02 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 15d387d..82461e3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,7 +59,7 @@ Until then, this section is the rule. FinLens is a **single-origin, zero-build, vanilla JavaScript personal finance dashboard** that runs entirely in the browser. There is no server, no npm, no bundler, no framework, and no TypeScript. The entry point is `index.html`; everything else is a plain `.js` file in the `js/` folder. -**Current version:** v1.16.1 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release). +**Current version:** v1.16.2 (updated in `index.html` footer, `README.md`, and `CHANGELOG.md` on every release). **What it does:** Imports CSV exports from a bank (checking + savings) and credit cards, parses and normalizes them into a unified transaction model, and renders a multi-tab financial dashboard with budgeting, debt tracking, savings forecasting, investment planning, and Claude-powered AI analysis. @@ -563,7 +563,14 @@ debounce(fn, ms) // Returns debounced function parseDate(s) // Parse date string/object → Date | null monthKey(d) // 'YYYY-MM' string describeRange() // Human-readable date range label -loadFiles(files, assignments, { sample }?) // the import; without `sample`, it first clears sample data (State.sampleData) so your files replace it +loadFiles(files, assignments, { sample, quiet, beforeImport, unopened }?) // the import. Reads every file first and changes nothing + // (memory, storage, IndexedDB, sample data, the import pause) unless a transaction was read (v1.16.2). + // Then: without `sample`, it clears sample data (State.sampleData) so your files replace it. + // quiet: no alert, the caller reports; beforeImport(result): runs after the read, false stops it; + // unopened: [{ name }] files the caller couldn't open. Returns { imported, reason, transactions, files } +parseCsvFile(file, kind) // reads one CSV → { accountId, rows, txs, problem, … } and writes nothing; importParsedCsv(p) imports it +importProblemText(res) // why a loadFiles() result's files gave no transactions, in words (importFileProblem(f) for one) +reloadFromDirHandle(handle, opts?) // (views-data.js) the folder's CSVs → loadFiles(); opts as loadFiles()'s; returns its result ``` --- @@ -971,7 +978,7 @@ Multiple credit card files can be loaded simultaneously. Each gets a unique `acc --- -*Last updated: v1.16.1 — 2026-11-02 (Clear transaction edits works from memory: §4.2 `_originalCategories`, `_originalType` and what a transaction keeps of its row; §4.5 IndexedDB may not hold the loaded transactions; §10 a stored row holds the CSV's values; §20 Clear transaction edits; v1.16.0: Trends, AI Analysis, Settings and the Welcome rebuilt: §3 file roles; §4.1 `cardDetails`, `sampleData`, AI history `model`; §6 tile tokens and Insights/Settings/Welcome components; §7 rule 7 covers Trends; §8 `loadFiles()` sample option; §9 `shellHero()`, `NO_RANGE_VIEWS`, ☰ → ×, JSON export/import buttons; §12 Settings tabs renamed, the Welcome, range label hidden on Trends and Settings; §13 quick analyses, tab state, history; §20 new fields, Use my own files, Export/Import JSON; §21 quick analyses send `AI_PROMPTS`; v1.15.0: Plan screens rebuilt: §3 file roles; §4.1 `debtPlan` and the Budget / Bucket / Debt / Scenario shapes; §6 `--teal-text` and Plan components; §7 rule 7 covers the Plan charts; §8 plan helpers (`payoffPlan()`, `growthByYear()` …); §9 plan form helpers, `simulatePayoff()` kept unused; §12 Plan titles, Forecast tabs, range label hidden on Plan screens; §13 debt balances in the summary; §20 Clear forecast scenarios / investment mix, Clear debts resets `debtPlan`; v1.14.0: Finances screens rebuilt: §4.1 Bill shape and `recurringConfirmed`; §6 new tokens and Finances components; §7 rule 7 inline SVG charts; §8 new aggregation helpers; §9 drawer, `TX_VIEWS`, SVG/heatmap/format helpers, bill helpers, removed renderers; §11 keywords re-run detection, `transferPairs()`, card payments; §12 page actions and Transactions pills; §20 Clear bills also clears spotted-charge choices; v1.13.0: Clarity redesign — §6 tokens and components; §9 theme, chart colors and app bar helpers; §12 app bar navigation replaces the sidebar and the emoji rule; §17 footer; §20 Data & Reset; v1.12.0: bank-neutral account codes, names, samples and docs; §22 `migrate.js` added; §3, §4, §5, §10, §18, §21 updated; v1.10.5: no section changes: Import Logs race fix; v1.10.4: §8, §13 single `~` is literal; §21 excerpts escape `~`; v1.10.3: §13 refusals; §21 declined runs; v1.10.2: §13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)* +*Last updated: v1.16.2 — 2026-11-02 (Imports read every file before changing anything: §8 `loadFiles()` options and result, `parseCsvFile()` reads only, `importParsedCsv()`, `importProblemText()`, `reloadFromDirHandle()`; §20 Rebuild from your folder reads before it clears, read before clearing; v1.16.1: Clear transaction edits works from memory: §4.2 `_originalCategories`, `_originalType` and what a transaction keeps of its row; §4.5 IndexedDB may not hold the loaded transactions; §10 a stored row holds the CSV's values; §20 Clear transaction edits; v1.16.0: Trends, AI Analysis, Settings and the Welcome rebuilt: §3 file roles; §4.1 `cardDetails`, `sampleData`, AI history `model`; §6 tile tokens and Insights/Settings/Welcome components; §7 rule 7 covers Trends; §8 `loadFiles()` sample option; §9 `shellHero()`, `NO_RANGE_VIEWS`, ☰ → ×, JSON export/import buttons; §12 Settings tabs renamed, the Welcome, range label hidden on Trends and Settings; §13 quick analyses, tab state, history; §20 new fields, Use my own files, Export/Import JSON; §21 quick analyses send `AI_PROMPTS`; v1.15.0: Plan screens rebuilt: §3 file roles; §4.1 `debtPlan` and the Budget / Bucket / Debt / Scenario shapes; §6 `--teal-text` and Plan components; §7 rule 7 covers the Plan charts; §8 plan helpers (`payoffPlan()`, `growthByYear()` …); §9 plan form helpers, `simulatePayoff()` kept unused; §12 Plan titles, Forecast tabs, range label hidden on Plan screens; §13 debt balances in the summary; §20 Clear forecast scenarios / investment mix, Clear debts resets `debtPlan`; v1.14.0: Finances screens rebuilt: §4.1 Bill shape and `recurringConfirmed`; §6 new tokens and Finances components; §7 rule 7 inline SVG charts; §8 new aggregation helpers; §9 drawer, `TX_VIEWS`, SVG/heatmap/format helpers, bill helpers, removed renderers; §11 keywords re-run detection, `transferPairs()`, card payments; §12 page actions and Transactions pills; §20 Clear bills also clears spotted-charge choices; v1.13.0: Clarity redesign — §6 tokens and components; §9 theme, chart colors and app bar helpers; §12 app bar navigation replaces the sidebar and the emoji rule; §17 footer; §20 Data & Reset; v1.12.0: bank-neutral account codes, names, samples and docs; §22 `migrate.js` added; §3, §4, §5, §10, §18, §21 updated; v1.10.5: no section changes: Import Logs race fix; v1.10.4: §8, §13 single `~` is literal; §21 excerpts escape `~`; v1.10.3: §13 refusals; §21 declined runs; v1.10.2: §13 summary states its dates; §21 per-month derivations; v1.10.1: §13 `AI_MAX_TOKENS`, thinking and cut-off answers; v1.10.0: §21 AI Eval added; §13 `callClaude()` raw mode; v1.9.6: §14 folder access lasts until the last tab closes, not one page load; v1.9.5: §7 chart guard; v1.9.4: views-db.js changelog reads CRLF checkouts; v1.9.3: §14 folder permission: re-request on click, Reconnect banner, browser copy wins; v1.9.2: §0 branch, PR and approval workflow added; v1.9.1: §4.5 rehydrate transactions loaded from JSON; v1.9.0: §20 Data & Reset added; v1.8.1: §10 re-upload dedup rewritten for the occurrence-suffix fix; v1.8.0: AI Analysis restored with API-key and output hardening; `renderMarkdown()` added; `samples/` added; §1–3, §13, §14, §17 corrected to match the code)* --- @@ -1021,7 +1028,7 @@ CSV files on disk are **never** modified. | Clear import log | Empties `import_log` | Everything else | | Clear transaction edits | `txOverrides` + `tx_overrides` store; rebuilds each loaded transaction without its edit, from memory (`_unedited()`), then `detectTransfers()` | Every loaded transaction; the `transactions` store as it is | | Clear budgets / bills / buckets / debts / forecast scenarios / investment mix / AI history | `State[key] = []` (`{}` for `investAlloc`; the empty value is `PLAN_ITEMS[i][2]`) + `lsSet`. Bills also resets `recurringConfirmed` (confirmed/dismissed spotted charges); Debts also resets `debtPlan` (`PLAN_ALSO`) | — | -| Rebuild from folder | Clears transactions, accounts, import log, then `reloadFromDirHandle()` | Plans, edits, settings | +| Rebuild from folder | `reloadFromDirHandle()` reads every CSV first; only if they can all be read (`_rebuildBlockers()`) it clears transactions, accounts and import log (in `beforeImport`), then imports them | Plans, edits, settings; everything, when a CSV can't be read | | Disconnect folder | `forgetDataFolder()` — stops CSV auto-reload and state-file writes | Files on disk | | Use my own files (sample banner) | `clearSampleData()`: transactions, accounts and import log (memory + stores), `sampleData` off, state file rewritten, back to the Welcome | Plans made while trying it | | Export / Import JSON | `exportAllData()` / `importJsonFile()` — not a clear; listed because it lives here | — | @@ -1038,6 +1045,10 @@ CSV files on disk are **never** modified. - **The originals:** `_originalDesc`, `_originalCategories` and `_originalType` on the transaction (§4.2), flags from the rules. A store row only fills in the category and type when the transaction doesn't record them. - **A file from ≤v1.16.0 with no row:** if a transaction's category or type was edited, or it was marked as a transfer, they are worked out again from the description and amount (`autoCategory()`, debit/credit by sign), as an import of a file without Category and Type columns does. That is exact for checking and savings; a card's own Category and Type (Sale/Payment/Return) can't be recovered. Rebuild from your folder gets them back. Worked-out values aren't recorded as `_original*`. - **The `transactions` store isn't written.** When it's empty, the transactions aren't copied into it: filling it is an import's job (accounts, coverage, import log), and the state file, rewritten by `_finish()`, holds all of them. +- **Read before clearing** (v1.16.2). Up to v1.16.1, Rebuild from your folder cleared first, so a folder with no CSV FinLens could read left 0 transactions, and `_finish()` saved that to the file. + - **An import that reads nothing changes nothing.** `loadFiles()` reads every file (`parseCsvFile()` writes nothing) before it touches memory, storage, IndexedDB, the sample data or the import pause. When no transaction was read it stops there, and says why (an alert, or the caller's message with `quiet`). That covers every import: Rebuild, Reload, Reconnect, picking a folder, choosing files, boot. + - **Rebuild clears only when nothing would be lost.** It passes `beforeImport` to `loadFiles()`, which calls it after the read. `_rebuildBlockers()` lists the files that would cost transactions: one the browser couldn't open or parse, one in a known format (by columns or name) with no readable row, one imported before whose columns are no longer recognised. If there are any, nothing is cleared and the tab names them. A CSV in another format that was never imported is skipped, as every import skips it, and the success message names it. + - **After the clear, the import finishes.** `importParsedCsv()` logs a failing IndexedDB write and carries on, so `_finish()` still saves every transaction to the file. - **Start over must leave nothing behind.** It sets `State._resetInProgress` so a pending debounced save can't re-create the file, and `DB.open()` registers `onversionchange` so other tabs release the database. `idbGet`/`idbSet`/`idbDel` close their connections so `finlens_fs` can be deleted. - **Fields added in v1.16.0:** `cardDetails` and `sampleData` live in localStorage, IndexedDB `app_state` (`DB.loadAll()`) and `finlens-state.json`; Start over erases both with everything else. `cardDetails` is also in the JSON export. Importing your own files turns `sampleData` off after clearing the samples (`loadFiles()`). - `clearAllData()` is kept as a wrapper for `DataCleanup.startOver('archive')`. Up to v1.8.1 it only cleared `localStorage`, which IndexedDB and the state file then restored on reload. diff --git a/README.md b/README.md index 1781de5..b431167 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > A zero-install, browser-native finance dashboard. Drop in your bank CSVs, get instant clarity on your money. -**Version:** v1.16.1 · **Stack:** Vanilla JS, HTML, CSS · **Install:** None +**Version:** v1.16.2 · **Stack:** Vanilla JS, HTML, CSS · **Install:** None --- @@ -261,4 +261,4 @@ See [CHANGELOG.md](CHANGELOG.md) for the full release history. --- -*FinLens v1.16.1 — Built for one person's finances, designed to be useful for anyone's.* +*FinLens v1.16.2 — Built for one person's finances, designed to be useful for anyone's.* diff --git a/index.html b/index.html index 8b4be31..1011cb2 100644 --- a/index.html +++ b/index.html @@ -1164,7 +1164,7 @@
- + diff --git a/js/core.js b/js/core.js index 320ab4b..c622cee 100644 --- a/js/core.js +++ b/js/core.js @@ -626,133 +626,221 @@ function mkTx(o) { }; } +// Reads one CSV file and turns its rows into transactions. It changes nothing: no State, +// browser storage or IndexedDB writes (v1.16.2). loadFiles() reads every file this way +// first and imports (importParsedCsv()) only when at least one transaction was read. +// Resolves to { file, name, schema, kind, accountId, accountName, rows, txs, problem }: +// `rows` is the file's data rows, `txs` its transactions, and `problem` null or why the +// file gives none: 'unknown-format' (no schema matched its columns or name), 'no-rows' +// (none of its rows has a date and an amount that can be read), 'unreadable' (the +// browser couldn't read it). A file with only a header has no problem and no transactions. function parseCsvFile(file, forcedKind) { return new Promise((resolve) => { + const out = { file, name: file.name, schema: null, kind: null, accountId: null, accountName: null, rows: 0, txs: [], problem: null }; Papa.parse(file, { header: true, skipEmptyLines: true, transformHeader: h => h.trim(), // KEY: trim whitespace from bank CSV headers (checking/savings exports have leading spaces) - complete: async (res) => { - const rows = res.data || []; - const headers = rows.length ? Object.keys(rows[0]) : []; - - // Determine schema — forced kind takes precedence over auto-detection - let schema = null; - if (forcedKind && forcedKind !== 'unknown' && forcedKind !== 'other') { - schema = BANK_SCHEMAS.find(s => s.id === forcedKind) || null; - } - if (!schema) { - schema = detectSchema(headers, file.name); - } - if (!schema && rows.length) { - // Final fallback: header content matching - const lk = headers.map(h => h.toLowerCase()); - if (lk.includes('transaction date') || lk.includes('post date')) - schema = BANK_SCHEMAS.find(s => s.id === 'credit_card'); - else if (lk.includes('time') && lk.includes('type') && lk.includes('amount')) - schema = BANK_SCHEMAS.find(s => s.id === 'bank_checking'); - } - // Unknown schema — skip with warning (Column Mapping UI is a future enhancement) - if (!schema) { - console.warn('parseCsvFile: no schema matched for', file.name, '— skipping. Headers:', headers); - resolve([]); - return; - } + complete: (res) => { + try { + const rows = res.data || []; + const headers = rows.length ? Object.keys(rows[0]) : []; + out.rows = rows.length; - const kind = schema.id; - - // Build accountId and display name (migrate.js maps the ids older versions used) - let accountId, accountName; - if (kind === 'credit_card') { - accountId = 'card_' + file.name.replace(/\.csv$/i,'').toLowerCase().replace(/[^a-z0-9]+/g,'_'); - accountName = 'Bank Credit Card (' + file.name.replace(/\.csv$/i,'') + ')'; - } else if (kind === 'bank_savings') { - accountId = 'bank_savings'; accountName = 'Bank Savings'; - } else if (kind === 'bank_checking') { - accountId = 'bank_checking'; accountName = 'Bank Checking'; - } else { - accountId = 'acct_' + file.name.replace(/\.csv$/i,'').replace(/[^a-z0-9]+/g,'_'); - accountName = file.name; - } + // Determine schema — forced kind takes precedence over auto-detection + let schema = null; + if (forcedKind && forcedKind !== 'unknown' && forcedKind !== 'other') { + schema = BANK_SCHEMAS.find(s => s.id === forcedKind) || null; + } + if (!schema) { + schema = detectSchema(headers, file.name); + } + if (!schema && rows.length) { + // Final fallback: header content matching + const lk = headers.map(h => h.toLowerCase()); + if (lk.includes('transaction date') || lk.includes('post date')) + schema = BANK_SCHEMAS.find(s => s.id === 'credit_card'); + else if (lk.includes('time') && lk.includes('type') && lk.includes('amount')) + schema = BANK_SCHEMAS.find(s => s.id === 'bank_checking'); + } + // Unknown schema — skip with warning (Column Mapping UI is a future enhancement) + if (!schema) { + console.warn('parseCsvFile: no schema matched for', file.name, '— skipping. Headers:', headers); + out.problem = 'unknown-format'; + resolve(out); + return; + } - // Initialise account in State (in-memory) - State.accounts[accountId] = State.accounts[accountId] || { - id: accountId, name: accountName, kind, balance: null, limit: null, sourceFile: file.name - }; - State.accounts[accountId].sourceFile = file.name; - // Restore persisted custom label if any - if (State.accountLabels && State.accountLabels[accountId]) { - State.accounts[accountId].name = State.accountLabels[accountId]; - } + const kind = schema.id; - // Normalize all rows. Truly identical rows (same hash inputs) are counted within - // this file: the 2nd copy gets an _2 suffix, the 3rd _3, and so on. Counting per - // file (never against the database) keeps IDs identical across re-uploads, so - // DB.importTransactions() can simply skip IDs it already holds. Schemas with a - // bank-assigned Time column (checking/savings) opt out: an identical row there is a true - // duplicate, so only the first copy is kept. - const allowSuffix = schema.allowOccurrenceSuffix ?? !schema.columns.time; - const txs = []; - const seen = {}; - for (const r of rows) { - let tx = normalizeRow(r, schema, accountId); - if (!tx) continue; - const n = seen[tx.id] = (seen[tx.id] || 0) + 1; - if (n > 1) { - if (!allowSuffix) continue; - tx = normalizeRow(r, schema, accountId, n); + // Build accountId and display name (migrate.js maps the ids older versions used) + let accountId, accountName; + if (kind === 'credit_card') { + accountId = 'card_' + file.name.replace(/\.csv$/i,'').toLowerCase().replace(/[^a-z0-9]+/g,'_'); + accountName = 'Bank Credit Card (' + file.name.replace(/\.csv$/i,'') + ')'; + } else if (kind === 'bank_savings') { + accountId = 'bank_savings'; accountName = 'Bank Savings'; + } else if (kind === 'bank_checking') { + accountId = 'bank_checking'; accountName = 'Bank Checking'; + } else { + accountId = 'acct_' + file.name.replace(/\.csv$/i,'').replace(/[^a-z0-9]+/g,'_'); + accountName = file.name; } - txs.push(tx); - } - - // Persist to IndexedDB (if DB is ready) - let importResult = null; - if (typeof DB !== 'undefined' && DB._db) { - await DB.upsertAccount({ - id: accountId, - name: State.accounts[accountId].name, - kind, - schemaId: schema.id, - sourceFile: file.name, - displayOrder: State.accountOrder.indexOf(accountId), - manualBalance: State.manualBalances[accountId] != null ? Number(State.manualBalances[accountId]) : null, - creditLimit: State.manualBalances[accountId+'_limit'] != null ? Number(State.manualBalances[accountId+'_limit']) : null, - initialBalance: State.manualBalances[accountId+'_initial'] != null ? Number(State.manualBalances[accountId+'_initial']) : null, - }); - - importResult = await DB.importTransactions(txs, accountId, schema.id); - - await DB.logImport({ - filename: file.name, - accountId, - schemaId: schema.id, - rowsFound: rows.length, - rowsInserted: importResult.inserted, - rowsSkipped: importResult.skipped, - rowsRemoved: importResult.removed, - dateRangeMin: importResult.dateRangeMin, - dateRangeMax: importResult.dateRangeMax, - }); - } - State.fileMeta.push({ - name: file.name, kind, account: State.accounts[accountId].name, rows: txs.length, - dateRange: txs.length ? [txs[txs.length-1].date, txs[0].date] : null, - importResult, // may be null if DB not ready - }); + // Normalize all rows. Truly identical rows (same hash inputs) are counted within + // this file: the 2nd copy gets an _2 suffix, the 3rd _3, and so on. Counting per + // file (never against the database) keeps IDs identical across re-uploads, so + // DB.importTransactions() can simply skip IDs it already holds. Schemas with a + // bank-assigned Time column (checking/savings) opt out: an identical row there is a true + // duplicate, so only the first copy is kept. + const allowSuffix = schema.allowOccurrenceSuffix ?? !schema.columns.time; + const txs = []; + const seen = {}; + for (const r of rows) { + let tx = normalizeRow(r, schema, accountId); + if (!tx) continue; + const n = seen[tx.id] = (seen[tx.id] || 0) + 1; + if (n > 1) { + if (!allowSuffix) continue; + tx = normalizeRow(r, schema, accountId, n); + } + txs.push(tx); + } - resolve(txs); + Object.assign(out, { schema, kind, accountId, accountName, txs }); + if (rows.length && !txs.length) out.problem = 'no-rows'; + resolve(out); + } catch(e) { + console.warn('parseCsvFile: could not read', file.name, '—', e.message); + out.problem = 'unreadable'; out.txs = []; + resolve(out); + } }, - error: () => resolve([]) + error: (e) => { out.problem = 'unreadable'; resolve(out); }, }); }); } +// Imports a file parseCsvFile() read: creates or updates its account (State and IndexedDB), +// adds its new rows to IndexedDB (DB.importTransactions() skips ids it already holds), +// logs the import and adds it to State.fileMeta. Returns the file's transactions; +// loadFiles() puts them in State.transactions. A failing database write is logged and the +// import carries on, so the transactions still reach memory and finlens-state.json. +async function importParsedCsv(p) { + const { file, schema, kind, accountId, accountName, txs } = p; + // Initialise account in State (in-memory) + State.accounts[accountId] = State.accounts[accountId] || { + id: accountId, name: accountName, kind, balance: null, limit: null, sourceFile: file.name + }; + State.accounts[accountId].sourceFile = file.name; + // Restore persisted custom label if any + if (State.accountLabels && State.accountLabels[accountId]) { + State.accounts[accountId].name = State.accountLabels[accountId]; + } + + // Persist to IndexedDB (if DB is ready) + let importResult = null; + if (typeof DB !== 'undefined' && DB._db) { + try { + await DB.upsertAccount({ + id: accountId, + name: State.accounts[accountId].name, + kind, + schemaId: schema.id, + sourceFile: file.name, + displayOrder: State.accountOrder.indexOf(accountId), + manualBalance: State.manualBalances[accountId] != null ? Number(State.manualBalances[accountId]) : null, + creditLimit: State.manualBalances[accountId+'_limit'] != null ? Number(State.manualBalances[accountId+'_limit']) : null, + initialBalance: State.manualBalances[accountId+'_initial'] != null ? Number(State.manualBalances[accountId+'_initial']) : null, + }); + + importResult = await DB.importTransactions(txs, accountId, schema.id); + + await DB.logImport({ + filename: file.name, + accountId, + schemaId: schema.id, + rowsFound: p.rows, + rowsInserted: importResult.inserted, + rowsSkipped: importResult.skipped, + rowsRemoved: importResult.removed, + dateRangeMin: importResult.dateRangeMin, + dateRangeMax: importResult.dateRangeMax, + }); + } catch(e) { + console.warn('[FinLens] import: the browser database didn\'t take', file.name, '—', e.message); + importResult = null; + } + } + + State.fileMeta.push({ + name: file.name, kind, account: State.accounts[accountId].name, rows: txs.length, + dateRange: txs.length ? [txs[txs.length-1].date, txs[0].date] : null, + importResult, // may be null if DB not ready + }); + + return txs; +} + +// Why a file gave no transactions, in words (loadFiles()'s result → importProblemText()). +const IMPORT_PROBLEMS = { + 'unknown-format': () => 'its columns don\'t match a format FinLens reads', + 'no-rows': (f) => `none of its ${f.rows.toLocaleString()} row${f.rows === 1 ? '' : 's'} has a date and an amount FinLens can read`, + 'unreadable': () => 'the browser couldn\'t read it', + 'skipped': () => 'set to Skip', + 'empty': () => 'it has no transactions', +}; +function importFileProblem(f) { + const k = f.problem || (f.transactions ? '' : 'empty'); + return k && IMPORT_PROBLEMS[k] ? IMPORT_PROBLEMS[k](f) : ''; +} +// "a.csv: its columns don't match …; b.csv: …" for the files of a loadFiles() result +// that gave no transactions. +function importProblemText(res, sep = '; ') { + return res.files.filter(f => importFileProblem(f)).map(f => `${f.name}: ${importFileProblem(f)}`).join(sep); +} + +// Imports CSV files: reads every file first (parseCsvFile()), then imports them +// (importParsedCsv()), detects transfers and renders (v1.16.2). +// - assignments: { filename -> kind } (optional; if not provided, auto-detect; 'skip' skips the file) +// - opts.sample: these are the synthetic samples/ files (loadSampleData() in views-data.js) +// - opts.quiet: no alert when nothing could be imported; the caller says why (Rebuild) +// - opts.unopened: [{ name, error }] CSV files the caller found but couldn't open (reloadFromDirHandle()) +// - opts.beforeImport(result): called after the files are read, before anything changes; +// returning false stops the import (Rebuild from your folder checks and clears here) +// Nothing changes (State, browser storage, IndexedDB, finlens-state.json) unless at least +// one transaction was read. Up to v1.16.1 it emptied the loaded transactions first, so a +// folder with no CSV it could read left 0 transactions, and the next save wrote that to +// finlens-state.json. +// Returns { imported, reason: null|'no-csv'|'nothing-read'|'stopped', transactions, +// files: [{ name, kind, accountId, rows, transactions, problem }] }. async function loadFiles(fileList, assignments, opts = {}) { - // assignments: { filename -> kind } (optional; if not provided, auto-detect) - // opts.sample: these are the synthetic samples/ files (loadSampleData() in views-data.js) const csvs = Array.from(fileList).filter(f=>f.name.toLowerCase().endsWith('.csv')); - if (!csvs.length) { alert('No CSV files found.'); return; } + const unopened = (opts.unopened || []).map(u => ({ name: u.name, kind: null, accountId: null, rows: 0, transactions: 0, problem: 'unreadable' })); + const result = { imported: false, reason: null, transactions: 0, files: [] }; + if (!csvs.length && !unopened.length) { + if (!opts.quiet) alert('No CSV files found.'); + return { ...result, reason: 'no-csv' }; + } + // 1. Read every file. Nothing is changed yet. + const read = []; + const skipped = []; + for (const f of csvs) { + const kind = assignments ? assignments[f.name] : undefined; + if (kind==='skip') { skipped.push({ name: f.name, kind: null, accountId: null, rows: 0, transactions: 0, problem: 'skipped' }); continue; } + read.push(await parseCsvFile(f, kind)); + } + result.files = read.map(p => ({ name: p.name, kind: p.kind, accountId: p.accountId, rows: p.rows, transactions: p.txs.length, problem: p.problem })) + .concat(unopened, skipped); + result.transactions = read.reduce((n, p) => n + p.txs.length, 0); + if (!result.transactions) { + result.reason = 'nothing-read'; + if (!opts.quiet) alert(`FinLens couldn't read any transactions from ${result.files.length === 1 ? 'this file' : 'these files'}, so nothing was changed.\n\n${importProblemText(result, '\n')}`); + return result; + } + if (opts.beforeImport && (await opts.beforeImport(result)) === false) { result.reason = 'stopped'; return result; } + + // 2. Import them. // Any import (Reload, Reconnect, Rebuild, picking a folder or files) ends the pause // set by "Clear transactions & accounts". Boot never gets here while paused. if (State.csvAutoImportPaused) { State.csvAutoImportPaused = false; lsSet('csvAutoImportPaused', false); } @@ -767,10 +855,9 @@ async function loadFiles(fileList, assignments, opts = {}) { } State.fileMeta = []; State.transactions = []; - for (const f of csvs) { - const kind = assignments ? assignments[f.name] : undefined; - if (kind==='skip') continue; - const txs = await parseCsvFile(f, kind); + for (const p of read) { + if (!p.schema || p.problem) continue; // a file that gave no transactions creates no account + const txs = await importParsedCsv(p); State.transactions.push(...txs); } // Edits still saved under a transaction's id from before v1.12.0 (e.g. kept while the @@ -792,6 +879,8 @@ async function loadFiles(fileList, assignments, opts = {}) { updateRefreshLabel(); buildAccountFilter(); renderActive(); + result.imported = true; + return result; } function detectTransfers() { diff --git a/js/views-cleanup.js b/js/views-cleanup.js index 7c1a75a..48e3d8e 100644 --- a/js/views-cleanup.js +++ b/js/views-cleanup.js @@ -114,11 +114,41 @@ const DataCleanup = { } if (!canRead) return this._done('bad', `FinLens doesn't have access to “${State.lastFolderName}”. Nothing was cleared.`); - this._dropImported(); - _importLogCache = []; - await DB.clearStores(['transactions', 'accounts', 'import_log']); - await reloadFromDirHandle(handle); // re-imports every CSV, detects transfers, renders - await this._finish(`Rebuilt from “${State.lastFolderName}”: ${State.transactions.length.toLocaleString()} transactions re-imported.`); + // Read and check every CSV before clearing anything (v1.16.2). Up to v1.16.1 this + // cleared first, so a folder with no CSV FinLens could read (only finlens-state.json, + // or files it can't parse) left 0 transactions, and _finish() wrote that to the file. + // loadFiles() reads the files, then calls beforeImport: only then is the data cleared, + // and only when nothing would be lost (_rebuildBlockers()). + const name = State.lastFolderName; + let cleared = false, blockers = [], res; + try { + res = await reloadFromDirHandle(handle, { // re-imports every CSV, detects transfers, renders + quiet: true, + beforeImport: async (r) => { + blockers = this._rebuildBlockers(r); + if (blockers.length) return false; + this._dropImported(); + _importLogCache = []; + await DB.clearStores(['transactions', 'accounts', 'import_log']); + cleared = true; + return true; + }, + }); + } catch(e) { + if (cleared) throw e; + return this._done('bad', `Nothing was cleared: FinLens couldn't read “${name}” (${e.message || e}).`); + } + const list = (fs) => fs.map(f => `${f.name} (${importFileProblem(f)})`).join(', '); + if (!cleared) { + const one = blockers.length === 1; + return this._done('bad', res.reason === 'no-csv' ? `Nothing was cleared: “${name}” has no CSV files.` + : res.reason === 'nothing-read' ? `Nothing was cleared: FinLens couldn't read any transactions from the CSV files in “${name}”. ${importProblemText(res)}.` + : `Nothing was cleared: FinLens couldn't read ${list(blockers)}. Rebuild re-reads every CSV from scratch, so ${one ? 'that file\'s' : 'those files\''} transactions would be left out. Fix or replace ${one ? 'it' : 'them'}, or move ${one ? 'it' : 'them'} out of the folder, then rebuild again.`); + } + // CSVs in other formats (never imported) are skipped, as every import skips them: say which. + const left = res.files.filter(f => f.problem && f.problem !== 'skipped'); + await this._finish(`Rebuilt from “${name}”: ${State.transactions.length.toLocaleString()} transactions re-imported.` + + (left.length ? ` Not read: ${list(left)}.` : '')); }, // "Use my own files" on the sample-data banner: removes the samples (transactions, @@ -212,6 +242,18 @@ const DataCleanup = { }, {}); }, + // The files that stop Rebuild from your folder (v1.16.2): CSVs FinLens would import but + // couldn't read, whose transactions a rebuild would leave out. That is a file the browser + // couldn't open or parse, a file in a known format (by its columns or name) with no row + // FinLens can read, or a file it imported before whose columns it no longer recognises. + // Other CSVs in a format it doesn't read are skipped, as every import skips them. + // `r` is loadFiles()'s result, before anything is cleared. + _rebuildBlockers(r) { + const imported = new Set(Object.values(State.accounts).map(a => a.sourceFile).filter(Boolean)); + return r.files.filter(f => f.problem === 'unreadable' || f.problem === 'no-rows' + || (f.problem === 'unknown-format' && imported.has(f.name))); + }, + // Forget imported data in memory (the caller clears the matching DB stores). _dropImported() { State.transactions = []; diff --git a/js/views-data.js b/js/views-data.js index b63895d..d6b9b91 100644 --- a/js/views-data.js +++ b/js/views-data.js @@ -151,6 +151,7 @@ registerView('import','Import',(root)=>{