diff --git a/.changeset/drizzle-encrypted-indexes.md b/.changeset/drizzle-encrypted-indexes.md new file mode 100644 index 00000000..d9f39025 --- /dev/null +++ b/.changeset/drizzle-encrypted-indexes.md @@ -0,0 +1,26 @@ +--- +'@cipherstash/stack-drizzle': minor +--- + +New `encryptedIndexes` helper on the `/v3` entry: spread +`...encryptedIndexes(t)` in `pgTable`'s third-argument callback and it derives +the recommended functional indexes for every encrypted column in the table — +named `__`, tracked by `drizzle-kit generate` like +any other index. The mapping comes from the same per-domain capability record +the operator layer gates on, so the emitted indexes and the operators that +engage them cannot drift: equality → btree on `eql_v3.eq_term`, ordering → +btree on `eql_v3.ord_term` (on the numeric/date/timestamp `_ord` domains one +index serves `=` and range — their injective ordering term answers equality +and no `eq_term` overload exists; the non-injective `text_ord` / `text_ord_ore` +also carry `hm` and get an `eq_term` index alongside), ORE ordering → +`eql_v3.ord_term_ore`, free-text → +GIN on `eql_v3.match_term`, encrypted JSON → GIN on +`(eql_v3.to_ste_vec_query(col)::jsonb) jsonb_path_ops`. Storage-only and +non-encrypted columns emit nothing. Closes the #753 gap where integrations +emitted query operators but no index DDL, so encrypted predicates +sequential-scanned by default. + +Also fixed: `isEqlV3Column` / `getEqlV3Column` no longer blow the stack when +handed a column from `pgTable`'s extras callback — drizzle-orm ≤0.45's +`ExtraConfigColumn.getSQLType()` recurses into itself, so the domain is now +recovered from the column's custom-type params instead of calling it. diff --git a/.changeset/stash-indexing-skill.md b/.changeset/stash-indexing-skill.md new file mode 100644 index 00000000..6303d9d6 --- /dev/null +++ b/.changeset/stash-indexing-skill.md @@ -0,0 +1,23 @@ +--- +'stash': minor +'@cipherstash/wizard': minor +--- + +New bundled agent skill: `stash-indexing` — how to index EQL v3 encrypted +columns. Integrations that were otherwise correct shipped with no index on any +encrypted predicate because nothing in the installed skills said encrypted +columns *can* be indexed (#753). The skill covers the functional-index recipes +over the term extractors (`eql_v3.eq_term` / `ord_term` / `ord_term_ore` / +`match_term` / `to_ste_vec_query`) mapped to the `types.*` domains, what works +without superuser on Supabase and managed Postgres versus the ORE opclass +restriction, which domains are storage-only by design, the query shapes that +engage an index (`ORDER BY` sort-key and `GROUP BY` traps), building indexes on +large tables, an `EXPLAIN` verification checklist, and when to create indexes +during an encryption rollout (after backfill, before switching reads). + +`stash init` / `stash impl` handoffs — and the `@cipherstash/wizard` skills +prompt — now install it for **every** integration (Drizzle, Supabase, Prisma +Next, plain PostgreSQL) — the gap is cross-cutting. +The existing per-integration skills gained pointers to it (including the +missing `stash-prisma-next` one-line purpose in the setup prompt, which +previously rendered "(no description)"). diff --git a/AGENTS.md b/AGENTS.md index 69ca5508..8c0566c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,7 +92,7 @@ If these variables are missing, tests that require live encryption will fail or - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). -- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma-next`, `stash-supply-chain-security`) +- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-indexing`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma-next`, `stash-supply-chain-security`) ## Agent Skills — these ship to customers @@ -119,6 +119,7 @@ nothing type-checks them, and the damage lands in a customer's repo, not ours. | `packages/stack` encryption API, schema builders, subpath exports | `skills/stash-encryption` | | Drizzle / Supabase / Prisma Next / DynamoDB integrations | `skills/stash-drizzle`, `skills/stash-supabase`, `skills/stash-prisma-next`, `skills/stash-dynamodb` | | The rollout/cutover lifecycle (`packages/migrate`, `stash encrypt *`) | `skills/stash-encryption` and `skills/stash-cli` | +| The `@cipherstash/eql` pin, `eql install`/`eql migration` behaviour, or index-related SQL guidance | `skills/stash-indexing` | | pnpm config, CI workflows, dependency policy | `skills/stash-supply-chain-security` | | The durable agent rules themselves | `packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md` | diff --git a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md index b8989919..411114a0 100644 --- a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md +++ b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md @@ -47,4 +47,4 @@ The CipherStash setup skills carry the API details. Use them when you need speci - `.codex/skills//SKILL.md` (Codex) - Under `## Skill references` at the bottom of this file (editor agents, or when a skills directory could not be written — if the section exists, it is the current copy) -Skills relevant to this project depend on the integration. Common ones: `stash-encryption` (encryption API), `stash-cli` (`stash` commands), and one of `stash-drizzle` / `stash-supabase` / `stash-dynamodb` for the chosen ORM. +Skills relevant to this project depend on the integration. Common ones: `stash-encryption` (encryption API), `stash-indexing` (indexes on encrypted columns), `stash-cli` (`stash` commands), and one of `stash-drizzle` / `stash-supabase` / `stash-prisma-next` / `stash-dynamodb` for the chosen ORM. diff --git a/packages/cli/src/commands/init/lib/__tests__/build-agents-md.test.ts b/packages/cli/src/commands/init/lib/__tests__/build-agents-md.test.ts index 8bef16d2..6599656b 100644 --- a/packages/cli/src/commands/init/lib/__tests__/build-agents-md.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/build-agents-md.test.ts @@ -27,6 +27,7 @@ describe('buildAgentsMdBody', () => { expect(out).toContain('# CipherStash') expect(out).toContain('# Skill: stash-encryption') expect(out).toContain('# Skill: stash-drizzle') + expect(out).toContain('# Skill: stash-indexing') expect(out).toContain('# Skill: stash-cli') // Frontmatter from individual skill files should be stripped — the // `name: ` line is part of YAML frontmatter and should not leak. diff --git a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts index 62e76f26..e4a2e3aa 100644 --- a/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/install-skills.test.ts @@ -55,6 +55,16 @@ describe('SKILL_MAP', () => { } }) + // #753: integrations shipped without a single CREATE INDEX example, so + // agent-built integrations left every encrypted predicate unindexed. The + // indexing skill is cross-cutting (Drizzle, Prisma Next, Supabase, and raw + // SQL all need it), so every integration must install it. + it('always includes stash-indexing for every integration', () => { + for (const [integration, skills] of Object.entries(SKILL_MAP)) { + expect(skills, integration).toContain('stash-indexing') + } + }) + it('drizzle includes stash-drizzle', () => { expect(SKILL_MAP.drizzle).toContain('stash-drizzle') }) @@ -79,6 +89,7 @@ describe('skillsFor', () => { expect(skillsFor('prisma-next')).toEqual([ 'stash-encryption', 'stash-prisma-next', + 'stash-indexing', 'stash-cli', ]) }) @@ -86,7 +97,7 @@ describe('skillsFor', () => { it('falls back to the base skills for an unmapped integration (never crashes)', () => { // Simulate a future Integration variant with no SKILL_MAP entry. const skills = skillsFor('mystery-orm' as Integration) - expect(skills).toEqual(['stash-encryption', 'stash-cli']) + expect(skills).toEqual(['stash-encryption', 'stash-indexing', 'stash-cli']) }) }) @@ -104,7 +115,12 @@ describe('installSkills', () => { it('copies the per-integration skills into destDir', () => { const { copied, failed } = installSkills(tmp, '.claude/skills', 'drizzle') - expect(copied).toEqual(['stash-encryption', 'stash-drizzle', 'stash-cli']) + expect(copied).toEqual([ + 'stash-encryption', + 'stash-drizzle', + 'stash-indexing', + 'stash-cli', + ]) expect(failed).toEqual([]) for (const name of copied) { expect( @@ -141,7 +157,12 @@ describe('installSkills', () => { }).not.toThrow() expect(result).toEqual({ copied: [], - failed: ['stash-encryption', 'stash-drizzle', 'stash-cli'], + failed: [ + 'stash-encryption', + 'stash-drizzle', + 'stash-indexing', + 'stash-cli', + ], }) expect(warnings()).toContain('Could not create .codex/skills/') }) @@ -162,7 +183,7 @@ describe('installSkills', () => { ) const { copied, failed } = installSkills(tmp, '.codex/skills', 'drizzle') - expect(copied).toEqual(['stash-encryption', 'stash-cli']) + expect(copied).toEqual(['stash-encryption', 'stash-indexing', 'stash-cli']) expect(failed).toEqual(['stash-drizzle']) expect(warnings()).toContain('Failed to install skill stash-drizzle') }) @@ -177,7 +198,12 @@ describe('installSkills', () => { const { copied, failed } = installSkills(tmp, '.codex/skills', 'drizzle') expect(copied).toEqual([]) expect(failed).toEqual(availableSkills('drizzle')) - expect(failed).toEqual(['stash-encryption', 'stash-drizzle', 'stash-cli']) + expect(failed).toEqual([ + 'stash-encryption', + 'stash-drizzle', + 'stash-indexing', + 'stash-cli', + ]) }) it('is idempotent — re-running does not throw and yields the same result', () => { diff --git a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts index c0b499e0..b1b68491 100644 --- a/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts +++ b/packages/cli/src/commands/init/lib/__tests__/setup-prompt.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from 'vitest' +import { SKILL_MAP } from '../install-skills.js' import { renderSetupPrompt, type SetupPromptContext } from '../setup-prompt.js' const baseCtx: SetupPromptContext = { @@ -12,7 +13,12 @@ const baseCtx: SetupPromptContext = { handoff: 'claude-code', mode: 'implement', skills: { - installed: ['stash-encryption', 'stash-drizzle', 'stash-cli'], + installed: [ + 'stash-encryption', + 'stash-drizzle', + 'stash-indexing', + 'stash-cli', + ], inlined: [], failed: [], }, @@ -134,13 +140,30 @@ describe('renderSetupPrompt — orient + route (implement mode)', () => { const out = renderSetupPrompt(baseCtx) expect(out).toContain('`stash-encryption`') expect(out).toContain('`stash-drizzle`') + expect(out).toContain('`stash-indexing`') expect(out).toContain('`stash-cli`') // Each skill line should explain what the skill is for, not just name it. + // A skill missing from SKILL_PURPOSES silently renders with no purpose + // line, so every skill SKILL_MAP can install must assert one here. expect(out).toMatch(/`stash-encryption`.*lifecycle/i) expect(out).toMatch(/`stash-drizzle`.*Drizzle/i) + expect(out).toMatch(/`stash-indexing`.*index recipes/i) expect(out).toMatch(/`stash-cli`.*command reference/i) }) + it('has a purpose line for every skill SKILL_MAP can install', () => { + // renderSkillIndex falls back to "(no description)" for a skill missing + // from SKILL_PURPOSES — a silent gap no per-skill assertion catches when + // a new skill lands (stash-prisma-next shipped that way). Render the + // union of every installable skill and require zero fallbacks. + const everyInstallable = [...new Set(Object.values(SKILL_MAP).flat())] + const out = renderSetupPrompt({ + ...baseCtx, + skills: { installed: everyInstallable, inlined: [], failed: [] }, + }) + expect(out).not.toContain('(no description)') + }) + it('points each handoff at the right rule location', () => { const claude = renderSetupPrompt({ ...baseCtx, handoff: 'claude-code' }) const codex = renderSetupPrompt({ ...baseCtx, handoff: 'codex' }) diff --git a/packages/cli/src/commands/init/lib/install-skills.ts b/packages/cli/src/commands/init/lib/install-skills.ts index e55da02c..acd15818 100644 --- a/packages/cli/src/commands/init/lib/install-skills.ts +++ b/packages/cli/src/commands/init/lib/install-skills.ts @@ -11,14 +11,28 @@ import { findBundledDir } from './bundled-paths.js' * `dist/skills/` at build time. */ export const SKILL_MAP: Record = { - drizzle: ['stash-encryption', 'stash-drizzle', 'stash-cli'], - supabase: ['stash-encryption', 'stash-supabase', 'stash-cli'], - 'prisma-next': ['stash-encryption', 'stash-prisma-next', 'stash-cli'], - postgresql: ['stash-encryption', 'stash-cli'], + drizzle: ['stash-encryption', 'stash-drizzle', 'stash-indexing', 'stash-cli'], + supabase: [ + 'stash-encryption', + 'stash-supabase', + 'stash-indexing', + 'stash-cli', + ], + 'prisma-next': [ + 'stash-encryption', + 'stash-prisma-next', + 'stash-indexing', + 'stash-cli', + ], + postgresql: ['stash-encryption', 'stash-indexing', 'stash-cli'], } /** The skills every integration gets — the safe fallback for an unmapped one. */ -const BASE_SKILLS: readonly string[] = ['stash-encryption', 'stash-cli'] +const BASE_SKILLS: readonly string[] = [ + 'stash-encryption', + 'stash-indexing', + 'stash-cli', +] /** * Skills for an integration, resilient to an unmapped one. `SKILL_MAP` is diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 364728ee..ae3fa5aa 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -106,6 +106,10 @@ const SKILL_PURPOSES: Record = { 'Drizzle-specific patterns: declaring encrypted columns, query operators, the rollout/cutover walkthrough for an existing column', 'stash-supabase': 'Supabase-specific patterns: `encryptedSupabase` wrapper, encrypted query filters, transparent decryption, the rollout/cutover walkthrough', + 'stash-prisma-next': + 'Prisma Next-specific patterns: `cipherstash.*` field constructors, migration flow, encrypted query operators', + 'stash-indexing': + 'index recipes for encrypted columns — the `eql_v3` extractor functional indexes, Supabase/managed-Postgres constraints, EXPLAIN verification', 'stash-dynamodb': 'DynamoDB encryption: per-item encrypt/decrypt, HMAC attribute keys, audit logging', 'stash-cli': diff --git a/packages/stack-drizzle/README.md b/packages/stack-drizzle/README.md index e78a27ca..83e184a3 100644 --- a/packages/stack-drizzle/README.md +++ b/packages/stack-drizzle/README.md @@ -60,6 +60,37 @@ comparisons and ordering at a scalar JSONPath leaf. For example, `.orderBy(await ops.selector(users.profile, '$.age').asc())` lowers to `ORDER BY eql_v3.ord_term(...)` over the selected encrypted entry. +### Indexing encrypted columns + +Encrypted predicates only use an index if one exists over the matching +`eql_v3.*` term-extractor expression — otherwise every encrypted query +sequential-scans. `encryptedIndexes` derives the recommended indexes for +every encrypted column in a table; spread it into `pgTable`'s third-argument +callback and `drizzle-kit generate` picks the indexes up like any others: + +```ts +import { integer, pgTable } from 'drizzle-orm/pg-core' +import { encryptedIndexes, types } from '@cipherstash/stack-drizzle/v3' + +export const users = pgTable( + 'users', + { + id: integer('id').primaryKey(), + email: types.TextEq('email'), + bio: types.TextSearch('bio'), + }, + (t) => [...encryptedIndexes(t)], +) +``` + +Each column gets indexes matching its domain's capabilities, named +`
__` (equality btree, ordering btree, free-text +GIN, JSON containment GIN); storage-only and non-encrypted columns get none. +After the migration applies, run `ANALYZE
` — expression indexes have +no statistics until then. For custom names, subsets, or field-level selector +indexes on encrypted JSON, declare individual expression indexes instead; +the bundled `stash-indexing` agent skill has the full recipes. + ## EQL v2 (package root) — legacy The v2 integration predates the typed v3 domains and is kept for existing diff --git a/packages/stack-drizzle/__tests__/v3/exports.test.ts b/packages/stack-drizzle/__tests__/v3/exports.test.ts index 4ad6d98e..2b3b41f7 100644 --- a/packages/stack-drizzle/__tests__/v3/exports.test.ts +++ b/packages/stack-drizzle/__tests__/v3/exports.test.ts @@ -8,6 +8,7 @@ const PUBLIC_SURFACE = [ 'EncryptionOperatorError', 'EqlV3CodecError', 'createEncryptionOperatorsV3', + 'encryptedIndexes', 'extractEncryptionSchemaV3', 'getEqlV3Column', 'isEqlV3Column', diff --git a/packages/stack-drizzle/__tests__/v3/indexes.test.ts b/packages/stack-drizzle/__tests__/v3/indexes.test.ts new file mode 100644 index 00000000..d81a9bc8 --- /dev/null +++ b/packages/stack-drizzle/__tests__/v3/indexes.test.ts @@ -0,0 +1,159 @@ +import type { SQL } from 'drizzle-orm' +import { + getTableConfig, + integer, + PgDialect, + pgTable, +} from 'drizzle-orm/pg-core' +import { describe, expect, it } from 'vitest' +import { encryptedIndexes } from '../../src/v3/indexes' +import { types } from '../../src/v3/types' + +// #753: the integration emitted the encrypted operators but no index DDL, so +// encrypted predicates sequential-scanned by default. `encryptedIndexes` +// derives the functional indexes from the same per-domain capability record +// the operator layer gates on — these tests pin that mapping, the +// `
__` naming, and the exact extractor expressions +// (index engagement is STRUCTURAL: the expression must match what the +// planner inlines the operators to, so a drifted expression builds a real +// index that never engages). + +const dialect = new PgDialect() + +interface IndexConfigView { + name?: string + method?: string + columns: unknown[] +} + +/** The built index configs for a table, keyed by index name. */ +function indexConfigs(table: Parameters[0]) { + const { indexes } = getTableConfig(table) + const byName = new Map() + for (const idx of indexes) { + const config = (idx as unknown as { config: IndexConfigView }).config + expect(config.name).toBeDefined() + byName.set(config.name as string, config) + } + return byName +} + +function renderedExpression(config: IndexConfigView): string { + expect(config.columns).toHaveLength(1) + return dialect.sqlToQuery(config.columns[0] as SQL).sql +} + +describe('encryptedIndexes', () => { + const users = pgTable( + 'users', + { + id: integer('id').primaryKey(), + email: types.TextEq('email'), + createdOn: types.DateOrd('created_on'), + weight: types.IntegerOrdOre('weight'), + title: types.TextOrd('title'), + slug: types.TextOrdOre('slug'), + nickname: types.TextMatch('nickname'), + bio: types.TextSearch('bio'), + profile: types.Json('profile'), + notes: types.Text('notes'), + }, + (t) => encryptedIndexes(t), + ) + + it('emits one index per capability, named
__', () => { + const configs = indexConfigs(users) + expect([...configs.keys()].sort()).toEqual( + [ + // TextEq: hm + 'users_email_eq', + // DateOrd: op only — numeric/date/timestamp ordering terms are + // INJECTIVE, so those `_ord` domains carry no hm, the bundle defines + // no eq_term overload for them, and `eql_v3.eq` inlines to + // `ord_term(a) = ord_term(b)`: the single ordering btree serves + // equality AND range. (Also pins the DB column name, `created_on`, + // over the `createdOn` property.) + 'users_created_on_ord', + // IntegerOrdOre: ob only — same shape, via ord_term_ore + 'users_weight_ord_ore', + // TextOrd: hm + op — TEXT ordering terms are non-injective, so the + // domain carries hm too and equality rides eq_term, not the ordering + // term. Two indexes. + 'users_title_eq', + 'users_title_ord', + // TextOrdOre: hm + ob — same split, ORE flavour + 'users_slug_eq', + 'users_slug_ord_ore', + // TextMatch: bf only + 'users_nickname_match', + // TextSearch: hm + op + bf + 'users_bio_eq', + 'users_bio_ord', + 'users_bio_match', + // Json: ste_vec + 'users_profile_json', + // `id` (plain integer) and `notes` (storage-only types.Text): nothing + ].sort(), + ) + }) + + it('uses btree for equality/ordering and gin for match/json', () => { + const configs = indexConfigs(users) + expect(configs.get('users_email_eq')?.method).toBe('btree') + expect(configs.get('users_created_on_ord')?.method).toBe('btree') + expect(configs.get('users_weight_ord_ore')?.method).toBe('btree') + expect(configs.get('users_nickname_match')?.method).toBe('gin') + expect(configs.get('users_profile_json')?.method).toBe('gin') + }) + + it('builds each index over the matching eql_v3 extractor expression', () => { + const configs = indexConfigs(users) + const cases: Array<[string, string]> = [ + ['users_email_eq', 'eql_v3.eq_term('], + ['users_created_on_ord', 'eql_v3.ord_term('], + ['users_weight_ord_ore', 'eql_v3.ord_term_ore('], + ['users_nickname_match', 'eql_v3.match_term('], + ] + for (const [name, expected] of cases) { + const config = configs.get(name) + expect(config, name).toBeDefined() + expect(renderedExpression(config as IndexConfigView), name).toContain( + expected, + ) + } + }) + + it('rides the jsonb_path_ops opclass inside the json expression', () => { + const configs = indexConfigs(users) + const rendered = renderedExpression( + configs.get('users_profile_json') as IndexConfigView, + ) + expect(rendered).toContain('eql_v3.to_ste_vec_query(') + expect(rendered).toContain('::jsonb) jsonb_path_ops') + }) + + it('references the column, not a bound parameter, in every expression', () => { + // A column interpolated into sql`` must render as an identifier; if it + // ever rendered as a placeholder the index DDL would be unbuildable. + const configs = indexConfigs(users) + for (const [name, config] of configs) { + const query = dialect.sqlToQuery(config.columns[0] as SQL) + expect(query.params, name).toEqual([]) + expect(query.sql, name).toMatch(/"[a-z_]+"/) + } + }) + + it('emits nothing for a table with no encrypted columns', () => { + let captured: unknown[] = [] + pgTable('plain', { id: integer('id').primaryKey() }, (t) => { + captured = encryptedIndexes(t) + return captured + }) + expect(captured).toEqual([]) + }) + + it('ignores non-column values defensively', () => { + expect(encryptedIndexes({})).toEqual([]) + expect(encryptedIndexes({ nope: 42, other: { name: 'x' } })).toEqual([]) + }) +}) diff --git a/packages/stack-drizzle/src/v3/column.ts b/packages/stack-drizzle/src/v3/column.ts index 46af025f..efad7635 100644 --- a/packages/stack-drizzle/src/v3/column.ts +++ b/packages/stack-drizzle/src/v3/column.ts @@ -4,7 +4,8 @@ import { types as v3Types, } from '@cipherstash/stack/eql/v3' import type { Encrypted } from '@cipherstash/stack/types' -import { customType } from 'drizzle-orm/pg-core' +import { is } from 'drizzle-orm' +import { customType, ExtraConfigColumn } from 'drizzle-orm/pg-core' import { v3FromDriver, v3ToDriver } from './codec.js' /** The schema the concrete `eql_v3_*` domains are created in. */ @@ -78,6 +79,35 @@ function getCarrier(column: unknown): EqlV3ColumnCarrier | undefined { function getSqlType(column: unknown): string | undefined { if (!column || typeof column !== 'object') return undefined + + // The extras-callback columns (`pgTable(name, cols, (t) => …)`) are + // ExtraConfigColumn wrappers whose `getSQLType()` in drizzle-orm ≤0.45 is + // `return this.getSQLType()` — unconditional self-recursion (upstream bug), + // so calling it blows the stack on ANY column, encrypted or not. Recover the + // type the way `PgCustomColumn`'s constructor does instead: the wrapper + // shares the real column's config, so `customTypeParams.dataType()` yields + // the same bare domain name. A non-custom column has no `customTypeParams` + // and resolves to undefined — correctly "not an EQL column". + if (is(column, ExtraConfigColumn)) { + // `config` is `protected` on the class, so the read goes through a plain + // structural view of the instance rather than the narrowed class type. + const view: unknown = column + const config = ( + view as { + config?: { + customTypeParams?: { dataType?: (fieldConfig?: unknown) => unknown } + fieldConfig?: unknown + } + } + ).config + const viaCustomType = config?.customTypeParams?.dataType?.( + config?.fieldConfig, + ) + return typeof viaCustomType === 'string' + ? qualifyDomain(viaCustomType) + : undefined + } + const columnAny = column as { getSQLType?: () => unknown dataType?: unknown diff --git a/packages/stack-drizzle/src/v3/index.ts b/packages/stack-drizzle/src/v3/index.ts index 7c0426d4..5e6c042c 100644 --- a/packages/stack-drizzle/src/v3/index.ts +++ b/packages/stack-drizzle/src/v3/index.ts @@ -1,5 +1,6 @@ export { EqlV3CodecError, v3FromDriver, v3ToDriver } from './codec.js' export { getEqlV3Column, isEqlV3Column, makeEqlV3Column } from './column.js' +export { encryptedIndexes } from './indexes.js' export { createEncryptionOperatorsV3, EncryptionOperatorError, diff --git a/packages/stack-drizzle/src/v3/indexes.ts b/packages/stack-drizzle/src/v3/indexes.ts new file mode 100644 index 00000000..0649a090 --- /dev/null +++ b/packages/stack-drizzle/src/v3/indexes.ts @@ -0,0 +1,133 @@ +import { Column, is, type SQL, sql } from 'drizzle-orm' +import { type IndexBuilder, index } from 'drizzle-orm/pg-core' +import { getEqlV3Column } from './column.js' +import { getDrizzleTableName } from './schema-extraction.js' +import { EQL_V3_FN_SCHEMA } from './sql-dialect.js' + +/** + * Derive the recommended functional indexes for every encrypted column in a + * table, from the same per-domain capability record the operator layer gates + * on (`builder.build().indexes`) — so the emitted indexes and the operators + * that engage them cannot drift. + * + * Call it inside `pgTable`'s third-argument callback and spread the result: + * + * ```ts + * import { integer, pgTable } from 'drizzle-orm/pg-core' + * import { encryptedIndexes, types } from '@cipherstash/stack-drizzle/v3' + * + * export const users = pgTable( + * 'users', + * { + * id: integer('id').primaryKey(), + * email: types.TextEq('email'), + * bio: types.TextSearch('bio'), + * }, + * (t) => [...encryptedIndexes(t)], + * ) + * ``` + * + * Emitted per capability, named `
__`: + * + * - equality (`hm` term) → `USING btree (eql_v3.eq_term(col))` + * - ordering (`op` term) → `USING btree (eql_v3.ord_term(col))` + * - ORE ordering (`ob`) → `USING btree (eql_v3.ord_term_ore(col))` + * - free-text match (`bf`) → `USING gin (eql_v3.match_term(col))` + * - encrypted JSON → `USING gin ((eql_v3.to_ste_vec_query(col)::jsonb) jsonb_path_ops)` + * + * A `types.TextSearch` column therefore yields three indexes. Ordering + * columns differ by type: a numeric, date, or timestamp `*Ord` / `*OrdOre` + * column yields a single ordering index that also serves `=` (EQL compares + * those columns by their ordering term directly), while `types.TextOrd` / + * `TextOrdOre` yield an equality index alongside the ordering one (text + * equality runs on a separate term). A storage-only column (bare + * `types.Text`, `types.Boolean`, …) yields none — it has no queryable term — + * and non-encrypted columns are ignored. Field-level selector indexes on + * encrypted JSON cannot be derived here (the selector hash is data the + * crypto layer emits, not schema) — declare those by hand; see the + * `stash-indexing` skill for the recipe. + * + * Existing tables adopt these through a normal `drizzle-kit generate` + * migration; run `ANALYZE
` after it applies — an expression index + * gathers no statistics at `CREATE INDEX` time. The gap this closes (#753): + * the integration emitted operators but no index DDL, so encrypted + * predicates sequential-scanned by default. + */ +export function encryptedIndexes( + columns: Record, +): IndexBuilder[] { + const builders: IndexBuilder[] = [] + for (const [property, column] of Object.entries(columns)) { + if (!is(column, Column)) continue + const dbName = typeof column.name === 'string' ? column.name : property + const eqlColumn = getEqlV3Column(dbName, column) + if (!eqlColumn) continue + + const base = `${tableNameOf(column)}_${dbName}` + const indexes = eqlColumn.build().indexes + if (indexes.unique) { + builders.push( + index(`${base}_eq`).using('btree', extractor('eq_term', column)), + ) + } + if (indexes.ope) { + builders.push( + index(`${base}_ord`).using('btree', extractor('ord_term', column)), + ) + } + if (indexes.ore) { + builders.push( + index(`${base}_ord_ore`).using( + 'btree', + extractor('ord_term_ore', column), + ), + ) + } + if (indexes.match) { + builders.push( + index(`${base}_match`).using('gin', extractor('match_term', column)), + ) + } + if (indexes.ste_vec) { + // GIN over the ste_vec query shape; the opclass rides inside the + // expression because `.op()` exists only for plain column elements. + builders.push( + index(`${base}_json`).using( + 'gin', + sql`(${extractor('to_ste_vec_query', column)}::jsonb) jsonb_path_ops`, + ), + ) + } + } + return builders +} + +/** + * Build the SQL expression each index is defined over: the EQL term extractor + * applied to the column — e.g. `extractor('eq_term', users.email)` renders + * `eql_v3.eq_term("users"."email")`. Encrypted predicates compile to + * comparisons on these same expressions, which is what makes the index + * match. (Same `EQL_V3_FN_SCHEMA` as the query dialect.) + */ +function extractor(fn: string, column: Column): SQL { + return sql`${sql.raw(`${EQL_V3_FN_SCHEMA}.${fn}`)}(${column})` +} + +/** + * Read the name of the table a column belongs to — e.g. the `email` column + * of `pgTable('users', …)` gives `'users'`, so its indexes are named + * `users_email_eq`, `users_email_ord`, … (index names must be unique per + * Postgres schema, hence the table prefix). The owning table is set exactly + * when the helper runs where it belongs: inside the `pgTable` callback. + */ +function tableNameOf(column: Column): string { + const name = getDrizzleTableName((column as { table?: unknown }).table) + if (!name) { + throw new Error( + "[stack-drizzle]: encryptedIndexes could not read the column's table " + + "name. Call it inside pgTable's third-argument callback: " + + 'pgTable(name, columns, (t) => [...encryptedIndexes(t)]).', + ) + } + return name +} diff --git a/packages/wizard/src/__tests__/install-skills.test.ts b/packages/wizard/src/__tests__/install-skills.test.ts index 05122864..7eb60911 100644 --- a/packages/wizard/src/__tests__/install-skills.test.ts +++ b/packages/wizard/src/__tests__/install-skills.test.ts @@ -46,6 +46,7 @@ describe('maybeInstallSkills', () => { expect(result.failed).toEqual([ 'stash-encryption', 'stash-drizzle', + 'stash-indexing', 'stash-cli', ]) expect(warnings()).toContain('Could not create ./.claude/skills/') @@ -62,7 +63,11 @@ describe('maybeInstallSkills', () => { const result = await maybeInstallSkills('/project', 'drizzle') - expect(result.copied).toEqual(['stash-encryption', 'stash-cli']) + expect(result.copied).toEqual([ + 'stash-encryption', + 'stash-indexing', + 'stash-cli', + ]) expect(result.failed).toEqual(['stash-drizzle']) expect(warnings()).toContain('Failed to install skill stash-drizzle') }) diff --git a/packages/wizard/src/lib/install-skills.ts b/packages/wizard/src/lib/install-skills.ts index c9bf1285..474bc2bd 100644 --- a/packages/wizard/src/lib/install-skills.ts +++ b/packages/wizard/src/lib/install-skills.ts @@ -12,10 +12,15 @@ import type { Integration } from './types.js' * follow-up work. */ const SKILL_MAP: Record = { - drizzle: ['stash-encryption', 'stash-drizzle', 'stash-cli'], - supabase: ['stash-encryption', 'stash-supabase', 'stash-cli'], - prisma: ['stash-encryption', 'stash-cli'], - generic: ['stash-encryption', 'stash-cli'], + drizzle: ['stash-encryption', 'stash-drizzle', 'stash-indexing', 'stash-cli'], + supabase: [ + 'stash-encryption', + 'stash-supabase', + 'stash-indexing', + 'stash-cli', + ], + prisma: ['stash-encryption', 'stash-indexing', 'stash-cli'], + generic: ['stash-encryption', 'stash-indexing', 'stash-cli'], } /** diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index f6f8250d..430b4195 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: stash-cli -description: Drive CipherStash setup and encryption migrations through the `stash` CLI — `init`, `plan`, `impl`, `status`, `auth login`, `eql install/upgrade/status`, `db push/validate`, `encrypt backfill/cutover/drop`, `schema build`, and `manifest --json`. Covers the agent / non-interactive interface, the credential rules for `~/.cipherstash`, and the rollout-then-cutover lifecycle. Use when setting up CipherStash EQL in a database, running any `stash` command, creating `stash.config.ts`, or rolling encryption out to production. +description: Drive CipherStash setup and encryption migrations through the `stash` CLI — `init`, `plan`, `impl`, `status`, `auth login`, `eql install/upgrade/status`, `db validate`, `encrypt backfill/cutover/drop`, `schema build`, and `manifest --json`. Covers the agent / non-interactive interface, the credential rules for `~/.cipherstash`, and the rollout-then-cutover lifecycle. Use when setting up CipherStash EQL in a database, running any `stash` command, creating `stash.config.ts`, or rolling encryption out to production. --- # CipherStash CLI (`stash`) @@ -199,7 +199,7 @@ export default defineConfig({ | Option | Required | Default | Purpose | |---|---|---|---| | `databaseUrl` | yes | — | PostgreSQL connection string | -| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `db push`, `db validate`, `schema build`, `encrypt *` | +| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `db validate`, `schema build`, `encrypt *` | Resolved by walking up from `process.cwd()`, like `tsconfig.json`. `stash init` scaffolds it; `stash eql install` offers to. @@ -220,7 +220,7 @@ Seven mechanical steps, no agent handoff. It prompts only when it can't pick a s 1. **Authenticate** — silent when a valid token exists. 2. **Resolve database** — per the resolution order above; verifies the connection. -3. **Resolve proxy choice** — CipherStash Proxy or direct SDK access (the default). Stored as `usesProxy` in `context.json`; this is what decides whether `db push` is part of your flow at all. Set by `--proxy` / `--no-proxy`; non-TTY without a flag defaults to SDK. +3. **Resolve proxy choice** — CipherStash Proxy or direct SDK access (the default). Stored as `usesProxy` in `context.json`. Set by `--proxy` / `--no-proxy`; non-TTY without a flag defaults to SDK. 4. **Build schema** — auto-detects Drizzle (`drizzle.config.*`, `drizzle-orm`/`drizzle-kit`), Supabase (from the `DATABASE_URL` host), and Prisma Next. Writes a placeholder encryption client; prompts only if a file already exists there. 5. **Install dependencies** — one combined prompt for `@cipherstash/stack` and `stash`. Skipped when both are present. 6. **Install EQL** — always EQL v3. **Drizzle** projects generate a v3 install migration (the same output as `eql migration --drizzle`, including the `cs_migrations` tracking schema) so the install lands in your migration history — apply it with `drizzle-kit migrate`; requires `drizzle-kit` to be installed and configured. **Prisma Next** is skipped (it installs EQL via `prisma-next migrate`). Everything else runs `eql install` directly against the resolved database, and is skipped when EQL is already installed. @@ -394,7 +394,7 @@ After writing the migration, `--drizzle` sweeps the output directory for sibling #### `eql upgrade` -The install SQL is idempotent. `upgrade` checks the current version, re-runs it, and reports the new one. If EQL isn't installed it points you at `eql install`. Same `--supabase`, `--exclude-operator-family`, `--eql-version`, `--latest`, `--dry-run`, `--database-url` flags. +The install SQL is safe to re-run — columns and data survive — but it is not fully idempotent: it begins with `DROP SCHEMA IF EXISTS eql_v3 CASCADE`, which cascade-drops any **functional indexes** built on the `eql_v3` extractors (see `stash-indexing`). After an upgrade, recreate them: migration runners skip migrations already recorded as applied, so add a *new* migration that re-issues the `CREATE INDEX` statements (or run the DDL directly), then `ANALYZE` the affected tables. `upgrade` checks the current version, re-runs the install SQL, and reports the new one. If EQL isn't installed it points you at `eql install`. Same `--supabase`, `--exclude-operator-family`, `--eql-version`, `--latest`, `--dry-run`, `--database-url` flags. #### `eql status` @@ -411,42 +411,7 @@ Whether EQL is installed and at which version; database permission status; wheth | No indexes on an encrypted column | Info | | `searchableJson` without `dataType("json")` | Error | -Runs automatically before `db push`, where issues warn but don't block. Exits 1 on errors only. - -#### `db push` / `db activate` — EQL v2 + CipherStash Proxy only - -> **SDK users skip both.** Drizzle, Supabase, and plain-PostgreSQL SDK users keep their encryption config in application code; the database needs no copy. These commands exist for **CipherStash Proxy**, which reads `public.eql_v2_configuration` to know which columns to encrypt and decrypt. `stash init` records this as `usesProxy`. - -```bash -stash db push [--dry-run] -stash db activate -``` - -`db push` loads the encryption client, validates it, maps SDK data types to EQL `cast_as` values, then writes to `eql_v2_configuration`: - -- **No active config** (first push) → writes `active` directly. Encryption is live immediately. -- **Active config exists** → writes `pending`, replacing any prior pending. The active config keeps serving reads until you finalise. - -Finalising a `pending` push: - -| Situation | Command | -|---|---| -| Adding a new encrypted column, no rename | `stash db activate` | -| Cutting over from a `_encrypted` twin | `stash encrypt cutover --table T --column C` | - -`db activate` runs `eql_v2.migrate_config()` then `eql_v2.activate_config()` in one transaction, promoting `pending` → `active` and marking the prior active `inactive`. No physical rename. It errors when there is nothing pending. - -**SDK to EQL type mapping** - -| `dataType()` | EQL `cast_as` | -|---|---| -| `string`, `text` | `text` | -| `number` | `double` | -| `bigint` | `big_int` | -| `boolean` | `boolean` | -| `date` | `date` | -| `timestamp` | `timestamp` | -| `json` | `jsonb` | +Exits 1 on errors only. The "No indexes" Info finding applies to term-carrying (queryable) columns — resolve it with the functional-index recipes in the `stash-indexing` skill. Storage-only columns (bare `types.T`, `types.Boolean`) have no index option by design; for them the finding needs no action. #### `db test-connection` @@ -635,7 +600,7 @@ Required: `SUPERUSER`, **or** `CREATE` on the database *and* on the `public` sch **Supabase.** Always pass `--supabase` (or `supabase: true`). It selects a compatible install script and grants `anon`, `authenticated`, and `service_role`. -**`ORDER BY` on encrypted columns:** on EQL v3, ordering works on OPE-backed columns — Drizzle emits `ORDER BY eql_v3.ord_term(col)`, and the Supabase adapter's `order()` sorts by the `col->op` term. ORE-flavour (`*OrdOre`) domains need a superuser-only operator class (unavailable on managed Postgres/Supabase) and are rejected; storage-only and equality/match-only columns have no ordering term. For those, order by a plaintext column or sort application-side. (The legacy v2 surface — bare `eql_v2_encrypted` — cannot order encrypted columns without operator families.) +**`ORDER BY` on encrypted columns:** on EQL v3, ordering works on OPE-backed columns — Drizzle emits `ORDER BY eql_v3.ord_term(col)`, and the Supabase adapter's `order()` sorts by the `col->op` term. ORE-flavour (`*OrdOre`) domains need a custom operator class the installer creates with `CREATE OPERATOR CLASS` — supported on self-hosted Postgres and on AWS RDS/Aurora, but not on cloud-hosted Supabase (the one confirmed platform whose install role cannot create operator classes; the installer skips the opclass there and disables the `*OrdOre` domains). Storage-only and equality/match-only columns have no ordering term. For those, order by a plaintext column or sort application-side. (The legacy v2 surface — bare `eql_v2_encrypted` — cannot order encrypted columns without operator families.) The ordering extractors are also the index expressions — see the `stash-indexing` skill for the `CREATE INDEX` recipes. **The native binary won't load.** Run `stash doctor`. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 565f5d6e..7a9b3366 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -79,6 +79,8 @@ CREATE TABLE users ( You don't usually hand-write this: the `types.*` factories below emit the domain as the column's SQL type, so `drizzle-kit generate` produces the `ADD COLUMN "email" "eql_v3_text_search"` DDL for you. The generated type is **unqualified** (`eql_v3_text_search`, not `public.eql_v3_text_search`): drizzle-kit wraps a custom type's whole name in one pair of quotes, which would turn a schema-qualified name into the invalid identifier `"public.eql_v3_text_search"`. The bare name resolves via the search path because the domains live in `public` — so keep `public` on the search path (the default), and don't hand-edit the generated type back to a qualified name. +Encrypted predicates need functional indexes over the `eql_v3.*` extractors, and Drizzle does not add them on its own — spread `encryptedIndexes(t)` into the table definition to derive them per column; see [Indexing Encrypted Columns](#indexing-encrypted-columns) below. + ## Schema Definition Use the `types` namespace from `@cipherstash/stack-drizzle/v3` to define encrypted columns. Each factory maps 1:1 to a Postgres domain, and the column's query capabilities are fixed by the type: @@ -256,7 +258,7 @@ const results = await db .orderBy(ops.desc(usersTable.age)) ``` -`ops.asc`/`ops.desc` emit `ORDER BY eql_v3.ord_term(col)` (`ord_term_ore(col)` for the `*OrdOre` domains). The ORE-flavoured domains require a superuser install and are unavailable on managed Postgres (Supabase, RDS, etc.) — prefer the plain `Ord` domains there; ordering works everywhere EQL v3 installs. +`ops.asc`/`ops.desc` emit `ORDER BY eql_v3.ord_term(col)` (`ord_term_ore(col)` for the `*OrdOre` domains). The ORE-flavoured domains require the installer to create a custom operator class — supported on self-hosted Postgres and on AWS RDS/Aurora, but not on cloud-hosted Supabase (the one confirmed platform whose install role cannot create operator classes; the installer disables the `*OrdOre` domains there). Prefer the plain `Ord` domains when unsure; ordering works everywhere EQL v3 installs. ### Encrypted-JSONB Containment (`contains`) @@ -370,6 +372,51 @@ if (!decrypted.failure) { `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. Non-schema fields pass through unchanged. +## Indexing Encrypted Columns + +Drizzle emits the encrypted query operators, but **no index DDL** — without functional indexes over the `eql_v3.*` term extractors, every encrypted predicate sequential-scans. `encryptedIndexes` derives the recommended indexes for every encrypted column in the table from its domain, so a schema column can't be forgotten: + +```typescript +import { encryptedIndexes, types } from "@cipherstash/stack-drizzle/v3" +import { integer, pgTable } from "drizzle-orm/pg-core" + +export const users = pgTable( + "users", + { + id: integer("id").primaryKey(), + email: types.TextEq("email"), + createdAt: types.TimestampOrd("created_at"), + bio: types.TextSearch("bio"), + }, + (t) => [...encryptedIndexes(t)], +) +``` + +The indexes are named `
__` and ride the normal `drizzle-kit generate` → `migrate` flow like any other index. What each column yields is fixed by its domain: + +| Column type | Indexes emitted | +|---|---| +| `types.TextEq` / numeric/date/timestamp `*Eq` | `_eq` — btree on `eql_v3.eq_term` | +| numeric/date/timestamp `*Ord` | `_ord` — btree on `eql_v3.ord_term`; serves `=`, range, and `ORDER BY` (the injective ordering term answers equality — those domains have no `eq_term`) | +| numeric/date/timestamp `*OrdOre` | `_ord_ore` — btree on `eql_v3.ord_term_ore` (needs the ORE opclass — not on Supabase) | +| `types.TextOrd` | `_eq` **+** `_ord` — text ordering terms are non-injective, so equality rides `eq_term` | +| `types.TextOrdOre` | `_eq` **+** `_ord_ore` (needs the ORE opclass — not on Supabase) | +| `types.TextMatch` | `_match` — GIN on `eql_v3.match_term` | +| `types.TextSearch` | `_eq` + `_ord` + `_match` | +| `types.Json` | `_json` — GIN on `(eql_v3.to_ste_vec_query(col)::jsonb) jsonb_path_ops` | +| bare `types.T`, `types.Boolean` | none — storage-only, no term to index | + +To hand-pick instead (custom names, a subset, or a field-level selector index on encrypted JSON — those can't be derived, the selector hash is data, not schema), declare individual expression indexes with the same extractor expressions: + +```typescript +import { sql } from "drizzle-orm" +import { index } from "drizzle-orm/pg-core" + +;(t) => [index("users_email_eq").using("btree", sql`eql_v3.eq_term(${t.email})`)] +``` + +Run `ANALYZE
` after the migration applies — an expression index gathers no statistics at `CREATE INDEX` time. For when to create indexes during a rollout (after backfill, before switching reads), engagement rules, and `EXPLAIN` verification, see the `stash-indexing` skill. + ## Migrating an Existing Column to Encrypted The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. @@ -616,7 +663,7 @@ All comparison/containment operators auto-encrypt their operands and are async; | `asc(col)` | `ORDER BY eql_v3.ord_term(col)` ascending | order/range | | `desc(col)` | `ORDER BY eql_v3.ord_term(col)` descending | order/range | -(`ord_term_ore` for `*OrdOre` domains — superuser-only, unavailable on managed Postgres.) +(`ord_term_ore` for `*OrdOre` domains — needs the ORE opclass; available on RDS/Aurora and self-hosted, not on Supabase.) ### Logical Operators (async, concurrent) diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 506a6043..227df064 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -233,7 +233,7 @@ Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_ **`Ord` vs `OrdOre`:** prefer `Ord`. The `OrdOre` domains are backed by an ORE operator class whose installation requires **superuser** — unavailable on managed Postgres (including Supabase), where the install bundle skips the ORE opclass and disables the `_ord_ore` domains it cannot support. The two ordering flavours produce different, non-cross-comparable terms (`Ord`/`Search` extract via `eql_v3.ord_term`; `OrdOre` via `eql_v3.ord_term_ore`). +> **`Ord` vs `OrdOre`:** prefer `Ord`. The `OrdOre` domains are backed by an ORE operator class the installer creates with the superuser-gated `CREATE OPERATOR CLASS`. Platform support varies — AWS RDS and Aurora allow it; cloud-hosted Supabase does not (the one confirmed platform that refuses it), and there the install bundle skips the ORE opclass and disables the `_ord_ore` domains it cannot support. The two ordering flavours produce different, non-cross-comparable terms (`Ord`/`Search` extract via `eql_v3.ord_term`; `OrdOre` via `eql_v3.ord_term_ore`). **Domain families and plaintext types:** @@ -564,7 +564,7 @@ All values in the array must be non-null. ### On the Wire: Operators and Ordering -Scalar filters compare through each domain's `eql_v3.*` operators (`col = term`, `col > term`, ...), and `ORDER BY` on an encrypted column goes through the ordering extractors — `eql_v3.ord_term(col)` for OPE-backed (`Ord`/`Search`) domains, `eql_v3.ord_term_ore(col)` for `OrdOre`. The Drizzle v3 integration emits all of this for you (including `asc`/`desc`, which emit `ORDER BY eql_v3.ord_term(col)`). Over Supabase/PostgREST, the adapter's `order()` works on OPE-backed ordering columns (plain `*_ord`, `text_ord`, `text_search`) by sorting on the column's `col->op` term; `OrdOre`-flavour (`*_ord_ore`) domains and columns with no ordering term are rejected. See the `stash-drizzle` and `stash-supabase` skills. +Scalar filters compare through each domain's `eql_v3.*` operators (`col = term`, `col > term`, ...), and `ORDER BY` on an encrypted column goes through the ordering extractors — `eql_v3.ord_term(col)` for OPE-backed (`Ord`/`Search`) domains, `eql_v3.ord_term_ore(col)` for `OrdOre`. The Drizzle v3 integration emits all of this for you (including `asc`/`desc`, which emit `ORDER BY eql_v3.ord_term(col)`). Over Supabase/PostgREST, the adapter's `order()` works on OPE-backed ordering columns (plain `*_ord`, `text_ord`, `text_search`) by sorting on the column's `col->op` term; `OrdOre`-flavour (`*_ord_ore`) domains and columns with no ordering term are rejected. See the `stash-drizzle` and `stash-supabase` skills. These same extractor expressions are also what you index — the functional-index recipes are in the `stash-indexing` skill. ## Encrypted JSON (`types.Json`) @@ -859,7 +859,7 @@ Once dual-writes are recorded as live in `cs_migrations`: | Remove dual-write code | The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write logic. | | `stash encrypt drop` | Emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling. | -> **If you use CipherStash Proxy:** After the schema rename, run `stash db push` to register the renamed shape as `pending`. This is required for Proxy-based queries; SDK users skip this step. +**Create the functional indexes between backfill and the read switch** (EQL v3 columns). After `stash encrypt backfill` completes and before reads move to the encrypted column, create the `eql_v3.*` extractor indexes for every queried capability (and `ANALYZE`) — one bulk build instead of per-row maintenance during backfill, and the switched reads engage an index from the first query. Recipes in the `stash-indexing` skill. (Legacy v2 rollouts have no extractor indexes to create — skip this step.) ### State storage diff --git a/skills/stash-indexing/SKILL.md b/skills/stash-indexing/SKILL.md new file mode 100644 index 00000000..10b362d1 --- /dev/null +++ b/skills/stash-indexing/SKILL.md @@ -0,0 +1,273 @@ +--- +name: stash-indexing +description: Create and verify PostgreSQL indexes on EQL v3 encrypted columns — functional-index recipes over the term extractors (eql_v3.eq_term, ord_term, ord_term_ore, match_term, to_ste_vec_query) mapped to the types.* domains, what works without superuser on Supabase and managed Postgres versus the ORE opclass restriction, which domains have no index option, the ORDER BY / GROUP BY query shapes that engage an index, building indexes on large tables, and the EXPLAIN verification checklist. Use when creating or reviewing a schema migration that adds an encrypted column, adding an index to an encrypted column, diagnosing a slow encrypted query or an EXPLAIN plan showing Seq Scan, or answering whether encrypted columns can be indexed on Supabase or managed PostgreSQL. +--- + +# Indexing Encrypted Columns (EQL v3) + +Encrypted columns **can** be indexed, and on any non-trivial table they **should** be. The model is one rule, uniform across every encrypted domain: **index a functional expression over the column's term extractor — never an operator class on the column itself.** The extractors are inlinable SQL functions, so bare-form predicates (`WHERE col = $1`, `WHERE col < $1`, `col @@ $1`) engage the index with no query rewriting. + +This covers EQL v3 — the bundle `stash eql install` applies (`@cipherstash/eql`). An integration that is otherwise correct (encrypted at rest, searchable, exact round-trip) but has no index on its encrypted predicates will sequential-scan every encrypted query; that is the default outcome unless you put these indexes in place — the integrations emit query operators, not index DDL (see [Where the Index DDL Goes](#where-the-index-ddl-goes)). + +## When to Use This Skill + +- Writing or reviewing a schema migration that adds or changes an encrypted (`eql_v3_*`) column. +- Deciding which indexes an encrypted column supports — or explaining why a column has none. +- An encrypted query is slow, or `EXPLAIN` shows a `Seq Scan` where you expected an index. +- `stash db validate` reports "No indexes on an encrypted column". +- Answering whether encrypted columns can be indexed on Supabase or managed PostgreSQL (yes — see the superuser section). + +## Which Columns Support Which Index + +Capability is fixed by the column's domain type, chosen at schema definition via the `types.*` factories. `N` ranges over the numeric-and-time base types `Integer`, `Smallint`, `Bigint`, `Date`, `Timestamp`, `Numeric`, `Real`, `Double`; text is listed separately because its ordering domains behave differently (see the note below the table). `` is the lowercase SQL name (`eql_v3_integer_eq`, `eql_v3_text_ord`, …). + +| Schema factory | Postgres domain | Terms carried | Index recipes | +|---|---|---|---| +| `types.NEq`, `types.TextEq` | `public.eql_v3__eq` | `hm` | equality (`eq_term`) | +| `types.NOrd` | `public.eql_v3__ord` | `op` | **one** ordering index (`ord_term`) — serves equality, range, and `ORDER BY` | +| `types.NOrdOre` | `public.eql_v3__ord_ore` | `ob` | **one** ORE ordering index (`ord_term_ore`) — equality + range; superuser installs only | +| `types.TextOrd` | `public.eql_v3_text_ord` | `hm`, `op` | equality (`eq_term`) **+** ordering (`ord_term`) — two indexes | +| `types.TextOrdOre` | `public.eql_v3_text_ord_ore` | `hm`, `ob` | equality (`eq_term`) **+** ORE ordering (`ord_term_ore`; superuser) | +| `types.TextMatch` | `public.eql_v3_text_match` | `bf` | free-text match (`match_term`) | +| `types.TextSearch` | `public.eql_v3_text_search` | `hm`, `op`, `bf` | equality + ordering/range + match (three indexes) | +| `types.Json` | `public.eql_v3_json_search` | `sv` (ste_vec) | containment GIN + field-level ordering | +| bare `types.N` / `types.Text`, `types.Boolean` | `public.eql_v3_` | none | **none — storage-only by design** | + +**Why the numeric/text split**: the numeric-and-time ordering terms (OPE and ORE alike) are **injective** — distinct plaintexts produce distinct terms — so equality can ride the ordering term. Those domains carry no `hm`, the bundle defines **no `eq_term` overload** for them, and `eql_v3.eq` inlines to `ord_term(a) = ord_term(b)`: one ordering btree serves `=`, range, and `ORDER BY`. Text ordering terms are **non-injective** and cannot be relied on for equality, so `text_ord` / `text_ord_ore` also carry `hm` and answer `=` via `eq_term` — give those columns both indexes. Do not add an `eq_term` index to a numeric `_ord` / `_ord_ore` column; the overload does not exist. + +The last row is deliberate, not a gap: a bare `types.Text` / `types.Integer` / `types.Boolean` column carries no query terms, so there is nothing to index and nothing to query server-side. If a column needs an index, it needs a term-carrying domain first. + +## The Recipes + +Every recipe is a functional index over the extractor, followed by `ANALYZE` (see [Making a Query Engage the Index](#making-a-query-engage-the-index) for why `ANALYZE` is mandatory). Name indexes descriptively (`users_email_eq`, `events_at_ord`) — it makes `EXPLAIN` output and maintenance legible. + +### Equality — `eql_v3.eq_term` + +For the domains carrying `hm`: `_eq`, `text_ord`, `text_ord_ore`, and `text_search`. (On numeric/date/timestamp `_ord` / `_ord_ore` columns, equality rides the ordering index below instead — no `eq_term` overload exists for them.) + +```sql +CREATE INDEX users_email_eq ON users USING btree (eql_v3.eq_term(encrypted_email)); +ANALYZE users; + +SELECT * FROM users WHERE encrypted_email = $1; +-- Index Scan using users_email_eq +-- Index Cond: (eql_v3.eq_term(encrypted_email) = eql_v3.eq_term($1)) +``` + +`btree` is the safe default: it serves `=` exactly as well as `hash` with no query-side cost, and its build scales (see [Building Indexes at Scale](#building-indexes-at-scale)). `USING hash` is fine for small and mid-size tables, but a hash index *build* degrades badly past a few million rows. + +### Ordering and Range — `eql_v3.ord_term` + +For the OPE-backed ordering domains: `_ord` and `text_search`. + +```sql +CREATE INDEX events_at_ord ON events USING btree (eql_v3.ord_term(encrypted_at)); +ANALYZE events; + +SELECT * FROM events WHERE encrypted_at < $1; +``` + +`eql_v3.ord_term` returns a `bytea`-backed domain, so this btree binds PostgreSQL's **default** `bytea_ops` operator class — nothing to install, no privilege required, works on Supabase and managed Postgres. The `<` `<=` `>` `>=` operators inline to comparisons on the extractor, so natural-form range predicates match the index — and on the numeric/date/timestamp `_ord` domains so does `=`, since their injective ordering term also answers equality. One index, every scalar predicate. (A `text_ord` column answers `=` via `eq_term` instead — pair this index with the equality one. `ORDER BY` needs the extractor form — see [Query-Shape Traps](#query-shape-traps).) + +### ORE Ordering — `eql_v3.ord_term_ore` (superuser installs only) + +For the `_ord_ore` domains only: + +```sql +CREATE INDEX events_at_ord_ore ON events USING btree (eql_v3.ord_term_ore(encrypted_at)); +ANALYZE events; +``` + +This one depends on a custom operator class the EQL installer must be privileged enough to create — see [Supabase and Managed Postgres](#supabase-and-managed-postgres-what-actually-needs-superuser) for which platforms allow it and for the failure mode to check. Prefer `types.TOrd` unless you specifically need ORE ordering on a platform whose installer could create the opclass. + +### Free-Text Match — `eql_v3.match_term` + +For the bloom-filter domains: `text_match` and `text_search`. Engages the `@@` match operator: + +```sql +CREATE INDEX users_name_match ON users USING gin (eql_v3.match_term(encrypted_name)); +ANALYZE users; + +SELECT * FROM users WHERE encrypted_name @@ $1; +-- Bitmap Index Scan on users_name_match +``` + +### JSON Containment — `eql_v3.to_ste_vec_query` + +For `public.eql_v3_json_search` (`types.Json`) document containment (`@>`): + +```sql +CREATE INDEX orders_data_gin + ON orders USING gin ((eql_v3.to_ste_vec_query(data_encrypted)::jsonb) jsonb_path_ops); +ANALYZE orders; + +SELECT * FROM orders WHERE data_encrypted @> $1::eql_v3.query_json; +-- Bitmap Index Scan on orders_data_gin +``` + +The needle must be typed — `$1::eql_v3.query_json` or another `public.eql_v3_json_search` value. A bare untyped literal falls through to native `jsonb @>` and skips the index. Note `jsonb_path_ops` indexes `@>` only, not `<@`. + +### Field-Level Ordering Inside Encrypted JSON + +For ordered access to a single field of a `types.Json` document, index the ordering extractor over the path: + +```sql +CREATE INDEX orders_total_ord + ON orders USING btree (eql_v3.ord_term(data_encrypted -> ''::text)); +ANALYZE orders; + +SELECT * FROM orders ORDER BY eql_v3.ord_term(data_encrypted -> ''::text) LIMIT 10; +``` + +- `` is the deterministic selector hash the encryption client emits (each `sv` element's `s` field) — **not** a plaintext JSONPath. To obtain it, encrypt the path through the client — `encryptQuery('$.total', { table, column, queryType: 'steVecSelector' })` — and read the selector from the returned query envelope; it is stable for a given column + path, so it can be pasted into migration DDL. (Cross-check: the same hash appears as `s` in the `sv` entries of any stored row that has the field.) +- The `->` operand must be typed (`::text`); a bare literal falls through to native `jsonb ->`. +- The extracted term is `bytea`-backed like top-level `ord_term` — default btree opclass, no superuser. +- Entry-to-entry `=` / `<>` and exact `GROUP BY` / `DISTINCT` on extracted fields are **not supported** (an extracted entry carries no value selector). Use document containment with the GIN index above for exact field equality. + +## Supabase and Managed Postgres: What Actually Needs Superuser + +**Only one thing on this page needs superuser: the ORE operator class behind `_ord_ore`.** Everything else — equality btree/hash, `_ord`/`text_search` ordering btree, match GIN, JSON containment GIN, field-level ordering — installs and engages with a plain non-superuser role. Do not generalize the ORE warning into "encrypted columns can't be indexed on Supabase"; the default ordering path (`_ord`, via CLLW-OPE) binds Postgres's native `bytea` btree operator class and needs nothing installed. + +The `_ord_ore` restriction, precisely: its btree ordering depends on a hand-written operator class created by the EQL installer, and `CREATE OPERATOR CLASS` is a superuser-gated command in stock PostgreSQL. Whether that blocks ORE is per-platform, not a blanket managed-Postgres rule: **AWS RDS and Aurora fully support it** (their master role can create operator classes), while **cloud-hosted Supabase is the one confirmed platform that refuses it**. Where the install role can't create the opclass, the installer detects this and **disables the `_ord_ore` domains** — using one raises `feature_not_supported` with a hint naming the alternatives. + +**The silent-failure mode to check for:** if an `_ord_ore` column somehow exists without the opclass, `CREATE INDEX … USING btree (eql_v3.ord_term_ore(col))` does **not** fail — PostgreSQL binds the generic `record_ops` instead. The index builds, occupies space, and never engages. Verify which opclass an ORE index actually bound: + +```sql +SELECT i.relname, oc.opcname + FROM pg_index x + JOIN pg_class i ON i.oid = x.indexrelid + JOIN pg_opclass oc ON oc.oid = x.indclass[0] + WHERE i.relname = 'events_at_ord_ore'; +-- ore_block_256_operator_class → ORE ordering, index engages +-- record_ops → opclass was skipped at install; index is inert +``` + +`_ord` has no such failure mode. + +## Making a Query Engage the Index + +Functional-index engagement is **structural**: the planner inlines the operator into the same extractor expression the index was built on and matches the expression trees syntactically. All three of these must hold: + +1. **The value must carry the term the index extracts.** `eq_term` needs `hm`, `ord_term` needs `op`, `ord_term_ore` needs `ob`, `match_term` needs `bf`, containment needs the ste_vec. The domain rows in the table above tell you which terms a column's values carry; a value with only a bloom term will never drive an equality index. +2. **The index must be created after the data carries the term.** If you change which terms a column's values carry (e.g. re-encrypt under a different domain), recreate the index — a functional index built before the term existed will not match. +3. **The query operand must be typed** so the encrypted operator resolves, not the native `jsonb` one. A typed parameter (`$1`) or an explicit cast to the domain works; a bare `::jsonb` literal falls through to native jsonb semantics and skips the index. The Drizzle, Prisma Next, and Supabase integrations emit correctly-typed operands already — this requirement only bites hand-written SQL. + +And after **every** index build: **run `ANALYZE`**. `CREATE INDEX` on an expression gathers no statistics for that expression, so until `ANALYZE` runs the planner has no histogram for `eql_v3.eq_term(col)` and can misjudge — or ignore — the index it just built. + +## Query-Shape Traps + +**The `ORDER BY` sort-key trap.** The planner inlines operators in *predicates*, not *sort keys*: `ORDER BY col` adds a `Sort` node even when the ordering index exists and the `WHERE` clause is using it. To stream rows out of the index already ordered, write the sort key in extractor form: + +```sql +SELECT * FROM events + WHERE encrypted_at < $1 + ORDER BY eql_v3.ord_term(encrypted_at) DESC + LIMIT 10; +``` + +The natural-form Top-N sort scales linearly with the rows passing `WHERE`; at scale that is the difference between seconds and milliseconds. The Drizzle integration's `asc`/`desc` already emit `ORDER BY eql_v3.ord_term(col)` for you. + +**The `value::jsonb` projection trap.** `SELECT col::jsonb … ORDER BY col` folds the cast into the scan and sorts on `(col)::jsonb` — which matches no index. Project the column raw, wrap the ordered query in a subquery and cast outside the `LIMIT`, or sidestep it entirely with `ORDER BY eql_v3.ord_term(col)`. + +**`GROUP BY` / `DISTINCT` on the extractor, not the raw column.** `GROUP BY col` hashes the entire encrypted payload (1–2 KB per row); the estimated hash table blows past `work_mem`, so the planner falls back to `GroupAggregate` — sorting kilobyte rows and spilling to disk. Group on the term instead: + +```sql +SELECT eql_v3.eq_term(encrypted_email), count(*) + FROM users + GROUP BY eql_v3.eq_term(encrypted_email); +``` + +The term is small and deterministic, so `HashAggregate` fits in `work_mem` with no tuning. If an ORM insists on grouping the raw column, raising `work_mem` is the rescue knob — but the extractor form is the design. + +Pick the extractor the domain actually has: `eq_term` on the `hm`-carrying domains (`types.*Eq`, `types.TextOrd*`, `types.TextSearch`). The numeric/date/timestamp `types.*Ord` / `*OrdOre` domains have **no** `eq_term` — group on `eql_v3.ord_term(col)` (or `ord_term_ore(col)`); their ordering term is injective, so it is an exact grouping key, and the ordering btree covers it. + +## Building Indexes at Scale + +Query performance and *build* performance are separate axes; on large encrypted tables the build is the one that bites. + +- **Raise `maintenance_work_mem` for the build session** — the single highest-leverage knob. The 64 MB default spills a multi-million-row build to disk early: + + ```sql + SET maintenance_work_mem = '2GB'; + CREATE INDEX ...; + ANALYZE ...; + ``` + +- **Prefer `btree` over `hash` for equality at scale.** Build characteristics differ sharply: + + | Access method | Build | Scales past cache? | Parallel build? | + |---|---|---|---| + | btree | sort, then sequential bulk-load | yes | yes | + | GIN | batched buffer build | yes | no | + | hash | random bucket fill | **no** | no | + + A hash build scatters rows to random buckets; once the index outgrows cache it goes random-I/O-bound (a 10M-row hash build has been observed to stall after 17 hours; the btree equivalent built without drama). A btree on `eql_v3.eq_term(col)` serves `=` identically. + +- **The de-TOAST floor.** A functional index build de-TOASTs the whole stored value once per row to evaluate the extractor — for large `eql_v3_json_search` documents this sets an unavoidable floor on build rate, identical across access methods. Run large builds on fast native storage (containerized Postgres on a virtualized filesystem — e.g. Docker Desktop on macOS — is the worst case). + +- **Diagnose a slow build** from a second session: + + ```sql + SELECT phase, tuples_done, tuples_total, + round(100.0 * tuples_done / nullif(tuples_total, 0), 1) AS pct + FROM pg_stat_progress_create_index; + ``` + + A steady `tuples_done` rate is healthy; a rate that decays over time is the cache/memory wall — raise `maintenance_work_mem`, and if it's a hash index, rebuild as btree. + +## Verifying with EXPLAIN + +The first move on any slow encrypted query is `EXPLAIN (COSTS OFF)`: + +- ✓ `Index Scan using ` — the functional index is engaged. +- ✓ `Bitmap Index Scan on ` — same, for set-style predicates (`@@`, `@>`). +- ✓ `Index Cond:` referencing the extractor (`eql_v3.eq_term(…)`, `eql_v3.ord_term(…)`) — the inlined predicate matched. +- ✗ `Seq Scan` — no index used; work through [Troubleshooting](#troubleshooting). +- ✗ `Filter:` showing the raw operator (`col < '…'`) — inlining did not happen. Usual causes: a pinned `search_path` on a customized extractor function, a `plpgsql` body where a `sql` one is expected, or the planner genuinely judging another plan cheaper. +- ✗ A `Sort` node above an Index Scan — natural-form `ORDER BY`; switch the sort key to the extractor form. + +Once the plan shape is right, `EXPLAIN ANALYZE` for actual timings. + +## Troubleshooting + +Index not being used: + +1. **Verify the value carries the term:** + + ```sql + SELECT encrypted_email::jsonb ? 'hm' AS has_hmac, + encrypted_email::jsonb ? 'op' AS has_ope, + encrypted_email::jsonb ? 'ob' AS has_ore_block, + encrypted_email::jsonb ? 'bf' AS has_bloom + FROM users LIMIT 1; + ``` + +2. **Verify the operand is typed** (`$1` or `$1::eql_v3.query_text_eq` — not `$1::jsonb`, and not the column domain `public.eql_v3_text_eq`: query payloads are term-only, and the column domains' CHECK requires the ciphertext key `c` that query payloads deliberately omit). +3. **Recreate the index** if the column's term composition changed after it was built. +4. **Run `ANALYZE`.** Also note: on very small tables a `Seq Scan` is the *correct* plan — don't chase it below a few thousand rows. + +**`=` returns zero rows**: equality needs the domain's equality-serving term — `hm` where the domain carries it (`_eq`, `text_ord`, `text_ord_ore`, `text_search`), the injective ordering term (`op` / `ob`) on the numeric/date/timestamp `_ord` / `_ord_ore` domains. A bare storage-only domain has neither; confirm the column's domain and that the client is emitting the term. + +**ORE index never engages:** run the `pg_opclass` query from the [superuser section](#supabase-and-managed-postgres-what-actually-needs-superuser) — a `record_ops` binding means the index is inert. + +## Where the Index DDL Goes + +**The integrations emit the query operators for you — none applies index DDL on its own. Making sure these indexes exist is always your job.** This skill is the general model — recipes, engagement rules, verification. How to apply it in a specific integration lives in that integration's skill: + +- **Drizzle** — `encryptedIndexes(t)` from `@cipherstash/stack-drizzle/v3` derives the recommended indexes for every encrypted column in the table, or declare individual expression indexes in the schema DSL. See `stash-drizzle` § Indexing Encrypted Columns. +- **Prisma Next** — Prisma's schema language cannot express functional indexes; the DDL goes in a migration in the adapter's flow. See `stash-prisma-next`. +- **Supabase** — a `supabase/migrations/` file; no superuser needed (see above). See `stash-supabase`. +- **Raw SQL / plain PostgreSQL** — the recipes in this skill, in whatever migration tool owns the schema. Never ad-hoc in production. + +## When to Create Indexes During an Encryption Rollout + +- **Fresh encrypted column (new table or new field):** ship the `CREATE INDEX` in the **same migration** that adds the column. Every value written carries its terms from day one, so the index is correct from the first row. +- **Encrypting an existing column** (the `stash encrypt` lifecycle): create the indexes **after `stash encrypt backfill` completes and before switching reads** to the encrypted column. Building after backfill is one bulk pass instead of per-row index maintenance across the whole backfill, and the reads you cut over to engage an index from the first query. Remember `ANALYZE` after the build. See `stash-encryption` § "Rolling Encryption Out to Production" for the full lifecycle. + +**These indexes do not survive an EQL reinstall or upgrade.** The install SQL begins with `DROP SCHEMA IF EXISTS eql_v3 CASCADE`, and every functional index on an extractor depends on that schema — so `stash eql upgrade` (or `eql install --force`, or re-applying the bundle by hand) cascade-drops all of them. Columns and data are untouched (the `public.eql_v3_*` domains deliberately don't depend on the `eql_v3` schema); only the indexes vanish, and queries fall back to sequential scans without erroring. After any EQL upgrade or reinstall, recreate them — migration runners skip already-applied migrations, so add a *new* migration re-issuing the `CREATE INDEX` statements (or run the DDL directly), `ANALYZE`, and confirm with the `EXPLAIN` checklist above. + +## Reference + +- `stash-encryption` — the `types.*` domain catalog, wire-format operators and ordering, and the rollout/cutover lifecycle. +- `stash-cli` — `stash eql install`, `stash db validate` (its "No indexes on an encrypted column" Info finding is resolved by this skill), `stash encrypt backfill` / `cutover`. +- `stash-drizzle`, `stash-supabase`, `stash-prisma-next` — per-integration query patterns; index DDL placement per the section above. diff --git a/skills/stash-prisma-next/SKILL.md b/skills/stash-prisma-next/SKILL.md index 702ade86..a19bae60 100644 --- a/skills/stash-prisma-next/SKILL.md +++ b/skills/stash-prisma-next/SKILL.md @@ -143,6 +143,65 @@ standalone installer for exactly this reason. The CLI enforces this: `stash eql install` detects a Prisma Next project and refuses (pointing you at `prisma-next migrate`) unless you pass `--force`. +## Indexing encrypted columns + +The adapter emits the encrypted query operators, but **no index DDL** — without +functional indexes over the `eql_v3.*` extractors, every encrypted predicate +sequential-scans. Two facts shape where the DDL goes: + +- **`schema.prisma` cannot express functional indexes** (`@@index` takes + fields, not expressions), so the schema file is not an option. +- Prisma Next migrations execute **raw SQL operations**, so an index migration + is just an operation whose statements are the `CREATE INDEX` recipes — + authored in the same migration history that installs the EQL bundle, applied + by the same `prisma-next migrate`. Never run index DDL out-of-band. + +One index per capability the column's domain carries: + +```sql +-- cipherstash.TextEq / TextSearch: equality +CREATE INDEX users_email_eq ON users USING btree (eql_v3.eq_term(email)); +-- cipherstash.*Ord / TextSearch: ordering + range (on numeric/date/timestamp +-- _ord domains this one index serves = too; TextOrd needs the eq_term index +-- above as well) +CREATE INDEX users_created_at_ord ON users USING btree (eql_v3.ord_term(created_at)); +-- cipherstash.TextMatch / TextSearch: free-text match +CREATE INDEX users_bio_match ON users USING gin (eql_v3.match_term(bio)); +-- cipherstash.Json: containment +CREATE INDEX users_profile_json + ON users USING gin ((eql_v3.to_ste_vec_query(profile)::jsonb) jsonb_path_ops); + +ANALYZE users; +``` + +The `ANALYZE` is part of the recipe — an expression index has no statistics +until it runs. Works as a non-superuser role (Supabase included); only the +ORE-flavour (`_ord_ore`) ordering opclass is superuser-gated. For the full +model — which domains take which index, engagement rules, `EXPLAIN` +verification, rollout timing — see the `stash-indexing` skill. + +In a migration, the recipes ride a raw-SQL operation (`rawSql` from +`@prisma-next/postgres/migration`) in the migration's `operations`: + +```typescript +rawSql({ + id: 'index.users.encrypted', + label: 'Index encrypted columns on users', + operationClass: 'additive', + target: { + id: 'postgres', + details: { schema: 'public', objectType: 'index', name: 'users_email_eq', table: 'users' }, + }, + precheck: [], + execute: [ + { description: 'equality index', + sql: 'CREATE INDEX IF NOT EXISTS users_email_eq ON "public"."users" USING btree (eql_v3.eq_term(email))' }, + { description: 'refresh statistics', sql: 'ANALYZE "public"."users"' }, + ], + postcheck: [], +}) +``` + ## Writing and reading encrypted values At the value boundary you wrap plaintext in a **runtime envelope** (primitive-named, diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index a0ebe696..57f7099d 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -74,6 +74,38 @@ No **Exposed schemas** change is needed: the column domains and their operators live in `public`, so bare `col = term` filters resolve under Supabase's default PostgREST configuration. Do not expose `eql_v3_internal`. +### Indexing encrypted columns (no superuser needed) + +Encrypted columns can and should be **indexed** on Supabase. Index creation +needs no superuser — only the ORE opclass behind the `_ord_ore` domains is +restricted (and those domains are disabled on non-superuser installs anyway); +the default equality / ordering / match / containment indexes all install as +a normal role. Do not read the ORE warning as "encrypted columns can't be +indexed on Supabase." + +Put the `CREATE INDEX` statements in a `supabase/migrations/` file, one index +per capability the column's domain carries: + +```sql +-- eql_v3_text_eq / eql_v3_text_search: equality +CREATE INDEX users_email_eq ON users USING btree (eql_v3.eq_term(email)); +-- eql_v3__ord / eql_v3_text_search: ordering + range (on numeric/date/ +-- timestamp _ord domains this one index serves = too; text_ord needs the +-- eq_term index above as well) +CREATE INDEX users_created_at_ord ON users USING btree (eql_v3.ord_term(created_at)); +-- eql_v3_text_match / eql_v3_text_search: free-text match +CREATE INDEX users_bio_match ON users USING gin (eql_v3.match_term(bio)); +-- eql_v3_json_search: containment +CREATE INDEX users_profile_json + ON users USING gin ((eql_v3.to_ste_vec_query(profile)::jsonb) jsonb_path_ops); + +ANALYZE users; +``` + +The `ANALYZE` is part of the recipe — an expression index has no statistics +until it runs. For the full model (which domains take which index, engagement +rules, `EXPLAIN` verification, rollout timing), see the `stash-indexing` skill. + ### 2. Database schema (per-domain columns) Each encrypted column is declared with a concrete `public.eql_v3_*` domain —