diff --git a/README.md b/README.md index dfab227..5ba630c 100644 --- a/README.md +++ b/README.md @@ -292,7 +292,7 @@ auto report = recovery.recover(); │ ▼ │ ResultSet │ - └─── Tier 2 (lightweight): DDL, transactions, SHOW, GRANT, USE ... + └─── Tier 2 (lightweight): DDL, COMMIT/ROLLBACK, SHOW, GRANT, USE ... ``` ### Execution layer @@ -330,7 +330,7 @@ auto report = recovery.recover(); ### Parser -- **Tier 1 deep parse:** SELECT, INSERT, UPDATE, DELETE, SET, REPLACE, EXPLAIN, CALL, DO, LOAD DATA +- **Tier 1 deep parse:** SELECT, INSERT, UPDATE, DELETE, SET, REPLACE, EXPLAIN, CALL, DO, LOAD DATA, BEGIN, START TRANSACTION - **Compound queries:** UNION / INTERSECT / EXCEPT with SQL-standard precedence and parenthesized nesting - **CTEs:** `WITH ... [RECURSIVE] AS (...)` — non-recursive materialized, recursive planned - **Window functions:** ROW_NUMBER, RANK, DENSE_RANK, SUM/COUNT/AVG/MIN/MAX OVER (PARTITION BY ... ORDER BY ...) @@ -341,7 +341,7 @@ auto report = recovery.recover(); - **Query reconstruction:** Parse → modify AST → emit valid SQL (round-trip) - **Digest:** Normalize for fingerprinting (literals → `?`, IN-list collapse, keyword upper-case) + 64-bit FNV-1a hash - **Prepared-statement cache:** LRU keyed by SQL text; one parser arena per cached plan -- **Tier 2 classification** for all other statements (DDL, transactions, SHOW, GRANT, ...) +- **Tier 2 classification** for all other statements (DDL, COMMIT/ROLLBACK/SAVEPOINT, SHOW, GRANT, ...) ### Query engine diff --git a/docs/superpowers/specs/2026-08-16-transaction-statement-node-design.md b/docs/superpowers/specs/2026-08-16-transaction-statement-node-design.md new file mode 100644 index 0000000..eb78aec --- /dev/null +++ b/docs/superpowers/specs/2026-08-16-transaction-statement-node-design.md @@ -0,0 +1,202 @@ +# Transaction Statement Node — Design Specification + +## Overview + +Promotes `BEGIN` and `START TRANSACTION` from Tier 2 to Tier 1 and introduces `NODE_TRANSACTION_STMT`, so that the transaction characteristics they carry (`READ ONLY`, `READ WRITE`, `ISOLATION LEVEL ...`) survive the parse. Today no mode reaches the AST, so a consumer has no structured way to tell a read-only transaction from a writable one. + +### Goals + +- **Every transaction mode reaches the AST:** `READ ONLY` / `READ WRITE` in both dialects, plus PostgreSQL's four isolation levels and `[NOT] DEFERRABLE` and MySQL's `WITH CONSISTENT SNAPSHOT`. A mode no consumer reads is still parsed, so it cannot hide a mode behind it. +- **`BEGIN` distinguishable from `BEGIN READ ONLY`:** the node is emitted even with no modes, so its absence is not overloaded to mean "no modes". +- **Digest stability:** modes are stored and re-emitted under their canonical spelling, so the digest text is unchanged for canonically written input, and casing, internal spacing, and PostgreSQL's optional commas all normalize onto that one form instead of producing a digest each. No mode is dropped, so no transaction form shortens its own digest text. +- **No behavior change for `COMMIT` / `ROLLBACK` / `SAVEPOINT`:** they stay Tier 2. + +### Constraints + +- **Each dialect accepts only its own grammar:** a mode one dialect does not define is not parsed for it, so the node never asserts semantics for a statement the server would reject. +- `BEGIN` is a benchmarked statement (README publishes 29 ns); added cost must stay well inside that. +- `scan_to_end()` must still run, so multi-statement `remaining` handling is untouched. + +--- + +## Problem + +`extract_transaction()` classifies `BEGIN`, `START TRANSACTION`, `COMMIT`, `ROLLBACK`, and `SAVEPOINT` as Tier-2 statements: it sets `stmt_type`, then calls `scan_to_end()` to consume the rest of the input without parsing it. No AST node is produced. + +For `COMMIT`, `ROLLBACK`, and `SAVEPOINT` that is sufficient; the statement type carries all the meaning. For the two transaction-*starting* statements it is not, because the modes that follow decide whether the transaction may write (needed for our read-only classification work). No mode is parsed, so the statement type and the AST are the same whether the transaction can write or not. The modes survive only as an unparsed text tail, which also clears `full_input`: + +| input | status | stmt_type | ast | full_input | remaining | +|---|---|---|---|---|---| +| `BEGIN` | OK | BEGIN | `nullptr` | true | `""` | +| `BEGIN READ ONLY` | OK | BEGIN | `nullptr` | false | `"READ ONLY"` | +| `START TRANSACTION READ ONLY` | OK | START_TRANSACTION | `nullptr` | false | `"READ ONLY"` | + +`SET TRANSACTION READ ONLY` — the same characteristics on a different statement — *is* parsed, producing `NODE_SET_TRANSACTION` with the mode as an identifier child (`set_parser.h`, `parse_set_transaction()`). The grammar is already implemented, it is simply not reachable from the two statements where it matters most. + +### Motivating consumer + +ProxySQL routes queries to backend hostgroups. Routing a read-only transaction to a read replica requires knowing, at `BEGIN` time, that the transaction cannot write. A transaction is the unit a pooler would pin to a replica, so this is the case that matters most for that feature. + +--- + +## Chosen Approach + +Promote `BEGIN` and `START TRANSACTION` to Tier 1, with a dedicated `NODE_TRANSACTION_STMT` whose children are the parsed modes as `NODE_IDENTIFIER` nodes. + +The node is produced even when no modes are present, so `BEGIN` yields an empty `NODE_TRANSACTION_STMT` rather than `nullptr`. That makes "plain `BEGIN`" and "`BEGIN READ ONLY`" distinguishable, and keeps the node's presence a property of the statement type rather than of its arguments. + +### Classifier Updates + +The switch in `Parser::classify_and_dispatch()`: + +- `TK_BEGIN` → `parse_transaction()` (was `extract_transaction()`) +- `TK_START` → `parse_transaction()` (was `extract_transaction()`) +- `TK_COMMIT`, `TK_ROLLBACK`, `TK_SAVEPOINT` → `extract_transaction()` (unchanged) + +Those three take no arguments, so Tier 2 remains correct for them — the five only shared a +function because they shared a *lack* of parsing. + +--- + +## New NodeType Additions + +```cpp +// TRANSACTION +NODE_TRANSACTION_STMT, +``` + +Flags for a `NODE_TRANSACTION_STMT` mode child: + +```cpp +static constexpr uint16_t FLAG_TXN_MODE_ISOLATION = 0x01; +``` + +The above flag is set on a mode child that is an isolation level, so the emitter re-inserts the `ISOLATION LEVEL` keywords the parser consumed. Without it the emitter would have to guess from the value, which breaks once modes other than the access modes are stored. + +## New Token Additions + +**None.** Every mode this spec parses is spelled with existing tokens: `TK_READ`, `TK_ONLY`, `TK_WRITE`, `TK_ISOLATION`, `TK_LEVEL`, `TK_SERIALIZABLE`, `TK_REPEATABLE`, `TK_COMMITTED`, `TK_UNCOMMITTED`, `TK_COMMA`, `TK_TRANSACTION`, plus `TK_NOT` for `NOT DEFERRABLE` and `TK_WITH` for `WITH CONSISTENT SNAPSHOT`. + +`WORK` has no token either and is matched on identifier text, as `set_parser.h` already does for `CHARACTERISTICS` and `AUTHORIZATION`. It is a pure noise word — PostgreSQL's `opt_transaction` production carries no semantic action, so the server discards it too — which is why it needs no form flag and re-emits as plain `BEGIN`. + +`DEFERRABLE`, `CONSISTENT` and `SNAPSHOT` are matched the same way. Unlike `WORK` they are not noise words, so they are stored as mode children and reproduced by the emitter, each is gated by `if constexpr` to the dialect whose grammar has it: `[NOT] DEFERRABLE` for PostgreSQL, `WITH CONSISTENT SNAPSHOT` for MySQL. + +--- + +## PostgreSQL Syntax + +``` +BEGIN [ WORK | TRANSACTION ] [ transaction_mode [, ...] ] +START TRANSACTION [ transaction_mode [, ...] ] + +transaction_mode: + ISOLATION LEVEL { SERIALIZABLE | REPEATABLE READ | READ COMMITTED | READ UNCOMMITTED } + READ WRITE | READ ONLY + [ NOT ] DEFERRABLE +``` + +Modes are comma-separated, though PostgreSQL accepts them with the commas omitted, so the parser treats the separator as optional. `WORK` and `TRANSACTION` are noise words after `BEGIN`. + +## MySQL Syntax + +``` +BEGIN [ WORK ] +START TRANSACTION [ transaction_characteristic [, ...] ] + +transaction_characteristic: + WITH CONSISTENT SNAPSHOT + READ WRITE | READ ONLY +``` + +MySQL has no `DEFERRABLE`, and sets the isolation level through `SET TRANSACTION` rather than on `START TRANSACTION`. `WITH CONSISTENT SNAPSHOT` is MySQL-only. + +One mode loop serves both dialects, but only `READ ONLY` / `READ WRITE` are common to them. Everything else is gated by `if constexpr` to the dialect whose grammar has it: + +| construct | PostgreSQL | MySQL | +|---|---|---| +| `READ ONLY` / `READ WRITE` | after `BEGIN` or `START TRANSACTION` | after `START TRANSACTION` only | +| `ISOLATION LEVEL …` | ✅ | ✗ | +| `[NOT] DEFERRABLE` | ✅ | ✗ | +| `WITH CONSISTENT SNAPSHOT` | ✗ | ✅ | +| `TRANSACTION` after `BEGIN` | ✅ | ✗ (`WORK` only) | +| any mode after bare `BEGIN` | ✅ | ✗ | +| comma between modes | optional | required | + +A construct the dialect does not define terminates the loop and falls to `scan_to_end()`, so it lands in `remaining` rather than becoming a mode child. This follows `set_parser.h`, which gates `SET LOCAL`, `SET ROLE`, `SET CONSTRAINTS`, `SET SCHEMA`, `SET SEED` and `SET TIME ZONE` to PostgreSQL for the same reason: a shared parse would emit a node for syntax the other server rejects. + +--- + +## AST Structure + +``` +BEGIN READ ONLY (PostgreSQL) +└── NODE_TRANSACTION_STMT "BEGIN" + └── NODE_IDENTIFIER "READ ONLY" + +BEGIN ISOLATION LEVEL READ COMMITTED, READ ONLY (PostgreSQL) +└── NODE_TRANSACTION_STMT "BEGIN" + ├── NODE_IDENTIFIER "READ COMMITTED" flags = FLAG_TXN_MODE_ISOLATION + └── NODE_IDENTIFIER "READ ONLY" + +BEGIN ISOLATION LEVEL SERIALIZABLE, READ ONLY, DEFERRABLE (PostgreSQL) +└── NODE_TRANSACTION_STMT "BEGIN" + ├── NODE_IDENTIFIER "SERIALIZABLE" flags = FLAG_TXN_MODE_ISOLATION + ├── NODE_IDENTIFIER "READ ONLY" + └── NODE_IDENTIFIER "DEFERRABLE" + +START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY (MySQL) +└── NODE_TRANSACTION_STMT "START TRANSACTION" + ├── NODE_IDENTIFIER "WITH CONSISTENT SNAPSHOT" + └── NODE_IDENTIFIER "READ ONLY" + +START TRANSACTION READ ONLY (both) +└── NODE_TRANSACTION_STMT "START TRANSACTION" + └── NODE_IDENTIFIER "READ ONLY" + +START TRANSACTION (both) +└── NODE_TRANSACTION_STMT "START TRANSACTION" +``` + +Mode values are stored under their **canonical spelling**, not as a span of the input, so a consumer can compare a child against `"READ ONLY"` without first normalizing case or internal whitespace. `select_parser.h` already does this for `NOWAIT` and `SKIP LOCKED`. Spanning the source instead would make `READ ONLY` a different string from `READ ONLY`, which every consumer would then have to work around. + +`ISOLATION LEVEL` is not stored, the level alone is — the same storage convention as `NODE_SET_TRANSACTION`. That node's emitter recovers the keywords by inference: anything that is not `READ ONLY` or `READ WRITE` is assumed to be an isolation level. This node records it instead, on `FLAG_TXN_MODE_ISOLATION`, because the inference breaks as soon as a mode that is neither is stored, which `[NOT] DEFERRABLE` and `WITH CONSISTENT SNAPSHOT` are. + +### Recording the introducing keywords + +`stmt_type` distinguishes `BEGIN` from `START TRANSACTION`, but not `BEGIN` from `BEGIN TRANSACTION`. The emitter needs that to round-trip, so the introducing keywords are stored as the node's **value**, under their canonical spelling — following `NODE_SET_OPERATION`, which likewise holds its mutually exclusive operator (`UNION` / `INTERSECT` / `EXCEPT`) in the value and reserves `flags` for the independent `ALL` modifier. A canonical literal rather than a source span keeps casing out of the digest: `begin read only` and `BEGIN READ ONLY` must normalize to the same digest text, as they did when the statement had no AST and fell through to the token-level digest path, which uppercases keyword tokens. + +--- + +## Emitter Extensions + +One new method, `emit_transaction_stmt()`, plus its dispatch case. It writes the node's value — the introducing keywords — then each mode. A single mode is emitted without a comma so the common forms round-trip exactly; multiple modes are comma-separated, which PostgreSQL accepts and MySQL requires. No case folding happens here as the parser already stored each mode canonically. + +| input | emitted | dialect | +|---|---|---| +| `BEGIN` | `BEGIN` | both | +| `begin read only` | `BEGIN READ ONLY` | PostgreSQL | +| `BEGIN TRANSACTION READ ONLY` | `BEGIN TRANSACTION READ ONLY` | PostgreSQL | +| `START TRANSACTION READ WRITE` | `START TRANSACTION READ WRITE` | both | +| `BEGIN ISOLATION LEVEL SERIALIZABLE` | `BEGIN ISOLATION LEVEL SERIALIZABLE` | PostgreSQL | + +--- + +## Scope Boundaries + +`COMMIT`, `ROLLBACK`, and `SAVEPOINT` keep their Tier-2 treatment and produce no AST. Their statement type is their entire meaning. + +`scan_to_end()` still runs after the modes are parsed, so multi-statement handling is unaffected: `BEGIN READ ONLY; SELECT 1` continues to report `remaining = "SELECT 1"`, and `full_input` stays false for it while every single-statement form sets it. + +An input that starts a mode without completing it — `BEGIN READ`, `BEGIN ISOLATION LEVEL` — reports `PARTIAL`, matching how the Tier-1 parsers treat unexpected EOF. + +--- + +## Implementation + +1. `NODE_TRANSACTION_STMT` and `FLAG_TXN_MODE_ISOLATION` in `common.h`. +2. `parse_transaction()` in `parser.cpp`, declared in the Tier-1 block of `parser.h`; `TK_BEGIN` / `TK_START` routed to it from `classify_and_dispatch()` and removed from `extract_transaction()`. +3. `parse_transaction_modes(ParseResult&, StringRef)` as the mode loop, following the Tier-1 conventions: `ERROR` if the node cannot be allocated, `PARTIAL` on an incomplete mode. Each mode is stored under its canonical spelling, as `select_parser.h` already does for `NOWAIT` and `SKIP LOCKED`, so a consumer never has to normalize before comparing. +4. `emit_transaction_stmt()` and its dispatch case in `emitter.h`. +5. Tests in `tests/test_misc_stmts.cpp`, beside the other Tier-1 statements that live in `parser.cpp`, plus digest coverage in `tests/test_digest.cpp`. + +--- \ No newline at end of file diff --git a/include/sql_parser/common.h b/include/sql_parser/common.h index 7cc651d..3bb4ae0 100644 --- a/include/sql_parser/common.h +++ b/include/sql_parser/common.h @@ -66,6 +66,11 @@ static constexpr uint16_t FLAG_SET_OP_ALL = 0x01; // which matters for SHOW search_path / SHOW canonical re-emission. static constexpr uint16_t FLAG_IDENT_DELIMITED = 0x01; +// -- Flags for a NODE_TRANSACTION_STMT mode child -- +// Set when the mode is an isolation level, so the emitter re-inserts the +// ISOLATION LEVEL keywords the parser consumed. +static constexpr uint16_t FLAG_TXN_MODE_ISOLATION = 0x01; + // -- Statement type (always set, even for PARTIAL/ERROR) -- enum class StmtType : uint8_t { @@ -237,6 +242,9 @@ enum class NodeType : uint16_t { NODE_USER_VARIABLE, NODE_LITERAL_HEX, NODE_LITERAL_BIT, + + // TRANSACTION + NODE_TRANSACTION_STMT, }; } // namespace sql_parser diff --git a/include/sql_parser/emitter.h b/include/sql_parser/emitter.h index 39f6596..46c9746 100644 --- a/include/sql_parser/emitter.h +++ b/include/sql_parser/emitter.h @@ -100,6 +100,9 @@ class Emitter { case NodeType::NODE_LOAD_DATA_STMT: emit_load_data_stmt(node); break; case NodeType::NODE_LOAD_DATA_OPTIONS: /* emitted inline */ break; + // ---- TRANSACTION ---- + case NodeType::NODE_TRANSACTION_STMT: emit_transaction_stmt(node); break; + // ---- UPDATE statement ---- case NodeType::NODE_UPDATE_STMT: emit_update_stmt(node); break; case NodeType::NODE_UPDATE_SET_CLAUSE: emit_update_set_clause(node); break; @@ -1046,6 +1049,24 @@ class Emitter { if (has_cols) sb_.append_char(')'); } + // ---- TRANSACTION ---- + + void emit_transaction_stmt(const AstNode* node) { + emit_value(node); // introducing keywords (BEGIN, BEGIN TRANSACTION, START TRANSACTION) + + if (node->first_child) sb_.append_char(' '); + bool first = true; + for (const AstNode* child = node->first_child; child; child = child->next_sibling) { + if (!first) sb_.append(", "); + first = false; + // The parser strips ISOLATION LEVEL and flags the child; restore it + if (child->flags & FLAG_TXN_MODE_ISOLATION) { + sb_.append("ISOLATION LEVEL "); + } + emit_node(child); + } + } + // ---- Compound query ---- void emit_compound_query(const AstNode* node) { diff --git a/include/sql_parser/parser.h b/include/sql_parser/parser.h index c8d7f7b..e83e071 100644 --- a/include/sql_parser/parser.h +++ b/include/sql_parser/parser.h @@ -62,6 +62,7 @@ class Parser { ParseResult parse_call(); ParseResult parse_do(); ParseResult parse_load_data(); + ParseResult parse_transaction(const Token& first); // Tier 2 extractors ParseResult extract_insert(const Token& first); @@ -88,6 +89,12 @@ class Parser { // Scan forward to semicolon or EOF, set result.remaining void scan_to_end(ParseResult& result); + + // Parse the transaction modes after BEGIN / START TRANSACTION, set result.ast. + // 'introducer' is the canonical spelling of the keywords that opened the + // statement, an unrecognized mode ends the loop and is left to scan_to_end(). + void parse_transaction_modes(ParseResult& result, StringRef introducer, + bool allow_modes); }; } // namespace sql_parser diff --git a/src/sql_parser/parser.cpp b/src/sql_parser/parser.cpp index cd7de68..4d351a9 100644 --- a/src/sql_parser/parser.cpp +++ b/src/sql_parser/parser.cpp @@ -64,7 +64,7 @@ ParseResult Parser::classify_and_dispatch() { case TokenType::TK_DELETE: return parse_delete(); case TokenType::TK_REPLACE: return parse_insert(true); case TokenType::TK_BEGIN: - case TokenType::TK_START: + case TokenType::TK_START: return parse_transaction(first); case TokenType::TK_COMMIT: case TokenType::TK_ROLLBACK: case TokenType::TK_SAVEPOINT:return extract_transaction(first); @@ -910,6 +910,40 @@ ParseResult Parser::parse_load_data() { return r; } +// ---- TRANSACTION ---- + +template +ParseResult Parser::parse_transaction(const Token& first) { + ParseResult r; + bool is_begin = (first.type == TokenType::TK_BEGIN); + r.stmt_type = is_begin ? StmtType::BEGIN : StmtType::START_TRANSACTION; + + StringRef introducer = is_begin ? StringRef{"BEGIN", 5} + : StringRef{"START TRANSACTION", 17}; + Token next = tokenizer_.peek(); + if (next.type == TokenType::TK_TRANSACTION) { + if (!is_begin) { + tokenizer_.skip(); + } else if constexpr (D == Dialect::PostgreSQL) { + // MySQL's BEGIN takes WORK but not TRANSACTION. + tokenizer_.skip(); + introducer = StringRef{"BEGIN TRANSACTION", 17}; + } + } else if (is_begin && next.type == TokenType::TK_IDENTIFIER && + next.text.equals_ci("WORK", 4)) { + // WORK is a noise word after BEGIN in both dialects; no form of its own. + tokenizer_.skip(); + } + + // MySQL carries modes on START TRANSACTION only; its BEGIN takes none. + bool allow_modes = (D == Dialect::PostgreSQL) || !is_begin; + + r.status = ParseResult::OK; + parse_transaction_modes(r, introducer, allow_modes); + scan_to_end(r); + return r; +} + // ---- Helpers ---- template @@ -972,6 +1006,122 @@ void Parser::scan_to_end(ParseResult& result) { } } +template +void Parser::parse_transaction_modes(ParseResult& result, StringRef introducer, + bool allow_modes) { + AstNode* root = make_node(arena_, NodeType::NODE_TRANSACTION_STMT, introducer); + if (!root) { result.status = ParseResult::ERROR; return; } + + while (allow_modes) { + Token t = tokenizer_.peek(); + + if (t.type == TokenType::TK_ISOLATION) { + // MySQL sets the isolation level with SET TRANSACTION, not here. + if constexpr (D == Dialect::MySQL) break; + tokenizer_.skip(); + if (tokenizer_.peek().type == TokenType::TK_LEVEL) tokenizer_.skip(); + + Token level = tokenizer_.next_token(); + if (level.type == TokenType::TK_EOF) { + result.status = ParseResult::PARTIAL; + break; + } + StringRef value = level.text; + if (level.type == TokenType::TK_SERIALIZABLE) { + value = StringRef{"SERIALIZABLE", 12}; + } else if (level.type == TokenType::TK_READ || + level.type == TokenType::TK_REPEATABLE) { + // READ COMMITTED / READ UNCOMMITTED / REPEATABLE READ + Token second = tokenizer_.next_token(); + if (second.type == TokenType::TK_EOF) { + result.status = ParseResult::PARTIAL; + break; + } + if (second.type == TokenType::TK_COMMITTED) { + value = StringRef{"READ COMMITTED", 14}; + } else if (second.type == TokenType::TK_UNCOMMITTED) { + value = StringRef{"READ UNCOMMITTED", 16}; + } else if (second.type == TokenType::TK_READ) { + value = StringRef{"REPEATABLE READ", 15}; + } else { + value = StringRef{level.text.ptr, + static_cast((second.text.ptr + second.text.len) - level.text.ptr)}; + } + } + AstNode* mode = make_node(arena_, NodeType::NODE_IDENTIFIER, value); + if (mode) mode->flags = FLAG_TXN_MODE_ISOLATION; + root->add_child(mode); + } else if (t.type == TokenType::TK_READ) { + tokenizer_.skip(); + Token rw = tokenizer_.next_token(); // ONLY or WRITE + if (rw.type == TokenType::TK_EOF) { + result.status = ParseResult::PARTIAL; + break; + } + StringRef value; + if (rw.type == TokenType::TK_ONLY) { + value = StringRef{"READ ONLY", 9}; + } else if (rw.type == TokenType::TK_WRITE) { + value = StringRef{"READ WRITE", 10}; + } else { + value = StringRef{t.text.ptr, + static_cast((rw.text.ptr + rw.text.len) - t.text.ptr)}; + } + root->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER, value)); + } else if (t.type == TokenType::TK_NOT || + (t.type == TokenType::TK_IDENTIFIER && + t.text.equals_ci("DEFERRABLE", 10))) { + // PostgreSQL: [ NOT ] DEFERRABLE. Parsed so it cannot hide a later mode. + if constexpr (D == Dialect::PostgreSQL) { + bool is_not = (t.type == TokenType::TK_NOT); + tokenizer_.skip(); + if (is_not) { + Token d = tokenizer_.next_token(); + if (d.type != TokenType::TK_IDENTIFIER || + !d.text.equals_ci("DEFERRABLE", 10)) { + result.status = ParseResult::PARTIAL; + break; + } + } + root->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER, + is_not ? StringRef{"NOT DEFERRABLE", 14} + : StringRef{"DEFERRABLE", 10})); + } else { + break; + } + } else if (t.type == TokenType::TK_WITH) { + // MySQL: WITH CONSISTENT SNAPSHOT, same reason. + if constexpr (D == Dialect::MySQL) { + tokenizer_.skip(); + Token c = tokenizer_.next_token(); + Token sn = tokenizer_.next_token(); + if (c.type != TokenType::TK_IDENTIFIER || + !c.text.equals_ci("CONSISTENT", 10) || + sn.type != TokenType::TK_IDENTIFIER || + !sn.text.equals_ci("SNAPSHOT", 8)) { + result.status = ParseResult::PARTIAL; + break; + } + root->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER, + StringRef{"WITH CONSISTENT SNAPSHOT", 24})); + } else { + break; + } + } else { + break; + } + + // PostgreSQL allows the commas to be omitted; MySQL requires them. + if (tokenizer_.peek().type == TokenType::TK_COMMA) { + tokenizer_.skip(); + } else if constexpr (D == Dialect::MySQL) { + break; + } + } + + result.ast = root; +} + // ---- Tier 2 Extractors ---- template @@ -1060,15 +1210,6 @@ ParseResult Parser::extract_transaction(const Token& first) { r.status = ParseResult::OK; switch (first.type) { - case TokenType::TK_BEGIN: - r.stmt_type = StmtType::BEGIN; - break; - case TokenType::TK_START: - r.stmt_type = StmtType::START_TRANSACTION; - // consume TRANSACTION if present - if (tokenizer_.peek().type == TokenType::TK_TRANSACTION) - tokenizer_.skip(); - break; case TokenType::TK_COMMIT: r.stmt_type = StmtType::COMMIT; break; diff --git a/tests/test_digest.cpp b/tests/test_digest.cpp index f22972c..fa313e0 100644 --- a/tests/test_digest.cpp +++ b/tests/test_digest.cpp @@ -278,6 +278,21 @@ class PgSQLDigestTest : public ::testing::Test { protected: Parser parser; + // AST-based digest (parses SQL, invalidates previous arena allocations) + StableDigest digest_ast(const char* sql) { + auto r = parser.parse(sql, strlen(sql)); + Digest digest(parser.arena()); + DigestResult dr; + if (r.ast) { + dr = digest.compute(r.ast); + } else { + dr = digest.compute(sql, strlen(sql)); + } + return StableDigest{std::string(dr.normalized.ptr, dr.normalized.len), dr.hash}; + } + + // Token-level digest (uses arena but does NOT call parse, so arena is stable + // within a single call but may be invalidated by subsequent parse calls) StableDigest digest_token(const char* sql) { parser.reset(); Digest digest(parser.arena()); @@ -285,6 +300,10 @@ class PgSQLDigestTest : public ::testing::Test { return StableDigest{std::string(dr.normalized.ptr, dr.normalized.len), dr.hash}; } + std::string normalized(const char* sql) { + return digest_ast(sql).normalized; + } + std::string normalized_token(const char* sql) { return digest_token(sql).normalized; } @@ -310,6 +329,20 @@ TEST_F(PgSQLDigestTest, ReturningDigest) { "INSERT INTO t (a) VALUES (?) RETURNING *"); } +// ========== Transaction modes ========== + +TEST_F(PgSQLDigestTest, TransactionCharacteristicsUppercased) { + EXPECT_EQ(normalized("begin read only"), "BEGIN READ ONLY"); + EXPECT_EQ(normalized("begin isolation level serializable"), + "BEGIN ISOLATION LEVEL SERIALIZABLE"); +} + +TEST_F(PgSQLDigestTest, TransactionCasingDoesNotChangeHash) { + auto d1 = digest_ast("BEGIN READ ONLY"); + auto d2 = digest_ast("begin read only"); + EXPECT_EQ(d1.hash, d2.hash); +} + // ========== Token-level digest for various Tier 2 statements ========== TEST_F(MySQLDigestTest, TokenLevelGrant) { diff --git a/tests/test_misc_stmts.cpp b/tests/test_misc_stmts.cpp index 1195ffe..319442d 100644 --- a/tests/test_misc_stmts.cpp +++ b/tests/test_misc_stmts.cpp @@ -445,6 +445,310 @@ TEST_F(MySQLLoadDataTest, RoundTripLocal) { "LOAD DATA LOCAL INFILE '/tmp/data.csv' INTO TABLE users"); } +// ===================================================================== +// TRANSACTION tests (MySQL) +// ===================================================================== + +class MySQLTransactionTest : public ::testing::Test { +protected: + Parser parser; + + std::string txn_modes(const ParseResult& r) { + std::string out; + if (!r.ast || r.ast->type != NodeType::NODE_TRANSACTION_STMT) return out; + for (const AstNode* c = r.ast->first_child; c; c = c->next_sibling) { + if (!out.empty()) out += ", "; + out.append(c->value_ptr ? c->value_ptr : "", c->value_len); + } + return out; + } +}; + +TEST_F(MySQLTransactionTest, StartTransactionReadOnly) { + const char* sql = "START TRANSACTION READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.stmt_type, StmtType::START_TRANSACTION); + ASSERT_NE(r.ast, nullptr); + EXPECT_EQ(r.ast->type, NodeType::NODE_TRANSACTION_STMT); + EXPECT_EQ(txn_modes(r), "READ ONLY"); +} + +TEST_F(MySQLTransactionTest, WithConsistentSnapshot) { + const char* one = "START TRANSACTION WITH CONSISTENT SNAPSHOT"; + auto r = parser.parse(one, strlen(one)); + EXPECT_EQ(txn_modes(r), "WITH CONSISTENT SNAPSHOT"); + + const char* both = "START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY"; + auto r2 = parser.parse(both, strlen(both)); + EXPECT_EQ(txn_modes(r2), "WITH CONSISTENT SNAPSHOT, READ ONLY"); +} + +TEST_F(MySQLTransactionTest, PostgresOnlyFormsAreNotMySQLModes) { + const char* cases[] = { + "BEGIN DEFERRABLE", + "BEGIN READ ONLY", + "BEGIN TRANSACTION READ ONLY", + "START TRANSACTION ISOLATION LEVEL REPEATABLE READ", + "START TRANSACTION READ ONLY WITH CONSISTENT SNAPSHOT", + }; + for (const char* sql : cases) { + SCOPED_TRACE(sql); + auto r = parser.parse(sql, strlen(sql)); + EXPECT_FALSE(r.full_input); + } +} + +TEST_F(MySQLTransactionTest, FullInputOnlyWhenEveryModeIsModelled) { + struct Case { const char* sql; bool full; const char* remaining; }; + const Case cases[] = { + {"BEGIN", true, ""}, + {"BEGIN WORK", true, ""}, + {"START TRANSACTION", true, ""}, + {"START TRANSACTION READ ONLY", true, ""}, + {"START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY", true, ""}, + {"START TRANSACTION READ ONLY GARBAGE", false, "GARBAGE"}, + {"BEGIN DEFERRABLE", false, "DEFERRABLE"}, + {"START TRANSACTION ISOLATION LEVEL REPEATABLE READ", false, + "ISOLATION LEVEL REPEATABLE READ"}, + {"BEGIN READ ONLY", false, "READ ONLY"}, + {"BEGIN TRANSACTION", false, "TRANSACTION"}, + {"START TRANSACTION READ ONLY WITH CONSISTENT SNAPSHOT", false, + "WITH CONSISTENT SNAPSHOT"}, + {"START TRANSACTION READ ONLY; SELECT 1", false, "SELECT 1"}, + }; + + for (const auto& tc : cases) { + SCOPED_TRACE(tc.sql); + auto r = parser.parse(tc.sql, strlen(tc.sql)); + EXPECT_EQ(r.full_input, tc.full); + EXPECT_EQ(std::string(r.remaining.ptr ? r.remaining.ptr : "", r.remaining.len), + tc.remaining); + } +} + +// Only BEGIN and START TRANSACTION moved to Tier 1; the other three verbs share +// the same dispatch switch and must still classify without an AST. +TEST_F(MySQLTransactionTest, OtherTransactionVerbsStayTier2) { + struct Case { const char* sql; StmtType type; }; + const Case cases[] = { + {"COMMIT", StmtType::COMMIT}, + {"COMMIT AND CHAIN", StmtType::COMMIT}, + {"ROLLBACK", StmtType::ROLLBACK}, + {"ROLLBACK TO SAVEPOINT s1", StmtType::ROLLBACK}, + {"SAVEPOINT s1", StmtType::SAVEPOINT}, + }; + + for (const auto& tc : cases) { + SCOPED_TRACE(tc.sql); + auto r = parser.parse(tc.sql, strlen(tc.sql)); + EXPECT_EQ(r.status, ParseResult::OK); + EXPECT_EQ(r.stmt_type, tc.type); + EXPECT_EQ(r.ast, nullptr); + } +} + +// ===================================================================== +// TRANSACTION tests (PostgreSQL) +// ===================================================================== + +class PgSQLTransactionTest : public ::testing::Test { +protected: + Parser parser; + + std::string txn_modes(const ParseResult& r) { + std::string out; + if (!r.ast || r.ast->type != NodeType::NODE_TRANSACTION_STMT) return out; + for (const AstNode* c = r.ast->first_child; c; c = c->next_sibling) { + if (!out.empty()) out += ", "; + out.append(c->value_ptr ? c->value_ptr : "", c->value_len); + } + return out; + } + + std::string round_trip(const char* sql) { + auto r = parser.parse(sql, strlen(sql)); + if (!r.ast) return "[PARSE_FAILED]"; + Emitter emitter(parser.arena()); + emitter.emit(r.ast); + StringRef result = emitter.result(); + return std::string(result.ptr, result.len); + } +}; + +TEST_F(PgSQLTransactionTest, BeginHasNoModes) { + const char* sql = "BEGIN"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.status, ParseResult::OK); + EXPECT_EQ(r.stmt_type, StmtType::BEGIN); + ASSERT_NE(r.ast, nullptr); + EXPECT_EQ(r.ast->type, NodeType::NODE_TRANSACTION_STMT); + EXPECT_EQ(txn_modes(r), ""); +} + +TEST_F(PgSQLTransactionTest, BeginReadOnly) { + const char* sql = "BEGIN READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.stmt_type, StmtType::BEGIN); + EXPECT_EQ(txn_modes(r), "READ ONLY"); +} + +TEST_F(PgSQLTransactionTest, BeginReadWrite) { + const char* sql = "BEGIN READ WRITE"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(txn_modes(r), "READ WRITE"); +} + +TEST_F(PgSQLTransactionTest, BeginTransactionKeywordAccepted) { + const char* sql = "BEGIN TRANSACTION READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.stmt_type, StmtType::BEGIN); + EXPECT_EQ(txn_modes(r), "READ ONLY"); +} + +TEST_F(PgSQLTransactionTest, BeginWorkKeywordAccepted) { + const char* sql = "BEGIN WORK READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.status, ParseResult::OK); + EXPECT_EQ(r.stmt_type, StmtType::BEGIN); + EXPECT_EQ(txn_modes(r), "READ ONLY"); + EXPECT_EQ(round_trip("BEGIN WORK"), "BEGIN"); + EXPECT_EQ(round_trip("BEGIN WORK READ ONLY"), "BEGIN READ ONLY"); +} + +TEST_F(PgSQLTransactionTest, StartTransactionReadOnly) { + const char* sql = "START TRANSACTION READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.stmt_type, StmtType::START_TRANSACTION); + EXPECT_EQ(txn_modes(r), "READ ONLY"); +} + +TEST_F(PgSQLTransactionTest, BeginIsolationLevelSerializable) { + const char* sql = "BEGIN ISOLATION LEVEL SERIALIZABLE"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(txn_modes(r), "SERIALIZABLE"); +} + +TEST_F(PgSQLTransactionTest, BeginIsolationLevelTwoWordLevels) { + struct Case { const char* sql; const char* mode; }; + const Case cases[] = { + {"BEGIN ISOLATION LEVEL REPEATABLE READ", "REPEATABLE READ"}, + {"BEGIN ISOLATION LEVEL READ COMMITTED", "READ COMMITTED"}, + {"BEGIN ISOLATION LEVEL READ UNCOMMITTED", "READ UNCOMMITTED"}, + }; + for (const auto& tc : cases) { + SCOPED_TRACE(tc.sql); + auto r = parser.parse(tc.sql, strlen(tc.sql)); + EXPECT_EQ(txn_modes(r), tc.mode); + } +} + +TEST_F(PgSQLTransactionTest, BeginCommaSeparatedModes) { + const char* sql = "BEGIN ISOLATION LEVEL READ COMMITTED, READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(txn_modes(r), "READ COMMITTED, READ ONLY"); +} + +TEST_F(PgSQLTransactionTest, BeginDeferrable) { + const char* sql = "BEGIN READ ONLY DEFERRABLE"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.status, ParseResult::OK); + EXPECT_EQ(txn_modes(r), "READ ONLY, DEFERRABLE"); +} + +TEST_F(PgSQLTransactionTest, BeginDeferrableBeforeReadOnly) { + const char* cases[] = { + "BEGIN DEFERRABLE, READ ONLY", + "BEGIN NOT DEFERRABLE, READ ONLY", + "BEGIN ISOLATION LEVEL SERIALIZABLE, DEFERRABLE, READ ONLY", + }; + for (const char* sql : cases) { + SCOPED_TRACE(sql); + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.status, ParseResult::OK); + EXPECT_NE(txn_modes(r).find("READ ONLY"), std::string::npos); + } +} + +TEST_F(PgSQLTransactionTest, SnapshotIsNotAPostgresMode) { + const char* sql = "START TRANSACTION WITH CONSISTENT SNAPSHOT"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(txn_modes(r), ""); +} + +TEST_F(PgSQLTransactionTest, IncompleteModeIsPartial) { + const char* cases[] = {"BEGIN READ", "BEGIN ISOLATION LEVEL", "BEGIN ISOLATION LEVEL READ"}; + for (const char* sql : cases) { + SCOPED_TRACE(sql); + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.status, ParseResult::PARTIAL); + } +} + +TEST_F(PgSQLTransactionTest, FullInputOnlyWhenEveryModeIsModelled) { + struct Case { const char* sql; bool full; const char* remaining; }; + const Case cases[] = { + {"BEGIN READ ONLY", true, ""}, + {"BEGIN TRANSACTION READ ONLY", true, ""}, + {"BEGIN WORK READ ONLY", true, ""}, + {"BEGIN ISOLATION LEVEL SERIALIZABLE, READ ONLY", true, ""}, + {"BEGIN NOT DEFERRABLE, READ ONLY", true, ""}, + // Longest mode lists PostgreSQL's own regression suite uses. + {"BEGIN TRANSACTION READ ONLY, READ WRITE, DEFERRABLE, NOT DEFERRABLE", true, ""}, + {"START TRANSACTION ISOLATION LEVEL SERIALIZABLE, READ WRITE, NOT DEFERRABLE", true, ""}, + {"BEGIN READ ONLY GARBAGE", false, "GARBAGE"}, + {"BEGIN GARBAGE, READ ONLY", false, "GARBAGE, READ ONLY"}, + {"BEGIN WITH CONSISTENT SNAPSHOT", false, "WITH CONSISTENT SNAPSHOT"}, + }; + + for (const auto& tc : cases) { + SCOPED_TRACE(tc.sql); + auto r = parser.parse(tc.sql, strlen(tc.sql)); + EXPECT_EQ(r.full_input, tc.full); + EXPECT_EQ(std::string(r.remaining.ptr ? r.remaining.ptr : "", r.remaining.len), + tc.remaining); + } +} + +TEST_F(PgSQLTransactionTest, BeginPreservesRemaining) { + const char* sql = "BEGIN READ ONLY; SELECT 1"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(r.stmt_type, StmtType::BEGIN); + EXPECT_EQ(txn_modes(r), "READ ONLY"); + ASSERT_TRUE(r.has_remaining()); + EXPECT_EQ(std::string(r.remaining.ptr, r.remaining.len), "SELECT 1"); +} + +// ========== TRANSACTION round-trip ========== + +TEST_F(PgSQLTransactionTest, RoundTripCanonicalIntroducer) { + EXPECT_EQ(round_trip("BEGIN"), "BEGIN"); + EXPECT_EQ(round_trip("begin"), "BEGIN"); + EXPECT_EQ(round_trip("BEGIN TRANSACTION READ ONLY"), "BEGIN TRANSACTION READ ONLY"); + EXPECT_EQ(round_trip("START TRANSACTION"), "START TRANSACTION"); + EXPECT_EQ(round_trip("START TRANSACTION READ WRITE"), "START TRANSACTION READ WRITE"); +} + +TEST_F(PgSQLTransactionTest, RoundTripCanonicalModes) { + EXPECT_EQ(round_trip("BEGIN READ ONLY"), "BEGIN READ ONLY"); + EXPECT_EQ(round_trip("begin read only"), "BEGIN READ ONLY"); + EXPECT_EQ(round_trip("BEGIN READ ONLY"), "BEGIN READ ONLY"); + EXPECT_EQ(round_trip("BEGIN ISOLATION LEVEL SERIALIZABLE"), + "BEGIN ISOLATION LEVEL SERIALIZABLE"); + EXPECT_EQ(round_trip("begin isolation level serializable"), + "BEGIN ISOLATION LEVEL SERIALIZABLE"); + EXPECT_EQ(round_trip("BEGIN ISOLATION LEVEL REPEATABLE READ"), + "BEGIN ISOLATION LEVEL REPEATABLE READ"); +} + +TEST_F(PgSQLTransactionTest, ModesWithoutCommas) { + EXPECT_EQ(round_trip("BEGIN ISOLATION LEVEL SERIALIZABLE READ ONLY"), + "BEGIN ISOLATION LEVEL SERIALIZABLE, READ ONLY"); + const char* sql = "BEGIN ISOLATION LEVEL SERIALIZABLE READ ONLY"; + auto r = parser.parse(sql, strlen(sql)); + EXPECT_EQ(txn_modes(r), "SERIALIZABLE, READ ONLY"); +} + + // ===================================================================== // Bulk data-driven tests // ===================================================================== @@ -486,6 +790,12 @@ static MiscStmtTestCase mysql_misc_cases[] = { {"LOAD DATA INFILE '/tmp/data.csv' INTO TABLE users", StmtType::LOAD_DATA, ParseResult::OK}, {"LOAD DATA LOCAL INFILE '/tmp/data.csv' INTO TABLE users", StmtType::LOAD_DATA, ParseResult::OK}, {"LOAD DATA INFILE '/tmp/data.csv' REPLACE INTO TABLE users", StmtType::LOAD_DATA, ParseResult::OK}, + // BEGIN / START TRANSACTION + {"BEGIN", StmtType::BEGIN, ParseResult::OK}, + {"BEGIN WORK", StmtType::BEGIN, ParseResult::OK}, + {"START TRANSACTION", StmtType::START_TRANSACTION, ParseResult::OK}, + {"START TRANSACTION READ ONLY", StmtType::START_TRANSACTION, ParseResult::OK}, + {"START TRANSACTION WITH CONSISTENT SNAPSHOT", StmtType::START_TRANSACTION, ParseResult::OK}, }; TEST_P(MySQLMiscStmtBulk, ClassifyAndParse) {