diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 09db4fc36..fc1e9bf2e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,6 +31,16 @@ jobs: - run: bunx playwright install --with-deps chromium - run: bun run test + hcon_grammar: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '20' + - name: Strict HCON grammar (Node only, no Playwright) + run: node --test hcon/test/*.test.js + website_tests: runs-on: ubuntu-latest defaults: diff --git a/hcon/GRAMMAR.md b/hcon/GRAMMAR.md new file mode 100644 index 000000000..ff3294f1c --- /dev/null +++ b/hcon/GRAMMAR.md @@ -0,0 +1,103 @@ +# HCON grammar (strict) + +A closed, non-Turing-complete map language for htmx attributes. +Implemented by recursive descent in `src/hcon.js`. Regex is not part of +the grammar; `test/no-regex.test.js` fails the build if a regex literal +or `RegExp` call appears in the parser. + +This is a proposal, not a description of shipping htmx 4.0.0. The 4.0.0 +parser is a global regex plus per-attribute peelers. This document is +the language we think HCON should be. + +## Non-goals + +- CSS. Selectors are string values (`target:'#table tbody'` or + `from:<#table tbody/>`). The grammar does not parse combinators. +- JavaScript. Trigger filters (`click[shiftKey]`) are outside HCON. + They `eval`. That is a host concern, not this language. +- Arrays and nested `{...}` inside HCON. JSON remains the full-fidelity + form (`parse` of a string that starts with `{` is `JSON.parse`). +- Mixing JSON and HCON in one string. + +## Start symbols + +Comma cannot mean two things. Two start symbols share one tokenizer: + +| Start | Used by | Pair separator | Item separator | +|---|---|---|---| +| `Map` | `hx-swap` modifiers, `hx-vals`, `hx-config`, `hx-headers`, `hx-swap-oob` | whitespace | — | +| `List` | `hx-trigger` | whitespace inside each map | comma | + +So `delay:100ms throttle:200ms` is one map, and +`click delay:500ms, keyup` is a list of two maps. +`delay:100ms, throttle:200ms` as a **Map** is a syntax error (comma). +As a **List** it is two maps. Hosts pick the start symbol. + +## EBNF + +``` +input := ws* ( json | map ) ws* +list := ws* map ( comma ws* map )* ws* + +json := '{' … '}' (* JSON.parse of the whole input *) + +map := pair ( ws+ pair )* +pair := key ( ':' ws* value )? (* missing value => true *) + +key := string | dotted +dotted := ident ( '.' ident )* +value := string | number | boolean | ident + +string := '"' chars '"' | "'" chars "'" | '<' hs-chars '/>' +ident := ident-char+ +ident-char:= any char except whitespace, ':' , ',' + +number := '-'? digits ( '.' digits )? ( [eE] [+-]? digits )? +boolean := 'true' | 'false' + +ws := space | tab | CR | LF +comma := ',' +``` + +`hs-chars` is any run that does not contain the two-character closer +`/>`. That is the existing hyperscript quoting form, kept because it +does not fight HTML attribute quotes. + +Quoted values are **always strings**. `count:42` is a number; +`count:"42"` is the string `"42"`. The shipping parser JSON-parses +every value, so quotes do not control type. + +Leftover input that is not a pair is an error. Unquoted spaces do **not** +produce leftover: `beforeend:#table tbody` is a well-formed Map +`{beforeend:"#table", tbody:true}`. That is why colon form cannot be +rejected in the grammar. `interpretSwap` / `interpretOob` reject a +swap-style key whose value is not `true` — hosts interpret, they do +not re-tokenize. + +The shipping parser `matchAll`s and drops unmatched text. + +## Interpretation (not parsing) + +A map is data. Attribute hosts look up vocabularies: + +- `hx-swap="innerHTML swap:200ms target:'#table tbody'"` + → `{innerHTML: true, swap: "200ms", target: "#table tbody"}` + → the unique key that is a swap style becomes `style`. +- `hx-swap-oob="beforeend target:'#table tbody'"` is the same map. + Colon form `beforeend:#table tbody` is **not** HCON. +- `hx-trigger` uses `List`. Each map’s unique event-name flag is + `name`. `every:2s` is a pair, not the shipping `every 2s` peel. + +## Complexity + +The grammar is regular over tokens plus one-token lookahead (LL(1)). +No recursion in values except JSON (a different language). No loops, +functions, or substitutions. That is the non-Turing bound: HCON cannot +express computation; hosts may still `eval` filters, which this parser +refuses to tokenize. + +## Compatibility + +`src/legacy.js` is a copy of the 4.0.0 regex parser. `test/corpus.json` +classifies strings as `both`, `legacy_only`, or `strict_error`. CI +fails if the classifications drift without an explicit corpus edit. diff --git a/hcon/README.md b/hcon/README.md new file mode 100644 index 000000000..78581d97e --- /dev/null +++ b/hcon/README.md @@ -0,0 +1,25 @@ +> RFC. This directory is a proposed grammar, parser, and Node test +> corpus. It is **not** wired into `src/htmx.js`. Shipping HCON is +> unchanged. See GRAMMAR.md. + +# Strict HCON + +A testable, LL(1), non-Turing-complete grammar for htmx's attribute config +language. Recursive descent. No regex. Shipping htmx 4.0.0 HCON is a +global regex plus per-attribute peelers; this is the language we think +it should be. + +```bash +make hcon # from the community wrapper root +# or +cd hcon && npm test +``` + +- Spec: [`GRAMMAR.md`](GRAMMAR.md) +- Parser: [`src/hcon.js`](src/hcon.js) +- Hosts interpret maps, they do not re-tokenize: [`src/interpret.js`](src/interpret.js) +- 4.0.0 regex copy, comparison only: [`src/legacy.js`](src/legacy.js) +- Compatibility classifications: [`test/corpus.json`](test/corpus.json) + +CI is `.github/workflows/hcon.yml` (Node, no Playwright). The associated +htmx RFC branch adds the same job next to `htmx_tests`. diff --git a/hcon/package.json b/hcon/package.json new file mode 100644 index 000000000..484fca1d6 --- /dev/null +++ b/hcon/package.json @@ -0,0 +1,12 @@ +{ + "name": "hcon-grammar", + "private": true, + "type": "module", + "description": "Strict, regex-free HCON grammar for htmx attributes", + "scripts": { + "test": "node --test test/*.test.js" + }, + "engines": { + "node": ">=20" + } +} diff --git a/hcon/src/hcon.js b/hcon/src/hcon.js new file mode 100644 index 000000000..b15bc2a2b --- /dev/null +++ b/hcon/src/hcon.js @@ -0,0 +1,315 @@ +/** + * Strict HCON: recursive-descent map language. + * No regex. Spec: ../GRAMMAR.md + */ + +export class HconError extends Error { + constructor(message, index, input) { + super(message) + this.name = 'HconError' + this.index = index + this.input = input + } +} + +const FORBIDDEN_KEYS = new Set(['__proto__', 'constructor', 'prototype']) + +function isWs(c) { + return c === ' ' || c === '\t' || c === '\n' || c === '\r' +} + +function isIdentChar(c) { + return c != null && c !== ':' && c !== ',' && !isWs(c) +} + +function isDigit(c) { + return c >= '0' && c <= '9' +} + +class Parser { + constructor(input) { + this.s = input == null ? '' : String(input) + this.i = 0 + } + + peek() { + return this.s[this.i] + } + + eof() { + return this.i >= this.s.length + } + + eat() { + return this.s[this.i++] + } + + fail(message) { + throw new HconError(message + ' at ' + this.i, this.i, this.s) + } + + skipWs() { + while (isWs(this.peek())) this.i++ + } + + parseInput() { + this.skipWs() + if (this.eof()) return {} + if (this.peek() === '{') return this.parseJson() + let map = this.parseMap() + this.skipWs() + if (!this.eof()) this.fail('leftover input ' + JSON.stringify(this.s.slice(this.i))) + return map + } + + parseList() { + this.skipWs() + if (this.eof()) return [] + if (this.peek() === '{') return [this.parseJson()] + let maps = [this.parseMap()] + this.skipWs() + while (this.peek() === ',') { + this.eat() + this.skipWs() + if (this.eof()) this.fail('trailing comma in list') + maps.push(this.parseMap()) + this.skipWs() + } + if (!this.eof()) this.fail('leftover input ' + JSON.stringify(this.s.slice(this.i))) + return maps + } + + parseJson() { + try { + return JSON.parse(this.s) + } catch (e) { + throw new HconError('invalid JSON: ' + e.message, this.i, this.s) + } + } + + parseMap() { + let map = {} + let pair = this.parsePair() + this.assign(map, pair) + for (;;) { + let start = this.i + this.skipWs() + if (this.eof() || this.peek() === ',') { + this.i = start + break + } + if (!isIdentChar(this.peek()) && this.peek() !== '"' && this.peek() !== "'") { + this.i = start + break + } + pair = this.parsePair() + this.assign(map, pair) + } + return map + } + + parsePair() { + let keyInfo = this.parseKey() + this.skipWs() + let value = true + if (this.peek() === ':') { + this.eat() + this.skipWs() + value = this.parseValue() + } + return { keyInfo, value } + } + + parseKey() { + if (this.peek() === '"' || this.peek() === "'") { + return { key: this.parseQuoted(), dotted: false } + } + let ident = this.parseIdent() + if (ident.includes('.')) { + return { key: ident, dotted: true } + } + return { key: ident, dotted: false } + } + + parseValue() { + let c = this.peek() + if (c === '"' || c === "'") return this.parseQuoted() + if (c === '<') return this.parseHyperscript() + if (c === '-' || isDigit(c)) { + let num = this.tryNumber() + if (num !== null) return num + } + let ident = this.parseIdent() + if (ident === 'true') return true + if (ident === 'false') return false + return ident + } + + parseQuoted() { + let q = this.eat() + let out = '' + while (!this.eof()) { + let c = this.eat() + if (c === q) return out + if (c === '\\') { + if (this.eof()) this.fail('unterminated escape') + let n = this.eat() + if (n === q || n === '\\') out += n + else out += n + } else { + out += c + } + } + this.fail('unterminated ' + q + ' string') + } + + parseHyperscript() { + this.eat() + let out = '' + while (!this.eof()) { + if (this.peek() === '/' && this.s[this.i + 1] === '>') { + this.i += 2 + return out + } + out += this.eat() + } + this.fail('unterminated <.../> string') + } + + parseIdent() { + if (!isIdentChar(this.peek())) this.fail('expected ident') + let start = this.i + while (isIdentChar(this.peek())) this.eat() + return this.s.slice(start, this.i) + } + + tryNumber() { + let start = this.i + if (this.peek() === '-') this.eat() + if (!isDigit(this.peek())) { + this.i = start + return null + } + while (isDigit(this.peek())) this.eat() + if (this.peek() === '.') { + let dot = this.i + this.eat() + if (!isDigit(this.peek())) { + this.i = start + return null + } + while (isDigit(this.peek())) this.eat() + void dot + } + if (this.peek() === 'e' || this.peek() === 'E') { + let e = this.i + this.eat() + if (this.peek() === '+' || this.peek() === '-') this.eat() + if (!isDigit(this.peek())) { + this.i = start + return null + } + while (isDigit(this.peek())) this.eat() + void e + } + if (isIdentChar(this.peek())) { + this.i = start + return null + } + return Number(this.s.slice(start, this.i)) + } + + assign(map, { keyInfo, value }) { + if (keyInfo.dotted) { + let segs = keyInfo.key.split('.') + if (segs.some((s) => FORBIDDEN_KEYS.has(s))) return + let cur = map + for (let i = 0; i < segs.length - 1; i++) { + let seg = segs[i] + if (cur[seg] == null || typeof cur[seg] !== 'object') cur[seg] = {} + cur = cur[seg] + } + let last = segs[segs.length - 1] + if (!FORBIDDEN_KEYS.has(last)) cur[last] = value + return + } + if (FORBIDDEN_KEYS.has(keyInfo.key)) return + map[keyInfo.key] = value + } +} + +export function parse(input) { + return new Parser(input).parseInput() +} + +export function parseList(input) { + return new Parser(input).parseList() +} + +export function stringify(map) { + if (map == null || typeof map !== 'object' || Array.isArray(map)) { + return JSON.stringify(map) + } + let parts = [] + stringifyInto(map, '', parts) + return parts.join(' ') +} + +function stringifyInto(obj, prefix, parts) { + for (let [k, v] of Object.entries(obj)) { + if (FORBIDDEN_KEYS.has(k)) continue + let path = prefix ? prefix + '.' + k : k + if (v && typeof v === 'object' && !Array.isArray(v)) { + stringifyInto(v, path, parts) + } else { + parts.push(formatPair(path, v)) + } + } +} + +function formatPair(key, value) { + let k = needsQuote(key) ? '"' + escapeStr(key) + '"' : key + if (value === true) return k + if (value === false) return k + ':false' + if (typeof value === 'number') return k + ':' + String(value) + if (typeof value === 'string') { + if (value === 'true' || value === 'false' || looksLikeNumber(value) || needsQuote(value)) { + return k + ':"' + escapeStr(value) + '"' + } + return k + ':' + value + } + return k + ':' + JSON.stringify(value) +} + +function needsQuote(s) { + if (s.length === 0) return true + for (let i = 0; i < s.length; i++) { + let c = s[i] + if (isWs(c) || c === ':' || c === ',' || c === '"' || c === "'") return true + } + return false +} + +function looksLikeNumber(s) { + if (s.length === 0) return false + let i = 0 + if (s[i] === '-') i++ + if (i >= s.length || !isDigit(s[i])) return false + while (i < s.length && isDigit(s[i])) i++ + if (s[i] === '.') { + i++ + if (!isDigit(s[i])) return false + while (i < s.length && isDigit(s[i])) i++ + } + return i === s.length +} + +function escapeStr(s) { + let out = '' + for (let i = 0; i < s.length; i++) { + let c = s[i] + if (c === '"' || c === '\\') out += '\\' + c + else out += c + } + return out +} diff --git a/hcon/src/interpret.js b/hcon/src/interpret.js new file mode 100644 index 000000000..32803ed7e --- /dev/null +++ b/hcon/src/interpret.js @@ -0,0 +1,87 @@ +/** + * Attribute hosts interpret maps. They do not re-tokenize. + */ + +import { HconError, parse, parseList } from './hcon.js' + +export const SWAP_STYLES = new Set([ + 'innerHTML', 'outerHTML', 'textContent', + 'beforebegin', 'afterbegin', 'beforeend', 'afterend', + 'before', 'after', 'prepend', 'append', + 'innerMorph', 'outerMorph', 'outerSync', + 'delete', 'none', 'upsert', +]) + +export function normalizeSwapStyle(style) { + return style === 'before' ? 'beforebegin' + : style === 'after' ? 'afterend' + : style === 'prepend' ? 'afterbegin' + : style === 'append' ? 'beforeend' + : style +} + +export function interpretSwap(map, defaultStyle = 'innerHTML') { + if (map == null || typeof map !== 'object') { + throw new HconError('interpretSwap expects a map', 0, '') + } + for (let k of Object.keys(map)) { + if (k !== 'style' && SWAP_STYLES.has(normalizeSwapStyle(k)) && map[k] !== true) { + throw new HconError( + "colon form is not HCON: " + k + ":" + map[k] + " — write `" + k + " target:'…'`", + 0, + '', + ) + } + } + let rest = { ...map } + let style + if (typeof rest.style === 'string') { + style = rest.style + delete rest.style + } else { + let flags = Object.keys(rest).filter((k) => rest[k] === true && SWAP_STYLES.has(normalizeSwapStyle(k))) + if (flags.length > 1) { + throw new HconError('multiple swap styles: ' + flags.join(', '), 0, '') + } + if (flags.length === 1) { + style = flags[0] + delete rest[flags[0]] + } + } + return { style: normalizeSwapStyle(style || defaultStyle), ...rest } +} + +export function interpretOob(map, defaultTarget, defaultStyle = 'outerHTML') { + let spec = interpretSwap(map, defaultStyle) + let target = spec.target || defaultTarget + delete spec.target + if (!target) throw new HconError('oob swap has no target', 0, '') + return { ...spec, target } +} + +export function interpretTrigger(map) { + if (typeof map.every === 'string') { + let { every, ...rest } = map + return { name: 'every', interval: every, ...rest } + } + let flags = Object.keys(map).filter((k) => map[k] === true) + if (flags.length === 0) { + throw new HconError('trigger map has no event name', 0, '') + } + let name = flags[0] + let rest = { ...map } + delete rest[name] + return { name, ...rest } +} + +export function parseSwap(input, defaultStyle = 'innerHTML') { + return interpretSwap(parse(input), defaultStyle) +} + +export function parseOob(input, defaultTarget) { + return interpretOob(parse(input), defaultTarget) +} + +export function parseTriggers(input) { + return parseList(input).map(interpretTrigger) +} diff --git a/hcon/src/legacy.js b/hcon/src/legacy.js new file mode 100644 index 000000000..1dd1880b2 --- /dev/null +++ b/hcon/src/legacy.js @@ -0,0 +1,46 @@ +/** + * Shipping htmx 4.0.0 HCON, copied for corpus comparison only. + * Not used by the strict parser. + */ +export const legacy = { + parse(string) { + if (!string) return {} + if (string.startsWith('{')) return JSON.parse(string) + let pattern = /(?:"([^"]+)"|'([^']+)'|([^\s,:]+))(?:\s*:\s*(?:"([^"]*)"|'([^']*)'|<((?:[^/]|\/(?!>))+)\/>|([^\s,]+)))?(?=\s|,|$)/g + let result = {} + for (let match of string.matchAll(pattern)) { + let [, + doubleQuotedKey, + singleQuotedKey, + bareKey, + doubleQuotedValue, + singleQuotedValue, + hyperscriptValue, + bareValue, + ] = match + let key = doubleQuotedKey ?? singleQuotedKey ?? bareKey + let value = (doubleQuotedValue ?? singleQuotedValue ?? hyperscriptValue ?? bareValue ?? 'true').trim() + try { value = JSON.parse(value) } catch {} + let isDottedPath = bareKey?.includes('.') + let pair = isDottedPath + ? key.split('.').reduceRight((acc, segment) => ({ [segment]: acc }), value) + : { [key]: value } + legacy.merge(pair, result) + } + return result + }, + split(string) { + return string.split(/,(?![^\[]*\])(?![^(]*\))(?![^<]*\/>)(?=(?:[^"']|"[^"]*"|'[^']*')*$)/) + }, + merge(source, target) { + if (typeof source === 'string') source = legacy.parse(source) + for (let [key, val] of Object.entries(source)) { + if (['__proto__', 'constructor', 'prototype'].includes(key)) continue + let sourceIsObject = val?.constructor === Object + let targetIsObject = target[key]?.constructor === Object + if (sourceIsObject && targetIsObject) legacy.merge(val, target[key]) + else target[key] = val + } + return target + }, +} diff --git a/hcon/test/corpus.json b/hcon/test/corpus.json new file mode 100644 index 000000000..623b8512d --- /dev/null +++ b/hcon/test/corpus.json @@ -0,0 +1,61 @@ +{ + "both": [ + ["delay:100ms", {"delay": "100ms"}], + ["once", {"once": true}], + ["once:false", {"once": false}], + ["count:42", {"count": 42}], + ["delay:100ms throttle:200ms", {"delay": "100ms", "throttle": "200ms"}], + ["target:\"#foo .bar\"", {"target": "#foo .bar"}], + ["target:'#foo .bar'", {"target": "#foo .bar"}], + ["sse.mode:once", {"sse": {"mode": "once"}}], + ["sse.mode:once sse.maxRetries:5", {"sse": {"mode": "once", "maxRetries": 5}}], + ["{\"delay\":\"100ms\",\"throttle\":\"200ms\"}", {"delay": "100ms", "throttle": "200ms"}], + ["foo:bar", {"foo": "bar"}], + ["\"a.b\":1", {"a.b": 1}], + ["from:<.foo/>", {"from": ".foo"}], + ["from:
", {"from": "div p"}], + ["from: