A headless browser for AI agents, written in Rust. No paint, no fonts, no GPU — but everything needed to answer what is on this page and what can I click: a DOM, a JavaScript engine, a network stack, CSS cascade and box layout, and a semantic view of the page.
Independent clean-room work, Apache-2.0. Architecturally inspired by Lightpanda (Zig, AGPL-3.0); no code from that project was read or copied, and it is used here only as a compiled binary for benchmarking.
Status: walking skeleton. It loads real pages, runs their JavaScript, and produces agent-usable output. It is not a spec-compliant browser and does not try to be yet — see Known gaps.
A rendering browser spends most of its budget on things an agent never looks at: layout, rasterisation, compositing, font shaping, image decode, the accessibility tree. Deleting all of it is worth more than any amount of optimising the rest.
URL ─▶ lb-net ─▶ lb-html ─▶ lb-dom ─▶ lb-css ─▶ lb-js ─▶ lb-layout ─▶ lb-semantic
HTTP html5ever arena cascade V8 taffy visible
parallel TreeSink DOM + UA bindings boxes elements
prefetch sheet + handles
| Crate | Job |
|---|---|
lb-dom |
Arena DOM + CSS selector engine + serialization |
lb-html |
html5ever TreeSink writing straight into the arena |
lb-css |
Stylesheet parser, UA stylesheet, cascade → computed style |
lb-js |
V8 isolate, DOM bindings, virtual-time event loop |
lb-layout |
Box layout via taffy + font-free text measurement |
lb-net |
HTTP with parallel subresource prefetch |
lb-semantic |
The page as an agent should see it |
lb-core |
The page pipeline |
1. The DOM is a flat arena, addressed by index.
Nodes live in one Vec<Node>; parent/child/sibling links are Option<NodeId>
where NodeId is a u32. No Rc, no RefCell, no per-node allocation for
tree structure. A whole document is freed by dropping one Vec. Element and
attribute names are string_cache atoms — 8 bytes, pointer-comparable, and
pre-interned at compile time for every HTML tag.
2. JS wrappers hold a NodeId, not a pointer.
Every DOM object exposed to JavaScript is an instance of one shared
ObjectTemplate with a single internal field containing the node index. The
wrapper does not own the node, so there is no reference cycle between the V8
heap and the Rust heap, and no cross-heap tracing to get right. A wrapper cache
keyed by NodeId keeps JS object identity stable, so a === b holds for the
same element.
(rusty_v8 does expose cppgc — GarbageCollected, Member, Visitor — which
is the Blink-style unified-heap approach. That is the upgrade path when the DOM
needs to own JS-visible state; the index indirection is deliberately simpler
for now.)
3. Time is virtual.
setTimeout(f, 5000) costs microseconds. The event loop pops the
earliest-deadline timer, jumps the clock to it, and runs it — draining until no
work remains. That gives a real "the page has settled" signal instead of a
guessed sleep, makes runs deterministic, and is bounded by an explicit budget so
a page that reschedules itself forever cannot spin.
4. Subresources are fetched in parallel.
The HTML spec requires classic scripts to execute in order; it says nothing
about the order they are downloaded in. So every src is collected up front
and downloaded concurrently — a preload scanner — then executed in document
order. On a page with 40 external scripts this is the difference between 1681ms
and 257ms.
5. Layout runs, painting does not. The CSS cascade resolves the ~40 properties that decide whether a box exists, where it is, and how big it is; taffy then does block, flex and grid layout. Nothing is rasterised, no font file is ever opened. Text is measured from a per-character advance table calibrated against Chrome — wrong in the third decimal place, right in the first, which is all that "does this button have a box and where is it" requires.
This exists for one reason: without it you cannot tell what is visible. A
display:none set by a stylesheet is invisible to any amount of
attribute-sniffing, and on a real page most hidden things are hidden that way —
collapsed menus, inactive tabs, modals, screen-reader-only text. An agent
working from the DOM alone is choosing among targets that are not there.
6. Layout is lazy and invalidated, not recomputed.
Every DOM mutation marks layout dirty; nothing is recomputed until something
asks. A script making 500 changes triggers one layout pass, not 500 — but a
script that reads getBoundingClientRect after each change gets the same
thrashing it would in a real browser, because it asked for it.
Machine: 8-core x86-64, Linux 6.8, Chrome 150.0.7871.181, rustc 1.94. Pages served from localhost so neither engine is network-bound. Memory is peak summed PSS across the process tree — not RSS, which double-counts the shared pages of Chrome's many processes and overstates it by ~4x.
All numbers below include full CSS cascade and layout.
| page | lightbrowser | Chrome --dump-dom |
speedup | ||
|---|---|---|---|---|---|
| ms | MB | ms | MB | ||
| static (300 articles) | 14.4 | 18.6 | 543.9 | 291.7 | 38x |
| ecommerce (250 cards + JS) | 15.8 | 20.8 | 753.8 | 309.4 | 48x |
| clientrender (400-row JS table) | 23.3 | 28.8 | 576.2 | 300.9 | 25x |
| compute (JS-heavy) | 30.9 | 33.3 | 512.0 | 283.4 | 17x |
| deep (60-level nesting) | 10.1 | 16.8 | 497.2 | 278.9 | 49x |
| median | 15.8 | 20.8 | 543.9 | 291.7 | 34x |
34x faster, 14x less memory — with layout. Layout roughly doubles page cost
on DOM-heavy pages (static: 6.1ms → 14.4ms) and is skippable with
--no-layout.
| engine | wall | pages/s | PSS | output |
|---|---|---|---|---|
| lightbrowser (batch) | 0.25s | 79.4 | 45 MB | complete |
| Chrome CDP, 300ms settle | 8.57s | 2.3 | 360 MB | complete |
| Chrome CDP, no settle | 2.88s | 6.9 | 341 MB | truncated |
34x faster, 8x less memory against a reused Chrome instance driven over CDP — the configuration most favourable to Chrome.
The third row is the interesting one. Chrome has no "the page has gone quiet"
signal, so an automation script must guess a settle time. Guess low and you
silently capture a half-rendered page: at 0ms settle Chrome reports 800
elements on clientrender where the correct answer is 802 — it misses the row
added by a setTimeout. lightbrowser gets the complete answer at full speed
because it drains the timer queue rather than waiting on a wall clock.
40 external scripts, 40ms latency each, on a server that closes every connection:
| concurrency | total | subresource phase |
|---|---|---|
| 1 (serial) | 1681ms | 1633ms |
| 2 | 876ms | 831ms |
| 4 | 470ms | 425ms |
| 8 (default) | 257ms | 212ms |
6.5x end-to-end. All 40 scripts execute in document order at every setting.
Every element matched by a, button, input, select, textarea, compared
element-by-element against Chrome at the same 1280×800 viewport. y is also
reported as drift relative to page height, because an absolute pixel tolerance
is meaningless on a page 40,000px tall.
| page | elements | visibility | x ±5px | w ±5px | h ±5px | y drift |
|---|---|---|---|---|---|---|
| static | 303 | 100% | 100% | 100% | 100% | 0.05% |
| ecommerce | 253 | 100% | 99% | 99% | 100% | 3.6% |
| hidden (adversarial) | 8 | 88% | 33% | 100% | 100% | — |
| wikipedia (live) | 1979 | 81% | 1% | 88% | 95% | 45% |
Visibility — the thing agents actually need — agrees with Chrome on 100% of elements across the synthetic corpus and 81% on a live Wikipedia article.
The one disagreement on the adversarial page is the .sr-only clip pattern,
where a naive getComputedStyle check calls a clipped 1×1 element visible and
we call it hidden. Ours is the answer an agent wants.
Wikipedia is the honest picture of a complex real page: 81% agreement, split
roughly evenly between elements we hide that Chrome shows and vice versa, and
absolute y drifts badly because floats and table layout are not modelled.
Here is what layout buys, on a page built from the standard hiding patterns:
$ lightbrowser hidden.html --geometry $ lightbrowser hidden.html --no-layout
[8] document "Visibility" [8] document "Visibility"
[10] link "Real link" @8,10 62x20 [10] link "Real link"
[26] link "Shown on desktop" @212,10 131x20 [13] link "Screen reader only" <- not visible
[32] button "Visible button" @343,8 96x22 [17] link "Inside hidden modal" <- not visible
[20] link "Faded out" <- not visible
[23] link "Collapsed" <- not visible
[26] link "Shown on desktop"
[29] link "Hidden on desktop" <- not visible
[32] button "Visible button"
Five of eight targets were phantoms. An agent working from the right-hand column spends its turns clicking things that are not on the screen.
What an agent pays for in tokens:
| page | raw HTML | semantic | reduction |
|---|---|---|---|
| bbc.com/news | 391,657 B | 12,598 B | 31.1x |
| wikipedia (Rust article) | 1,165,761 B | 158,133 B | 7.4x |
| docs.python.org | 19,313 B | 7,336 B | 2.6x |
| news.ycombinator.com | 34,929 B | 14,576 B | 2.4x |
9 of 10 load and produce a usable tree. Serial → parallel prefetch:
| site | nodes | semantic | serial | parallel |
|---|---|---|---|---|
| github.com/rust-lang/rust | 3,252 | 263 | 6696ms | 474ms |
| docs.python.org | 854 | 132 | 1224ms | 308ms |
| rust-lang.org | 639 | 99 | 1693ms | 428ms |
| wikipedia | 18,850 | 1,621 | 1457ms | 1178ms |
| news.ycombinator.com | 1,303 | 232 | 15252ms | 4023ms† |
| bbc.com/news | 1,692 | 146 | 9267ms | 5929ms |
| crates.io | 115 | 1 | — | — ‡ |
| stackoverflow.com | — | — | — | — § |
† Also needed a bounded TCP connect timeout — HN's first DNS answer is a blackholed IPv6 address, and the OS connect timeout was costing 10s per connection. curl avoids this with Happy Eyeballs; we approximate with a 1.5s connect deadline. 15.2s → 4.0s. ‡ Ember SPA — renders essentially nothing without fuller framework support. A real gap, not a crash. § HTTP 403 bot block, not an engine failure.
Measured against Web Platform Tests,
the suite browser vendors are held to. 1,315 files across dom, domparsing,
html/dom and css/cssom, run through testharness.js unmodified via its
vendor reporting hook. Setup and caveats: bench/WPT.md.
| area | subtest pass rate | subtests reached | lightpanda |
|---|---|---|---|
| dom (core DOM) | 93.9% | 50,683 | 97.3% |
| html/dom (IDL reflection) | 93.3% | 26,581 | 93.7% |
| css/cssom | 47.8% | 1,487 | 51.8% |
| domparsing | 9.4% | 1,533 | 16.0% |
| overall | 91.3% | 80,284 | 93.0% |
Read the counts, not just the rate. The engine went from passing 7,541 subtests
to 73,262, while the rate moved 21.3% → 91.3% — a smaller-looking gain,
because most of dom/ used to abort at setup and never register a test, so
early rates were computed over a much easier denominator. It now reaches 80,284
subtests against Lightpanda's 83,014, so the two are measured over comparable
ground.
Read that with two caveats. First, the suite is run against a plain static
server rather than the full wptserve, so tests needing custom headers,
subdomains or TLS fail for reasons unrelated to the engine. Second, and more
importantly: an engine that aborts early runs fewer subtests, so its pass
rate is computed over an easier denominator. Quote the counts alongside the
rate.
domparsing is the weakest area by a wide margin and is next: it is fragment
parsing, serialization and the sanitizer, and the dom work barely touched it.
css/cssom is dominated by shorthand and computed-value serialization rather
than by the object model, which is implemented. The largest single gap in dom
is now XML: several test files load XML documents into frames, and this engine
has only an HTML parser.
The reflection layer is generated from the metadata WPT itself uses
(tools/gen_reflect_table.py), so it stays honest about which specification it
implements.
git clone --filter=blob:none --sparse --depth 1 \
https://github.com/web-platform-tests/wpt.git /tmp/wpt
cd /tmp/wpt && git sparse-checkout set resources dom domparsing html/dom css/cssom common
cp /path/to/lightbrowser/bench/testharnessreport.js resources/testharnessreport.js
python3 -m http.server 18731 --directory /tmp/wpt &
WPT_PORT=18731 ENGINE=ours python3 bench/wpt_run.py dom domparsing html/dom css/cssomConformance says whether an engine is right; this says what it costs. Median over eight WPT files spanning 40 to 10,920 subtests, three runs each, peak summed PSS across the process tree.
| engine | median wall | median peak PSS |
|---|---|---|
| lightbrowser | 854 ms | 108 MB |
| lightpanda | 1752 ms (2.1x) | 111 MB (1.0x) |
| Chrome 150 headless | 2026 ms (2.4x) | 537 MB (5.0x) |
Per file, the shape matters more than the median:
| file | subtests | lightbrowser | lightpanda | Chrome |
|---|---|---|---|---|
| Node-nodeName | 40 | 34 ms / 20 MB | 1248 ms / 24 MB | 610 ms / 283 MB |
| Element-classlist | 1,420 | 134 ms / 39 MB | 1537 ms / 73 MB | 1129 ms / 386 MB |
| Range-comparePoint | 5,580 | 1492 ms / 118 MB | 1966 ms / 148 MB | 2924 ms / 689 MB |
| Range-compareBoundaryPoints | 9,313 | 2370 ms / 218 MB | 3469 ms / 291 MB | 6126 ms / 1026 MB |
| Range-set | 10,920 | 5624 ms / 238 MB | 4515 ms / 352 MB | 6795 ms / 1151 MB |
Three honest readings of that table:
- The advantage on small pages is startup, not throughput. Lightpanda pays
roughly 600 ms of fixed cost per invocation; on a 40-subtest file that is the
entire difference. Its
servemode amortises it away. - On the largest file Lightpanda is faster (4.5 s against 5.6 s). The lead is not uniform.
- Chrome's 5x memory buys layout, paint and compositing, none of which the other two do. It is the cost of being a real browser, not waste.
Firefox 154 is installed here but would not complete a headless page load in
this environment, under --headless or Xvfb, so it is absent rather than
estimated.
cargo build --release
lightbrowser https://example.com # semantic tree
lightbrowser https://example.com --dump text # visible text
lightbrowser https://example.com --dump html # settled DOM
lightbrowser https://example.com --select '.price' # CSS query
lightbrowser https://example.com --dump json # timing/size stats
lightbrowser https://example.com --geometry # semantic tree with boxes
lightbrowser https://example.com --explain '#login' # why is this hidden?
lightbrowser --batch urls.txt # many pages, one process
lightbrowser --file page.html --no-external # offline--step runs a script after the page settles. If a step navigates — a form
submit, a link click, an assignment to location — the navigation is followed
and the next step runs on the page it landed on. Cookies persist across the
whole flow, so a session survives.
That is the whole model: observe, act, observe.
lightbrowser https://www.flipkart.com \
--step 'document.querySelector("input[name=q]").value = "iphone 15";
document.querySelector("button[type=submit]").click(); ""' \
--step 'JSON.stringify([].map.call(
document.querySelectorAll("a[href*=\"/p/\"]"),
function (a) { return a.textContent; }))'A worked version is in examples/product-search.sh,
which types a query into a real shopping site, clicks search, follows the
navigation, and prints the products with prices as JSON:
$ ./examples/product-search.sh "iphone 15"
{
"query": "iphone 15",
"found": 24,
"results": [
{ "name": "Apple iPhone 15 (Black, 128 GB)", "price": "₹56,900",
"url": "https://www.flipkart.com/apple-iphone-15-black-128-gb/p/itm6ac6485515ae4" },
...
The whole two-page flow takes about 3.3 s and peaks at 108 MB, most of which is network wait.
Flags: --no-js, --no-external, --no-layout, --concurrency N,
--viewport WxH, --heap MB, --timeout SEC, --quiet.
lightbrowser serve speaks the Chrome DevTools Protocol, so existing tooling
drives it without knowing what it is:
lightbrowser serve --port 9222import { chromium } from 'playwright-core';
const browser = await chromium.connectOverCDP('http://127.0.0.1:9222');
const page = browser.contexts()[0].pages()[0];
await page.goto('https://example.com');
await page.fill('input[name=q]', 'hello');
await page.click('button[type=submit]');
console.log(await page.title());Puppeteer connects the same way, via puppeteer.connect({ browserURL }).
examples/playwright-search.mjs is the
product search above, rewritten as an ordinary Playwright script — it runs
unchanged against Chrome by pointing the endpoint at Chrome's debugging port.
What works: goto, evaluate, $/$$/$eval/$$eval, locators and their
count/textContent/innerText/getAttribute/isVisible/isEnabled,
waitForSelector, waitForLoadState, fill, type, click, title,
content, inputValue, boundingBox, and element handles across worlds.
Two things make this work rather than merely connect. Clicking is a real hit
test: DOM.getContentQuads reports boxes from the layout engine, and
Input.dispatchMouseEvent resolves the coordinates back to an element through
elementFromPoint — so a click lands on whatever is actually on top, the same
way it would in a browser with a screen. And Runtime.callFunctionOn keeps a
table of live JavaScript handles, which is what lets a driver inject its own
bundle, hold element references, and call methods on them later.
Unimplemented: screenshots and PDF (there is no renderer), Network.*
interception, downloads, multiple tabs, and Page.goBack/goForward (no
history yet). A CDP command with no equivalent here answers emptily rather than
erroring, because a driver aborts the whole connection over an unrecognised
setting.
--explain answers the question that actually comes up, by walking the
ancestor chain until it finds what hid the element:
$ lightbrowser https://en.wikipedia.org/... --explain '#searchInput'
html.client-js... -> visible @0,0 1280x18421
body.skin-vector... -> visible @0,0 1280x18421
div.vector-header-container -> visible @0,0 1280x66
div#p-search.vector-search-box -> visible @264,8 972x50
form#searchform.cdx-search-input -> visible @264,8 500x32
div.cdx-text-input -> HIDDEN @264,8 448x0
input#searchInput -> NO BOX (display:none, or not laid out)
Semantic output looks like this — stable handles, roles, accessible names,
wrapper divs lifted away:
[69] document "Rust (programming language) - Wikipedia"
[72] link "Jump to content" href=#bodyContent
[85] checkbox "Main menu" type=checkbox id=vector-main-menu-dropdown-checkbox
[105] button "Move Main menu to sidebar"
[122] link "Visit the main page [z]" href=/wiki/Main_Page
python3 -m http.server 8731 --directory bench/pages &
SP=/tmp REPS=9 python3 bench/run.pybench/measure.py reports wall time and peak memory, combining
getrusage(RUSAGE_CHILDREN) (exact for one process) with process-tree sampling
of both RSS and PSS (needed for Chrome's multi-process model).
cargo test # 54 testsSelectors (type/id/class/attribute operators/combinators/structural
pseudo-classes/:not()/lists), id-index maintenance across mutation, CSS
parser recovery (comments, nested at-rules, @keyframes, strings containing
braces, truncated input), media-query evaluation, layout and visibility
(stylesheet display:none, inherited visibility, opacity, zero-size,
ancestor hiding, offscreen text-indent, specificity, !important, inline
overrides, flex placement, text wrapping, viewport membership), and the full
pipeline (implied tags, script mutation, innerHTML reparse, timer quiescence,
JS object identity, event dispatch, error isolation, runaway-timer budget,
Unicode round-tripping).
Six real bugs were found by these tests and fixed:
setAttributeappended a duplicate attribute instead of replacing it, on every parsed element. Element names are HTML-namespaced but attribute names are not; building both with the same helper meant theQualNames never compared equal. Reads kept returning the stale value.clearInterval(h)called from inside its own callback was a no-op, because a repeating timer is popped off the queue before it runs and so was rescheduled after being cancelled.- Subresources were silently dropped when a peer closed a pooled keep-alive connection — a missing script, and a page that renders wrong with no error.
- CSS comments were skipped but still captured in the returned text, so
a /* c */ > breached the selector parser with the comment embedded. - An unterminated
:not(panicked on an inverted slice range. - A blackholed IPv6 address cost 10s per connection before falling back.
Honest list. These are the reasons this is a skeleton and not a browser.
No inline flow. Inline boxes are laid out as a wrapping flex row rather
than as real line boxes. Widths and heights land within a few pixels; what
suffers is anything depending on true line-box behaviour — floats, vertical
alignment, and y on very long documents, where sub-pixel error accumulates.
No floats, no tables. float and real table layout are not implemented,
which is most of the residual Wikipedia disagreement.
Text metrics are estimated. Two calibrated advance tables (serif for content, sans for form controls) rather than font files. Accurate in aggregate, individually wrong for unusual scripts and any page whose web font differs markedly from the default metrics.
Web API surface. Present: DOM core and traversal, 397 reflected IDL
attributes, selectors, classList, cloneNode, namespaces, innerHTML/
textContent, insertAdjacent*, ParentNode/ChildNode methods, events, timers,
fetch with promises, DOMException, DOMParser, URL/URLSearchParams,
TextEncoder/TextDecoder, atob/btoa, geometry APIs, getComputedStyle,
document.cookie, localStorage/sessionStorage, IntersectionObserver
(reports intersecting, so lazy content loads).
Also present: Ranges, TreeWalker/NodeIterator, MutationObserver, frames,
document.activeElement and focus(), elementFromPoint/elementsFromPoint
hit testing, matches/closest, and text selection on form controls.
Absent, and visible in the conformance numbers: Shadow DOM, custom element
upgrades, ES modules, streams, XMLHttpRequest, FormData, Blob, and rule
editing in CSSOM (cssRules, insertRule).
This list is not cosmetic. Sites feature-detect it and deliberately serve the
degraded path when it comes up short: MediaWiki reverted Wikipedia to its
no-JavaScript mode — hiding the search box and the whole interactive
chrome — purely because localStorage was missing. Adding it moved visibility
agreement from 74% to 81% in one step. On real sites, layout fidelity is
bounded by JavaScript coverage, not by the layout engine.
Event model. Listeners fire on the target only; no capture, bubbling, or propagation.
<template> contents are not separated from the element.
Single-threaded per page. By design — one isolate, one arena, one thread. Scaling is thread-per-core, which is not yet wired up.
Not written yet: session persistence to disk, page snapshot/fork, deterministic replay, back/forward history.
Apache-2.0 — see LICENSE and NOTICE. Dependencies:
html5ever/markup5ever (MIT/Apache-2.0), v8 (MIT; V8 itself BSD-3),
taffy (MIT), ureq (MIT/Apache-2.0), url (MIT/Apache-2.0).
taffy (MIT) does block/flex/grid layout. selectors/cssparser (MPL-2.0)
are not used — the selector engine and CSS parser here are original — but
are the natural upgrade path, and MPL's file-level copyleft permits linking
into a permissively licensed binary.