This is for calling the compiler from an application, or from a language other than Rust. It covers the things that only matter once the compiler is inside something long-lived.
Every entry point is the same shape: give it source, give it options, get HTML back.
use wikitext::{compile, Options};
let html = compile(source, &options);Nothing is retained, and nothing is cached between calls. There is no global state, so threads can compile at the same time with nothing shared.
One detail matters when you spread work across threads: Options is not Send,
because the Lua module provider inside it holds an Rc. Build one per thread rather
than sharing one. Document is Send + Sync, so a parsed tree can move between
threads freely, and any TemplateResolver is Send + Sync by its trait bound.
Options::default() allocates. For a server that compiles many pages, build the
common part once and clone it per call:
let base = Options { title: Some("Home".into()), ..Options::default() };
for page in pages {
let options = Options { title: Some(page.title.clone()), ..base.clone() };
let html = compile(&page.text, &options);
println!("{} -> {} bytes", page.title, html.len());
}Options is Clone, and the resolver behind it is an Arc, so cloning does not
duplicate a template store. Clone it per call rather than per page if you can.
Neither is built in, and that is a deliberate boundary: a compiler has no business
knowing where your pages live. Implement TemplateResolver for templates and
ModuleProvider for Lua, and hand them over.
- Templates covers the resolver.
- Lua modules covers the provider.
A resolver is called once per transclusion and runs on the compiling thread, so it
must not block for long and must not re-enter the same Options.
compile never fails. A page with an unclosed table, an unclosed tag or a broken
comment compiles, and the damaged part stays as literal text. Problems are collected
as Diagnostic values when options.diagnostics is on:
use wikitext::{compile_with_diagnostics, Options};
let (html, document) = compile_with_diagnostics(source, &Options::default());
for diagnostic in &document.diagnostics {
eprintln!(
"{:?} {} at {}:{} {}",
diagnostic.severity,
diagnostic.code.as_str(),
diagnostic.position.line,
diagnostic.position.column,
diagnostic.message
);
}This is the same stance MediaWiki takes, and it is why importing a wiki dump does not
stop at the first bad page. If you need strictness, treat any Error severity
diagnostic as a failure yourself.
The input to this compiler is untrusted by assumption. What that means in practice, in order of importance:
- Bound the work. The parser is linear in input size, and Lua execution, template expansion and nesting depth all have ceilings. Add a wall-clock timeout anyway; a caller-side timeout is cheap and catches the case neither of you thought of.
- Cap the input size before compiling, if your users can submit large pages.
- Sanitise the output if you serve it to other users. The compiler emits HTML5
fragments and does not remove
<script>, exactly as a wiki would not.
SECURITY.md has the full threat model and the limits in force.
Inline nesting is capped at 40 levels, matching MediaWiki's order of magnitude. Beyond that the inner content is kept as literal text rather than being dropped, so nothing is lost, but the output is not what a page that deep intended. If you are compiling generated markup that legitimately nests deeper than that, raise the ceiling and check what it does to your stack.
Stack depth is the reason the ceiling exists: a link label is re-parsed, so nesting
recurses. Deep tags and deep trees are safe because Drop and the serializer are
iterative, but the inline parser is not.
The C ABI is the foundation. Everything else builds on it or mirrors it.
| Language | Page |
|---|---|
| C and C++ | c.md |
| Go | go.md |
| Node.js | node.md |
| Anything else | ffi.md |
Two invariants hold at that boundary, both with tests:
- No call aborts the process. Every exported function is wrapped so a panic
becomes
WIKITEXT_ERROR_PANICwith a readable message, because unwinding into a foreign caller is undefined behaviour. - Allocation and deallocation stay on one side. A buffer the library allocated is freed by the library's own free function; your data is only borrowed for the duration of the call.
Handles are not thread safe in any binding. Give each thread or worker its own.
When the HTML is not enough, keep the tree and walk it:
use wikitext::{parse, Options};
let document = parse(source, &Options::default());
for heading in &document.headings {
println!("h{} {}", heading.level, heading.title);
}
for category in &document.categories {
println!("category {} sort {:?}", category.name, category.sort_key);
}
println!("{} references", document.references.len());Document::text_content returns the page's plain text, useful for search indexing
without a separate HTML-to-text pass.
Document is plain data, so it can be serialized as often as you like with different
settings:
use wikitext::{parse, serialize, Options, OutputMode};
let document = parse(source, &Options::default());
let fragment = serialize(&document);Both parse and serialize are public, so a caller can also stop between them and
inspect or rewrite the tree before serializing.
An empty page compiles in about a microsecond, a 19 KB article in about 1.35 ms, and a 150 KB page in about 11.8 ms. Parsing is roughly 94% of that, so there is little to gain from optimizing serialization.
Absolute numbers vary by machine, and the same configuration has been measured to
vary by a factor of two on one machine. Measure with
cargo run -p wikitext --release --example latency rather than trusting a figure
from here. AGENTS.md §9 has the breakdown by pipeline stage.