`base-markdown`

A Markdown parser for Rust that follows CommonMark, apart from a documented restriction on angle-bracket autolinks, with pull events, a source-backed AST, extensions, and HTML and Markdown renderers. It builds on markdown-rs, with compact CommonMark parsing and a single source-backed document representation. That restriction is deliberate: only a fixed table of schemes, such as https://, mailto: and tel:, and email addresses form angle-bracket autolinks, so text such as Option<std::io::Error> stays text.

Motivation

Socketry's documentation tooling needs to inspect, edit, and render Markdown sections as structured data. This fork builds on markdown-rs with a pull-based event parser, a single source-backed document AST with editing helpers, shared HTML and Markdown renderers, and optional syntax extensions for richer documentation.

Usage

cargo add base-markdown

The library supports Rust 1.87 and later.

For direct conversion from Markdown source to HTML:

let html = base_markdown::to_html("## Hello, *world*!");
assert_eq!(html, "<h2>Hello, <em>world</em>!</h2>");

The default options are CommonMark with GFM tables, task list items, footnotes, ~~ strikethrough and literal autolinks, plus frontmatter. Options::gfm() follows GitHub more closely, with single-tilde strikethrough and the tag filter, and Options::commonmark() enables no extensions. The Options documentation lists each preset and how it differs from its namesake:

use base_markdown::{to_html, to_html_with_options, Options};

assert_eq!(to_html("~~done~~"), "<p><del>done</del></p>");
assert_eq!(to_html_with_options("~~done~~", &Options::commonmark()), "<p>~~done~~</p>");

See the API documentation for pull events, document editing, parser options, and frontmatter, the architecture guide for the parser design, and the performance comparison for benchmarks against other Markdown engines.

Releasing

Prepare a release with cargo bake cargo:release:patch (or minor, major, or bump --version X.Y.Z), which bumps the version, commits it with a Release: vVERSION trailer, validates it and opens the release pull request. After review and merge, GitHub Actions publishes the release when the configured crates-io environment approves it, then creates or updates the matching GitHub Release from releases.md. Follow the shared Releasing skill for the standard process.

Releases

See releases.md for the full release history.

v0.2.0

0.2.0 makes Document an editable tree with one content model and named references, and makes normalize and to_markdown round trip; the contract states what they guarantee. Upgrading from 0.1.x needs the changes under Breaking changes: most callers need only call Document::normalize before to_markdown, and update code that matches EventKind::TaskMarker, Tag::Document, code block language, math or reference targets.

Breaking changes

A reference is a name and a definition a binding of it, so a document stores a reference by its label alone and finds where it leads when asked, and it follows every edit to the definitions. Tag::Link { destination, title, reference } and Tag::Image { … } become Tag::Link(stream::Target) and Tag::Image(stream::Target): Target::Resource { destination, title } for a link of its own, Target::Reference(Box<Reference>), its kind and label, which is how a document stores a reference, and Target::Resolved(Box<Resolved>), the reference with the destination and title of the definition it is bound to, which the event stream of EventParser, to_html and IncrementalParser carries as before. A reference leads where the first link reference definition of its label in document order does, as the parser matches labels, and nothing stores the binding: a document keeps an index of the definitions and footnote definitions of the attached document by label, with the binding of each first, which every edit keeps, so Document::definition, footnote_definition and destination take constant time usually and at most time logarithmic in the number of labels, whatever they are; destination looks a detached node up in its own tree, in time linear in that tree, so calling it, or a per-subtree operation, on each node of a large detached tree takes quadratic time (#226). A footnote call calls the first footnote definition of its identifier to end, which is one nested in another of its identifier before the other, as the HTML renderer shows that one. A document may hold copies of a binding, later definitions of a label or identifier, as the parser and AstBuilder accept them, but no edit adds one to the attached document: Document::append_child, attach, set_kind and parse_children reject, with StreamError::InvalidValue and before changing anything, a definition or footnote definition of a label or identifier the attached document already defines, a subtree defining one twice, and moving within the document a definition of a label that has copies, which could change which one binds; insert_from and replace_heading_section copy no link reference definitions, turning references into resources, and relabel a copied footnote definition whose identifier is taken, or reject it under FootnotePolicy::Reject. A detached tree has no index, so edits inside one are not checked for copies, which would make them quadratic (#226), but attaching a tree that holds copies is rejected. Removing a binding makes its next copy the binding. Document::normalize removes copies, which render nothing, keeping a definition in a footnote definition that is a copy where it is when it binds its own label, so a normalized document defines each label once, and, as for other nodes it removes, leaves what else a copy held unspecified, its IDs no longer in the tree; Document::is_normalized reports a copy. Removing copies that leave the only item of a loose list with fewer than two blocks makes the list tight, as such a list cannot be written loose, which changes how the item's paragraph renders. Removing a copy, or hoisting a definition out of one, can leave blocks in an item of a tight list that cannot be written as they are, which get the same treatment as after any edit: two paragraphs in a row are joined with a soft break, an ordered list directly after a paragraph is made to start at 1, which drops its start in the HTML, and otherwise the list is made loose, which changes how that list's paragraphs render. Each place a copy leaves, or a definition hoisted out of one fills, is given the separators the parser puts there, and so is the end of the document, so the output reads back as the tree, separators too, and normalizing what it reads back as writes the same; where a copy after a list or at the end of the document goes, the list is then written next to what follows, or at the end of the output, which can add or remove a line ending in the HTML, as - [c]: /6\n\n[c]: /6\n, which renders <ul>\n<li></li>\n</ul>, normalizes to - [c]: /6\n, which renders a line ending after the list. Separators that an edit left, which the parser could not have given, stay. An item above a copy keeps its spread. Removing a footnote definition that is a copy removes the footnote calls in it, which render nothing but which the renderer counted when numbering footnotes by first call, so footnote numbering, anchors and entries then follow only the visible calls (#225): call numbers, anchor ids and the order of entries can change, back references to calls that are not in the output go, and so does the entry of a footnote called only there. An item whose only block is a copy is left empty, and removed, as an empty item is, where it would start a list directly after a paragraph in an item of a tight list, which cannot be written. A footnote definition that is a copy of one it holds, or of one around it, is removed in one pass in document order, with what binds in it taking its place, so normalizing then changes nothing. Attaching or detaching a subtree, and Document::extract_children and replace_heading_section, take time linear in what moves, to mark its nodes attached or detached and to index or unindex its definitions; a node's lookups in the attached document take constant time whatever its depth, as each node records whether it is attached. The new Document::bindings builds a Bindings table of the attached document in one pass, with definition, footnote_definition and destination lookups in constant time usually and at most time logarithmic in the number of labels. Document::events, subtree_events, text_content, render_node and the serializer look each reference up in the index, so calling them on each section of a document takes time linear in the document, and give each bound reference as Target::Resolved, so a parsed document's events and HTML are the parser's. An event stream carries resolved targets only: render_events rejects a Target::Reference with StreamError::InvalidValue. A footnote call that no footnote definition in a stream calls fails render_events with StreamError::Unbalanced, while a document built from that stream keeps it, as an unbound call, which renders as its text. AstBuilder keeps the name of a resolved reference only when its label binds to a definition with the same destination and title in the built document, and otherwise stores it as a resource to them, so it never loses its meaning; set_kind and append_child store one as a resource too, and deserialization rejects one with StreamError::InvalidValue; the serialized form of a reference has no destination. A reference or footnote call that no definition binds is valid but not normalized: the document's events give it as the text it reads back as, so it renders, gives text_content, headings and heading ids, that text, and a footnote call without its definition renders without an error or a footnote section, and Document::normalize makes it that text, [b][x] becoming text [b][x], as it makes a footnote call that no footnote definition binds text [^n], so being normalized is a property of the whole document. A detached subtree binds only its own references. The serializer writes a reference as one when it is bound to the first definition of its label in what is written, and otherwise as a resource to where it leads, as a subtree written without its definitions was before; a bound reference copied from another document by Document::replace_heading_section becomes a resource to where it led there, and an unbound reference or footnote call becomes the text it rendered as there, as Document::normalize makes it. A reference built with a destination of its own that did not match its definition, which was written as a resource, cannot be built.
Carry a reference's label, and a footnote's, in its event, as the parser read it, so every way of building a document from events builds the one Document::parse does (#192). Tag::Link and Tag::Image gain reference: Option<Box<stream::Reference>>, boxed so Event stays 96 bytes, the new struct of the reference's kind and label, which is None for a resource or autolink. Tag::FootnoteDefinition(identifier) becomes Tag::FootnoteDefinition { identifier, label } and EventKind::FootnoteReference(identifier) becomes EventKind::FootnoteReference { identifier, label }, where label is the label as authored, and None writes the identifier. NodeSyntax::Reference and NodeSyntax::Footnote are removed, along with their inference from a node's span: set the event with Document::set_kind instead. The label is the text the parser read without the prefixes of the lines it continues on in a container, such as a block quote's >, which only Document::parse recorded before, so AstBuilder over EventParser or IncrementalParser built a document whose label held the prefixes, and wrote > [x][a…\n> b] with a long label as a resource. A link copied by Document::replace_heading_section has no reference, and is written as a resource as before. The serialized form changes with the events, and a Reference or Footnote syntax entry is rejected when deserialized. Where a block quote or list item consumes only part of a tab before a line of a label, the label keeps the tab's other columns as spaces at the start of that line, as code does, so > [x][a\n>\t ```] has the label a\n ```` and is written back with them.
Check reference, definition and footnote labels with one rule, the labels the parser can read, with StreamError::InvalidValue, wherever an event enters a document or renderer (#192). A label must hold something other than whitespace, no carriage return, no blank line, no bracket without a backslash before it, not end in a backslash that would escape the closing bracket,. The rule does not depend on parse options, and accepts every label the parser reads under any of them, so a line such as # b, which a parser with headings turned off reads as part of a label, is accepted; Document::normalize indents it so it reads back. A full reference's, a definition's and a footnote's label is at most 999 characters, counting a tab as the columns it spans, up to the next multiple of four, from the column that suits each of its lines best, where it counted one: a label of 300 tabs and a letter, which was written as a reference that read back as text, is rejected. A shortcut or collapsed reference's label is not limited, since the parser reads one at any size; one longer than a full reference's that no longer matches the link's text is written as a resource. A footnote definition's label also holds no whitespace. A footnote identifier must be the normalized form of its label, as the parser gives it, or of itself when there is no label, so a lowercase identifier such as n is rejected, where it was written as a call that read back as N. An identifier is not limited, so a parsed identifier that case folding makes longer than 999 bytes, such as one of 499 ΐ, is accepted. An empty label, which was written as [] and read back as text, and a label with a carriage return are now rejected.
One content model decides what each node can hold, so an invalid tree cannot be built (#194). The document root, block quotes, items and footnote definitions hold blocks and separators; paragraphs, headings, table cells, emphasis, strong, strikethrough, links and images hold phrasing; a list holds only items; a table holds heads and bodies, which hold rows, with separators between them; a row holds only cells. A root whose first child is phrasing is an inline document and holds only phrasing, and phrasing and blocks never share a parent. A footnote reference cannot be inside a link at any depth, nor a soft or hard break inside a table cell, which is one line of its row (#214). AstBuilder, the HTML renderer, Document::append_child, Document::attach, Document::replace_heading_section and serde deserialization check it against the parent, and reject a node that does not fit with the new StreamError::InvalidNesting. Text in a block quote or item, a block in a paragraph or heading, an item outside a list, a row directly in a table and a separator in phrasing, a list, a row or a cell were accepted, and are now rejected. A heading level outside one through six is now StreamError::InvalidValue rather than StreamError::Unbalanced. Document::render_node renders a subtree as a fragment, so its root's children need not fit a document. The Markdown serializer no longer returns the unsupported-node error, which only such trees could reach.
A task item's checkbox is now Tag::Item { checked: Option<bool> }, and EventKind::TaskMarker is removed. The checkbox is written before the item's first block other than definitions and separators, which must be a paragraph; Document::normalize makes an item without one an ordinary item.
Frontmatter is now the body in Tag::Document { line_ending, frontmatter }, which replaces Tag::Document(line_ending), with its fences and info as the root's NodeSyntax::Frontmatter, rather than an EventKind::Opaque first child, which is removed. The body is the lines between the fences, each with its line ending, so ---\na\n--- has the body a\n and ---\n\n--- the body \n. Document::normalize makes its CRLF and lone CR line endings LF, as the parser stores them, and adds LF to a body that is not empty and has no final line ending, so a body set to a becomes a\n; Document::is_normalized reports both. A root with frontmatter holds blocks: giving frontmatter to a root that holds phrasing, or appending phrasing to a root with frontmatter, returns StreamError::InvalidNesting, as does a document start inside a document, in Document::append_child, AstBuilder, the HTML writer and deserialization. Document::frontmatter returns the body and Document::frontmatter_mut edits it, replacing Document::frontmatter_value; IncrementalParser delivers it on the document start. Document::replace_heading_section no longer copies a replacement's frontmatter. The serialized form of a document no longer records edited frontmatter nodes.
Document::set_kind changes a node only where its children and its place stay valid: a container keeps its kind, except that emphasis, strong and strikethrough are interchangeable, and a leaf keeps its variant, with HTML staying a block or inline, and an inline root takes no frontmatter. A heading can change its level, a link its destination, code its language and a list whether it is ordered, but a paragraph can no longer become a heading, nor text inline code or a soft break; any other change returns StreamError::InvalidNesting. Create the new node, move the children with Document::attach and detach the old one instead.
Store the whole info string of a code block as one value, EventKind::CodeBlock::info, in place of language, as cmark and cmarkly do (#188). The parser trims it of the spaces and tabs at its edges and decodes it once, with backslash escapes and character references only when Constructs::character_escape and Constructs::character_reference are on. The language is derived from it with the new stream::code_language: the first word, up to the first space, tab or line ending of the decoded info string. NodeSyntax::Code is removed, along with its lazy inference from the source, which guessed the decoding because a document does not keep its parse options. A block parsed with escapes or references off now writes its info string as it was read: ```x & parsed without references was written ```x &, and is now written ```x \&, which reads back with the default options as the same info string. An empty info string is code without one, None, so Some empty info is rejected with StreamError::InvalidValue, as an empty language was. The serialized form of a document changes: a code block stores info rather than language, and a Code syntax entry is rejected when deserialized. A document serialized before this change deserializes with no info string on its code blocks, since the old language field is ignored.
The language in class="language-…" is now the first word of the decoded info string, as cmark splits it, so a space, tab or line ending written as a character reference ends it: ```a b c renders class="language-a" rather than class="language-a b", ```a	b and ```a
b render class="language-a" rather than a language holding a tab or line ending, and ``` b, whose first word is empty, renders no class rather than class="language- b". Info strings without encoded whitespace render as before.
Remove math. $ is now always text, which is never escaped when serializing Markdown, and a ```math fence is ordinary code, rendered with class="language-math". Removed are Constructs::math_flow and Constructs::math_text, ParseOptions::math_text_single_dollar, MarkdownOptions::single_dollar_text_math, EventKind::Math, and the math field of EventKind::CodeBlock, which serialized documents no longer carry. A serialized document with a Math event no longer deserializes, and a math field on a code block is ignored, so such a block reads as code.
Remove MarkdownOptions::setext. Headings are written as ATX headings, and as setext headings only when they hold a soft or hard break, or a line ending in text or inline HTML, which an ATX heading cannot hold; after a block it would continue in an item of a tight list, a line ending in text is written as 
 in an ATX heading instead (#187). Output with the default options is unchanged.
Options::default(), ParseOptions::default() and Constructs::default() now enable GFM tables, task list items, footnotes, ~~ strikethrough and literal autolinks, and frontmatter, on top of CommonMark. Single-tilde strikethrough stays off, so ~5 min is text. to_html uses these options.
Code-fence frontmatter is no longer recognized. A document that starts with a backtick or tilde fence, such as ```json, always has an ordinary code block; write --- json for frontmatter with a format hint. Frontmatter is only --- or +++ on the first line, optionally followed by a format hint. The Markdown serializer always writes --- or +++: a body with a line that would close the fence switches to the other delimiter instead of a tilde fence, a body with both a --- and a +++ line is an error, and a frontmatter node built with a backtick or tilde fence is written as --- <info>, or plain ---, which implies YAML, when its info is empty. The closing fence is the opening delimiter on a line of its own, optionally followed by spaces or tabs. The serializer's check for a body line that would close the fence recognizes LF, CRLF and lone CR line endings, as the parser does. This also fixes a tilde code fence at the start of a document being written as a backtick fence that read back as frontmatter (#107).
Remove ParseOptions::normalize. normalize_markdown parses with ParseOptions::default(), so it no longer parses math or single-tilde strikethrough; use normalize_markdown_with_options for single-tilde strikethrough.
Link a GFM www. literal to https:// rather than http://, in every preset. This is a deliberate divergence from GitHub. A bare www. link whose destination is http:// is written by the Markdown serializer as [www.a.b](http://www.a.b).
Reject an inline code language that ParseOptions::inline_code_info would not read back, with the new StreamError::InvalidValue, wherever an event enters a document or renderer: Document::set_kind, Document::append_child, AstBuilder, render_events and deserialization. A language is a nonempty ASCII token that starts with a letter or digit and continues with letters, digits, _, -, +, # and ., as the parser reads it. A hand-built language such as a\nb, a b, -a or the empty string was accepted and written as a prefix that read back as text (#138). The Markdown serializer also encodes the last character of text directly before a language prefix when it would join the language, so text xa before rust: is written xarust: rather than xarust:, which read back as the language xarust, and an escaped character there is encoded instead of escaped, as x\.rust: read back with no language.
Reject code block languages and info strings that cannot be written, with StreamError::InvalidValue, wherever they are assigned (#144). An empty code language, which was written as a fence with no info string and read back with no language, is rejected by Document::set_kind, Document::append_child, AstBuilder, render_events and deserialization. Every other code language is accepted. Until #188 removed NodeSyntax::Code, Document::set_syntax and deserialization rejected its info string on code that did not start with its language as the parser reads it: it had to be empty for code without a language, and otherwise the language, alone or followed by a space or tab and the metadata. Info y z on code with language x lost its metadata, and xy z was written as ```x y z, which read back with info x y z. Parsed documents always meet these rules: the parsed info string of code now has NUL replaced with U+FFFD, as its language and every other parsed value do, where it kept the NUL, which the serializer already wrote as U+FFFD. The Markdown serializer now encodes whitespace at the end of code metadata, and metadata of only whitespace after the first space or tab, so info x a is written as ```x a  and x as ```x   , where both lost the whitespace. The spaces and tabs between the language and the metadata are still written as one space. It also escapes a ~ at the start of the language of a tilde fence, so language ~x is written as ~~~\~x rather than ~~~~x, which read back with language x. The language field and NodeSyntax::Code this describes are since replaced by EventKind::CodeBlock::info (#188), above.
The Markdown serializer now expects a normalized tree: call Document::normalize on a parsed or edited tree before to_markdown, to_markdown_with_options or to_markdown_node. Other trees are still written without panicking, but may be written incorrectly or unstably: the serializer no longer skips empty text, reads ahead across several text nodes, or merges adjacent attention itself. normalize_markdown and normalize_markdown_with_options normalize before serializing. Constructed emphasis of a and emphasis of b with empty text between them are now merged, and written as *a b*.
Document::replace_heading_section now copies the replacement as insert_from does: it no longer copies the replacement's link reference definitions, which its references no longer need, as they are copied as resources, nor footnote definitions that the copies do not call, and a footnote definition whose identifier the document binds outside the section is relabelled instead of rejected with StreamError::InvalidValue. A replacement that defines a label or identifier twice, or one the document already defines, is therefore no longer an error. The replacement's blocks get the separators the parser writes, so in an item, such as - # A\n\n old\n- x\n replaced with new, the output is - # A\n\n new\n\n- x\n, which reads back as the edited tree, rather than without the blank line before the next item; in an item of a tight list its rendering ends as the parser's does.
Store every source line ending as LF in parsed values, whether the source used LF, CRLF or a lone CR, as line endings are structure rather than content. This covers text, soft breaks, code, HTML, link and image titles and destinations, definitions, reference and footnote labels, and frontmatter. Positions are unchanged offsets into the source, and a CR left in a value comes only from a character reference such as . A CRLF document therefore serializes as the same document with LF line endings, without , and a shortcut reference whose label spans lines, such as [a\r\nb], stays a shortcut rather than becoming [a\nb][a\nb]. HTML rendered from a CRLF or CR document is byte-identical to HTML from the same document with LF: content, separators between blocks, and the line ending added after code without a final one are all LF. Tag::Document carries Some(LineEnding::LineFeed) for any document with a line ending and None otherwise, so CompileOptions::default_line_ending applies only to a document without one. This is a breaking HTML output and event change: HTML with CRLF or CR line endings can no longer be produced from a document that has line endings, and Tag::Document no longer reports the source's line ending. Document::text_content and heading text use LF too. The Markdown serializer no longer rewrites CR in its output, and a CR in a constructed code, HTML or frontmatter value, which a parsed value never has, is written as it is. Documents persisted with the serde feature by earlier versions from a CRLF or CR source hold source ranges with those line endings, which the serializer now writes as they are, such as a raw CR in a frontmatter body or  in a title; reparse them from their source. A CR or CRLF in a constructed definition or reference label is written as LF, which matches the same definitions, since labels match after collapsing whitespace.
Document::append_child and Document::attach reject a parent ID that is not in the arena with StreamError::InvalidRange, as every other edit rejects an invalid ID, where they returned StreamError::Unbalanced, which is now only for a leaf parent, an end event, a cycle and moving the root.

Added

Add the contract module, mirrored in context/normalize-serialize.md, which states as numbered clauses what a Document holds and what Document::normalize, Document::is_normalized, to_markdown and HTML rendering guarantee: names and bindings, the order and guarantees of normalizing, the round trip, which HTML normalizing changes, when a document is settled, edit errors and atomicity, costs, and every open issue as a known gap. tests/contract.rs tests each clause, with an ignored repro for each gap that has one.
Add Document::insert_from(parent, after, &source, nodes, &InsertOptions), which copies nodes of another document, such as its root's children, into a node after one of its children, and Document::heading_section_last(heading, subsections), the node to insert after to insert at the end of a heading's section (#224). The copies render as they do in the source: bound references become resources and unbound ones text, so link reference definitions are not copied, and each footnote call brings the footnote definition it calls, once, placed after the copies, while footnote definitions nothing copied calls are left out. A copied footnote definition whose identifier the document already binds, or calls without binding, which the copy would bind, is given a fresh one, a deterministic free one of n-2, n-3 and so on, with its calls rewritten, under the default FootnotePolicy::Relabel, or rejected with StreamError::InvalidValue under FootnotePolicy::Reject. InsertOptions::heading_shift shifts each copied heading's level, and a level outside 1 to 6 is rejected with InvalidValue rather than clamped. Into a container of blocks, the copies get the separators the parser writes, so two paragraphs in an item of a tight list stay two. The insert checks the content model as other edits do, changes nothing when it fails, and takes time linear in the copied content plus one index lookup per reference, call and copied definition, and, when it copies a footnote definition into a detached subtree or a document edited since it was parsed or normalized, one pass over the tree. It returns Inserted { nodes, footnote_definitions }.
Add Document::parse_children(parent, markdown, &options), which parses Markdown and replaces all of a node's children with it (#121). A paragraph, heading, table cell, emphasis, strong, strikethrough, link or image takes one paragraph's content, and rejects a fragment that parses into anything else, such as a heading, two paragraphs, a list or a definition, with StreamError::InvalidNesting; the document root, a block quote, an item or a footnote definition takes blocks; a list, table, table head, table body or row is InvalidNesting. A reference or footnote call in the fragment is a name, as in a parsed document, which the parser reads as one where a definition binds its label: one of the fragment's own, or one of the document's outside the replaced children, in the tree that holds the parent; one that nothing binds stays text, as does a footnote call inside a link. As an edit adds no copy of a binding, a fragment parsed into the attached document that defines a label or footnote identifier twice, or one that a definition of the document outside the replaced children already has, is rejected with StreamError::InvalidValue, changing nothing. The fragment is parsed once, the parser asking the document's index once for each label it does not define. A block fragment's definitions become nodes of the document. A table cell reads its fragment as written in a cell, so | must be escaped and a fragment with a line ending is rejected, and a link reads its fragment as written in its text, so one that holds a link, or whose brackets or trailing backslash would end the text early, such as [a or a\, is rejected. In an item, the fragment adds no separator that writing it there would not: trailing blank lines are dropped in any parent but the root, and in an item of a tight list a blank line between a paragraph and another block is dropped; an item of a loose list is spread as the fragment's blocks are. Frontmatter is never read from a fragment. Parsed values are owned. Parsing takes time linear in the fragment and the replaced children, plus a lookup of each label the fragment uses, whatever the size of the document, or, in a detached subtree, one pass over it. Document::replace_heading_section into an item now refreshes the item once rather than after each copied block.
Add commonmark() presets to Options, ParseOptions, Constructs and CompileOptions, equal to the previous defaults: CommonMark with no extensions. The CommonMark specification suite runs with them. ParseOptions::commonmark() differs from ParseOptions::default() only in its constructs, so its strikethrough_single_tilde is false, where the previous default was true: strikethrough turned on afterward needs two tildes, as in the default. Parsing with the preset itself is unchanged, as strikethrough is off.
Record how a link that is not a reference was written as NodeSyntax::Link { form: LinkForm }, which is Bare for a GFM literal such as www.example.com, Angle for <https://a.b>, and Explicit for [text](url). Parsed documents derive it from the source on lookup, copying a link keeps it, and set_syntax changes it. The Markdown serializer writes a link in its stored form where that form holds the link. A bare link is written as its text only when that text, read alone, is one whole GFM literal with the link's destination, which is http:// and the text for www., mailto: and the text for an email address, and the text itself for a URL, so a link whose text was edited to click here or https://evil.example keeps its destination as [click here](http://www.a.b). It also needs the character before it is the start of a line or whitespace, and the character after it is the end of the paragraph, heading or table cell, a line ending, or a space or tab before more text, since nothing else next to it can change where it ends. No punctuation counts, so see https://a.b/c. is written as see <https://a.b/c>. and www.x.org. as [www.x.org](http://www.x.org).. Anywhere else, and in a table cell when its text contains |, a bare link is written as <…> when that reads back as the same link, which holds for a URL whose scheme is in the link-scheme table and for an email address, and as [text](url) otherwise. This includes a bare link directly inside emphasis, strong or strikethrough, so *https://a.b/c* is written as *<https://a.b/c>*. A link without a form, including one built by hand, is written as [text](url), so a link whose text equals its destination is no longer written as <…> unless it was authored that way.
Add Document::normalize, which removes empty text, joins adjacent text and merges adjacent emphasis, strong and strikethrough of the same kind in place, in time linear in the size of the document, and Document::is_normalized, which reports whether it would change anything. Merging moves the children of the second node into the first and normalizes where they meet, so *a*_b_ becomes one emphasis, written as *ab*, rather than *a**b*, which parses differently. Separators are block-level, so one inside a paragraph, heading, table cell, attention, link or image is removed, as the serializer writes nothing for it; separators between blocks stay in place and are skipped when looking for adjacent nodes. The IDs of joined text and merged attention no longer refer to nodes in the tree.

Documents and normalizing

Document::normalize gives labels a shape that reads back wherever they are written with the default parse options: a line of a label after the first that would start a block, such as # b read with headings turned off, or could be a table delimiter row, such as |-|, under whatever line is written before it, or whose leading tabs start a block at some columns only, has its leading whitespace written as four spaces, a label longer than the parser reads at some column has its other tabs written as one space, a link or image whose label is longer than a full reference's can be at some column becomes a resource, which renders the same, and in an ATX heading or table cell each run of whitespace with a line ending in a label becomes one space. A footnote call whose label is still too long there, such as one of 999 letters and a line ending in a table cell, loses its label and is written as its identifier. Document::is_normalized reports each. A parsed shortcut reference of 251 tabs and a letter is now normalized to a resource.
Document::normalize unwraps a link inside another link's text, at any depth, into its children, unless it is an autolink that reads back as one, whatever the serializer's resource_link option, and removes one without children, as the parser reads no other link there. A parsed [<http://a.b/\|>](m) in a table cell, whose autolink would be written as a resource, now reads back. It also gives a table the shape the parser gives one: a head of the first row and a body of the others, every row as wide as the widest, with empty cells added, header flags and alignment following the head, and parts in order; a table without a cell is removed. Document::is_normalized reports both.
Document::normalize now also rewrites shapes that Markdown cannot write, or can write only with character references, into ones that render the same. Adjacent inline code is joined when the second has no language. Whitespace and soft breaks at either edge of emphasis, strong and strikethrough, and hard breaks at the end, move out through any attention nested at that edge, so text x, emphasis of a and text y become text x , emphasis of a and text y, written as x *a* y rather than x * a * y; attention left empty is removed. Links and images keep whitespace at their edges, and a hard break at the start of attention stays. Soft breaks at either edge of a paragraph, heading or table cell are removed, and so are hard breaks at the end of a paragraph or heading. A task marker at the start of a paragraph, heading or table cell followed only by breaks and at most one text that is a single space or tab, the separator the parser keeps in the text after the marker, has nothing written after it, so it read back as text [ ] or changed on every write; it is removed, and the rules above then apply to what is left, so an item of a marker and a hard break becomes an empty item, and one of a marker, text and a hard break is written -  . Anything else after the marker keeps it, and breaks at the end are trimmed, so a marker, text \u{a0} and a hard break is written - [ ] \u{a0}. A task item whose text starts with a tab separator, as in - [ ]\ta, is now written with the tab, where it was written with a space and read back with different text. A paragraph left with no children is removed with one separator beside it. Normalizing never changes text otherwise: whitespace at the edges of a block stays, and the serializer encodes it so that it reads back, so a paragraph of hi is written as  hi . Document::is_normalized reports each of these shapes. The Markdown serializer now also encodes a space or tab at either edge of a table cell, which it previously wrote raw and the parser trimmed (#109), so a hand-built cell of Text("www.a.b. ") is written as | www.a.b.  | and keeps its space. Since every line ending in a table cell is written as a reference, whitespace next to one and text that would start a line after one are no longer encoded there, so a cell of a, a soft break and b is written as | a
 b | and reads back the same. It no longer encodes whitespace inside the edges of attention, and no longer treats a hard break at the end of attention as a line ending when choosing and encoding its markers, so trees built with these shapes must be normalized before serializing, as normalize_markdown does. This supersedes the earlier entry that writes whitespace at the edges of attention as a character reference.
Append children in amortized constant time per child when building in document order. Document::append_child walked the parent's children to find the last one, and refreshed a list item from all of its children, so building a node with many children took quadratic time.
Document::normalize also removes emphasis, strong and strikethrough that have no children once their own children are normalized, such as emphasis of empty text or strong that holds only empty emphasis, and unwraps strikethrough inside other strikethrough at any depth, so that its children take its place, as the Markdown serializer writes every strikethrough with ~~ and cannot write one inside another. Text and attention that meet where a node was removed or unwrapped are joined or merged as before. ~a~~b~~c~ parses as <del>a<del>b</del>c</del> and was written as ~~a~~b~~c~~, which parses as <del>a</del>b<del>c</del>; it is now normalized to strikethrough of abc and written as ~~abc~~, which renders as <del>abc</del>. Text a, empty emphasis and text b are normalized to the text ab, rather than written as a**b. Document::is_normalized reports both shapes, and the IDs of removed attention and unwrapped strikethrough no longer refer to nodes in the tree. The root of a detached subtree is never removed or unwrapped. The serializer no longer avoids encoding characters inside empty attention.
Keep the looseness of a list when Document::normalize removes an empty paragraph from its item. A loose list whose only item is left with fewer than two blocks, such as an item of an empty paragraph and b, a soft break alone and b, or a lone task marker and b, cannot be written loose, so it was written - b and read back tight, rendering b rather than b; normalizing now makes such a list tight, and Document::is_normalized reports it (#128). The Markdown serializer now writes the only item of a loose list with a blank line between its blocks when none would be written otherwise, so an item of b and a nested list, or of two code blocks, reads back loose whatever spread it records. This also fixes a parsed item of a footnote definition and a paragraph after a blank line, which records no spread, being written without the blank line, so the paragraph read back as part of the definition.
Remove the leading empty items of a list that directly follows a paragraph in an item of a tight list in Document::normalize, and the list when none is left. An empty item cannot interrupt a paragraph, and - alone under one is a setext underline, so a hand-built item of p and a list of an empty item was written - p\n -, which reads back as a heading p, and an ordered one - p\n 1., which reads back as text p 1.; it is now written - p, and an item of p and a list of an empty item and a is written - p\n - a. Later empty items stay, as - p\n - a\n - reads back the same. Loose lists are left alone, since a blank line before the nested list makes the shape writable. This includes an item left empty by removing a task marker with nothing after it, such as a nested - [ ] built by hand, and lists in block quotes. Document::is_normalized reports the shape (#129).
Make an ordered list that directly follows a paragraph in an item of a tight list start at 1 in Document::normalize. Only an ordered list that starts at 1 can interrupt a paragraph, so a hand-built item of p and an ordered list of a starting at 3 was written - p\n 3. a, which reads back as text p 3. a; it is now written - p\n 1. a, keeping every item and its text. This also covers a list left starting at another number once its leading empty items are removed, such as a list starting at 2 of an empty item and a, which is written - p\n 1. a rather than - p\n 2. a. Loose lists, block quotes, footnote definitions and the document, where a blank line is written after the paragraph, keep the start, and Document::is_normalized reports the shape (#152).
Make a tight list loose in Document::normalize for the other blocks in a row in an item that can be written apart only with a blank line between them, which read back loose, and in some cases the output changed on the next write (#168): a link reference definition or a heading with a soft or hard break, which only a setext heading can write, after a paragraph, and a table, definition or such a heading after a list or footnote definition that ends in a paragraph or definition, or after a block quote whose last block is such a list or definition. The serializer and Document::normalize now decide which joins need a blank line with one rule, so they agree, and Document::is_normalized reports these shapes. In those places in an item of a tight list, the serializer writes any other heading as an ATX heading, as - a\n # h, with a line ending in its text written as 
, where it wrote a setext heading after a blank line, which made the list loose and changed on the next write, as for a parsed heading with a line ending from a character reference, such as - a\n # x
y, which is now written - a\n # x
y.
Remove HTML and lists that Markdown cannot write in Document::normalize, which Document::is_normalized reports (#149). Inline HTML without a value writes nothing, so the nodes on either side met as if it were not there, though normalizing kept them apart: a hand-built paragraph of Html(""), a shortcut reference [r] and text : /u was written [r]: /u, which reads back as a definition, and empty HTML between emphasis of a and emphasis of b kept them from merging, so they were written *a*_b_, which reads back as one emphasis once normalized; it is now removed, so text joins and attention merges, and the paragraph is written [r]\: /u (#146). An HTML block whose value is empty or holds only spaces, tabs and line endings was written as blank lines, which read back as nothing, or ended a list item or made a tight list loose: a loose item of Html("") and b was written -\n\n b, which reads back as an empty item and a paragraph. A list without items was written as nothing, or as a lone - in an item, so an item of p and a list whose only item holds a list without items was written - p\n -, which reads back as a heading p. Each is now removed with one separator beside it, as an empty paragraph is, and the rules for what that leaves apply in turn: that item is written - p, paragraphs that meet in an item of a tight list are joined, and a loose list whose only item is left with one block is made tight. Inline HTML of whitespace alone stays, as it is written as it is between text, and so does any HTML block with other characters. The parser produces none of these shapes, so parsed documents are unchanged. The Markdown serializer also no longer panics on hand-built inline HTML without a value at the start of a line, and treats inline HTML there as starting an HTML block only when its value starts with <.

Parsing

Only form angle-bracket autolinks for the schemes in the link-scheme table, described in the next entry, and the email form <user@host>. This is a deliberate divergence from CommonMark, which accepts any scheme of 2–32 characters and an empty target, such as <http:>, so prose such as Option<std::io::Error> became a link to std::io::Error. Other <scheme:…> text is now raw HTML when it is valid HTML, such as <xs:element> with html_tag_namespaces, and text otherwise. Markdown serialization writes links with other schemes, such as gopher://a, as [gopher://a](gopher://a) instead of <gopher://a>, and writes <…> only when it parses back as the same link. Autolink text is now written without backslash escapes, which autolinks do not process, so <http://a\b> no longer gains a backslash on each serialization. In GFM table cells, links whose text contains | keep the [text](url) form, because a | in an autolink would split the cell.
Use one table of link schemes for angle-bracket autolinks, GFM literal autolinks and the HTML renderer's safe href check, which previously disagreed. Each scheme has a kind that defines what must follow it: http, https, ftp, sftp, ssh, irc and ircs need :// and a host, mailto and xmpp need an email address, and tel and sms need one or more of +, digits, -, ., ( and ) with at least one digit. Schemes match in any case. Angle-bracket autolinks now accept all of them, so <irc://a.b>, <ssh://a.b> and <tel:+1-555-0100> are links, while <mailto:a>, which has no @, and <mailto:a@b.c,d@e.f>, which has several addresses, are now text. So are <mailto:a@b.c#x>, since an address may be followed only by a query, though with GFM the mailto:a@b.c inside it is still a literal link, and <mailto:@b.c>, which has no user. An address may be followed by a ?query, such as <mailto:a@b.c?subject=x>, and an xmpp address also by a /resource, such as <xmpp:a@b.c/phone?message>. GFM literals now also link ftp://, sftp://, ssh://, irc:// and ircs://, but never tel: or sms:. Ending a literal at a backslash applies to every host scheme. With html_tag_namespaces, inline <tel:1> is now an autolink rather than a namespaced tag, though alone on a line it still starts an HTML block. The safe href list gains ftp, sftp, ssh, tel and sms; image src stays http and https. CommonMark example 596, <irc://foo.bar:2233/baz>, is a link again, as CommonMark expects.
End GFM www., http:// and https:// literal autolinks at a backslash, as email literals already did, because a backslash is never valid in a host name or URI and browsers read it as /. The text after it is parsed normally, so www.com\_ links www.com followed by an escaped _, and a backslash hard break after a URL is no longer swallowed into the link. In GFM table cells, www.a\|b now links www.a followed by an escaped pipe. What precedes the backslash must still form a valid literal, so www.\ is not a link. A backslash directly before an email address now blocks the email literal, as / does, so x\a@b.c and http://x\a@b.c no longer link a@b.c. This is a deliberate divergence from GitHub, which links www.com\ as http://www.com%5C.
Require a domain after www. in GFM literal autolinks. Text such as www. a, www.), www.*, www. before a line ending, www.-, www.#a and www./a was linked, as www before trailing punctuation or as www. and a path, and is now text, so www.*a* is text followed by emphasis and www.[^1] keeps its footnote reference. A domain is what follows www. up to the first character that cannot be in one, so www.a still links. This is a deliberate divergence from GitHub, which links all of these, www.#a and www./a as www. followed by a path.
Require the domain of a GFM literal autolink to be a host name, as RFC 952 and RFC 1123 define it, for www., scheme and email literals: labels of letters, digits and -, separated by ., with no _, no empty label and no label that starts or ends with -. Non-ASCII letters, digits and other characters that are not Unicode punctuation or whitespace are allowed in labels for internationalized names (RFC 5890), so www.例.jp still links, while non-ASCII punctuation ends the domain as ASCII punctuation does. The domain still runs up to the first character that cannot be in one, or the trailing punctuation before the end of the literal, and an invalid domain gives no link at all, rather than a shorter one. So www._a.b.c, www.a_b.c.d, www.-a.b, www.a-.b, www.a..b, http://a-, https://a_b.com, a@b_c.com and a@-b.com are now text, as are www.a.# and https://a.bc.)d, whose domain ends in an empty label before the rest of the literal. www.a.b., www.a.b_ and _www.a.b_ still link www.a.b, since trailing punctuation is not part of the domain. The local part of an email, the /resource of an xmpp: address and the path after a domain may still have _. This is a deliberate divergence from GitHub, which links all of these.
Parse GFM literal autolinks in linear time. A www after _ inside a domain, which the _ makes invalid, scanned the rest of that domain again, so text such as www.a_, _www.a or www.例_ repeated took seconds to parse at a few hundred kilobytes; each domain is now scanned once, and one that starts inside it is decided from that scan, with the same result (#123).
Parse text into one event per source slice. Markers that stay literal, such as ., :, _, *, ~, @ and <, previously split text into several events, so a.b gave a, . and b; text is now split only at backslash escapes and character references, whose values are not one slice of the source. Document::parse does not normalize, so a parsed document can still have adjacent text, at escapes and references, and adjacent attention of the same kind, such as *a*_b_.
Apply CommonMark's emphasis rules unchanged next to GFM strikethrough. A * or _ run next to ~ is no longer treated as flanking on that side, which neither CommonMark nor GFM specifies, so a*~b~*c renders as a*<del>b</del>*c, as a*.b.*c stays literal. The rule of 3 now always compares the lengths of the original delimiter runs. A run next to ~ previously used what remained of its run after a partial match while the run it paired with used its original length, so .*****~~a~~_b_***** lost its outer emphasis. Both are deliberate divergences from micromark and markdown-rs, which treat such runs as flanking and apply the rule of 3 to remaining lengths, and the flanking change also diverges from cmark-gfm, which looks through ~ to the next character when classifying flanking.
Record spread for a parsed list item with a blank line after a nested list or footnote definition and before its next block. The nested block takes that blank line into its span, so Document::syntax found no blank line between the two and the item recorded no spread, though the parser counts it, as CommonMark does, and makes the list loose. Document::syntax now looks through the last blocks of nested lists, items and footnote definitions for a blank line at their end, also when it is a line of > alone in a nested list's last item in a block quote; a block quote's own lines of > alone still do not count. So - a\n - b\n\n c is written - a\n\n - b\n\n c, with a blank line between each block of the item, which renders the same. The Markdown serializer now writes an item of a loose list spread whenever it writes a blank line between two of its blocks, as that blank line reads back as spread. A hand-built loose list of an item that records spread and holds a list of an empty item and a heading, written - 1.\n\n ## h\n\n-, and an item that records no spread and holds two paragraphs and a list, now give the same Markdown when written again. After a block quote that ends in a list or footnote definition, the serializer writes a blank line before a paragraph, table, definition or setext heading, where it wrote a line of > alone, which is a blank line in that list's last item or in the definition and read back as spread (#157).
Parse footnote definitions nested in each other in linear time. A footnote definition cannot start in a heading or image, and the HTML renderer and the Document builder checked that by looking through every open container at each footnote definition or heading start, so - [^1]: repeated 8,000 times took 26 ms in to_html and 88 ms in Document::parse, four times as long at each doubling. They now count the open headings and images, and take 4 ms and 2 ms, with the same output (#171).
Start a list item, and a list, after a tab where its indent starts. The indent before a marker is counted in columns, and the start was found by taking that count from the marker's byte offset, so after a tab, which is one byte of up to four columns, it started too early: in - a\n\n\t- the nested list started on the blank line before it, which Document::syntax then did not count, so the outer item recorded no spread in a loose list. With code_indented off, a tab before a marker near the start of the document, as in \t- a, made the offset negative, which panicked. HTML is unchanged otherwise (#171).

Serializing Markdown

Write labels as they are when serializing Markdown, since every label accepted reads back. The serializer escaped characters at the start of a label's lines, so [x][a\n#b] was written [x][a\n\#b], which reads back with a different label and does not match its definition, and likewise for =, < and the lines of definitions; these now read back. Labels with a line ending in a heading of rank 1 or 2 are written setext, as an ATX heading cannot hold one, so [x][a\nb] followed by === reads back, where it was written as an ATX heading, which read back as a heading and a paragraph.
Write image text as phrasing when serializing Markdown, like link text, so it reads back as the same tree (#190). A footnote call in image text was dropped, so ![see [^1]](/u) was written ![see ](/u), and links and images in it were written as their text, so ![a [b](/v)](/i) was written ![a b](/i) and ![a ![b](/v)](/i) as ![a b](/i). They are now written as they are, as are references, autolinks and attention in image text, so the HTML, whose alt is the plain text, is unchanged. Text in image text is escaped as before, with literals forming outside link text, and [ and ] in it are escaped as in link text. A literal link in image text is written bare only where it would be in a paragraph, so ![a www.x.com.](/i) is written ![a [www.x.com](https://www.x.com).](/i). Image text that starts with ^, parsed from ![\^a](u), was written ![^a](u), which reads as ! and a link or footnote call, and is now written with the escape. A ( after a shortcut reference in image text, as in ![[r]\(x)](u), kept its escape only by chance; it now always does. Nodes in image text now count toward the nesting limit, so images nested 127 or more deep in a paragraph return the nesting-too-deep error, as emphasis and block quotes nested that deep already do, where they were written flat and lost their content. Line endings in image text now follow the usual rules: LineWrapping::Unwrap, which normalize_markdown uses, writes a soft break in image text as a space, and Document::normalize replaces a line ending in inline code in image text with a space, which Document::is_normalized reports, where both were kept. A literal link at the end of image text, before spaces or tabs, is written bare, since image text keeps them, so ![x@y.z\n](u) is written ![x@y.z ](u) with LineWrapping::Unwrap on every write.
Write the info string of a code fence exactly when serializing Markdown, including the spaces and tabs between its words, which were written as one space: ```x a was written ```x a and is now written as it is, and ```x  a  as ```x a . Spaces and tabs at the edges, line endings, backticks in a backtick fence, a ~ at the start in a tilde fence, and escapes and references that would decode are still encoded. Whitespace from a character reference inside the info string is written raw, so ```a b	c d is written ```a b c d with a raw tab after b. Code with an info string but no language, such as x, is still written fenced.
Escape text that would start a line after a line ending written as 
, such as the second of two soft breaks in a row, as after a raw line ending. A paragraph of a, two soft breaks and # was written as a\n
#, which reads back as a, a soft break and text \n#, so the next write escaped #; it is now written as a\n
\# on the first write. The same holds for >, -, +, *, =, ~, `, _, [, <, and . or ) after digits, in paragraphs, headings, list items and block quotes (#125).
Escape < in text before characters other than letters, !, / and ? when the rest of the same text forms an email autolink. Text such as \<1@a.b> was previously written as <1@a.b>, which parsed as an email link. Text such as a <= b, <0.5% and -> Poll<T> is still written unchanged.
Escape text where GFM would read it as a literal autolink when serializing Markdown, so text reads back as the same text. A backslash goes before the colon after a scheme, the dot after www or the @ of an email, as in www\.example.com, https\://a.b/c and a\@b.co, and only where a literal would form: www must start the text or follow whitespace, (, *, _, [, ] or ~, and the domain must be a host name, so awww.a.b, www.a_b.c and x.org are unchanged. The @ is escaped rather than the colon of mailto: or xmpp:, since the email still links without them. Each @ that would form an email once the one before it is escaped is escaped too, so a@b.c.d@e.f is written as a\@b.c.d\@e.f. Text in link text is not escaped, since literals do not form there, and bare links are written as they were. Text in image alt is escaped the same way, so alt text such as www\.a.b stays text, except in an image inside link text, while a link parsed from alt text, such as ![see www.a.b](u), is written as it was and still links. This fixes text before a code span, such as https://a.b before `<script>`, becoming a literal that took in the backtick and left the code as raw HTML (#97), and whitespace that normalizing trims before www at the start of a paragraph leaving a link. An escaped @ also keeps an email autolink from forming, so <1@b.c> is written as <1\@b.c> rather than \<1@b.c>. Escaping is always applied, as the serializer writes GFM, so a document parsed without GFM gains a backslash in text such as https\://a.b. An email domain with _, such as x@a_b.com, is no literal, so it is not escaped. When the escape before an email's @ ends a www. or scheme literal before it, and so leaves out trailing punctuation that made its domain invalid, that literal is escaped too, so www.a..@b.cd is written as www\.a..\@b.cd.
Serialize inline code with many line endings in linear time. A line ending in their content that would start a line before a block marker, such as the one in a\n#, is written as a space, and writing each one scanned the content again from its start, so code of a\n# repeated 32,000 times took two seconds to serialize. It now takes under a millisecond, with the same output (#115).
Write each line ending in inline code as a space when serializing Markdown, and replace each one with a space in Document::normalize, which Document::is_normalized reports. The parser reads every line ending in inline code as one space, CRLF and lone CR too, so a value that holds one cannot be written. Only a line ending before a block marker was written as a space, so code of a\nb after x was written as x `a\nb` , which reads back as a b and was written differently the next time (#136), and code of a\n|-|\nb was written with the |-| on its own line, which reads back as a table (#135). Code in image text is written the same way, as image text is phrasing (#190). A heading whose only line endings are in code is now written as an ATX heading rather than a setext one. This supersedes the previous entry's rule for which line endings are written as spaces.
Escape & before what would read as a character reference in the info string of a code fence when serializing Markdown. The parser decodes escapes and references there, so a code block parsed from ```x \& has the info x &, which was written as ```x & and read back as x &; it is now written as ```x \&, and \	 likewise (#131).
Pad inline code that starts and ends with a space and holds a tab or other character that is not a space, such as \t . The parser strips one space from each side unless the content is only spaces, and a tab is not a space, so x ` \t ` read back as \t, and a parsed x ` \t ` was written as x ` \t ` and then as x `\t` . It is now written as x ` \t ` .
Serialize text in linear time. Escape scans restarted from the start of the paragraph for every text node, so text such as long runs of a_ or [a\|b] took tens of seconds to serialize at a few hundred kilobytes; they now take tens of milliseconds.
Escape every backslash before ASCII punctuation, including one that follows another escaped backslash, as mdast-util-to-markdown does. Text such as \\. was written with one escape too few, so it rendered as \..
Write reference and definition labels with their backslashes as authored, since labels match by their raw text. Labels such as a\*b or a\|b gained backslashes on every serialization, and a reference in a table cell could stop matching its definition. Shortcut references such as [a\*b] now stay shortcuts.
Escape a ) or . after digits based on where the text sits in its line, not on how the parser split it into text nodes, and text right after an opening emphasis, strong or strikethrough marker no longer counts as the start of a line. 1)a and 1\)a previously swapped on every serialization, and 1\. a was written as 1. a, which parses as a list.
Escape a ~ that is the first or last character of strikethrough content, wherever the strikethrough sits. ~~\~5 min~~ was written as ~~~5 min~~, which at the start of a line opens a code fence.
Write whitespace at the start or end of emphasis, strong and strikethrough content as a character reference, in parsed and constructed trees and wherever the text sits in its line, as mdast-util-to-markdown does. When the character directly outside that marker is neither whitespace nor punctuation, such as a letter or digit, it is written as a reference too, so the marker still opens or closes. * a* was written as * a*, which parses as a list item, and x * a* as x * a*, which is text.
Choose * or _ for each emphasis and strong from its immediate neighbours when serializing Markdown, so touching sequences open and close as intended, and encode characters next to attention as character references where needed. Tildes at the edges of attention are escaped so they cannot pair with others.
Keep whitespace after the space that follows a task-list marker when serializing Markdown. - [ ] a was written as - [ ] a, losing a space, and is now written as - [ ]  a.
Choose the other marker for emphasis or strong that shares one edge with enclosing attention of the same marker when the rule of 3 would otherwise keep the enclosing sequences from pairing in Markdown serialization. **a*~** and **a*.**, emphasis in emphasis followed by a literal *, were written as **a*\~*\* and **a*.*\*, which parse as *a~**, and are now written as *_a_\~*\* and *_a_.*\*. The closing edge is handled the same way: emphasis of . and emphasis of a between two . was written as .*.*a**., which parses as .*.a*., and is now written as .*._a_*.. So is strong in strong at either edge, and sequences still join where the rule of 3 does not apply, so **a* b* and *a **b*** are unchanged.
Escape a text ~ wherever it could pair as strikethrough, not only at the start of a line and at the edges of strikethrough content. Each run of ~ is judged by CommonMark's flanking rules against the characters written around it. A run of one or two tildes that can close is escaped, and so is a run of two that can open, since a strikethrough sequence can close it. x \~a~ and [\~a~](u) were written as x ~a~ and [~a~](u), which parse as strikethrough, and are now written as x ~a\~ and [~a\~](u). A run is escaped whole, so   ~~a~ is written as   \~\~a\~ rather than   \~~a~. Tildes inside strikethrough are escaped by the same rule, so a constructed Delete[Text("a~~b")] is written as ~~a\~\~b~~ rather than ~~a~~b~~, and a backslash in text is literal, so Text("\~a~") is written as \\~a\~ rather than \\~a~. A single ~ that can only open stays unescaped, so prose such as about ~5 min, ~/path, (~3 GB) and a ~ b is written as it is.
Encode line endings in text that a raw line ending would change when serializing Markdown, so any text reads back exactly. A line ending after another one is written as 
, since two in a row are a blank line that ends the paragraph, so text a\n\nb is written as a, a line ending and 
b, and three in a row as a\n
\nb. So is one directly before the end of a paragraph, heading or other block, which was trimmed, so a
 and *a*
 are kept, and so is one directly after a hard break, which would otherwise make a blank line, so text a, a hard break and text \nb are written as a\\, a line ending and 
b. Every line ending in a GFM table cell is written as a reference, because a table row is one line, so a cell of a
b no longer splits the table (#116). A CR in text or a title, which comes from  now that parsed line endings are LF, is written as , since serialized line endings are LF and a raw CR would read back as one, so [a]: u "t" keeps its CR. This applies in list items, block quotes, setext headings, links, image text and attention, wherever the line falls (#112).
Write a code block whose body is one blank line with that line when serializing Markdown. Its value is a single line ending, which the serializer dropped along with the terminal one, so ~~~x\n\n~~~, also after a paragraph or in a list item or block quote, was written as an empty block and read back with an empty value (#130).
Escape a : in text directly after a shortcut reference or footnote reference when serializing Markdown, where it would read back as a definition. [r]\://a.b and [^1]\: x were written as [r]://a.b and [^1]: x, which read back as a definition and a footnote definition, also in a list item, block quote or footnote definition (#122). A footnote definition can interrupt a paragraph, so a footnote reference at the start of any line, including after a soft or hard break, as in a\\, a line ending and [^1]\: x, is escaped whenever its label is one a footnote definition can have, which ^ 1 is not. A shortcut reference, or a footnote reference with such a label, can only start a definition at the start of a paragraph or setext heading, and is escaped there when the paragraph, as written, starts with a definition by the parser's own rules: a destination, which can come from a later node, as in [r]\: <https://a.b>, followed on its line only by an optional title. So [r]: a b, as in a changelog's - [Issue 1]: Fixed a crash, is written as it was. A full or collapsed reference, an image reference, a reference after a task marker, as in - [ ] [r]: /u, and a reference that does not start a line, or is in an ATX heading or table cell, cannot start a definition, so [r][]:x, ![r]:x, a [r]:x, a second line [r]:x and # [r]:x are written as they were too.
Write a blank line between a footnote definition and a paragraph after it in a list item when the definition ends in a paragraph, as after a nested list, whatever spread the item records. A parsed footnote definition takes in the blank line after it, so an item of a definition and a paragraph recorded no spread, and in a list of more than one item - [^1]: a\n\n b\n- c was written - [^1]: a\n b\n\n- c, which reads back with b in the definition. The same held for a definition that ends in a block quote or list, or before a definition or table, which cannot interrupt a paragraph either. A definition that ends in another block, or is empty, is ended by the next line, so - [^1]: # h\n b is written as it was (#148).
Write a blank line between a paragraph and a list after it in an item of a loose list when the list cannot interrupt a paragraph, as it starts with an empty item or is ordered and starts at a number other than 1, whatever spread the item records. In a loose list of more than one item whose items record no spread, as a hand-built tree may leave them, an item of p and a list of an empty item was written - p\n -, which reads back as a heading, and one of p and a list starting at 3 was written - p\n 3. a, which reads back as text; they are now written - p\n\n - and - p\n\n 3. a. The blank line makes the item read back spread, so its other blocks are written spread too, which the list being loose already renders. The only item of a loose list followed by a block after such a list is fixed the same way (#151).
With fences: false, write code that is the first block of a footnote definition on the line after the label, and code directly after a footnote definition as a fence. Indented code cannot start on the label's line, so a definition of code a was written [^a]: a, which reads back as a paragraph; it is now written [^a]:\n a. Indented code after a definition continues it, so in a list item [^1]: a followed by code x was written - [^1]: a\n x, which reads back with x in the definition; the same held at the document level and in a block quote. Other code is still indented (#142, #157).
Write the blocks of an item of a tight list that need a blank line between them so that they read back the same (#154). Document::normalize joins two paragraphs in a row in such an item, not counting separators, into the first, with a soft break between them, so a hand-built item of a and b is written - a\n b rather than - a\n\n b, which read back loose; a tight list has no , so this renders almost the same. It makes the list loose when an item holds two block quotes in a row, which read back as one, or a paragraph after a list or footnote definition that ends in a paragraph or a link reference definition, which a following line continues alike, through the last blocks of any containers in it, or after a block quote whose last block is one, which would continue that paragraph lazily. In an item of a tight list, the serializer writes a paragraph, table, definition or setext heading after a list that does not end in a paragraph or link reference definition on the next line, where it wrote a blank line that made the list loose, so * *\n p, a tight list whose item holds a list of an empty item and p, is written - -\n p rather than - -\n\n p. Document::is_normalized reports these shapes. The Markdown serializer writes a paragraph, table, definition or setext heading after any other block quote that ends in a paragraph or link reference definition, in an item that is not written spread, after a line of > alone, so - > a\n >\n p stays a tight list of a quote and a paragraph, where it was written - > a\n p, which reads back as one quote of a p. An item of a loose list with two block quotes in a row is written spread, as the blank line between them makes it read back spread.
Return an error from to_markdown, to_markdown_with_options, to_markdown_node and normalize_markdown for content nested more than 128 levels deep, with the rule id nesting-too-deep, instead of overflowing the stack and aborting. Every block and inline node counts one level below the serialized node, including those in an image's text (#190). Parsed input can nest this deep, as 16 KB of > or a list nested 8,000 deep does, and so can a hand-built tree (#139). Serializing 128 levels still recurses, but fits on a 2 MiB thread, the default for spawned threads, in debug and release builds: it needs at most about 290 KiB of stack in a release build and 1.5 MiB in a debug build, for links nested in links.
Return an error from to_markdown, to_markdown_with_options, to_markdown_node and normalize_markdown for an ordered list item whose number would have more than 9 digits, with the rule id list-number-too-long, as CommonMark limits a list marker to 9 digits. Such a marker was written anyway and read back as text, so a hand-built list that starts at 1000000000 lost its list, and a list that starts at 999999999, parsed or built, merged its second item, written as 1000000000. b, into the first item's paragraph. An item's number is its list's start plus, when increment_list_marker is on, its position among the list's items, so with it off only the start must fit; the number is checked as each item is written, without wrapping for any start, and an item serialized on its own is numbered by its position in its list. Document::normalize never raises a list's numbers, so this applies to trees built or edited by hand, and to a parsed list whose numbers increment past 999999999 (#155).
With tight_definitions, the Markdown serializer joins blocks other than adjacent definitions as it does without it. It wrote a blank line between every other pair of blocks, so a tight list was written loose and rendered with paragraphs: - c\n- c was written - c\n\n- c, and - a\n # h was written - a\n\n # h. With fences = false, it also wrote no  before indented code after a list, so the code read back in the list's last item. Adjacent definitions in a spread item are still joined by a line ending, unless they are the item's only blocks, where they are joined by blank lines so the item reads back spread (#160).

v0.1.4

Give setext underlines, thematic breaks and other interrupting lines precedence over a following GFM delimiter row, as cmark-gfm does. Previously a\n===\n|- rendered the underline and delimiter row as a second paragraph instead of a heading followed by |-. After a link reference definition, an underline-shaped line followed by a delimiter row now forms a table header, and on a line that closes or opens containers it is a thematic break rather than paragraph text.
Add IncrementalParser, which accepts Markdown in pieces and emits the events of each document-level block once appended text can no longer change it. Its combined output equals EventParser events without the document start and end events, except that references resolve only against definitions before the end of their document-level block.
Escape Markdown syntax in GFM table cells as in paragraphs, as mdast-util-to-markdown does. Cell text such as \_a\_ or \*b\* was previously written unescaped and re-parsed as emphasis.
Escape pipes inside code and math in GFM table cells. GFM splits cells before parsing code, so an unescaped pipe in code split the cell, and usually stopped the table from parsing at all.

v0.1.3

Add normalize_markdown and normalize_markdown_with_options, which parse Markdown with GFM, frontmatter, and math and serialize it with default formatting and soft line breaks unwrapped. This is the normalization bake-markdown applies, so tools that generate Markdown can produce output it leaves unchanged.
Add ParseOptions::normalize, the syntax normalize_markdown parses, so tools can inspect a document with the same structure the normalizer sees.
Indent the content of GFM task list items to the list marker's content column rather than past the checkbox. Nested lists, paragraphs, and code in a task item were previously re-parsed as paragraph text or indented code, so repeated serialization changed or grew the document.
Keep the blank line between a nested list and a following paragraph, table, definition, or setext heading in the same list item. Without it, the next parse continued the nested list's last paragraph, merging the two.
Write a heading with soft line breaks as an ATX heading when soft breaks are unwrapped. It was previously written as a setext heading, which the next normalization turned into an ATX heading.
Leave underscores between letters or digits unescaped when serializing Markdown, as mdast-util-to-markdown does, so io_uring and snake_case no longer become io\_uring and snake\_case. Such underscores cannot open or close emphasis, so the rendered output is unchanged.

Contributing

Please open an issue or pull request on GitHub.

Agent Context

Run cargo bake agent:context:install to install shared context and skills. Use cargo bake agent:context:list and cargo bake agent:context:show to discover relevant guides, follow agents.md if present, and apply skills under .agents/skills/. The installer preserves repository-owned agents.md; it does not create or regenerate that file.

Name		Name	Last commit message	Last commit date
.github		.github
bake		bake
benches		benches
context		context
design		design
examples		examples
fuzz		fuzz
generate		generate
releases		releases
src		src
tests		tests
.editorconfig		.editorconfig
.gitignore		.gitignore
.mailmap		.mailmap
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
agents.md		agents.md
codecov.yml		codecov.yml
funding.yml		funding.yml
license.md		license.md
readme.md		readme.md
releases.md		releases.md

Navigation Menu

Repository navigation

About

Repository files navigation

`base-markdown`

Motivation

Usage

Releasing

Releases

v0.2.0

Breaking changes

Added

Documents and normalizing

Parsing

Serializing Markdown

v0.1.4

v0.1.3

See Also

Contributing

Agent Context

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

base-markdown

Motivation

Usage

Releasing

Releases

v0.2.0

Breaking changes

Added

Documents and normalizing

Parsing

Serializing Markdown

v0.1.4

v0.1.3

See Also

Contributing

Agent Context

About

Topics

Resources

Code of conduct

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`base-markdown`