Various general questions

[Hibou57]

Hi all,

Landed here while seeking for a base standard format to represent AST of various languages.

Wondering if Unist is standard enough is a matter of how widely it is used, so I guess it's difficult to answer this.

For JavaScript at least, there already is ESTree. It seems well adopted as a standard but it's only for JavaScript. How a JavaScript specialization of Unist (if there is one) compares with ESTree? Edit: ESTree seems to be more designed as an API, not as a representation, so this comparison may be difficult. Edit2: yes, there is one: esast.

Is there a directory where all language specializations are registered?

Is there something to help diverging by language specialization to converge? I mean with a place to talk about what is missing or may not be up to date regarding a language’s reference, etc. The question is about the standard representation, not about the toolings. Edit: seems syntax-tree is already about this?

Is there a way to keep comments with their original formatting?

Is there some common documentation idiom to mention semantics? Or may be generated nodes (I’ve seen mentioned in the README) already have this purpose? Some (not all) semantic aspects of most languages, could be represented by simple relations, indeed (ex. that name refers to this declaration or object, this name belongs to that scope, are examples).

Is it expected to be used outside of the web languages world? (CSS, HTML, JavaScript, Markdown, etc)

I feel to remember, a long time ago, I learned about an XML standard to represent AST of numerous languages, but cannot find it anymore. Do anyone remember this? May be it has disappeared … :-/ Or may be it's me confusing with something else.

With many thanks …

[ChristianMurphy]

Welcome @Hibou57 ! 👋
I'd recommend starting by reading it answers a lot of these questions https://unifiedjs.com/learn/

Wondering if Unist is standard enough is a matter of how widely it is used, so I guess it's difficult to answer this.

What do you mean by standard enough?
It is widely used? Matter of perspective? remark (mdast) alone gets 19 million downloads a week. I'd say that's pretty well adopted.

Is there a directory where all language specializations are registered?

Many are on the unist topic on GitHub https://github.com/topics/unist

seems syntax-tree is already about this?

It is

Is there a way to keep comments with their original formatting?

That's language specific.
Which language are you interested in?

Is it expected to be used outside of the web languages world? (CSS, HTML, JavaScript, Markdown, etc)

Sure it can be.
It is more document oriented, which works well with web formats.
It isn't designed to be a general purpose syntax tree that can represent any context free grammar (ESAST needs more richness than base Unist provides, and so add a lot of attributes compared with say mdast or hast)

I feel to remember, a long time ago, I learned about an XML standard to represent AST of numerous languages, but cannot find it anymore. Do anyone remember this? May be it has disappeared … :-/ Or may be it's me confusing with something else.

We aren't that XML standard, unist is JSON based. 🙂
And there are a bunch you could be referring to.
Web search is your friend 😉 , the first few results are:

• https://www.omg.org/spec/ASTM/1.0/About-ASTM
• https://www.srcml.org/about.html
• https://www.sciencedirect.com/science/article/pii/S0167642305001176

[Hibou57]

This answer is important too: https://github.com/orgs/syntax-tree/discussions/158#discussioncomment-17607613

[Hibou57]

Hello and thanks for the responses,

Is there a way to keep comments with their original formatting?

That's language specific.
Which language are you interested in?

An examples is JavaScript and JSDoc, and the JSDoc are using TypeScript types syntax but it is JavaScript, not TypeScript which is only used for type checking. It's for documentation, not type checking which is left to TypeScript.

It isn't designed to be a general purpose syntax tree that can represent any context free grammar (ESAST needs more richness than base Unist provides, and so add a lot of attributes compared with say mdast or hast)

As an example, it would not be well suited to a language like SML ?

Will have a look at the three links you suggested (searched, but did not find these, while feel to remember the first one a long time ago)

I had another question and may avoid opening another discussion for that. Since there is a format to represent an AST for a language, there a parsers producing this. It makes think about test suits. This would be profitable to everyone and could help discussing in case of differing output. I noted Unist is more JSON than XML, but with XML there is Canonical XML to ease comparison. May be there is something similar with JSON or could be defined. Test suits, does it already exists or is there any provision for it? Edit: it would be useful even to projects not necessarily based on unist, but which could be related to it in this way, and contribute to it in that aspect.

Have nice day and wish you the weather is not too hot

[ChristianMurphy]

Is there a way to keep comments with their original formatting?

An examples is JavaScript and JSDoc, and the JSDoc are using TypeScript types syntax but it is JavaScript, not TypeScript which is only used for type checking. It's for documentation, not type checking which is left to TypeScript.

Define "original formatting".
There will still be a comment there (doc, inline, or block), that has the same text inside.
The byte for byte spacing and alignment may not be identical.
Unist trees are generally abstract syntax trees, they remove non-structural syntax details, and are useful for formatting, linting, and rendering.
Different from concrete syntax tree, which keep exact byte details and are useful for style preserving edits.

As an example, it would not be well suited to a language like SML ?

It could handle SML.
I'd step back, and ask "what are you trying to solve or build?"

SML already has several parsers.

• https://github.com/rossberg/hamlet/tree/succ
• http://mlton.org/SuccessorML
• https://github.com/azdavis/millet

What are you trying to do, beyond what those already do?
More linting? Embedding syntax in another document like markdown?

[Hibou57]

What are you trying to do, beyond what those already do? More linting? Embedding syntax in another document like markdown?

A standard format because data representation is more important than tool. It gives interoperability and optionally persistence. As an example, yes, for SML there are parsers (while Hamlet and MLTon are more interpreter and compiler than parsers), but there is no interoperable format for the syntax tree (and they are not intended to output it). At least, if a standard format exists, it is possible to convert a specific format to a standard format and have it applied with a wrapper.

It will be more specific to answer your question. There are two things I wanted to achieve first. The first was when I wanted to document JavaScript sources with Sphinx and I encountered issues with autodoc and the JavaScript extension whose JSDoc implementation does not understand TypeScript type syntax and additionally, I got an exception asking me to submit it as a bug while I was not even sure it is indeed a bug. To have an input in a standard format rather than relying on an extension for parsing, could solve the case. Then there is SML. I have something which is both simple and complex that I would like to map with some documentations to not loose track of anything important. But I would need to ensure the documentation is indeed mapping the SML program and that any divergence is caught as an error. Having parsers generating standard output tree, would also allow to use any documentation system (ex. DITA instead of Sphinx, because Sphinx has no support for SML anyway), instead of choosing one because this is the only one with a support for some language.

This is separation of concern.

[ChristianMurphy]

A standard format because data representation is more important than tool. It gives interoperability and persistence

I'm still not sure what the saved tree would be for. What would read it later, and what would it do with it? A formatter? A doc generator? Long-term storage? Depending on the answer, "standard" means quite different things, so it's worth figuring that out before picking one.

The reason I ask: a syntax tree usually lives inside a tool while it runs, in a linter, a formatter, a transformer, and is thrown away afterwards. It rarely gets saved and passed around as a file. And which tree fits depends on the job. For formatting and highlighting, something like tree-sitter or prettier is often the better fit, because they keep the exact source details (a concrete syntax tree), comments and spacing included. Unified is at its best with document-style languages, where you want to transform a tree and embed the result into markdown, HTML, or MDX. In every case, though, the useful part isn't the file format, it's the agreed set of node types for each language and the tools built on top. Agreeing on those node types, and keeping them current as the language changes, is also where the older universal-AST attempts (ASTM, srcML) got stuck.

it is possible to convert a specific format to a standard format and have it applied with a wrapper

It is, and that's how much of the unist ecosystem already works. Several syntaxes are wrappers around existing parsers rather than parsers of their own: hast trees are built from parse5 output via hast-util-from-parse5, and esast wraps ESTree parsers like acorn via esast-util-from-estree. An "smlast" could be done the same way: pick an existing SML parser, write a small *-util-from-* style converter, and write down what the resulting nodes look like. Just know that almost all the work is in deciding and documenting the node types, not in the JSON.

Some SML parsers that would suit this better than HaMLet or MLton, since the tree is their whole output:

• smlfmt, a standalone parser with a full concrete syntax tree, comments included, SuccessorML extensions supported
• Millet's parser crates, usable as a Rust library, and a thin wrapper could write the trees out as JSON
• tree-sitter-sml, a tree-sitter grammar maintained by MLton's lead, if you'd rather use tree-sitter

I would need to ensure the documentation is indeed mapping the SML program and that any divergence is caught as an error

You don't need a standard to exist first for this. Parse with one of the above, pull out the declarations and doc comments, compare them against your docs, and raise an error when they don't match. If you publish the SML-to-unist converter as part of that, even better.

The first was when I wanted to document JavaScript sources with Sphinx

This one is a separate problem from unist or syntax-tree. The extension is sphinx-js, and its README covers your exact case: for TypeScript type syntax it uses TypeDoc in place of JSDoc. If that mode still throws the exception you saw, their tracker has a typescript label for it. Please raise it there, or in its own discussion, so it doesn't get lost in this thread. 🙂

[Hibou57]

* [smlfmt](https://github.com/shwestrick/smlfmt), a standalone parser with a full concrete syntax tree, comments included, SuccessorML extensions supported
* [Millet](https://github.com/azdavis/millet)'s parser crates, usable as a Rust library, and a thin wrapper could write the trees out as JSON
* [tree-sitter-sml](https://github.com/MatthewFluet/tree-sitter-sml), a tree-sitter grammar maintained by MLton's lead, if you'd rather use tree-sitter

Greatly valuable! thanks 😊

(aside, will have a second look at Sphinx, if I ever missed something)

[Hibou57]

More about the last question.

More limited, but may be an option: the Language Server Protocol. It does not expose an AST (and it does not on purpose), but allows queries on a source. Providing the possible queries are enough for some intent, it is at least interoperable.

For the last question, I will have a look into that direction, different from the one I was devising first.

[ChristianMurphy]

More limited, but may be an option: the Language Server Protocol.

LSP could work for peeking into the SML source (definitions, references, symbols), depending on what you actually need out of it.

But it doesn't check docs any more than a tree does. Either way you're writing a wrapper: pull the declarations from LSP (or a parser), compare against your docs, error on drift.

Better still, prevent the drift instead of detecting it. For the JSDoc case, remark-usage generates a readme's Usage section from a real example that runs in Node, so there's no mismatch to catch.
https://github.com/remarkjs/remark-usage