unified: Add Swift parser based on swift-syntax

[tausbn]

Adds an alternative parser to tree-sitter-swift, using the Swift-native swift-syntax package instead.

The current implementation exposes this parser through a Rust wrapper, using the C FFI to communicate with Swift. The Rust component passes the source file contents to the Swift component, which in turn responds with a JSON string containing the AST.

Note that at present, the AST is quite verbose (though some effort has been made to make it less so). In particular, it contains detailed location information about every single token in the input, which is almost certainly overkill for our purposes.

This PR does not attempt to replace the existing yeast rules to be based on the swift-syntax AST. That is left as follow-up work.

Can be reviewed commit-by-commit.

[Copilot]

Pull request overview

Adds a Swift-native swift-syntax parser exposed through Rust and prepares unified extraction to consume its JSON AST.

Changes:

• Adds Swift/Rust FFI parser and Bazel/Cargo builds.
• Adds JSON-to-yeast AST adaptation and minimal Swift rules.
• Extends yeast for externally constructed ASTs and owned locations.

Show a summary per file

File	Description
MODULE.bazel	Registers Swift dependencies and toolchains.
Cargo.toml	Adds the parser crate to the workspace.
Cargo.lock	Locks the new crate.
shared/yeast-schema/src/schema.rs	Supports merging schema names.
shared/yeast-macros/src/parse.rs	Uses yeast-owned ranges.
shared/yeast/src/build.rs	Migrates build contexts to owned ranges.
shared/yeast/src/lib.rs	Adds external AST construction/desugaring APIs.
shared/yeast/src/range.rs	Defines owned point and range types.
shared/yeast/tests/test.rs	Tests desugaring hand-built ASTs.
shared/tree-sitter-extractor/src/extractor/mod.rs	Converts yeast points for extraction.
unified/extractor/src/languages/mod.rs	Exposes the Swift JSON adapter.
unified/extractor/src/languages/swift/adapter.rs	Converts JSON into a yeast AST.
unified/extractor/src/languages/swift/swift.rs	Adds minimal swift-syntax rules.
unified/extractor/tests/fixtures/let_x.swiftsyntax.json	Provides parser output fixture.
unified/extractor/tests/swift_syntax_pipeline.rs	Tests the adapter/desugarer pipeline.
unified/swift-syntax-rs/.gitignore	Ignores generated outputs.
unified/swift-syntax-rs/.swift-version	Pins Swift 6.3.2.
unified/swift-syntax-rs/BUILD.bazel	Defines Bazel Swift/Rust targets.
unified/swift-syntax-rs/Cargo.toml	Defines the Rust crate and CLI.
unified/swift-syntax-rs/README.md	Documents building and usage.
unified/swift-syntax-rs/build.rs	Builds and links the Swift shim for Cargo.
unified/swift-syntax-rs/src/lib.rs	Implements safe Rust FFI bindings.
unified/swift-syntax-rs/src/main.rs	Adds the parser CLI.
unified/swift-syntax-rs/swift/Package.resolved	Locks swift-syntax.
unified/swift-syntax-rs/swift/Package.swift	Defines the Swift FFI package.
unified/swift-syntax-rs/swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift	Serializes SwiftSyntax ASTs as JSON.
unified/swift-syntax-rs/xcode_transition.bzl	Isolates macOS Xcode configuration.

Review details

• Files reviewed: 26/27 changed files
• Comments generated: 9
• Review effort level: Medium

[jketema]

I mostly skimmed the Rust changes.

For the Swift code I wonder whether we really need to write the complete serializer ourselves, or whether we could leverage things that are already there in swift-syntax. This is not blocking though.

Do we need someone with more Bazel knowledge to review those parts?

[jketema]

Is it common to have something like this in rust builds that also compile code in other languages? Note that I'm not suggesting to change this.

[tausbn]

I don't think we do it anywhere else, so it's a valid concern. Originally I had swift-syntax-rs much more tightly integrated with the extractor, in which case it made sense to build both at the same time. However, now that we have (for the time being) moved to a model where we call an external binary for the parsing, we could also decouple the builds themselves.

[jketema]

This seems potentially fragile if swift-syntax changes?

[tausbn]

Agreed. This was a "proof of concept" test that was useful at the time, but it is brittle. We can probably just get rid of it once the whole extraction is using swift-syntax-rs.

[jketema]

I think we should do the following (apologies for this being split in two comments).

[redsun82]

Hey, thanks for this, seems like it was quite some work to get this working! I've looked at the bazel and build side of things, and didn't look at the actual compiled code at all. I don't have major comments on the overall direction, but I do have some comments I'd like answers to before we go ahead with this.

[redsun82]

This branch can just be return {} — on the Bazel version we pin, omitted transition outputs keep their incoming value, so it's equivalent to echoing the two settings back.

If you do that, _XCODE_VERSION_CONFIG is no longer read anywhere (the macOS branch sets it to a constant without reading the incoming value), so you can also drop it from the transition's inputs and keep only _EXTRA_TOOLCHAINS.

[jketema]

Fixed in bf57586

[redsun82]

so @swift_toolchain_ubuntu24.04//:files is glob(["**/*"], exclude=["BUILD.bazel", "*.pkg"]), which is the entire swift.org toolchain, ~4.3 GB uncompressed. That's 3.3 GB of usr/ (compiler + all the static/dynamic libs), plus the 1 GB swift-6.3.2-RELEASE-ubuntu24.04.tar.gz download archive itself (the glob only excludes *.pkg, not *.tar.gz, so the tarball ships too).

At runtime though, the binary only actually needs the Swift runtime shared libs it rpaths into, i.e. usr/lib/swift/linux/, which is 142 MB total (~91 MB of .sos). So as it stands this pulls in 30-45x more than we need as runfiles, and I'm a bit worried it'll push us over the total codeql bundle size limit once we try to release it.

Could we point data at a narrow filegroup of just the runtime libs instead of //:files (something like glob(["usr/lib/swift/linux/**"]))? That'd get it down to ~90-140 MB and drop the tarball entirely.

Couple of related things, mostly for @tausbn:

• the legacy Swift extractor already bundles a runtime this way (@swift-resource-dir-{os} → resource-dir/{CODEQL_PLATFORM}), so I wonder if we could reuse that rather than ship a second one. The catch is it's built for the legacy compiler fork, and the Linux Swift runtime ABI isn't stable across versions, so that only works if both end up pinned to the exact same Swift release.
• it also depends on the rollout plan: if the new extractor replaces legacy in the same release, only one runtime ships either way and just narrowing is enough. But if the two coexist in the bundle for a while, we'd be shipping two runtimes unless we share one.

[jketema]

I have this as follow-up at the moment:

$ cat unified/swift-syntax-rs/swift_runtime.bzl
"""Filter the Swift toolchain down to just its Linux runtime shared objects.

The standalone toolchain's `:files` bundles the compiler, host libraries, clang
runtime, plugins, etc. For *running* a Swift-linked binary we only need the
runtime shared objects in `usr/lib/swift/linux/`. Flattened packaging
(codeql_pkg_runfiles) also can't tolerate the whole toolchain, since duplicate
basenames (e.g. clang's inttypes.h, or lib_InternalSwiftScan.so appearing in
multiple host dirs) collide once directory structure is stripped.
"""

def _swift_runtime_libs_impl(ctx):
    libs = [
        f
        for f in ctx.files.toolchain
        if "/usr/lib/swift/linux/" in f.path and f.path.endswith(".so")
    ]
    return [DefaultInfo(files = depset(libs))]

swift_runtime_libs = rule(
    implementation = _swift_runtime_libs_impl,
    attrs = {"toolchain": attr.label(allow_files = True)},
)

and:

--- a/unified/swift-syntax-rs/BUILD.bazel
+++ b/unified/swift-syntax-rs/BUILD.bazel
@@ -1,37 +1,22 @@
 load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_test")
 load(":xcode_transition.bzl", "xcode_transition_swift_library")
+load(":swift_runtime.bzl", "swift_runtime_libs")
 
 package(default_visibility = ["//visibility:public"])
 
+swift_runtime_libs(
+    name = "swift_runtime_libs",
+    toolchain = "@swift_toolchain_ubuntu24.04//:files",
+ )

and then do:

-        "@platforms//os:linux": ["@swift_toolchain_ubuntu24.04//:files"],
+        "@platforms//os:linux": [":swift_runtime_libs"],

Does the above make sense?

Regarding the follow-ups: note that we do not ship any Swift on Linux at the moment so this does look like a real problem. Also I we would ship this, this would currently depend on the runtime libraries already being present on any macOS system on which this get run. I don't know if that's a valid assumption?

[redsun82]

yeah, that looks like a good solution! and yes, you're right about Linux not being in in any case.

[jketema]

Fixed in cf8eeb8

[redsun82]

could we use a 22.04 toolchain, for a broader glibc compatibility range?

[jketema]

Yes, assuming it exists.

[jketema]

Fixed in 38c9c84

[redsun82]

could we make this degrade gracefully when there's no working swift toolchain, instead of hard-panicking? our actual builds go through bazel, and cargo day-to-day is mostly just fmt/check for us, but right now build.rs runs swift build unconditionally, so with no swift on PATH even those blow up. and since the crate is a workspace member now, a plain cargo check at the repo root fails for anyone who doesn't have swift installed, not just in here.

check/clippy don't link, so they don't actually need the swift lib to exist. so I think if swift build isn't runnable we could just emit a cargo:warning=... and skip the link directives rather than panic. that keeps the cargo fmt/check loop we rely on working swift-free, and only cargo build/cargo test would fail after that (at link, which is fair enough: those genuinely need swift, and we get them through bazel anyway).

[redsun82]

(speaking from a Linux box perspective that doesn't have Swift preinstalled 😉)

[jketema]

I'll leave this one for @tausbn to fix.