Skip to content

Fallback Ident is not normalized #500

Description

@kennytm

Starting from Rust 1.42.0 (rust-lang/rust#66670) Unicode identifiers are normalized in NFC form, that means the following proc-macro will produce true:

extern crate proc_macro;
use proc_macro::{Ident, Span, TokenStream};

#[proc_macro]
pub fn check(_: TokenStream) -> TokenStream {
    let decomposed = Ident::new("e\u{301}", Span::call_site());
    let composed = Ident::new("\u{e9}", Span::call_site());
    (composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

However, the proc_macro2 fallback does not perform NFC normalization, meaning the equivalent code in proc_macro2 will produce false:

use proc_macro2::{Ident, Span, TokenStream};

pub fn check(_: TokenStream) -> TokenStream {
    let decomposed = Ident::new("e\u{301}", Span::call_site());
    let composed = Ident::new("\u{e9}", Span::call_site());
    (composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

fn main() {
    dbg!(check(TokenStream::default()));
    // Ident { sym: false, span: bytes(1..6) }
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions