macOS: improve IME state management

[ksqsf]

• Tested on all platforms changed
• Added an entry to the changelog module if knowledge of this change could be valuable to users
• Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• Created or updated an example program if it would help users understand this functionality

Fixes #3925
Fixes #3814

The basic idea of this change is: when IME is allowed, we always forward key events to the IME first, and use the result from the IME. Simpler keyboards, like the US keyboard, is also correctly supported by this.

Summary of changes:

• Send Ime::Enabled when set_ime_allowed(true). Do not manage it elsewhere. This is a huge simplification, and it should be harmless to current conforming applications.
• Do not prevent inserting text when there's no preedit.
• Do not try to forward key to app even when no text is committed. This is safe because the IME now receive all keys and forwards back unhandled keys.
• Some code becomes useless after the change and is thus removed.

What is improved:

1. Any Chinese or Japanese input methods should now commit full-width punctuation just fine. Previously, this never worked.
2. SKK Japanese IME should work now. Also, the toggle key q (toggles katakana) should work properly. Previously, SKK is not usable at all.
3. The behavior of Korean IMEs doesn't change.

(Korean IMEs work mostly fine but #3095 is not fixed by this PR: the first key will generate a call to insertText: instead of setMarkedText:, which seems unrelated to the problems this PR tries to fix.)

[fredizzimo]

I'm unable to test this myself, but it should probably mark #3814 as fixed too.

[madsmtm]

I hate our IME code with passion (another way of say I don't understand it at all ;) ), and will defer to Kirill's judgement as to whether this is a good idea or not. On the surface it looks fine.

[kchibisov]

All regular input now goes though IME, which is wrong. You can see that every character is going through Ime::Commit and not through regular keyboard input, thus nothing really works anymore when it comes to bindings and key combinations.

Regular input shouldn't go through IME.

[ksqsf]

Hmmm, then things are much more complicated. How about enable IME only when a text entry is focused?

Another way is perhaps to handle all shortcuts in performKeyEquivalent, supposing we know which shortcuts we are interested in.

[kchibisov]

What text entry when we have none of that, so it won't really help as you can see? Like the issue is that normal input is not being sent at all. I'd suggest to figure out why punctuation is not sent in the first place, like there's clearly a case with the old code that makes it ignored, which should likely be changed to Commit, since no key input is also generated. It's just it seems like there should be a proper AND condition.

[lifei]

what's the status of this issue?

[kchibisov]

Someone with macOS should figure out how to make it work, so it doesn't blow up all over the place. The comment I stated above still applies.

[lifei]

Thing here: 👇

1. macOS App Neovim.app use winit, when using Squirrel as IME, no luck.
2. Neovim.app, when using Apple IME, ok.
3. Wezterm built with Rust too, when using Squirrel as IME, OK.

App	IME	Result
Neovim.app	Squirrel	❌
Neovim.app	Apple IME	✔
Wezterm	Squirrel	✔

So, it's ok when fixing IME, it's also ok when fixing GUI framework…

[kchibisov]

I mean the point is not that there's no issue on our side, it's just someone should solve it in a way that is not distractable, by stating more that there's an issue, you don't help anyone, since we know that information. The issue is who can write code in a way that it works, arguing won't go anywhere.

You may provide guidance on how to fix it, so one could pick it up, but any discussion that is not around how to write code won't really move anything.

[lifei]

wezterm/wezterm@180a991 👈 similar commit in wezterm, hope helpful.

I can't write Rust and have no knowledge of macOS GUI programming. Maybe using AI can help you guys to better understand the code.

[kchibisov]

it's irrelevant though to us. The problem is that all macOS stuff (IME as well) is reverse engineering pretty much. If you don't know what you're talking about, then please, don't make an extra noise here for everyone else.

[ShikiSuen]

@lifei Mentioning Apple IME (the macOS Built-in Simplified-Chinese Pinyin IME (macOS 26)) is not helpful at all.

Despite that the 。 is correctly committed by this IME, the base Ukelele keyLayout of this IME hardcoded the 。 symbol to the period key. This proves that this IME is not suitable for testing issues like this one.

Please always use Sogou IME or WeChat IME to test this. Test results using RIME Squirrel may vary depending on what input schema is loaded in RIME.

[ksqsf]

I think this is superseded by #4651, hence closing.