Repository navigation
fix: wrap Chinese, Japanese and Korean text by display width - #112
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Text in Chinese, Japanese and Korean has no spaces to break at, so long lines never wrapped, and wide characters counted as one column although they take two. Line length is now measured in display columns: CJK and other East Asian wide characters and common emoji take two, combining marks take none. A line holding wide characters may also break between CJK characters, keeping closing punctuation and small kana off the start of a line and opening brackets off the end.
Heading rules and data table columns use the same widths, so a
<h2>日本語</h2>rule is six columns and CJK table cells line up.Lines without wide characters keep the existing fast path. Finding wide characters costs one extra pass over each line longer than the line length: on
BenchmarkWordWrap, ASCII prose is ~1µs (16%) slower per 15KB and prose with curly quotes ~2µs. Full conversions of the benchmark corpus are unchanged within noise, and their output is byte-identical. CJK text costs about 8ns a character now that it actually wraps.The width table is a hand-written approximation rather than
golang.org/x/text/width, to avoid a new dependency; emoji joined with U+200D count each part. It is marked with aponytail:comment.