[feat] add regex_replace_feature and align tokenize/text_normalizer with fg

[tiankongdeguiji]

Add regex_replace_feature, a wrapper around the pyfg op of the same name, which replaces text matched by one or more RE2 patterns. It works as a scalar feature, as a flat sequence_regex_replace_feature, and as a sub-feature of a grouped sequence_feature.

Two things are worth calling out in the implementation:

• fg has no sequence_regex_replace_feature; the sequence version is activated by is_sequence (the same 「特殊情况1」 family as custom_feature). So the class builds its fg json in fg_json() rather than _fg_json(), otherwise BaseFeature.fg_json would rename the type to something fg does not register.
• value_dim defaults to 1 instead of fg's 0. With 0 the output column type is array<string>, which tokenize_feature rejects with has invalid input type.

The motivating use case is truncating a text and appending an EOS token before tokenization, which tokenize_feature can not do on its own: the tokenizer is called without special tokens, and a truncation block in tokenizer.json runs after tokenization and cuts the freshly appended EOS off. One regex_replace_feature with (?s)^(.{0,N}).*$ / \1<|im_end|> does both in a single op, upstream of the tokenizer. FAQ Q19 describes the chain for a single text and for a sequence of texts, and Q20 covers why padding in tokenizer.json is usually the wrong tool.

num_buckets is kept for parity with the other string features, with a documented caveat: it only works while every replaced string parses as an integer in [0, num_buckets), otherwise fg raises.

Tokenize / text_normalizer fixes

While writing the docs above, the tokenize path turned out to describe fg incorrectly in several places. Everything below was checked against the fg C++ and re-measured with pyfg 1.0.6:

• norm_options were summed, not or-ed (parameter += ...): a repeated option silently set an unrelated bit — [TEXT_FILTER, TEXT_FILTER] produced 64, which is the unimplemented __NORMALIZED_SYNONYMS__, so filtering was simply off. Now or-ed.
• The documented default normalization was wrong: the docs and the proto comment said TEXT_LOWER2UPPER, ..., but nmConstant.h defines the default as UPPER2LOWER|SBC2DBC|BIG52GBK|FILTER (=60) — upper→lower. The example config carried the same wrong option.
• TEXT_REMOVE_SPACE alone is a no-op switch: it is not a bit, so configuring only it emits parameter: 0, and fg treats 0 as "use the default set". "Only remove spaces" is not expressible; documented.
• TEXT_FILTER replaces the character with a space, it does not delete it (spaces are merged and trimmed afterwards).
• stop_char_file must be GBK-encoded with one character per line, and it replaces the built-in table — a UTF-8 file raises no error and matches nothing.
• tokenizer_type: bpe really means "a huggingface tokenizers json", whose algorithm (BPE, WordPiece, …) is decided by the json; the doc implied WordPiece was unsupported.
• tokenize_feature.default_value is text that gets tokenized, and sequence-mode features silently rewrite an empty default to "0", i.e. the token for the literal string 0.
• regex_pattern is required by fg, but proto can not mark a repeated field required. An empty list compiles to (?:), which matches the empty string everywhere and inserts the replacement between every character, with no error anywhere. It now raises.

Test Plan

• New tzrec/features/regex_replace_feature_test.py (16 cases): fg-encoded parsing, FG_NORMAL for replace_all / replace_first / multi-pattern / icase / num_buckets / default_value (which fg emits verbatim, without the regex), the missing-regex_pattern error, hash_bucket_size through fg, a multi-value array input with value_dim: 0, the flat sequence form asserting the fg json keeps regex_replace_feature + is_sequence, the grouped sequence sub-feature, and two end-to-end truncate+EOS+tokenize tests that run the whole DAG through create_fg_json + pyfg.FgArrowHandler with data/test/tokenizer.json and assert the EOS id lands at the end of each text.
• New case in tokenize_feature_test.py asserting [TEXT_UPPER2LOWER, TEXT_UPPER2LOWER, TEXT_FILTER] yields 4 | 32 rather than the summed 40.
• Checked the two configs in the new docs against a real Qwen/Qwen3.5-0.8B tokenizer.json: every text ends with exactly one <|im_end|>, both scalar and per sequence element, and the text is cut at the configured character count.
• Each documented fg behavior above was re-measured through pyfg: the default normalization on "ABC EFG！" → "abc efg", the parameter: 0 fallback → "abcefg", and the empty-pattern corruption "abc" → "XaXbXcX".
• tzrec/features/feature_test.py, tokenize_feature_test.py, id_feature_test.py, custom_feature_test.py, bool_mask_feature_test.py and tzrec/datasets/data_parser_test.py pass (the only shared change is adding RegexReplaceFeature to SINGLE_INPUT_FEATURE_CLASSES).
• Every feature_configs { ... } example in faq.md was parsed with text_format and serialized, to catch missing proto2 required fields (the Q19 grouped example was missing sequence_delim), and the doc table's regex_pattern cells were rendered with markdown-it and fed back through text_format to confirm they are copy-pasteable.
• pre-commit run --files <changed> and pyrefly check clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VkWBjeAQP8rcsutwL6rkfV

[github-actions]

Minor: SequenceFeature.sequence_delim is proto2 required (feature.proto:1071), but this example omits it. The current loader (text_format.Merge) tolerates it and the getter falls back to ";", but strict paths (text_format.Parse / IsInitialized, C++-side consumers) reject the message. Every other sequence_feature example in the docs (faq.md Q5, feature.md) sets it explicitly, as does this PR's own grouped-sequence test. Suggest adding sequence_delim: ";".

[github-actions]

Minor: the regex_pattern cells are one escaping level short. Raw ["\\\|"] renders in the table as ["\|"] (markdown eats \\→\ and \|→|), and \| / \( are invalid escapes in pbtxt strings — a user copying the rendered table into a config gets a ParseError. The fenced example above correctly shows ["\\|", "#"]. Either add one more backslash level in the raw markdown (e.g. ["\\\\\|"] → renders ["\\|"]) or label this column as the effective regex rather than the config value. The input/output columns themselves are correct.

[github-actions]

Coverage gap in the bucketize/modifier branches fg_json() copies from IdFeature:

• hash_bucket_size — the primary config in the feature.md example — never runs through fg: the fg-encoded test above is FG_NONE (which skips fg_json() entirely), and the missing-pattern test raises before the bucketize chain. Deleting the elif HasField("hash_bucket_size") branch would keep CI green.
• The value_dim: 0 + separator multi-value mode that the new docs promise is untested; all current cases are single-value, so both the separator branch and the fg_cfg["value_dim"] emission are unobservable.

One FG_NORMAL case with hash_bucket_size, or with value_dim=0 + separator=" " and a replacement producing a multi-value string (asserting lengths > 1), would pin both — alternatively fg_json dict asserts like the sequence tests do. Same applies to a lesser extent to the zch/vocab_dict/vocab_file branches and the SINGLE_INPUT_FEATURE_CLASSES line in feature.py, which nothing exercises with a user:-side sequence expression.

[github-actions]

Code Review Summary

Reviewed across five areas (code quality, performance, test coverage, documentation accuracy, security). Overall this is a solid, well-verified PR — no bugs found in the implementation, and the fg-behavior claims in the docs check out.

Verified correct:

• The fg_json() full override (instead of _fg_json()) matches the established CustomFeature/BoolMaskFeature precedent for the is_sequence-activated feature family; the empty-default_value → "0" fallback is not lost (the BaseFeature.default_value property already applies it), and both flat and grouped sequence shapes are pinned by real-pyfg tests.
• The += → |= norm-options fix is correct (NORM_OPTION_MAPPING values are distinct powers of two, so the change only affects repeated options — exactly the corrupt-bit case), and the new test genuinely discriminates (40 vs 36).
• The empty-regex_pattern guard fires on every path that materializes fg json (FG_NORMAL construction, FG_DAG, export), symmetrically across ranks; the FG_NONE non-firing path is benign since fg never runs there.
• Docs vs. upstream: RE2 semantics in Q19 ((?s), $ = end-of-text, . = code point) confirmed against the RE2 syntax reference; proto comments, feature.md, and the FAQ are mutually consistent on defaults, TEXT_REMOVE_SPACE, stop_char_file encoding, and num_buckets; Q19/Q20 numbering and the fg-json shapes all line up with the code.
• Class registration (auto_import + metaclass), proto field layout/numbering (mirrors IdFeature), and the 1.4.9 version bump all follow repo conventions. Performance and security reviews found nothing noteworthy (all new Python runs at config-build time; RE2 is linear-time).

Posted 3 inline comments (all minor):

1. faq.md Q19 grouped example omits required sequence_delim — not a valid proto2 message under strict parsing.
2. feature.md example table: the regex_pattern cells are one markdown-escaping level short — they render as invalid pbtxt (\|, \().
3. regex_replace_feature_test.py: the bucketize branches copied from IdFeature have no coverage through this class — hash_bucket_size (the primary config in the docs) never runs through fg, and the documented value_dim: 0 + separator multi-value mode is untested.

Optional nits (no action required):

• The output_type reference in Q20 is a generated-fg-json field, not a user-facing config — a clarifying clause ("TorchEasyRec生成的FG配置中output_type固定为word_id") would save users a futile config search.
• Since this PR aligns text_normalizer docs with measured fg, the TEXT_FILTER enum comment in feature.proto ("filter speicial chars", pre-existing typo) could also be updated to match the new "特殊符号替换成空格" wording in feature.md.

🤖 Generated with Claude Code