diff --git a/docs/source/faq.md b/docs/source/faq.md index 93503b25..73bc73b4 100644 --- a/docs/source/faq.md +++ b/docs/source/faq.md @@ -391,3 +391,158 @@ export NUMPY_MANUAL_SEED=100007 export TORCH_MANUAL_SEED=100007 # 同时会设置所有CUDA设备的种子 export USE_DETERMINISTIC_ALGORITHMS=1 # 已包含cudnn的确定性行为 ``` + +______________________________________________________________________ + +**Q19: tokenize_feature如何截断文本并保留EOS token** + +`tokenize_feature`不会自动添加EOS等特殊token。这里有三个机制容易混淆：**按字符截断发生在分词之前，tokenizer的truncation发生在分词之后，EOS在哪一步加入决定了它会不会被截掉**。如果希望文本截断后仍然以EOS结尾，需要根据截断方式选择不同的方案。 + +**先确定需要哪种截断方式** + +| 需求 | 推荐方案 | 说明 | +| ------------------------------------------ | ---------------------------------------------------------- | --------------------------------------------------------------------- | +| 只需要限制token数量，不需要EOS | 在`tokenizer.json`中配置`truncation`，用`direction: Right` | 直接按token数截断，不需要`regex_replace_feature` | +| 需要EOS，并保留文本开头 | 在分词前用`regex_replace_feature`按字符截断并追加EOS | 推荐方案；不要再配置`direction: Right`的tokenizer truncation | +| 需要EOS，可以丢弃文本开头 | 上游追加EOS，再配置`direction: Left` | Left truncation保留文本末尾，因此EOS不会被截掉 | +| 既要精确的token数量上限，又要EOS且保留开头 | 当前配置方式无法同时严格保证 | 可以按字符数保守截断；如果再用Right truncation兜底，超长样本仍会丢EOS | + +**1. 推荐方案：分词前截断文本并追加EOS** + +例如，需要： + +``` +原始 title + ↓ 最多保留前200个字符 +截断后的 title + <|im_end|> + ↓ tokenize +title_token +``` + +可以通过`regex_replace_feature`和`tokenize_feature`串联实现： + +``` +feature_configs { + regex_replace_feature { + feature_name: "title_eos" + expression: "item:title" + regex_pattern: "(?s)^(.{0,200}).*$" + replacement: "\\1<|im_end|>" + replace_all: false + stub_type: true + } +} +feature_configs { + tokenize_feature { + feature_name: "title_token" + expression: "feature:title_eos" + vocab_file: "tokenizer.json" + embedding_dim: 128 + tokens_as_sequence: true + sequence_length: 64 + } +} +``` + +这里： + +- `regex_replace_feature`先截取最多200个字符，再追加`<|im_end|>`。`.`按字符（UTF-8）计数，不是字节也不是token；`(?s)`让`.`可以匹配换行符；`$`匹配的是文本结尾而不是行结尾，配合`replace_all: false`保证只追加一个EOS +- `stub_type: true`表示`title_eos`只是FG的中间结果，不会作为特征输出给模型 +- `tokenize_feature`通过`feature:title_eos`消费上一步的结果 +- 特征之间通过`feature:`输入域串联，因此`data_config.fg_mode`需要配置为`FG_DAG` + +**2. EOS token的注意事项** + +- EOS字面量必须已经存在于`tokenizer.json`的`added_tokens`中，例如Qwen的`<|im_end|>`，否则会被BPE拆成多个token +- `tokenizer_type: sentencepiece`不支持上述方式 +- 如果已经用`regex_replace_feature`在文本末尾追加了EOS，就不要再在`tokenizer.json`中配置`direction: Right`的truncation，tokenizer的截断发生在分词之后，会把末尾的EOS再截掉 +- 输入为空时，可以不给`regex_replace_feature`配置`default_value`，由后面的`tokenize_feature.default_value`兜底 + +**3. 序列特征的多段文本** + +对于分组序列特征，也可以用相同的方式： + +``` +feature_configs { + sequence_feature { + sequence_name: "click_50_seq" + sequence_length: 50 + sequence_delim: ";" + features { + regex_replace_feature { + feature_name: "title_eos" + expression: "item:title" + regex_pattern: "(?s)^(.{0,200}).*$" + replacement: "\\1<|im_end|>" + replace_all: false + stub_type: true + } + } + features { + tokenize_feature { + feature_name: "title_token" + expression: "feature:title_eos" + sequence_fields: ["title_eos"] + vocab_file: "tokenizer.json" + embedding_dim: 128 + } + } + } +} +``` + +这里需要用`sequence_fields: ["title_eos"]`声明`title_eos`是序列字段，FG会把输入改写成`feature:__`，从而引用到同一个序列下的中间特征。 + +**4. 如果需要按token数截断** + +如果不要求“保留文本开头的同时保证EOS存在”，可以直接用`tokenizer.json`的`truncation`： + +```json +"truncation": { + "max_length": 128, + "strategy": "LongestFirst", + "direction": "Right", + "stride": 0 +} +``` + +`tokenize_feature`是直接调用tokenizer做Encode的，因此这里的`max_length`限制的是**分词后的token数量**，而不是原始文本的字符数。需要特别区分： + +- `direction: Right`：保留前面的token，截掉末尾，因此可能把EOS截掉 +- `direction: Left`：保留末尾的token，EOS可以保留，但会丢掉文本开头 + +`strategy`主要影响文本对的截断方式，单段文本保持默认即可。 + +**5. 容易混淆的两个参数** + +- `text_normalizer`的`max_length`不是文本截断参数，文本超过该长度时它只是跳过normalization并原样输出 +- `tokens_as_sequence`时配置的`sequence_length`也不会传给tokenizer做token截断，token数量只能通过tokenizer的`truncation`或分词前的字符截断来控制 + +`tokenizer.json`中的`padding`见Q20。 + +______________________________________________________________________ + +**Q20: tokenize_feature是否应该在tokenizer.json中配置padding** + +**一般不建议。** `padding`确实会生效，但补齐出来的pad token在下游和真实token没有区别，TorchEasyRec也不需要定长的输入。 + +`strategy`配成`{"Fixed": N}`时每条文本都会补齐到N个token；配成`"BatchLongest"`则不起作用，因为FG是逐条调用Encode的，一个“batch”里只有一条文本。 + +```json +"padding": { + "strategy": { "Fixed": 128 }, + "direction": "Right", + "pad_to_multiple_of": null, + "pad_id": 248044, + "pad_type_id": 0, + "pad_token": "<|endoftext|>" +} +``` + +不建议配置的原因： + +- 默认的`tokenize_feature`会把补齐的pad token一起pooling，短文本的向量会被pad的embedding淹没 +- `tokens_as_sequence: true`时每条样本的序列长度都变成N，sequence_encoder拿到的长度也全是N，无法区分真实token和padding +- TorchEasyRec在需要稠密序列时会自己按batch内的最大长度padding，并保留每条样本真实的长度用于mask，在tokenizer里补齐反而会丢掉这个信息 + +如果确实需要定长输出，注意TorchEasyRec生成的FG配置中`output_type`固定为`word_id`，因此只有`pad_id`生效：`pad_token`不会和`pad_id`做一致性校验，配错了不会报错；`pad_id`也不会校验是否在词表范围内，超出词表大小时训练会在embedding查表时越界。 diff --git a/docs/source/feature/feature.md b/docs/source/feature/feature.md index 2e479e62..ad75e16c 100644 --- a/docs/source/feature/feature.md +++ b/docs/source/feature/feature.md @@ -1,6 +1,6 @@ # 特征 -TorchEasyRec多种类型的特征，包括IdFeature、RawFeature、ComboFeature、CombineFeature、LookupFeature、MatchFeature、ExprFeature、OverlapFeature、TokenizeFeature、KvDotProduct、BoolMaskFeature、CustomFeature、SequenceFeature。 +TorchEasyRec多种类型的特征，包括IdFeature、RawFeature、ComboFeature、CombineFeature、LookupFeature、MatchFeature、ExprFeature、OverlapFeature、TokenizeFeature、KvDotProduct、BoolMaskFeature、RegexReplaceFeature、CustomFeature、SequenceFeature。 **共用配置** @@ -576,7 +576,7 @@ feature_configs { vocab_file: "tokenizer.json" embedding_dim: 8 text_normalizer { - norm_options: [TEXT_LOWER2UPPER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER] + norm_options: [TEXT_UPPER2LOWER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER] } } } @@ -586,12 +586,16 @@ feature_configs { - **vocab_file**: 分词字典，完全兼容 https://github.com/mlc-ai/tokenizers-cpp 库的分词文件 -- **tokenizer_type**: 分词类型，支持bpe、sentencepiece，默认为bpe +- **tokenizer_type**: 分词类型，默认为`bpe`。`bpe`表示用huggingface tokenizers的json词典，具体是BPE还是WordPiece等由`tokenizer.json`的内容决定；`sentencepiece`表示用sentencepiece模型 + +- **default_value**: 输入为空时的默认值。注意该默认值是**文本**，会跟正常输入一样被分词。 + 序列特征（包括`tokens_as_sequence: true`）不支持空默认值，未配置时会被重置为`"0"` - **text_normalizer**: 可选，是否对文本进行归一化 - - **stop_char_file**: 停用词表路径，默认为系统内置，详见[stop_char](https://tzrec.oss-accelerate.aliyuncs.com/third_party/stop_char) - - **norm_options**: 归一化选项，默认为TEXT_LOWER2UPPER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER + - **max_length**: 可选，默认不限制。输入长度超过该值时**跳过归一化，原样输出原始文本**（不是截断），并且每条超长记录都会打一条ERROR日志。长度按**GBK编码的字节数**计算，中文和全角字符算2字节，ASCII算1字节，即`max_length: 512`约等于512个英文字符或256个汉字 + - **stop_char_file**: 特殊符号表路径，默认为系统内置，详见[stop_char](https://tzrec.oss-accelerate.aliyuncs.com/third_party/stop_char)。文件必须是**GBK编码**、每行一个字符，配置后会**替换**（而不是追加）内置的特殊符号表 + - **norm_options**: 归一化选项，默认为TEXT_UPPER2LOWER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER。注意`TEXT_REMOVE_SPACE`不是归一化选项而是单独的开关，只配它等价于没有配置归一化选项，FG会按默认选项归一化，也就是说无法表达“只去空格、不做其他归一化” | 方式 | 描述 | | ----------------- | ---------------------- | @@ -599,7 +603,7 @@ feature_configs { | TEXT_UPPER2LOWER | 大写转换成小写 | | TEXT_SBC2DBC | 全角到半角 | | TEXT_CHT2CHS | 繁体到简体 | - | TEXT_FILTER | 去除特殊符号 | + | TEXT_FILTER | 特殊符号替换成空格 | | TEXT_SPLITCHRS | 中文拆成单字(空格分隔) | | TEXT_REMOVE_SPACE | 去除空格 | @@ -680,6 +684,52 @@ feature_configs { | [1, 2, 3, 4] | [1, 0, 1, 0] | [1, 3] | | [1, 2, 3, 4] | "true,false,true,false" | [1, 3] | +## RegexReplaceFeature: 正则替换特征 + +`regex_replace_feature`用正则表达式（[RE2语法](https://github.com/google/re2/wiki/Syntax)）替换输入文本中匹配的片段，可以配置多个pattern，匹配任一pattern的片段都会被替换。 + +``` +feature_configs { + regex_replace_feature { + feature_name: "query_clean" + expression: "user:query" + regex_pattern: ["\\|", "#"] + replacement: " " + embedding_dim: 32 + hash_bucket_size: 100000 + } +} +``` + +- **expression**: 特征FG所依赖字段的来源 + +- **regex_pattern**: 必选项，正则表达式，`string`或`list`类型，配置多个时取并集 + +- **replacement**: 替换文本，可以用`\\1`引用pattern中的捕获组；为空时删除匹配的文本片段 + +- **replace_all**: 是否全局替换，默认为true；设为false时只替换第一次匹配到的片段 + +- **icase**: 匹配时是否忽略大小写，默认为false + +- **value_dim**: 输出值的维度，默认为1。为1时输出列的类型是单值`string`，为其他值时是`array`，`tokenize_feature`等下游算子只接受单值输入。注意它不会截断输出，多值（array）输入的每个元素都会被替换，输出值的个数由输入决定 + +- **default_value**: 输入为空时的默认值，默认值会**原样输出，不经过正则替换**；不配置时，输入为空则不输出任何值 + +- 分箱支持`hash_bucket_size`/`vocab_list`/`vocab_dict`/`vocab_file`/`num_buckets`，其中`num_buckets`要求替换后的文本都能转成`[0, num_buckets)`的整数，非数字的文本会导致FG报错 + +- 在分词前截断文本并追加EOS token的用法，见[FAQ](../faq.md) + +- **separator**: 只在序列场景下用来切分序列元素内的多值字符串，非序列的字符串输入不会按它切分 + +- 其余配置同IdFeature + +示例 + +| 输入 | regex_pattern | replacement | 未进行bucketize的输出 | +| ------------------ | --------------------------- | ----------- | --------------------- | +| 中华\|人民\|共和国 | `["\\\|"]` | `" "` | 中华 人民 共和国 | +| a\|b#c(d) | `["\\\|", "#", "\\(.*\\)"]` | `""` | abc | + ## CustomFeature: 自定义特征 自定义特征，自定义方式参考[自定义算子文档](https://help.aliyun.com/zh/airec/what-is-pai-rec/user-guide/custom-feature-operator) @@ -721,11 +771,11 @@ feature_configs { - 其余配置如果是类别型特征同IdFeature，如果是数值型特征同RawFeature -| 算子名称 | 算子功能 | 算子动态库 | 算子参数 | -| ------------ | -------- | ---------------------------- | -------------------------------------------------------------------------------- | -| EditDistance | 编辑距离 | pyfg/lib/libedit_distance.so | • encoding: 输入文本的编码，可选：utf-8, latin，默认值为latin | -| RegexReplace | 正则替换 | pyfg/lib/libregex_replace.so | • regex_patten: 正则表达式，匹配的文本片段将会被替换
• replacement: 替换文本 | -| | | | | +| 算子名称 | 算子功能 | 算子动态库 | 算子参数 | +| ------------ | --------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------- | +| EditDistance | 编辑距离 | pyfg/lib/libedit_distance.so | • encoding: 输入文本的编码，可选：utf-8, latin，默认值为latin | +| RegexReplace | 正则替换，建议直接用RegexReplaceFeature | pyfg/lib/libregex_replace.so | • regex_patten: 正则表达式，匹配的文本片段将会被替换
• replacement: 替换文本 | +| | | | | ## SequenceFeature：序列特征 diff --git a/tzrec/features/feature.py b/tzrec/features/feature.py index 5cd025c7..1fcd949f 100644 --- a/tzrec/features/feature.py +++ b/tzrec/features/feature.py @@ -74,6 +74,7 @@ "RawFeature", "TokenizeFeature", "CombineFeature", + "RegexReplaceFeature", ] diff --git a/tzrec/features/regex_replace_feature.py b/tzrec/features/regex_replace_feature.py new file mode 100644 index 00000000..7c076911 --- /dev/null +++ b/tzrec/features/regex_replace_feature.py @@ -0,0 +1,91 @@ +# Copyright (c) 2026, Alibaba Group; +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Any, Dict, List + +from tzrec.features.feature import ( + MAX_HASH_BUCKET_SIZE, +) +from tzrec.features.id_feature import IdFeature + + +class RegexReplaceFeature(IdFeature): + """RegexReplaceFeature class. + + Args: + feature_config (FeatureConfig): a instance of feature config. + """ + + @property + def value_dim(self) -> int: + """Fg value dimension of the feature.""" + # fg types the output column as array unless value_dim is 1, and + # tokenize_feature rejects an array input, so we default to 1 instead of + # IdFeature's 0. it has to be the property, the model side and the fg + # json would disagree otherwise. + if self.config.HasField("value_dim"): + return self.config.value_dim + else: + return 1 + + def fg_json(self) -> List[Dict[str, Any]]: + """Get fg json config.""" + if len(self.config.regex_pattern) == 0: + # fg compiles an empty pattern list into `(?:)`, which matches the + # empty string everywhere and inserts replacement between every char + raise ValueError( + f"{self.__class__.__name__}[{self.name}] must set regex_pattern." + ) + # fg has no sequence_regex_replace_feature, the sequence version is + # activated by is_sequence, so we do not use _fg_json here. + fg_cfg = { + "feature_type": "regex_replace_feature", + "feature_name": self.config.feature_name, + "default_value": self.default_value, + "expression": self.config.expression, + "regex_pattern": list(self.config.regex_pattern), + "replacement": self.config.replacement, + } + if not self.config.replace_all: + fg_cfg["replace_all"] = False + if self.config.icase: + fg_cfg["icase"] = True + if self.config.separator != "\x1d": + fg_cfg["separator"] = self.config.separator + if self.config.HasField("zch") or self.config.HasField("dynamicemb"): + fg_cfg["hash_bucket_size"] = MAX_HASH_BUCKET_SIZE + elif self.config.HasField("hash_bucket_size"): + fg_cfg["hash_bucket_size"] = self.config.hash_bucket_size + elif len(self.vocab_list) > 0: + fg_cfg["vocab_list"] = self.vocab_list + fg_cfg["default_bucketize_value"] = self.default_bucketize_value + elif len(self.vocab_dict) > 0: + fg_cfg["vocab_dict"] = self.vocab_dict + fg_cfg["default_bucketize_value"] = self.default_bucketize_value + elif len(self.vocab_file) > 0: + fg_cfg["vocab_file"] = self.vocab_file + fg_cfg["default_bucketize_value"] = self.default_bucketize_value + elif self.config.HasField("num_buckets"): + fg_cfg["num_buckets"] = self.config.num_buckets + fg_cfg["value_dim"] = self.value_dim + if self.config.HasField("stub_type"): + fg_cfg["stub_type"] = self.config.stub_type + + if self.is_sequence: + if self.is_grouped_sequence: + if len(self.config.sequence_fields) > 0: + fg_cfg["sequence_fields"] = list(self.config.sequence_fields) + else: + fg_cfg["sequence_delim"] = self.sequence_delim + fg_cfg["sequence_length"] = self.sequence_length + fg_cfg["is_sequence"] = True + + return [fg_cfg] diff --git a/tzrec/features/regex_replace_feature_test.py b/tzrec/features/regex_replace_feature_test.py new file mode 100644 index 00000000..2258b835 --- /dev/null +++ b/tzrec/features/regex_replace_feature_test.py @@ -0,0 +1,385 @@ +# Copyright (c) 2026, Alibaba Group; +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import unittest + +import numpy as np +import pyarrow as pa +import pyfg +from parameterized import param, parameterized + +from tzrec.features import regex_replace_feature as regex_replace_feature_lib +from tzrec.features.feature import FgMode, create_features, create_fg_json +from tzrec.protos import feature_pb2 +from tzrec.utils import test_util + +# `<|endoftext|>` is an added token of data/test/tokenizer.json +_EOS = "<|endoftext|>" +_EOS_ID = 0 + + +class RegexReplaceFeatureTest(unittest.TestCase): + @parameterized.expand( + [ + [["1\x032", "", None, "3"], [1, 2, 3], [2, 0, 0, 1]], + [[[1, 2], None, None, [3]], [1, 2, 3], [2, 0, 0, 1]], + ] + ) + def test_fg_encoded_regex_replace_feature( + self, input_feat, expected_values, expected_lengths + ): + regex_feat_cfg = feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="regex_feat", + embedding_dim=16, + hash_bucket_size=100, + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature(regex_feat_cfg) + self.assertEqual(regex_feat.output_dim, 16) + self.assertEqual(regex_feat.is_sparse, True) + self.assertEqual(regex_feat.inputs, ["regex_feat"]) + + parsed_feat = regex_feat.parse({"regex_feat": pa.array(input_feat)}) + self.assertEqual(parsed_feat.name, "regex_feat") + np.testing.assert_allclose(parsed_feat.values, np.array(expected_values)) + np.testing.assert_allclose(parsed_feat.lengths, np.array(expected_lengths)) + + @parameterized.expand( + [ + param( + "replace_all", + regex_pattern=["\\|"], + replacement=" ", + inputs=["中华|人民|共和国", "abc", None], + vocab_list=["中华 人民 共和国", "abc"], + expected_values=[2, 3], + ), + param( + "replace_first", + regex_pattern=["\\|"], + replacement=" ", + replace_all=False, + inputs=["中华|人民|共和国", "abc", None], + vocab_list=["中华 人民|共和国", "abc"], + expected_values=[2, 3], + ), + param( + "multi_pattern", + regex_pattern=["\\|", "#", "\\(.*\\)"], + replacement="", + inputs=["a|b#c(d)", "abc", None], + vocab_list=["abc"], + expected_values=[2, 2], + ), + param( + "icase", + regex_pattern=["abc"], + replacement="x", + icase=True, + inputs=["ABCd", "abcd", None], + vocab_list=["xd"], + expected_values=[2, 2], + ), + param( + "truncate_and_append_eos", + regex_pattern=["(?s)^(.{0,3}).*$"], + replacement="\\1" + _EOS, + replace_all=False, + inputs=["abcdef", "中华人民共和国", None], + vocab_list=["abc" + _EOS, "中华人" + _EOS], + expected_values=[2, 3], + ), + param( + "default_value_not_replaced", + regex_pattern=["\\|"], + replacement=" ", + inputs=["a|b", None], + default_value="x|y", + vocab_list=["a b"], + expected_values=[2, 0], + expected_lengths=[1, 1], + ), + ], + name_func=test_util.parameterized_name_func, + ) + def test_regex_replace_feature( + self, + name, + regex_pattern, + replacement, + inputs, + vocab_list, + expected_values, + expected_lengths=(1, 1, 0), + default_value="", + replace_all=True, + icase=False, + ): + regex_feat_cfg = feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="regex_feat", + embedding_dim=16, + expression="item:title", + regex_pattern=regex_pattern, + replacement=replacement, + replace_all=replace_all, + icase=icase, + default_value=default_value, + vocab_list=vocab_list, + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature( + regex_feat_cfg, fg_mode=FgMode.FG_NORMAL + ) + self.assertEqual(regex_feat.output_dim, 16) + self.assertEqual(regex_feat.is_sparse, True) + self.assertEqual(regex_feat.inputs, ["title"]) + self.assertEqual(regex_feat.num_embeddings, len(vocab_list) + 2) + + parsed_feat = regex_feat.parse({"title": pa.array(inputs)}) + self.assertEqual(parsed_feat.name, "regex_feat") + np.testing.assert_allclose(parsed_feat.values, np.array(expected_values)) + np.testing.assert_allclose(parsed_feat.lengths, np.array(expected_lengths)) + + def test_regex_replace_feature_with_num_buckets(self): + regex_feat_cfg = feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="regex_feat", + embedding_dim=16, + expression="item:title", + regex_pattern=["[^0-9]"], + replacement="", + num_buckets=100, + default_value="0", + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature( + regex_feat_cfg, fg_mode=FgMode.FG_NORMAL + ) + self.assertEqual(regex_feat.num_embeddings, 100) + + parsed_feat = regex_feat.parse({"title": pa.array(["id7", "id42", None])}) + np.testing.assert_allclose(parsed_feat.values, np.array([7, 42, 0])) + + def test_regex_replace_feature_with_hash_bucket_size(self): + regex_feat_cfg = feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="regex_feat", + embedding_dim=16, + expression="item:title", + regex_pattern=["\\|"], + replacement=" ", + hash_bucket_size=100, + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature( + regex_feat_cfg, fg_mode=FgMode.FG_NORMAL + ) + self.assertEqual(regex_feat.num_embeddings, 100) + + # "a|b" and "a b" hash to the same bucket after the replacement + parsed_feat = regex_feat.parse({"title": pa.array(["a|b", "a b", "c|d"])}) + values = parsed_feat.values.tolist() + self.assertEqual(values[0], values[1]) + self.assertNotEqual(values[0], values[2]) + self.assertTrue(all(0 <= v < 100 for v in values)) + + def test_regex_replace_feature_with_multival_input(self): + regex_feat_cfg = feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="regex_feat", + embedding_dim=16, + expression="item:title", + regex_pattern=["\\|"], + replacement=" ", + value_dim=0, + vocab_list=["a b", "c d"], + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature( + regex_feat_cfg, fg_mode=FgMode.FG_NORMAL + ) + self.assertEqual(regex_feat.fg_json()[0]["value_dim"], 0) + + parsed_feat = regex_feat.parse({"title": pa.array([["a|b", "c|d"], ["c|d"]])}) + np.testing.assert_allclose(parsed_feat.values, np.array([2, 3, 3])) + np.testing.assert_allclose(parsed_feat.lengths, np.array([2, 1])) + + def test_regex_replace_feature_without_regex_pattern(self): + regex_feat_cfg = feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="regex_feat", + embedding_dim=16, + expression="item:title", + replacement="x", + hash_bucket_size=100, + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature(regex_feat_cfg) + with self.assertRaises(ValueError): + regex_feat.fg_json() + + def test_tokenize_truncated_text_with_eos(self): + feature_cfgs = [ + feature_pb2.FeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="title_eos", + expression="item:title", + regex_pattern=["(?s)^(.{0,8}).*$"], + replacement="\\1" + _EOS, + replace_all=False, + stub_type=True, + ) + ), + feature_pb2.FeatureConfig( + tokenize_feature=feature_pb2.TokenizeFeature( + feature_name="title_token", + expression="feature:title_eos", + embedding_dim=16, + vocab_file="data/test/tokenizer.json", + tokens_as_sequence=True, + ) + ), + ] + features = create_features(feature_cfgs, fg_mode=FgMode.FG_DAG) + # pyre-ignore [16] + fg_handler = pyfg.FgArrowHandler(create_fg_json(features), 1) + fg_output, status = fg_handler.process_arrow( + {"title": pa.array(["abc efg hij klm", "hij", None])} + ) + self.assertTrue(status.ok(), status.message()) + + feat_data = fg_output["title_token"] + np.testing.assert_allclose( + feat_data.np_values, + np.array([19758, 299, 16054, 209, _EOS_ID, 73, 1944, _EOS_ID, 17]), + ) + np.testing.assert_allclose(feat_data.np_lengths, np.array([5, 3, 1])) + + +class SequenceRegexReplaceFeatureTest(unittest.TestCase): + def test_sequence_regex_replace_feature(self): + regex_feat_cfg = feature_pb2.FeatureConfig( + sequence_regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="click_50_seq_title", + embedding_dim=16, + expression="item:titles", + regex_pattern=["\\|"], + replacement=" ", + sequence_delim=";", + sequence_length=50, + vocab_list=["a b", "c d"], + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature( + regex_feat_cfg, fg_mode=FgMode.FG_NORMAL, is_sequence=True + ) + self.assertEqual(regex_feat.is_sequence, True) + self.assertEqual(regex_feat.inputs, ["titles"]) + # fg has no sequence_regex_replace_feature, is_sequence activates it + fg_cfg = regex_feat.fg_json()[0] + self.assertEqual(fg_cfg["feature_type"], "regex_replace_feature") + self.assertEqual(fg_cfg["is_sequence"], True) + self.assertEqual(fg_cfg["sequence_delim"], ";") + self.assertEqual(fg_cfg["sequence_length"], 50) + + parsed_feat = regex_feat.parse({"titles": pa.array(["a|b;c|d", "c|d"])}) + np.testing.assert_allclose(parsed_feat.values, np.array([2, 3, 3])) + np.testing.assert_allclose(parsed_feat.key_lengths, np.array([1, 1, 1])) + np.testing.assert_allclose(parsed_feat.seq_lengths, np.array([2, 1])) + + def test_grouped_sequence_regex_replace_feature(self): + regex_feat_cfg = feature_pb2.SeqFeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="title_clean", + embedding_dim=16, + expression="item:title", + regex_pattern=["\\|"], + replacement=" ", + vocab_list=["a b", "c d"], + ) + ) + regex_feat = regex_replace_feature_lib.RegexReplaceFeature( + regex_feat_cfg, + fg_mode=FgMode.FG_NORMAL, + is_sequence=True, + sequence_name="click_50_seq", + sequence_delim=";", + sequence_length=50, + ) + self.assertEqual(regex_feat.inputs, ["click_50_seq__title"]) + fg_cfg = regex_feat.fg_json()[0] + self.assertEqual(fg_cfg["feature_type"], "regex_replace_feature") + self.assertEqual(fg_cfg["is_sequence"], True) + self.assertNotIn("sequence_delim", fg_cfg) + + parsed_feat = regex_feat.parse( + {"click_50_seq__title": pa.array(["a|b;c|d", "c|d"])} + ) + self.assertEqual(parsed_feat.name, "click_50_seq__title_clean") + np.testing.assert_allclose(parsed_feat.values, np.array([2, 3, 3])) + np.testing.assert_allclose(parsed_feat.seq_lengths, np.array([2, 1])) + + def test_tokenize_truncated_sequence_text_with_eos(self): + feature_cfgs = [ + feature_pb2.FeatureConfig( + sequence_feature=feature_pb2.SequenceFeature( + sequence_name="click_50_seq", + sequence_length=50, + sequence_delim=";", + features=[ + feature_pb2.SeqFeatureConfig( + regex_replace_feature=feature_pb2.RegexReplaceFeature( + feature_name="title_eos", + expression="item:title", + regex_pattern=["(?s)^(.{0,8}).*$"], + replacement="\\1" + _EOS, + replace_all=False, + stub_type=True, + ) + ), + feature_pb2.SeqFeatureConfig( + tokenize_feature=feature_pb2.TokenizeFeature( + feature_name="title_token", + expression="feature:title_eos", + sequence_fields=["title_eos"], + embedding_dim=16, + vocab_file="data/test/tokenizer.json", + ) + ), + ], + ) + ) + ] + features = create_features(feature_cfgs, fg_mode=FgMode.FG_DAG) + # pyre-ignore [16] + fg_handler = pyfg.FgArrowHandler(create_fg_json(features), 1) + fg_output, status = fg_handler.process_arrow( + { + "click_50_seq": pa.array(["a;b"]), + "click_50_seq__title": pa.array(["abc efg hij;hij"]), + } + ) + self.assertTrue(status.ok(), status.message()) + + feat_data = fg_output["click_50_seq__title_token"] + np.testing.assert_allclose( + feat_data.np_values, + np.array([19758, 299, 16054, 209, _EOS_ID, 73, 1944, _EOS_ID]), + ) + np.testing.assert_allclose(feat_data.np_key_lengths, np.array([5, 3])) + np.testing.assert_allclose(feat_data.np_lengths, np.array([2])) + + +if __name__ == "__main__": + unittest.main() diff --git a/tzrec/features/tokenize_feature.py b/tzrec/features/tokenize_feature.py index c91ef2ba..07e6e04c 100644 --- a/tzrec/features/tokenize_feature.py +++ b/tzrec/features/tokenize_feature.py @@ -136,7 +136,7 @@ def _fg_json(self) -> List[Dict[str, Any]]: parameter = 0 for norm_option in norm_cfg.norm_options: if norm_option in NORM_OPTION_MAPPING: - parameter += NORM_OPTION_MAPPING[norm_option] + parameter |= NORM_OPTION_MAPPING[norm_option] if norm_option == TextNormalizeOption.TEXT_REMOVE_SPACE: norm_fg_cfg["remove_space"] = True norm_fg_cfg["parameter"] = parameter diff --git a/tzrec/features/tokenize_feature_test.py b/tzrec/features/tokenize_feature_test.py index 27a4cf8f..3b849443 100644 --- a/tzrec/features/tokenize_feature_test.py +++ b/tzrec/features/tokenize_feature_test.py @@ -252,6 +252,26 @@ def test_tokenize_feature_sentencepiece( np.testing.assert_allclose(parsed_feat.values, np.array(expected_values)) np.testing.assert_allclose(parsed_feat.lengths, np.array(expected_lengths)) + def test_text_norm_options_are_or_ed(self): + token_feat_cfg = feature_pb2.FeatureConfig( + tokenize_feature=feature_pb2.TokenizeFeature( + feature_name="token_feat", + vocab_file="data/test/tokenizer.json", + embedding_dim=16, + expression="user:token_input", + text_normalizer=feature_pb2.TextNormalizer( + norm_options=[ + feature_pb2.TEXT_UPPER2LOWER, + feature_pb2.TEXT_UPPER2LOWER, + feature_pb2.TEXT_FILTER, + ] + ), + ) + ) + token_feat = tokenize_feature_lib.TokenizeFeature(token_feat_cfg) + # summing the bits would give 4 + 4 + 32 = 40, which sets an unrelated bit + self.assertEqual(token_feat.fg_json()[0]["parameter"], 4 | 32) + class SequenceTokenizeFeatureTest(unittest.TestCase): @parameterized.expand( diff --git a/tzrec/protos/feature.proto b/tzrec/protos/feature.proto index da7503df..8535f56c 100644 --- a/tzrec/protos/feature.proto +++ b/tzrec/protos/feature.proto @@ -692,7 +692,7 @@ enum TextNormalizeOption { TEXT_SBC2DBC = 2; // traditional chinese to simple chinese TEXT_CHT2CHS = 3; - // filter speicial chars + // replace special chars with a space TEXT_FILTER = 4; // chinese split to chars with blanks TEXT_SPLITCHRS = 5; @@ -701,11 +701,14 @@ enum TextNormalizeOption { } message TextNormalizer { - // if text_length greater than max_length, will not do normalize + // if input length in gbk bytes greater than max_length, will not do + // normalize, and the raw text is output as-is optional uint32 max_length = 1; - // stop char file path, default will use built-in stop char + // stop char file path, must be gbk encoded with one char per line, and it + // replaces the built-in stop chars. default will use built-in stop char optional string stop_char_file = 2; - // text normalize options, default is TEXT_LOWER2UPPER & TEXT_SBC2DBC & TEXT_CHT2CHS & TEXT_FILTER + // text normalize options, default is TEXT_UPPER2LOWER & TEXT_SBC2DBC & TEXT_CHT2CHS & TEXT_FILTER. + // TEXT_REMOVE_SPACE is a standalone switch, setting only it still normalizes with the defaults repeated TextNormalizeOption norm_options = 3; } @@ -728,7 +731,9 @@ message TokenizeFeature { optional string pooling = 10 [default = "sum"]; // fg default value, default value before bucktize optional string default_value = 11 [default = ""]; - // tokenizer_type type, available is {bpe | sentencepiece} + // tokenizer_type type, available is {bpe | sentencepiece}. bpe means a + // huggingface tokenizers json vocab, whose real algorithm (BPE, WordPiece, ...) + // is decided by the json itself optional string tokenizer_type = 12 [default = "bpe"]; // embedding init function, e.g. "nn.init.uniform_,a=-0.01,b=0.01" optional string init_fn = 14; @@ -766,6 +771,79 @@ message TokenizeFeature { optional bool tokens_as_sequence = 104 [default = false]; } +message RegexReplaceFeature { + // feature name, e.g. title_clean + required string feature_name = 1; + // feature input, e.g. item:title + required string expression = 2; + // embedding name, feature with same embedding name will share embedding + optional string embedding_name = 3; + // embedding dimension, not needed when stub_type is true + optional uint32 embedding_dim = 4; + // number of hash size + optional uint64 hash_bucket_size = 5; + // number of id enumerators, replaced value should be an integer in + // [0, num_buckets), otherwise fg will raise an error + optional uint64 num_buckets = 6; + // id vocabulary list + repeated string vocab_list = 7; + // id vocabulary dict + map vocab_dict = 8; + // value dimensions, default = 1, set 0 to keep multi-value output + optional uint32 value_dim = 9; + // embedding pooling type, available is {sum | mean} + optional string pooling = 10 [default = "sum"]; + // fg default value, default value before bucktize + optional string default_value = 11 [default = ""]; + // fg multi-value separator + optional string separator = 12 [default = "\x1d"]; + // required, regular expressions, text matched any of them will be replaced + repeated string regex_pattern = 13; + // embedding init function, e.g. "nn.init.uniform_,a=-0.01,b=0.01" + optional string init_fn = 14; + // mask value in training progress + optional bool use_mask = 15; + // zero collision hash + optional ZeroCollisionHash zch = 16; + // id vocabulary file path + optional string vocab_file = 17; + // vocab file relative directory + optional string asset_dir = 18; + // dynamic embedding + optional DynamicEmbedding dynamicemb = 19; + // replacement text, matched text will be removed when it is empty + optional string replacement = 20; + // replace all matched text or only the first one + optional bool replace_all = 21 [default = true]; + // ignore case when matching or not + optional bool icase = 22 [default = false]; + + // default value when fg_mode = FG_NONE, + // when use pai-fg, you do not need to set the param. + // when use own fg and data contain null value, you can set the param for fill null + optional string fg_encoded_default_value = 30; + // out-of-vocab(OOV) id bucketize value when use vocab_list or vocab_dict + // when use default_bucketize_value, we will not add additional bucketize_value of + // `default_value`=0, bucketize_value of =1 into vocab_list or vocab_dict + optional uint64 default_bucketize_value = 31; + // embedding param trainable or not + optional bool trainable = 33 [default = true]; + // only used as fg dag intermediate result or not + optional bool stub_type = 34 [default = false]; + // embedding data type + optional string data_type = 35 [default = 'FP32']; + + // embedding param constraints + optional ParameterConstraints embedding_constraints = 50; + + // max sequence length, only take effect when use it as sequence + optional uint32 sequence_length = 101; + // sequence delimiter, only take effect when use it as sequence + optional string sequence_delim = 102 [default = ";"]; + // specify sequence type fields in inputs. default is item side inputs. + repeated string sequence_fields = 103; +} + message KvDotProduct { // feature name, e.g. kv_os_click_count @@ -1013,6 +1091,7 @@ message FeatureConfig { KvDotProduct kv_dot_product = 11; BoolMaskFeature bool_mask_feature = 12; CombineFeature combine_feature = 13; + RegexReplaceFeature regex_replace_feature = 14; IdFeature sequence_id_feature = 101; RawFeature sequence_raw_feature = 102; @@ -1026,6 +1105,7 @@ message FeatureConfig { KvDotProduct sequence_kv_dot_product = 111; BoolMaskFeature sequence_bool_mask_feature = 112; CombineFeature sequence_combine_feature = 113; + RegexReplaceFeature sequence_regex_replace_feature = 114; } } @@ -1044,5 +1124,6 @@ message SeqFeatureConfig { KvDotProduct kv_dot_product = 11; BoolMaskFeature bool_mask_feature = 12; CombineFeature combine_feature = 13; + RegexReplaceFeature regex_replace_feature = 14; } } diff --git a/tzrec/version.py b/tzrec/version.py index 7ecc3b05..5bf8aa73 100644 --- a/tzrec/version.py +++ b/tzrec/version.py @@ -9,4 +9,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -__version__ = "1.4.8" +__version__ = "1.4.9"