Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 155 additions & 0 deletions docs/source/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -391,3 +391,158 @@ export NUMPY_MANUAL_SEED=100007
export TORCH_MANUAL_SEED=100007 # 同时会设置所有CUDA设备的种子
export USE_DETERMINISTIC_ALGORITHMS=1 # 已包含cudnn的确定性行为
```

______________________________________________________________________

**Q19: tokenize_feature如何截断文本并保留EOS token**

`tokenize_feature`不会自动添加EOS等特殊token。这里有三个机制容易混淆:**按字符截断发生在分词之前,tokenizer的truncation发生在分词之后,EOS在哪一步加入决定了它会不会被截掉**。如果希望文本截断后仍然以EOS结尾,需要根据截断方式选择不同的方案。

**先确定需要哪种截断方式**

| 需求 | 推荐方案 | 说明 |
| ------------------------------------------ | ---------------------------------------------------------- | --------------------------------------------------------------------- |
| 只需要限制token数量,不需要EOS | 在`tokenizer.json`中配置`truncation`,用`direction: Right` | 直接按token数截断,不需要`regex_replace_feature` |
| 需要EOS,并保留文本开头 | 在分词前用`regex_replace_feature`按字符截断并追加EOS | 推荐方案;不要再配置`direction: Right`的tokenizer truncation |
| 需要EOS,可以丢弃文本开头 | 上游追加EOS,再配置`direction: Left` | Left truncation保留文本末尾,因此EOS不会被截掉 |
| 既要精确的token数量上限,又要EOS且保留开头 | 当前配置方式无法同时严格保证 | 可以按字符数保守截断;如果再用Right truncation兜底,超长样本仍会丢EOS |

**1. 推荐方案:分词前截断文本并追加EOS**

例如,需要:

```
原始 title
↓ 最多保留前200个字符
截断后的 title + <|im_end|>
↓ tokenize
title_token
```

可以通过`regex_replace_feature`和`tokenize_feature`串联实现:

```
feature_configs {
regex_replace_feature {
feature_name: "title_eos"
expression: "item:title"
regex_pattern: "(?s)^(.{0,200}).*$"
replacement: "\\1<|im_end|>"
replace_all: false
stub_type: true
}
}
feature_configs {
tokenize_feature {
feature_name: "title_token"
expression: "feature:title_eos"
vocab_file: "tokenizer.json"
embedding_dim: 128
tokens_as_sequence: true
sequence_length: 64
}
}
```

这里:

- `regex_replace_feature`先截取最多200个字符,再追加`<|im_end|>`。`.`按字符(UTF-8)计数,不是字节也不是token;`(?s)`让`.`可以匹配换行符;`$`匹配的是文本结尾而不是行结尾,配合`replace_all: false`保证只追加一个EOS
- `stub_type: true`表示`title_eos`只是FG的中间结果,不会作为特征输出给模型
- `tokenize_feature`通过`feature:title_eos`消费上一步的结果
- 特征之间通过`feature:`输入域串联,因此`data_config.fg_mode`需要配置为`FG_DAG`

**2. EOS token的注意事项**

- EOS字面量必须已经存在于`tokenizer.json`的`added_tokens`中,例如Qwen的`<|im_end|>`,否则会被BPE拆成多个token
- `tokenizer_type: sentencepiece`不支持上述方式
- 如果已经用`regex_replace_feature`在文本末尾追加了EOS,就不要再在`tokenizer.json`中配置`direction: Right`的truncation,tokenizer的截断发生在分词之后,会把末尾的EOS再截掉
- 输入为空时,可以不给`regex_replace_feature`配置`default_value`,由后面的`tokenize_feature.default_value`兜底

**3. 序列特征的多段文本**

对于分组序列特征,也可以用相同的方式:

```
feature_configs {
sequence_feature {
sequence_name: "click_50_seq"
sequence_length: 50

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: SequenceFeature.sequence_delim is proto2 required (feature.proto:1071), but this example omits it. The current loader (text_format.Merge) tolerates it and the getter falls back to ";", but strict paths (text_format.Parse / IsInitialized, C++-side consumers) reject the message. Every other sequence_feature example in the docs (faq.md Q5, feature.md) sets it explicitly, as does this PR's own grouped-sequence test. Suggest adding sequence_delim: ";".

sequence_delim: ";"
features {
regex_replace_feature {
feature_name: "title_eos"
expression: "item:title"
regex_pattern: "(?s)^(.{0,200}).*$"
replacement: "\\1<|im_end|>"
replace_all: false
stub_type: true
}
}
features {
tokenize_feature {
feature_name: "title_token"
expression: "feature:title_eos"
sequence_fields: ["title_eos"]
vocab_file: "tokenizer.json"
embedding_dim: 128
}
}
}
}
```

这里需要用`sequence_fields: ["title_eos"]`声明`title_eos`是序列字段,FG会把输入改写成`feature:<sequence_name>__<feature_name>`,从而引用到同一个序列下的中间特征。

**4. 如果需要按token数截断**

如果不要求“保留文本开头的同时保证EOS存在”,可以直接用`tokenizer.json`的`truncation`:

```json
"truncation": {
"max_length": 128,
"strategy": "LongestFirst",
"direction": "Right",
"stride": 0
}
```

`tokenize_feature`是直接调用tokenizer做Encode的,因此这里的`max_length`限制的是**分词后的token数量**,而不是原始文本的字符数。需要特别区分:

- `direction: Right`:保留前面的token,截掉末尾,因此可能把EOS截掉
- `direction: Left`:保留末尾的token,EOS可以保留,但会丢掉文本开头

`strategy`主要影响文本对的截断方式,单段文本保持默认即可。

**5. 容易混淆的两个参数**

- `text_normalizer`的`max_length`不是文本截断参数,文本超过该长度时它只是跳过normalization并原样输出
- `tokens_as_sequence`时配置的`sequence_length`也不会传给tokenizer做token截断,token数量只能通过tokenizer的`truncation`或分词前的字符截断来控制

`tokenizer.json`中的`padding`见Q20。

______________________________________________________________________

**Q20: tokenize_feature是否应该在tokenizer.json中配置padding**

**一般不建议。** `padding`确实会生效,但补齐出来的pad token在下游和真实token没有区别,TorchEasyRec也不需要定长的输入。

`strategy`配成`{"Fixed": N}`时每条文本都会补齐到N个token;配成`"BatchLongest"`则不起作用,因为FG是逐条调用Encode的,一个“batch”里只有一条文本。

```json
"padding": {
"strategy": { "Fixed": 128 },
"direction": "Right",
"pad_to_multiple_of": null,
"pad_id": 248044,
"pad_type_id": 0,
"pad_token": "<|endoftext|>"
}
```

不建议配置的原因:

- 默认的`tokenize_feature`会把补齐的pad token一起pooling,短文本的向量会被pad的embedding淹没
- `tokens_as_sequence: true`时每条样本的序列长度都变成N,sequence_encoder拿到的长度也全是N,无法区分真实token和padding
- TorchEasyRec在需要稠密序列时会自己按batch内的最大长度padding,并保留每条样本真实的长度用于mask,在tokenizer里补齐反而会丢掉这个信息

如果确实需要定长输出,注意TorchEasyRec生成的FG配置中`output_type`固定为`word_id`,因此只有`pad_id`生效:`pad_token`不会和`pad_id`做一致性校验,配错了不会报错;`pad_id`也不会校验是否在词表范围内,超出词表大小时训练会在embedding查表时越界。
72 changes: 61 additions & 11 deletions docs/source/feature/feature.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 特征

TorchEasyRec多种类型的特征,包括IdFeature、RawFeature、ComboFeature、CombineFeature、LookupFeature、MatchFeature、ExprFeature、OverlapFeature、TokenizeFeature、KvDotProduct、BoolMaskFeature、CustomFeature、SequenceFeature。
TorchEasyRec多种类型的特征,包括IdFeature、RawFeature、ComboFeature、CombineFeature、LookupFeature、MatchFeature、ExprFeature、OverlapFeature、TokenizeFeature、KvDotProduct、BoolMaskFeature、RegexReplaceFeature、CustomFeature、SequenceFeature。

**共用配置**

Expand Down Expand Up @@ -576,7 +576,7 @@ feature_configs {
vocab_file: "tokenizer.json"
embedding_dim: 8
text_normalizer {
norm_options: [TEXT_LOWER2UPPER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER]
norm_options: [TEXT_UPPER2LOWER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER]
}
}
}
Expand All @@ -586,20 +586,24 @@ feature_configs {

- **vocab_file**: 分词字典,完全兼容 https://github.com/mlc-ai/tokenizers-cpp 库的分词文件

- **tokenizer_type**: 分词类型,支持bpe、sentencepiece,默认为bpe
- **tokenizer_type**: 分词类型,默认为`bpe`。`bpe`表示用huggingface tokenizers的json词典,具体是BPE还是WordPiece等由`tokenizer.json`的内容决定;`sentencepiece`表示用sentencepiece模型

- **default_value**: 输入为空时的默认值。注意该默认值是**文本**,会跟正常输入一样被分词。
序列特征(包括`tokens_as_sequence: true`)不支持空默认值,未配置时会被重置为`"0"`

- **text_normalizer**: 可选,是否对文本进行归一化

- **stop_char_file**: 停用词表路径,默认为系统内置,详见[stop_char](https://tzrec.oss-accelerate.aliyuncs.com/third_party/stop_char)
- **norm_options**: 归一化选项,默认为TEXT_LOWER2UPPER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER
- **max_length**: 可选,默认不限制。输入长度超过该值时**跳过归一化,原样输出原始文本**(不是截断),并且每条超长记录都会打一条ERROR日志。长度按**GBK编码的字节数**计算,中文和全角字符算2字节,ASCII算1字节,即`max_length: 512`约等于512个英文字符或256个汉字
- **stop_char_file**: 特殊符号表路径,默认为系统内置,详见[stop_char](https://tzrec.oss-accelerate.aliyuncs.com/third_party/stop_char)。文件必须是**GBK编码**、每行一个字符,配置后会**替换**(而不是追加)内置的特殊符号表
- **norm_options**: 归一化选项,默认为TEXT_UPPER2LOWER, TEXT_SBC2DBC, TEXT_CHT2CHS, TEXT_FILTER。注意`TEXT_REMOVE_SPACE`不是归一化选项而是单独的开关,只配它等价于没有配置归一化选项,FG会按默认选项归一化,也就是说无法表达“只去空格、不做其他归一化”

| 方式 | 描述 |
| ----------------- | ---------------------- |
| TEXT_LOWER2UPPER | 小写转换成大写 |
| TEXT_UPPER2LOWER | 大写转换成小写 |
| TEXT_SBC2DBC | 全角到半角 |
| TEXT_CHT2CHS | 繁体到简体 |
| TEXT_FILTER | 去除特殊符号 |
| TEXT_FILTER | 特殊符号替换成空格 |
| TEXT_SPLITCHRS | 中文拆成单字(空格分隔) |
| TEXT_REMOVE_SPACE | 去除空格 |

Expand Down Expand Up @@ -680,6 +684,52 @@ feature_configs {
| [1, 2, 3, 4] | [1, 0, 1, 0] | [1, 3] |
| [1, 2, 3, 4] | "true,false,true,false" | [1, 3] |

## RegexReplaceFeature: 正则替换特征

`regex_replace_feature`用正则表达式([RE2语法](https://github.com/google/re2/wiki/Syntax))替换输入文本中匹配的片段,可以配置多个pattern,匹配任一pattern的片段都会被替换。

```
feature_configs {
regex_replace_feature {
feature_name: "query_clean"
expression: "user:query"
regex_pattern: ["\\|", "#"]
replacement: " "
embedding_dim: 32
hash_bucket_size: 100000
}
}
```

- **expression**: 特征FG所依赖字段的来源

- **regex_pattern**: 必选项,正则表达式,`string`或`list<string>`类型,配置多个时取并集

- **replacement**: 替换文本,可以用`\\1`引用pattern中的捕获组;为空时删除匹配的文本片段

- **replace_all**: 是否全局替换,默认为true;设为false时只替换第一次匹配到的片段

- **icase**: 匹配时是否忽略大小写,默认为false

- **value_dim**: 输出值的维度,默认为1。为1时输出列的类型是单值`string`,为其他值时是`array<string>`,`tokenize_feature`等下游算子只接受单值输入。注意它不会截断输出,多值(array)输入的每个元素都会被替换,输出值的个数由输入决定

- **default_value**: 输入为空时的默认值,默认值会**原样输出,不经过正则替换**;不配置时,输入为空则不输出任何值

- 分箱支持`hash_bucket_size`/`vocab_list`/`vocab_dict`/`vocab_file`/`num_buckets`,其中`num_buckets`要求替换后的文本都能转成`[0, num_buckets)`的整数,非数字的文本会导致FG报错

- 在分词前截断文本并追加EOS token的用法,见[FAQ](../faq.md)

- **separator**: 只在序列场景下用来切分序列元素内的多值字符串,非序列的字符串输入不会按它切分

- 其余配置同IdFeature

示例

| 输入 | regex_pattern | replacement | 未进行bucketize的输出 |
| ------------------ | --------------------------- | ----------- | --------------------- |
| 中华\|人民\|共和国 | `["\\\|"]` | `" "` | 中华 人民 共和国 |
| a\|b#c(d) | `["\\\|", "#", "\\(.*\\)"]` | `""` | abc |

## CustomFeature: 自定义特征

自定义特征,自定义方式参考[自定义算子文档](https://help.aliyun.com/zh/airec/what-is-pai-rec/user-guide/custom-feature-operator)
Expand Down Expand Up @@ -721,11 +771,11 @@ feature_configs {

- 其余配置如果是类别型特征同IdFeature,如果是数值型特征同RawFeature

| 算子名称 | 算子功能 | 算子动态库 | 算子参数 |
| ------------ | -------- | ---------------------------- | -------------------------------------------------------------------------------- |
| EditDistance | 编辑距离 | pyfg/lib/libedit_distance.so | • encoding: 输入文本的编码,可选:utf-8, latin,默认值为latin |
| RegexReplace | 正则替换 | pyfg/lib/libregex_replace.so | • regex_patten: 正则表达式,匹配的文本片段将会被替换 <br>• replacement: 替换文本 |
| | | | |
| 算子名称 | 算子功能 | 算子动态库 | 算子参数 |
| ------------ | --------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------- |
| EditDistance | 编辑距离 | pyfg/lib/libedit_distance.so | • encoding: 输入文本的编码,可选:utf-8, latin,默认值为latin |
| RegexReplace | 正则替换,建议直接用RegexReplaceFeature | pyfg/lib/libregex_replace.so | • regex_patten: 正则表达式,匹配的文本片段将会被替换 <br>• replacement: 替换文本 |
| | | | |

## SequenceFeature:序列特征

Expand Down
1 change: 1 addition & 0 deletions tzrec/features/feature.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@
"RawFeature",
"TokenizeFeature",
"CombineFeature",
"RegexReplaceFeature",
]


Expand Down
91 changes: 91 additions & 0 deletions tzrec/features/regex_replace_feature.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Copyright (c) 2026, Alibaba Group;
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from typing import Any, Dict, List

from tzrec.features.feature import (
MAX_HASH_BUCKET_SIZE,
)
from tzrec.features.id_feature import IdFeature


class RegexReplaceFeature(IdFeature):
"""RegexReplaceFeature class.

Args:
feature_config (FeatureConfig): a instance of feature config.
"""

@property
def value_dim(self) -> int:
"""Fg value dimension of the feature."""
# fg types the output column as array<string> unless value_dim is 1, and
# tokenize_feature rejects an array input, so we default to 1 instead of
# IdFeature's 0. it has to be the property, the model side and the fg
# json would disagree otherwise.
if self.config.HasField("value_dim"):
return self.config.value_dim
else:
return 1

def fg_json(self) -> List[Dict[str, Any]]:
"""Get fg json config."""
if len(self.config.regex_pattern) == 0:
# fg compiles an empty pattern list into `(?:)`, which matches the
# empty string everywhere and inserts replacement between every char
raise ValueError(
f"{self.__class__.__name__}[{self.name}] must set regex_pattern."
)
# fg has no sequence_regex_replace_feature, the sequence version is
# activated by is_sequence, so we do not use _fg_json here.
fg_cfg = {
"feature_type": "regex_replace_feature",
"feature_name": self.config.feature_name,
"default_value": self.default_value,
"expression": self.config.expression,
"regex_pattern": list(self.config.regex_pattern),
"replacement": self.config.replacement,
}
if not self.config.replace_all:
fg_cfg["replace_all"] = False
if self.config.icase:
fg_cfg["icase"] = True
if self.config.separator != "\x1d":
fg_cfg["separator"] = self.config.separator
if self.config.HasField("zch") or self.config.HasField("dynamicemb"):
fg_cfg["hash_bucket_size"] = MAX_HASH_BUCKET_SIZE
elif self.config.HasField("hash_bucket_size"):
fg_cfg["hash_bucket_size"] = self.config.hash_bucket_size
elif len(self.vocab_list) > 0:
fg_cfg["vocab_list"] = self.vocab_list
fg_cfg["default_bucketize_value"] = self.default_bucketize_value
elif len(self.vocab_dict) > 0:
fg_cfg["vocab_dict"] = self.vocab_dict
fg_cfg["default_bucketize_value"] = self.default_bucketize_value
elif len(self.vocab_file) > 0:
fg_cfg["vocab_file"] = self.vocab_file
fg_cfg["default_bucketize_value"] = self.default_bucketize_value
elif self.config.HasField("num_buckets"):
fg_cfg["num_buckets"] = self.config.num_buckets
fg_cfg["value_dim"] = self.value_dim
if self.config.HasField("stub_type"):
fg_cfg["stub_type"] = self.config.stub_type

if self.is_sequence:
if self.is_grouped_sequence:
if len(self.config.sequence_fields) > 0:
fg_cfg["sequence_fields"] = list(self.config.sequence_fields)
else:
fg_cfg["sequence_delim"] = self.sequence_delim
fg_cfg["sequence_length"] = self.sequence_length
fg_cfg["is_sequence"] = True

return [fg_cfg]
Loading
Loading