| Directory | Contents |
|---|---|
| src/cnlp_compiler/ | CNL-P parsing, static checking, control-flow processing, and Agent code generation. |
| src/converter/ | A converter and model request client for converting natural-language instructions into CNL-P, RISEN, or RODES. |
| experiments/RQ2/ | Experiments on model understanding and execution of CNL-P, RISEN, RODES, and natural-language instructions. |
| experiments/RQ3/ | Experiments on static checking of CNL-P syntax and semantic errors. |
| experiments/RQ4/ | Experiments on Agent code generation and execution. |
| experiment_results/RQ1/ | Participant programs, scores, and feedback from the user study. |
| experiment_results/RQ2/ | Task instances, instructions used in the experiments, human ratings, model responses, and accuracy results. |
| experiment_results/RQ3/ | Outputs and records from the static-checking experiments. |
| experiment_results/RQ4/ | Generated Agent programs, execution outputs, and evaluation records. |
Use Python 3.11.8.
Run the following command from the repository root, which contains requirements.txt:
python -m pip install -r requirements.txtCopy .env.example in the repository root to .env in the same directory, then enter your configuration. If .env already exists, edit that file directly.
if (-not (Test-Path -LiteralPath ".env")) { Copy-Item -LiteralPath ".env.example" -Destination ".env" }The basic configuration is shown below. Replace the example endpoint, API key, and model name with your actual values:
BASE_URL=https://your-provider.example/v1
OPENAI_API_KEY=your-api-key
OPENAI_MODEL=your-model-id| Variable | Purpose |
|---|---|
| BASE_URL | The model service endpoint. |
| OPENAI_API_KEY | The model service API key shared by the compiler, converter, and experiment code; configure it once. |
| OPENAI_MODEL | The default model name used by the compiler. |
| OPENAI_ORG | An optional organization identifier; omit it if the service does not require it. |
Both the compiler and converter read OPENAI_API_KEY. The compiler's default model is specified by OPENAI_MODEL, while the converter's model name is supplied as a call argument.
The repository includes the task instances used in RQ2:
- instances/natural_instructions/: Instances from six Natural Instructions tasks.
- instances/kor_bench/: Ten KOR-Bench rule instances.
The paths for both datasets are configured under paths.instances in experiments/RQ2/config.yaml.
RQ3 error instances are stored in semantic/tasks/ and syntaxical/task/. RQ4 specifications and test inputs are stored in cases/.
Run the following commands from the repository root.
python experiments/RQ2/main.pypython src/cnlp_compiler/run_linting.pypython src/cnlp_compiler/validate_templates.pypython experiments/RQ4/main.py| 目录 | 内容 |
|---|---|
| src/cnlp_compiler/ | CNL-P 解析、静态检查、控制流程处理及 Agent 代码生成。 |
| src/converter/ | 将自然语言指令转换为 CNL-P、RISEN 或 RODES 的转换器及模型请求客户端。 |
| experiments/RQ2/ | 模型对 CNL-P、RISEN、RODES 与自然语言指令的理解和执行实验。 |
| experiments/RQ3/ | CNL-P 语法错误和语义错误的静态检查实验。 |
| experiments/RQ4/ | Agent 代码生成与执行实验。 |
| experiment_results/RQ1/ | 用户研究中的参与者程序、评分和反馈。 |
| experiment_results/RQ2/ | 任务实例、实际指令、人工评分、模型回答及准确率结果。 |
| experiment_results/RQ3/ | 静态检查实验的输出与记录。 |
| experiment_results/RQ4/ | 生成的 Agent 程序、执行输出及评价记录。 |
使用 Python 3.11.8。
在仓库根目录,即包含 requirements.txt 的目录,执行:
python -m pip install -r requirements.txt将仓库根目录的 .env.example 复制为同一目录下的 .env,填写实际配置。已有 .env 时直接修改其中的配置。
if (-not (Test-Path -LiteralPath ".env")) { Copy-Item -LiteralPath ".env.example" -Destination ".env" }基本配置如下,示例中的服务地址、密钥和模型名称需要替换为实际值:
BASE_URL=https://your-provider.example/v1
OPENAI_API_KEY=your-api-key
OPENAI_MODEL=your-model-id| 变量 | 用途 |
|---|---|
| BASE_URL | 模型服务的接口地址。 |
| OPENAI_API_KEY | 编译器、转换器及实验代码共用的模型服务密钥,只需填写一次。 |
| OPENAI_MODEL | 编译器使用的默认模型名称。 |
| OPENAI_ORG | 可选的组织标识,服务不要求时无需填写。 |
编译器和转换器统一读取 OPENAI_API_KEY。编译器的默认模型由 OPENAI_MODEL 指定,转换器的模型名称由调用参数指定。
仓库已包含 RQ2 使用的任务实例:
- instances/natural_instructions/:Natural Instructions 的六项任务实例。
- instances/kor_bench/:KOR-Bench 的十条规则实例。
两个数据集的读取路径在 experiments/RQ2/config.yaml 的 paths.instances 中配置。
RQ3 的错误实例分别保存在 semantic/tasks/ 和 syntaxical/task/,RQ4 的需求文件与测试输入保存在 cases/。
以下命令在仓库根目录执行。
python experiments/RQ2/main.pypython src/cnlp_compiler/run_linting.pypython src/cnlp_compiler/validate_templates.pypython experiments/RQ4/main.py