CodeNet — C→Rust Translation Benchmark

200 C programs to translate to Rust, scored by pass@1: each translation is compiled and run against the reference C program's own I/O (command-line arguments or stdin), and a problem passes only if the Rust program's output matches the C program's output on every test case.

Nothing about the score is a judgement call — a problem either compiles and reproduces the reference output, or it does not.

Methodology. Decoding is held fixed for every model being compared (temperature 0.6, top_p 0.95, top_k 20). A single run at that temperature is worth roughly ±2–3 points, so a score is always the mean of 5 seeds; scripts/aggregate.py reports mean, standard deviation, and a bootstrap confidence interval over problems. No reference scores are published in this repository — the benchmark itself is the artifact.

---

How one problem is evaluated

raw_data/<mode>/<problem>.c          the reference C program
        │
        │  run_eval.py tells the prompt builder whether input arrives via argv or stdin
        │  (SACTOR_IO_MODE), so the model is told the I/O convention instead of guessing
        ▼
   sactor translate --unidiomatic-only  ──►  Rust translation (single stage, unsafe Rust)
        │
        │  compile; on failure the engine retries with the compiler error as feedback,
        │  up to max_translation_attempts (6 in the shipped configs)
        ▼
   test_tasks/<mode>/<problem>.c.json      one test command per problem
        │
        │  run the Rust binary on the inputs in generated_tests/, diff against the
        │  expected output captured from the reference C program
        ▼
   status.json  ──►  {"returncode": 0, "status": "success", "reason": "success", ...}

--unidiomatic-only means one translation stage: preserve the C interface, produce compiling Rust that behaves identically. The engine's second ("idiomatic") stage is not part of this benchmark.

Repository layout

CodeNet/                       the dataset, self-contained (see CodeNet/README.md)
  raw_data/{argv,scanf}/           200 reference C programs (92 argv + 108 scanf)
  generated_tests/{argv,scanf}/    input / expected-output cases
  test_tasks/{argv,scanf}/         the test command per problem
  manifest.json                    problem list, metric, decoding settings
scripts/run_eval.py            the runner: translate all 200 in parallel, write per-problem status
scripts/aggregate.py           pass@1 mean ± std + bootstrap CI across seeds
scripts/launch_model.sh        serve a local checkpoint with sglang
configs/generic_prompt.toml    harness + decoding config, generic translation prompt
configs/native_prompt.toml     same, SFT-native prompt (for instruction-tuned checkpoints)
configs/api_openrouter.toml    same, against an API endpoint instead of a local server
run_5seed.sh                   driver: 5 seeds against one model, then aggregate
fix_paths.sh                   one-time fixup after cloning (see step 1)
engine/                        the SACTOR translation engine, vendored as source

---

Setup (once per machine)

1. Re-point the tests

Every CodeNet/test_tasks/*.json embeds an absolute path to its generated_tests counterpart. They ship as /__DATASET__/... placeholders, so rewrite them to your checkout:

bash fix_paths.sh          # prints: re-pointed 200 test_tasks to <your path>

Skip this and every problem fails with Invalid test command.

2. Build the engine

engine/ ships as source only (no venv, no compiled artifacts — same policy as model weights):

cd engine
uv sync                        # creates engine/.venv from uv.lock
./update_rust_ast_parser.sh    # builds the rust_ast_parser extension module
cargo build --release          # rust_ast_parser + sactor_proc_macros
cd ..

The Rust toolchain version is pinned in engine/rust-toolchain.toml.

3. Check the tools SACTOR requires on PATH

The engine calls check_all_requirements() at import and raises OSError: Missing requirements unless all four resolve — crown, rustup, c2rust, and gcc or clang:

for t in crown rustup c2rust gcc; do printf '%-8s %s\n' "$t" "$(command -v $t || echo MISSING)"; done

run_eval.py prepends $SACTOR_HOME/.venv/bin, ~/.local/bin, $SACTOR_HOME/crown/target/release and ~/.cargo/bin to PATH, so a symlink in ~/.local/bin is enough for anything built elsewhere.

4. Point at the engine

export SACTOR_HOME=$PWD/engine     # default; set it elsewhere to use another install

SACTOR_HOME is both where the sactor CLI is found ($SACTOR_HOME/.venv/bin/sactor) and the working directory used for every translate call.

See SETUP.md for the serving stack, the environment variables launch_model.sh sets and why, and the failure modes that look like model quality but are not.

---

Running

Step 1 — serve the model

Local checkpoint:

export SERVE_VENV=/path/to/your/sglang-venv
./scripts/launch_model.sh /path/to/checkpoint 0,1 30878 2

Arguments are MODEL_PATH GPUS [PORT] [TP] [CHAT_TEMPLATE]. Loading a large checkpoint takes a while; wait for health 200 before starting an eval:

curl -s -o /dev/null -w "%{http_code}\n" localhost:30878/health     # want 200

The port must match api_base in the config you evaluate with (the shipped configs use 30878). Note the ptxas= field in the launcher's banner — see SETUP.md §3 for why it matters.

API endpoint instead: edit configs/api_openrouter.toml to set the model id, then export OPENROUTER_API_KEY=.... No server, no GPU.

Step 2 — smoke-test 2 problems first

Do this every time you move to a new machine. It separates environment failures from model failures, which otherwise look identical in a 200-problem log:

python3 scripts/run_eval.py configs/native_prompt.toml results/_smoke \
    --modes argv --limit 2 --workers 1

You want reason=success. Anything else — read results/_smoke/argv/*/translate.log and fix the environment before launching the full set. Then rm -rf results/_smoke.

Step 3 — the full run, 5 seeds

PORT=30878 CFG=configs/native_prompt.toml TAG=mymodel ./run_5seed.sh

The driver waits for /health, runs 5 seeds sequentially, writes one log per seed under results/_driverlogs/, and aggregates at the end. For an API-hosted model there is no server to wait for:

SKIP_HEALTH=1 CFG=configs/api_openrouter.toml TAG=mymodel_api ./run_5seed.sh

Or call the runner directly, if you want control over the loop:

for s in 1 2 3 4 5; do
  python3 scripts/run_eval.py configs/native_prompt.toml results/eval_mymodel_s$s \
      --root ./CodeNet --modes argv,scanf --workers 4
done

--workers 4 is the default for good reason — see Operational rules.

Step 4 — aggregate

python3 scripts/aggregate.py results eval_mymodel

Output format (numbers below are illustrative, not a result):

eval_mymodel: pass@1 = NN.N +- N.N  (per-seed ['NN.N', 'NN.N', 'NN.N', 'NN.N', 'NN.N']; n=200, 5 seeds)
  95% CI (bootstrap by problem): [NN.N, NN.N]

It reads results/<prefix>_s*/ and intersects the problem sets across seeds, so a partially finished seed lowers n rather than silently skewing the mean. Check that n=200.

---

Reading the output

results/
  _driverlogs/eval_mymodel_s1.log        driver log: one line per finished problem
  eval_mymodel_s1/
    summary.json                         totals + reason histogram + every per-problem record
    argv/codenet_argv_001.c/
      status.json                        the verdict for this problem
      translate.log                      full stdout/stderr of the sactor run  ← start here
      translated_code_unidiomatic/
        combined.rs                      the Rust the model produced
        functions/, clippy_stat.json
      llm_stat_unidiomatic.json          attempts / token counts
      unidiomatic_failure_info.json      per-function status report
      logs/*.jsonl                       prompt / response trace
      config.json                        the config as resolved for this run
    scanf/codenet_scanf_001.c/ ...

unidiomatic_failure_info.json is not a failure marker — it is written for successful problems too (with "status": "success" inside). Judge pass/fail by status.json only.

status.json is six fields:

{"mode": "argv", "file": "codenet_argv_001.c", "returncode": 0,
 "status": "success", "reason": "success", "time_sec": 30.7}

reason classifies what happened, which is where the useful signal lives:

reason	what it means
success	compiled and reproduced the reference output
max_attempts_unidiomatic	never produced a passing translation within max_translation_attempts — the normal way a weaker model fails
max_attempts_idiomatic	same, in the idiomatic stage (not used by the shipped configs)
timeout	the harness killed the whole process tree after --timeout
crash	engine traceback — most often the model server was unreachable
no_test_task	Invalid test command — fix_paths.sh was not run
struct_not_found, circular_deps	static-analysis dead ends on that C source
other	unclassified; read translate.log

A quick histogram of a finished seed:

python3 -c "import json;print(json.load(open('results/eval_mymodel_s1/summary.json'))['reasons'])"

Re-running

Runs resume: a problem that already has a status.json is skipped, so re-running the same command only fills in gaps. To redo work, delete it first:

rm -rf results/eval_mymodel_s1/argv/codenet_argv_042.c    # one problem
rm -rf results/eval_mymodel_s1                            # a whole seed

To re-run a specific subset, list mode/file entries one per line and pass --only:

printf 'argv/codenet_argv_042.c\nscanf/codenet_scanf_007.c\n' > /tmp/redo.txt
python3 scripts/run_eval.py configs/native_prompt.toml results/eval_mymodel_s1 --only /tmp/redo.txt

Environment variables

var	used by	meaning
SACTOR_HOME	run_eval.py	engine install holding .venv/bin/sactor; also the cwd for each translate call. Default <repo>/engine
SERVE_VENV	launch_model.sh	sglang serving venv (required)
CUDA_DIR	launch_model.sh	toolkit to borrow ptxas from; default /usr/local/cuda-13.0, used only if present. Set empty to skip
CFG, TAG	run_5seed.sh	config file and results-name prefix (both required)
PORT, SEEDS, WORKERS	run_5seed.sh	defaults 30878, 1 2 3 4 5, 4
SKIP_HEALTH=1	run_5seed.sh	skip the server health wait (API-hosted models)
BENCH_DIR	run_5seed.sh	benchmark root; defaults to the script's own directory
OPENROUTER_API_KEY	API config	read as os.environ/OPENROUTER_API_KEY

run_eval.py --help lists its own flags (--root, --sactor-home, --workers, --timeout, --modes, --only, --limit).

Operational rules

• Keep total concurrent workers ≤ ~15 across every job on the machine; watch cat /proc/loadavg. High parallelism spawns many rustc processes and can orphan looping test binaries → CPU overload → false timeouts that silently corrupt results. run_eval.py caps rustc with CARGO_BUILD_JOBS=1 and kills the whole process group on timeout, but still watch the load.
• Always average 5 seeds. One run is worth ±2–3 points; a single run is not a result.
• Never compare across settings. test_pass_threshold, CoT, thinking mode and max_tokens each move the number by several points. The shipped configs pin them (threshold 1, CoT off, thinking off, max_tokens=1536); change one and you can only compare within your own change.
• A config does not name a model. generic_prompt.toml and native_prompt.toml differ only in prompt mode; what you are measuring is decided by whichever checkpoint is served on the port. Keep the config fixed and swap the server.
• Prompt mode matters. sft_native_prompt=false is the generic prompt; true is the SFT-native prompt that instruction-tuned checkpoints expect. Using the wrong one is worth many points.
• run_eval.py sets CARGO_NET_OFFLINE=true, since compute nodes often have no network.

Troubleshooting

symptom	cause	fix
every problem reason=no_test_task	fix_paths.sh never ran	run it
OSError: Missing requirements	crown / rustup / c2rust / gcc not all on PATH	setup step 3
FileNotFoundError: 'sactor'	SACTOR_HOME wrong, or engine/.venv not built	setup steps 2 and 4
every problem reason=crash, log says Connection error	nothing serving on the port in the config's api_base	start the server, check /health
server starts, then dies on the first request	Triton's bundled ptxas does not support your GPU	use launch_model.sh; check its ptxas= banner and SETUP.md §3
many reason=timeout	CPU overload from too many workers	lower --workers, check /proc/loadavg
n= below 200 in the aggregate	a seed did not finish	re-run that seed; it resumes
all reason=max_attempts_unidiomatic with garbage in combined.rs	prompt mode mismatch, or a checkpoint that degenerates at this temperature	try the other config; check translated_code_unidiomatic/combined.rs

Credits & licenses

• Translation engine — engine/ is SACTOR by Tianyang Zhou et al. (paper), Apache-2.0, vendored as source with local modifications for this benchmark (the argv/stdin I/O-prompt handling, feedback and timeout knobs). Upstream license retained at engine/LICENSE; see NOTICE.
• Dataset — derived in part from IBM Project CodeNet (CDLA-Permissive-2.0). See CodeNet/README.md for provenance and terms.
• This repository — Apache-2.0, see LICENSE.