gridworld-learning-lab

Minimal CLI-only Rust project: a 2D Gridworld + tabular Q-learning trainer with reproducible metrics and deterministic GIF demos.

Research prototype focused on evaluation and reproducibility.

Evaluation mindset

• I measure success rate and steps-to-goal.
• I can modify reward shaping and verify impact on learning curves.
• I provide reproducible runs via fixed seeds and saved configs.

---

Demo (before / after)

Before training (random policy)

After training (learned policy)

---

What this project demonstrates

• Clean Rust 2024 CLI architecture (small modules, clear types, no engine dependencies).
• A minimal RL loop (environment, reward, terminal conditions, Q-table).
• Reproducibility (seeded RNG, persisted config, deterministic rendering).
• Practical outputs:
  • metrics.csv for analysis
  • qtable.bin for policy reuse
  • GIF rollouts for visual verification

---

Environment

• Grid size: width × height (default 24×24, configurable via CLI)
• Goal zone: rightmost goal_zone_width columns (default 4)
• Start state: random position in the left half (x ∈ [0 .. width/2))
• Actions: Up, Down, Left, Right, Stay
• Episode ends when:
  • agent enters goal zone ⇒ success
  • max_steps reached ⇒ failure
• Optional obstacles (OFF by default):
  • random walls with fixed seed (--world-seed)
  • layout is re-sampled until at least one reachable start exists
  • goal zone is kept wall-free for stability

Rewards

• +1.0 when entering goal zone (terminal)
• -0.01 per step (time penalty)
• Optional shaping (OFF by default):
  • +0.02 if x increases (closer to goal)
  • -0.02 if x decreases

---

Q-learning

• State: (x, y) mapped to state_id = y * width + x
• Q-table: Vec<f32> of length num_states * num_actions
• Update:
  • Q[s,a] = Q[s,a] + alpha * (target - Q[s,a])
  • target = r if terminal else r + gamma * max_a' Q[s',a']
• Exploration: epsilon-greedy with linear decay
  • start 1.0 → end 0.05
  • decays over first 70% of episodes

---

Quick start

1) Train

cargo run --release -- train \
--episodes 50000 \
--max-steps 96 \
--seed 1 \
--out runs/latest

Outputs in runs/latest/:

• metrics.csv - per-episode metrics
• qtable.bin - learned Q-table (bincode)
• config.json - full run configuration for reproducibility

2) Generate "before" GIF (random policy)

mkdir -p assets

cargo run --release -- demo \
--random \
--seed 2 \
--gif assets/demo_before.gif

3) Generate "after" GIF (learned policy)

cargo run --release -- demo \
--policy runs/latest/qtable.bin \
--seed 2 \
--gif assets/demo_after.gif

Notes:

• demo --policy auto-loads runs/latest/config.json if present next to qtable.bin.
• GIF rendering is deterministic given seeds + config.

---

Metrics

Training writes runs/<name>/metrics.csv with columns:

• episode - episode index (1-based)
• success - 1 if agent reached the goal zone, else 0
• steps - steps taken until terminal or max_steps
• epsilon - exploration rate used in that episode

The trainer logs progress every --log-every episodes:

• rolling success rate over the last --window episodes
• avg steps to success in that window (if any)
• current epsilon

Example log line:

ep   500/5000  success_rate(last 200)= 84.50%  avg_steps_to_success=  31.2  eps=0.336

---

Reproducibility

This repo is designed so you can reproduce results later:

• Training determinism:
  • --seed controls exploration + episode starts
  • saved config.json records environment and learning parameters
• Obstacle determinism:
  • --world-seed controls wall layout sampling
• Demo determinism:
  • demo --policy runs/latest/qtable.bin reuses saved config automatically

---

CLI reference

Help:

cargo run --release -- --help
cargo run --release -- train --help
cargo run --release -- demo --help

Useful flags:

• Environment:
  • --width, --height, --goal-zone-width, --max-steps
  • --shaping
  • --obstacles, --wall-density, --world-seed
• Training:
  • --episodes, --alpha, --gamma
  • --log-every, --window
• Rendering:
  • --cell-px, --gif-delay-cs

---

Development

Run formatting + lint + tests:

cargo fmt
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-targets --all-features

CI runs the same checks on every push/PR.

---

Repository layout

Cargo.toml
src/
  main.rs     # CLI entry (train/demo)
  env.rs      # gridworld dynamics + optional obstacles
  qlearn.rs   # Q-table + trainer
  render.rs   # deterministic GIF rendering
  metrics.rs  # CSV writer
  types.rs    # config + enums
assets/       # committed demo gifs for README
runs/         # local outputs (gitignored)
.github/
  workflows/ci.yml

---

License

MIT - see LICENSE.