Curriculum reinforcement learning for Minecraft combat — from zombies to the Ender Dragon.
MineCombat is a research project built on MineStudio and VPT. The agent controls the native keyboard/mouse action space — camera, movement, attack, jump, sprint — from 128×128 RGB frames. The long-term goal is an end-to-end policy that can fight the Ender Dragon; the first milestone is a reproducible zombie arena.
M0 — environment bring-up, M1 — zombie arena, and M2 — baselines are
complete. M3 — first PPO is in progress: the 2M flat PPO checkpoint reaches
90% on easy front, 64% at ±30°, and 62% on the combined distance + angle task
(3×50). Full random spawn fell to 10%, so the stage is now split into
Stage-1a visible combat (passed) and Stage-1b 360° search (open).
HH1/HH2 passed engineering validation on an RTX 4090D; behavioral work stays
paused until the team accepts the split gate.
Key results:
- M2 baselines: random 0% win / 96% death; zero-shot VPT 2% win / 28% death.
- M3 PPO smoke: 10k steps, 20 updates, 6.3 GiB peak GPU memory.
- Dense damage: first nonzero damage ratio (0.0090) at 10k.
- 100k flat PPO: 0% win, 16% death, damage ratio 0.0313; weak signal, not yet a usable policy.
- R1 reward ablation: stronger aggression reward did not help; deaths rose, damage fell.
- Easy front-spawn curriculum: first kill, but only 2/50 episodes dealt damage; attack behavior is still unreliable.
- Stationary-target curriculum: 0 kills; action diagnosis shows the policy presses attack in <1% of steps. Attack discovery is the bottleneck.
- Attack-shaped stationary run: 7/50 episodes dealt damage (up from 0/50), but still 0 kills.
- Phase 1 probes: inventory masking produced the first repeatable kills; 8% win, 0 deaths on the stationary task.
- 100k mask warm-start: 22% win on stationary, 30% win against a front AI zombie; random spawn still 0%.
- 100k easy-AI warm-start: still 30% win but 58% deaths; the easy curriculum plateaued and defense probes are next.
- Defense probes: far-spawn training reached 52% win / 40% deaths on the easy task; ±30° spawn angles are now the bottleneck, not distance.
- Angle curriculum: 100k on ±30° only reached 16%; a joint mixed-distance + angle run is next.
- Mixed + angle curriculum: joint 100k reached 34% at ±30° and 48% on mixed distance; widening to ±60° raised easy to 58% but collapsed the ±60° target to 8%. Camera entropy did not help; this is an off-screen search problem, so the gate definition is the next decision.
- Evaluation variance: the same checkpoint and seeds produced 10–24% on the combined task because mob AI is not reset between episodes; 3×50-seed runs are now required for claims.
- Stage-1 re-baseline: 3×50 means are 16% (10–22%) on the combined task and 10% (8–14%) on random spawn; the gate is not passed.
- 1M flat PPO: 72% easy, 52% at ±30°, 40.7% combined (range 32–52%), 22.7% random (range 20–26%); case B, a 1M continuation is running.
- 2M continuation: 90% easy, 64% ±30°, 62% combined (58–62%), 10% random; visible-combat gate passes, 360° search regresses.
- Face-curriculum probe: 100k face-assisted training lifts
random_facefrom 36% to 52%, but unassisted random stays at 10%; search must be handled explicitly. - Scan-at-reset probe: a fixed 360° reset scan produced 0% wins and 54% timeouts; rejected. Search needs a cue/observation or an auto-face/split-gate decision.
- Search dataset v1: 560 real RGB frames (255 face, 305 random), quality-checked and delivered to the shared directory for YOLO annotation.
- Closed-loop search acceptance: real camera calibration (
yaw_sign=1) and 9-step VisualSearchPolicy integration both passed on the 2M checkpoint. - Mob Radar zero-shot detector: third-party YOLO26 Nano loads and integrates, but only 56.4% frame recall at
imgsz=320(0% false positives); case B, fine-tuning needed. - HH1/HH2 4090D validation: engineering chain validated on a 24 GB GPU; no behavioral-effectiveness claim.
- 1M runbook and 4090D handoff: exact commands and preflight for the next host.
Terminology: see docs/glossary.md for the M, RQ, and HH naming rules. The single control-plus-extension roadmap is in docs/unified-roadmap.md. Shared training preflight is in docs/training-common.md.
The first baseline and PPO runs are tracked in Issues.
Most Minecraft agents are evaluated on exploration or item-collection tasks. Combat is a useful next step because it requires:
- visual target detection and camera control,
- approach and spacing decisions,
- timing of attacks and retreats,
- risk management under partial observability,
- transfer from one enemy type to another.
The project studies a curriculum from a single zombie to mixed enemies and, eventually, the Ender Dragon. Each stage is a self-contained RL experiment with measurable win rate, damage taken, and time-to-kill.
| Stage | Enemy | Skills | Status |
|---|---|---|---|
| 0 | Environment | Stable reset, rendering, VPT inference, logging | Complete |
| 1 | Zombie | Approach, aim, attack, avoid damage | Arena + baselines complete; PPO in progress |
| 2 | Skeleton | Projectile avoidance, distance control | Planned |
| 3 | Creeper + mixed | Explosion risk, multiple enemies, terrain | Planned |
| 4 | Ender Dragon | Long-horizon tactics, high-level harness | Planned |
- Policy: VPT 1x foundation model, fine-tuned with PPO.
- Observation: 128×128 RGB plus the previous action; HP is logged for reward and evaluation but is not a policy input in stage 1.
- Action space: MineStudio
agentaction space (native camera and button controls). - Reward: dense combat reward combining damage dealt, damage taken, kill bonus, death penalty, and a small time penalty.
- Evaluation: fixed-seed episodes with win rate, HP lost, time-to-kill, damage ratio, and stuck rate.
The initial reward is:
r_t = 0.01 * Δdamage_dealt # damage_dealt is in tenths of a HP
- 0.2 * Δhp_lost
- 0.001
+ 10 * kill
- 5 * death
Weights are starting points and will be ablated. See docs/rewards.md.
configs/ Environment, PPO, and evaluation configurations
src/minecombat/ Environment wrappers, callbacks, training and evaluation code
scripts/ Setup, smoke test, baseline evaluation, and training entry points
docs/ Research plan, environment notes, reward design, roadmap
experiments/ Run logs and reproducibility notes (large artifacts are ignored)
tests/ Fast unit tests for reward math, metrics, and configs
The project requires Python 3.10, JDK 8, and a working OpenGL renderer. See docs/environment.md.
git clone git@github.com:MAS-W-415/MineCombat.git
cd MineCombat
bash scripts/setup.sh
source .venv/bin/activate
# Random-action smoke test
python scripts/smoke_test.py --config configs/env/zombie_stage1.yaml --steps 100
# Baseline evaluation
python scripts/eval.py --policy random --episodes 5For VPT inference, download the 1x checkpoint first:
python - <<'PY'
from minestudio.models import VPTPolicy
VPTPolicy.from_pretrained("CraftJarvis/MineStudio_VPT.foundation_model_1x")
PYThen run:
python scripts/smoke_test.py --policy vpt --steps 600The local RTX 4060 Laptop has 8 GB VRAM, so the smoke profile uses one environment. The base config is sized for a 24 GB GPU with 8 environments, batch size 4, context length 64, and VPT 1x.
# Local pipeline smoke (one environment)
bash scripts/start_ray.sh
bash scripts/train_zombie.sh --smoke
# 1M-step run on a 24 GB GPU / 64 GB RAM host
bash scripts/train_zombie.sh --num-iterations 20 --num-envs 8 --batch-size 4The official MineStudio configuration assumes multiple GPUs. Use
configs/ppo/zombie_vpt1x.yaml for the single-GPU profile and see
docs/1m-run.md for the full runbook.
- RQ1: PPO from VPT 1x improves zombie win rate and reduces damage taken versus zero-shot VPT.
- RQ2: A zombie-pretrained policy transfers to skeletons more sample-efficiently than direct VPT fine-tuning.
- RQ3: A high-level harness improves long-horizon boss performance once the flat policy is strong.
The flat VPT + PPO policy is the control condition. The harness is intentionally deferred until the zombie stage is stable, so that any gain can be attributed to the hierarchy rather than to a stronger base policy.
| Milestone | Goal | Acceptance |
|---|---|---|
| M0 | Environment bring-up | Stable simulator, VPT 1x inference, FPS/VRAM recorded |
| M1 | Zombie arena | 20 reproducible resets, random spawn and yaw |
| M2 | Baseline and metrics | 50-seed zero-shot evaluation, metrics report |
| M3 | First PPO run | 1M steps, learning curve, stage-1 gate |
| M4 | Curriculum | Skeleton extension only after M3 passes |
See docs/roadmap.md for details.
This project builds on:
- MineStudio for the simulator, data, models, and online RL infrastructure.
- MineRL and Project Malmo for the underlying Minecraft environment.
- OpenAI VPT for the pretrained foundation model.
MIT. See LICENSE.