Skip to content

About

Curriculum RL for Minecraft combat — from zombies to the Ender Dragon, built on MineStudio + VPT.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

101 Commits

Folders and files

Repository files navigation

MineCombat

Curriculum reinforcement learning for Minecraft combat — from zombies to the Ender Dragon.

MineCombat is a research project built on MineStudio and VPT. The agent controls the native keyboard/mouse action space — camera, movement, attack, jump, sprint — from 128×128 RGB frames. The long-term goal is an end-to-end policy that can fight the Ender Dragon; the first milestone is a reproducible zombie arena.

中文说明

Status

M0 — environment bring-up, M1 — zombie arena, and M2 — baselines are complete. M3 — first PPO is in progress: the 2M flat PPO checkpoint reaches 90% on easy front, 64% at ±30°, and 62% on the combined distance + angle task (3×50). Full random spawn fell to 10%, so the stage is now split into Stage-1a visible combat (passed) and Stage-1b 360° search (open). HH1/HH2 passed engineering validation on an RTX 4090D; behavioral work stays paused until the team accepts the split gate.

Key results:

  • M2 baselines: random 0% win / 96% death; zero-shot VPT 2% win / 28% death.
  • M3 PPO smoke: 10k steps, 20 updates, 6.3 GiB peak GPU memory.
  • Dense damage: first nonzero damage ratio (0.0090) at 10k.
  • 100k flat PPO: 0% win, 16% death, damage ratio 0.0313; weak signal, not yet a usable policy.
  • R1 reward ablation: stronger aggression reward did not help; deaths rose, damage fell.
  • Easy front-spawn curriculum: first kill, but only 2/50 episodes dealt damage; attack behavior is still unreliable.
  • Stationary-target curriculum: 0 kills; action diagnosis shows the policy presses attack in <1% of steps. Attack discovery is the bottleneck.
  • Attack-shaped stationary run: 7/50 episodes dealt damage (up from 0/50), but still 0 kills.
  • Phase 1 probes: inventory masking produced the first repeatable kills; 8% win, 0 deaths on the stationary task.
  • 100k mask warm-start: 22% win on stationary, 30% win against a front AI zombie; random spawn still 0%.
  • 100k easy-AI warm-start: still 30% win but 58% deaths; the easy curriculum plateaued and defense probes are next.
  • Defense probes: far-spawn training reached 52% win / 40% deaths on the easy task; ±30° spawn angles are now the bottleneck, not distance.
  • Angle curriculum: 100k on ±30° only reached 16%; a joint mixed-distance + angle run is next.
  • Mixed + angle curriculum: joint 100k reached 34% at ±30° and 48% on mixed distance; widening to ±60° raised easy to 58% but collapsed the ±60° target to 8%. Camera entropy did not help; this is an off-screen search problem, so the gate definition is the next decision.
  • Evaluation variance: the same checkpoint and seeds produced 10–24% on the combined task because mob AI is not reset between episodes; 3×50-seed runs are now required for claims.
  • Stage-1 re-baseline: 3×50 means are 16% (10–22%) on the combined task and 10% (8–14%) on random spawn; the gate is not passed.
  • 1M flat PPO: 72% easy, 52% at ±30°, 40.7% combined (range 32–52%), 22.7% random (range 20–26%); case B, a 1M continuation is running.
  • 2M continuation: 90% easy, 64% ±30°, 62% combined (58–62%), 10% random; visible-combat gate passes, 360° search regresses.
  • Face-curriculum probe: 100k face-assisted training lifts random_face from 36% to 52%, but unassisted random stays at 10%; search must be handled explicitly.
  • Scan-at-reset probe: a fixed 360° reset scan produced 0% wins and 54% timeouts; rejected. Search needs a cue/observation or an auto-face/split-gate decision.
  • Search dataset v1: 560 real RGB frames (255 face, 305 random), quality-checked and delivered to the shared directory for YOLO annotation.
  • Closed-loop search acceptance: real camera calibration (yaw_sign=1) and 9-step VisualSearchPolicy integration both passed on the 2M checkpoint.
  • Mob Radar zero-shot detector: third-party YOLO26 Nano loads and integrates, but only 56.4% frame recall at imgsz=320 (0% false positives); case B, fine-tuning needed.
  • HH1/HH2 4090D validation: engineering chain validated on a 24 GB GPU; no behavioral-effectiveness claim.
  • 1M runbook and 4090D handoff: exact commands and preflight for the next host.

Terminology: see docs/glossary.md for the M, RQ, and HH naming rules. The single control-plus-extension roadmap is in docs/unified-roadmap.md. Shared training preflight is in docs/training-common.md.

The first baseline and PPO runs are tracked in Issues.

Why this project

Most Minecraft agents are evaluated on exploration or item-collection tasks. Combat is a useful next step because it requires:

  • visual target detection and camera control,
  • approach and spacing decisions,
  • timing of attacks and retreats,
  • risk management under partial observability,
  • transfer from one enemy type to another.

The project studies a curriculum from a single zombie to mixed enemies and, eventually, the Ender Dragon. Each stage is a self-contained RL experiment with measurable win rate, damage taken, and time-to-kill.

Curriculum

Stage Enemy Skills Status
0 Environment Stable reset, rendering, VPT inference, logging Complete
1 Zombie Approach, aim, attack, avoid damage Arena + baselines complete; PPO in progress
2 Skeleton Projectile avoidance, distance control Planned
3 Creeper + mixed Explosion risk, multiple enemies, terrain Planned
4 Ender Dragon Long-horizon tactics, high-level harness Planned

Method

  • Policy: VPT 1x foundation model, fine-tuned with PPO.
  • Observation: 128×128 RGB plus the previous action; HP is logged for reward and evaluation but is not a policy input in stage 1.
  • Action space: MineStudio agent action space (native camera and button controls).
  • Reward: dense combat reward combining damage dealt, damage taken, kill bonus, death penalty, and a small time penalty.
  • Evaluation: fixed-seed episodes with win rate, HP lost, time-to-kill, damage ratio, and stuck rate.

The initial reward is:

r_t = 0.01 * Δdamage_dealt   # damage_dealt is in tenths of a HP
    - 0.2 * Δhp_lost
    - 0.001
    + 10 * kill
    - 5 * death

Weights are starting points and will be ablated. See docs/rewards.md.

Repository layout

configs/        Environment, PPO, and evaluation configurations
src/minecombat/ Environment wrappers, callbacks, training and evaluation code
scripts/        Setup, smoke test, baseline evaluation, and training entry points
docs/           Research plan, environment notes, reward design, roadmap
experiments/    Run logs and reproducibility notes (large artifacts are ignored)
tests/          Fast unit tests for reward math, metrics, and configs

Quickstart

The project requires Python 3.10, JDK 8, and a working OpenGL renderer. See docs/environment.md.

git clone git@github.com:MAS-W-415/MineCombat.git
cd MineCombat
bash scripts/setup.sh
source .venv/bin/activate

# Random-action smoke test
python scripts/smoke_test.py --config configs/env/zombie_stage1.yaml --steps 100

# Baseline evaluation
python scripts/eval.py --policy random --episodes 5

For VPT inference, download the 1x checkpoint first:

python - <<'PY'
from minestudio.models import VPTPolicy
VPTPolicy.from_pretrained("CraftJarvis/MineStudio_VPT.foundation_model_1x")
PY

Then run:

python scripts/smoke_test.py --policy vpt --steps 600

Training

The local RTX 4060 Laptop has 8 GB VRAM, so the smoke profile uses one environment. The base config is sized for a 24 GB GPU with 8 environments, batch size 4, context length 64, and VPT 1x.

# Local pipeline smoke (one environment)
bash scripts/start_ray.sh
bash scripts/train_zombie.sh --smoke

# 1M-step run on a 24 GB GPU / 64 GB RAM host
bash scripts/train_zombie.sh --num-iterations 20 --num-envs 8 --batch-size 4

The official MineStudio configuration assumes multiple GPUs. Use configs/ppo/zombie_vpt1x.yaml for the single-GPU profile and see docs/1m-run.md for the full runbook.

Research questions

  • RQ1: PPO from VPT 1x improves zombie win rate and reduces damage taken versus zero-shot VPT.
  • RQ2: A zombie-pretrained policy transfers to skeletons more sample-efficiently than direct VPT fine-tuning.
  • RQ3: A high-level harness improves long-horizon boss performance once the flat policy is strong.

The flat VPT + PPO policy is the control condition. The harness is intentionally deferred until the zombie stage is stable, so that any gain can be attributed to the hierarchy rather than to a stronger base policy.

Roadmap

Milestone Goal Acceptance
M0 Environment bring-up Stable simulator, VPT 1x inference, FPS/VRAM recorded
M1 Zombie arena 20 reproducible resets, random spawn and yaw
M2 Baseline and metrics 50-seed zero-shot evaluation, metrics report
M3 First PPO run 1M steps, learning curve, stage-1 gate
M4 Curriculum Skeleton extension only after M3 passes

See docs/roadmap.md for details.

Acknowledgements

This project builds on:

License

MIT. See LICENSE.

About

Curriculum RL for Minecraft combat — from zombies to the Ender Dragon, built on MineStudio + VPT.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages