RL Minesweeper (PPO + CNN)

Train a PPO agent with a CNN state encoder for Minesweeper, then export the policy to ONNX for in-browser inference on the website.

Observation format (Python and JS must match)

• Shape: (3, H, W) in PyTorch; [1, 3, H, W] for ONNX (batch first).
• Channels:
  • 0: revealed mask (0 or 1).
  • 1: flagged mask (0 or 1).
  • 2: adjacent mine count for revealed cells, normalized to [0, 1] as adj/9; unrevealed cells use value 1.0 (sentinel 9/9).
• All values float32 in [0, 1].

Action format

• Action space: flat index 0 .. W*H - 1 (row-major: index = y * W + x).
• Meaning: reveal the cell at (index % W, index // W).
• Invalid actions (already revealed or flagged) are masked out during training; the JS side should only request an action when at least one valid cell exists, and can mask or re-sample if needed.

Reward modes

• info (default, recommended): Information-based reward. Values numbered cells by their adjacency count (4 > 1 > 0). Zeros contribute 0 — a flood of zeros only pays via the numbered ring it exposes. Formula: sum(adj_of_revealed_cells) / (8 * safe_total). Mine penalty: -1.0. Terminal win bonus: +2.0 (finishing is the objective, not farming info). Fixes the zero-flood farming problem.
• shaped: Area-based reward (newly_revealed_cells / safe_total). Over-rewards zero-floods — agent plateaus at ~60 cells, 0 wins, farming area instead of solving.
• sparse (old): Only terminal rewards (+1 win, -1 loss, 0 otherwise). Very slow to learn.

The shaped reward over-rewards zero-floods. The info reward values information, not just area.

Setup

cd rl-minesweeper
pip install -r requirements.txt

Architecture

New (spatial policy): Fully convolutional architecture preserves spatial dimensions (HxW) for per-cell decisions. Essential for Minesweeper because the agent needs to reason about individual cells (“this 1 has one closed neighbor”).

• Encoder: 5× 3x3 conv layers (3→32→64→64→64→64 channels, maintains HxW)
• Receptive field: ~11×11 (covers beginner 9×9 board)
• Actor head: 1×1 conv to 1 channel → flatten to n_actions logits (row-major)
• Critic head: AdaptiveAvgPool2d → Linear to scalar value

Old (GAP architecture): Previous model used Global Average Pooling after conv stack, which discarded all spatial information. Old checkpoints (e.g. checkpoints/best.pt from shaped-reward run) are incompatible with the new architecture.

Train (Apple Silicon / MPS)

Train on Apple Silicon GPU with information-based rewards and curriculum learning:

python train.py --preset beginner --total-timesteps 1000000 --seed 42 --device auto --save-dir checkpoints \
  --curriculum-start-mines 5 --curriculum-target-mines 10 --curriculum-win-threshold 0.7

Flags:

• --device auto: automatically uses MPS (Apple Silicon GPU), CUDA (NVIDIA), or CPU
• --reward-mode info (default): information-based reward (values numbered cells, zeros contribute 0)
• --reward-mode shaped: area-based (over-rewards zero-floods, not recommended)
• --reward-mode sparse: terminal-only (very slow)
• --curriculum-start-mines 5: start with easier 5-mine boards
• --curriculum-target-mines 10: gradually increase to full beginner (10 mines)
• --curriculum-win-threshold 0.7: increase mines when win rate ≥ 0.7 over recent window
• --curriculum-window 100: episode window for win rate threshold (default 100)

Training logs show:

• wr: win rate (rolling window)
• cells: mean cells revealed
• mean_r: mean reward per step
• info: mean information sum (for info mode)
• mines: current mine count (for curriculum)

Export to ONNX

After training, export the policy for the web:

python export_onnx.py --checkpoint checkpoints/best.pt --preset beginner

This writes ../assets/rl-minesweeper/policy.onnx. The default preset is beginner (9×9); use --preset intermediate or --preset expert to export for other board sizes (you need a checkpoint trained for that size).

Presets

The website loads policy.onnx and runs inference in the browser with the same observation encoding.