RL Minesweeper Training Notes

Architecture Evolution

Old: Global Average Pooling (GAP)

The original architecture used 3 conv layers followed by AdaptiveAvgPool2d(1) which collapsed spatial dimensions to a single point, then Linear layers for actor/critic. This discarded all spatial information — the agent couldn’t reason about individual cell positions.

Current: Fully Convolutional (Spatial Policy) with Variable Size Support

Preserves spatial dimensions (HxW) through the entire network for per-cell decisions:

Encoder: 5× 3x3 conv layers (maintains HxW spatial dims)
Input: 5 channels (was 3; extended for global features)
Receptive field: ~11×11 (covers beginner 9×9 board completely; weak for large boards)
Actor head: 1×1 conv to 1 channel → flatten to n_actions (row-major: index = y*W + x)
Critic head: AdaptiveAvgPool2d → Linear to scalar value
Size-agnostic: Actor is fully convolutional, works on any H×W at inference time

Why this matters: Minesweeper needs per-cell reasoning. “This 1 has exactly one closed neighbor in the top-right corner” requires spatial awareness. The old GAP architecture couldn’t distinguish between board positions.

Checkpoint compatibility: Old GAP checkpoints are incompatible with FCN spatial policy. Use transfer learning to load beginner FCN weights into variable-size models.

Observation Channels (5 channels, was 3)

The observation tensor has shape (5, H, W):

Channel 0: Revealed mask (0 or 1)
Channel 1: Flagged mask (0 or 1)
Channel 2: Adjacent mine count normalized (0-8 → 0/9..8/9, unrevealed → 1.0)
Channel 3: Remaining mine density (remaining_mines / remaining_hidden) — NEW
Channel 4: Fraction hidden (remaining_hidden / total_cells) — NEW

Why global channels matter: On large boards (16×16, expert 30×16), local 11×11 receptive field doesn’t see the whole board. Global mine density helps the agent estimate risk when it can’t count all remaining mines visually.

JS compatibility: inference.js must encode the exact same 5 channels. See encodeState() function.

Evolution of Reward Functions

V1: Sparse Rewards (Original - Failed)

The original reward structure only provided sparse terminal rewards:

Win: +1.0
Mine hit: -1.0
All other moves: 0.0

This gave the agent no learning signal for making progress. The agent only learned from wins/losses, making training extremely slow and inefficient.

V2: Shaped Rewards (Area-Based - Failed)

Attempted fix with incremental progress rewards based on area revealed.

Problem: Over-rewards zero-floods

reward = newly_revealed_cells / safe_total  # Area-based

What happened: Agent learned to farm area instead of solving. Plateaued at ~60 cells revealed, 0 wins. Opening a zero flood gives massive reward even though zeros provide no information for solving the puzzle.

V3: Info Rewards (Information-Based - Current)

Solution: Value information over area. Zeros contribute 0 (they tell you nothing about mine locations). Numbered cells contribute their adjacency count (a 4 is more informative than a 1).

# On each safe click:
info_sum = sum(adj_count for each newly_revealed_cell)
reward = info_sum / (8.0 * safe_total)

# Mine hit:
reward = -1.0

# Terminal win bonus (increased, finishing is the objective):
reward += 2.0

Why Info Rewards Work Better

Values information: High-number cells (4, 5, 6) > low-number cells (1) > zeros (0)
Zero-flood fix: A flood of zeros only pays via the numbered ring it exposes
Encourages solving: Agent must click strategically to gain information, not farm area
Strong terminal bonus: +2.0 win bonus ensures finishing the board is the actual objective

Curriculum Learning

Start training with easier boards (fewer mines), gradually increase difficulty as the agent improves:

--curriculum-start-mines 5      # Start with 5 mines (easier than beginner's 10)
--curriculum-target-mines 10    # Goal: full beginner difficulty
--curriculum-win-threshold 0.7  # Increase mines when wr ≥ 0.7
--curriculum-window 100         # Over last 100 episodes

How it works:

Train on 5-mine beginner boards
When win rate ≥ 0.7 (over last 100 episodes), increase to 6 mines
Repeat until reaching target (10 mines)
Continue training at full difficulty

Why curriculum helps: Agent learns basic Minesweeper mechanics on easier boards before tackling full difficulty. Without curriculum, random exploration on 10-mine boards rarely succeeds.

Prototype Results (Cloud VM, CPU, Short Runs)

Info vs Shaped Reward Behavior (5k steps, seed 42)

Example step: 35-cell zero-flood revealing 23 info-sum:

Shaped: reward = 0.4930 (35/71 cells, huge for one click!)
Info: reward = 0.0405 (23/(8×71), modest despite flood size)

Key insight: Info reward doesn’t over-reward zero-floods. It only rewards the information content (numbered cells), forcing the agent to play strategically.

Info Reward Training Metrics (5k steps)

Win rate: 0.000 (still learning, very short run)
Mean cells revealed: ~48-50
Mean reward: -0.20 to -0.21 (normalized differently than shaped)
Mean info: 0.0 (agent dying before revealing useful info)

Curriculum Test (3k steps, 4→6 mines)

Started with 4 mines
Win rate varied 0.10-0.30 (learning)
Successfully tracked “mines=4” in logs
Info metric shows 0.1-3.8 (information being gathered)

Transfer Learning: Variable Board Sizes

Why Transfer Learning?

Training from scratch on large boards is slow and sample-inefficient. The beginner 9×9 checkpoint already learned:

Basic Minesweeper tactics (1s, 2s, corner reasoning)
Pattern recognition (3 conv layers worth ~100k params)
Opening strategy (flood zeros first)

Transfer learning loads the 9×9 weights into a new model, then trains on progressively larger boards via board size curriculum.

How It Works

Old checkpoint: 3 input channels (revealed, flagged, adj)
New model: 5 input channels (+ mine_density, frac_hidden)
Transfer: Copy the 3 old channels to first 3 weights of new model; initialize extra 2 channels with small noise (~0.01 std)
Result: Agent starts with 9×9 tactics, learns global features during size curriculum

Important: FCN spatial architecture transfers across board sizes naturally. Actor head is 1×1 conv (no fixed size). Critic Linear may need retraining (different feature pooling), but that’s cheap.

Board Size Curriculum

Start on beginner 9×9, grow to intermediate 16×16 → expert 30×16 as the agent improves.

Default curriculum steps:

9×9 with 10 mines (beginner)
12×12 with 20 mines (transition)
16×16 with 40 mines (intermediate)
30×16 with 99 mines (expert)

Advance when rolling win rate ≥ 0.7 over last 100 episodes. Agent must master each size before scaling up.

Why curriculum helps:

Agent learns basic tactics on small boards (fast episodes)
Gradually exposes it to longer-horizon planning (more cells)
Mine density stays balanced (not too easy/hard at each step)
Transfer from 9×9 checkpoint gives it a head start

Production Training Command (Apple Silicon)

Recommended: Variable Size with Transfer Learning

cd rl-minesweeper
python train.py --preset beginner --total-timesteps 1000000 --seed 42 --device auto \
  --save-dir checkpoints_spatial_size_curriculum \
  --load-checkpoint checkpoints_spatial_info_curriculum/best.pt \
  --curriculum-board \
  --curriculum-win-threshold 0.7

Parameters:

--device auto: Auto-detects MPS (Apple Silicon GPU), CUDA, or CPU
--load-checkpoint: Transfer from beginner 9×9 FCN checkpoint
--curriculum-board: Enable board size curriculum (9→12→16→30x16)
--curriculum-win-threshold 0.7: Advance when win rate ≥ 0.7 (over 100 episodes)
--save-dir checkpoints_spatial_size_curriculum: NEW directory (won’t overwrite beginner)
--total-timesteps 1000000: 1M steps should reach intermediate; may need more for expert

Metrics tracked:

wr: Win rate (rolling window)
cells: Mean cells revealed
mean_r: Mean reward per step
info: Mean information sum
size: Current board size (e.g. 9×9, 12×12)
mines: Current mine count

Checkpoints:

best.pt: Highest win rate (may be on intermediate size, not expert)
last.pt: Final model (likely trained on expert size)

Custom Curriculum Steps

python train.py --preset beginner --total-timesteps 2000000 --seed 42 --device auto \
  --save-dir checkpoints_spatial_size_custom \
  --load-checkpoint checkpoints_spatial_info_curriculum/best.pt \
  --curriculum-board \
  --curriculum-board-steps "9x9x10,16x16x40,30x16x99" \
  --curriculum-win-threshold 0.65

Skip 12×12 transition; jump directly to intermediate then expert.

Old: Mine Curriculum (Fixed Size)

If you want to train on beginner 9×9 with just mine curriculum (no size changes):

python train.py --preset beginner --total-timesteps 1000000 --seed 42 --device auto \
  --save-dir checkpoints_info_curriculum \
  --curriculum-start-mines 5 --curriculum-target-mines 10 --curriculum-win-threshold 0.7

This is the old approach (still works for 9×9). Board size curriculum supersedes this for variable-size training.

After Training: Export to ONNX

Variable Size Model (Dynamic Axes)

python export_onnx.py --checkpoint checkpoints_spatial_size_curriculum/best.pt --preset beginner

Exports with dynamic spatial axes (default). Model can run on any board size at inference time. Writes to ../assets/rl-minesweeper/policy.onnx.

Important: JS inference.js must pass 5-channel observations and the correct board size. See encodeState() and getRLAction().

Fixed Size Model (e.g. beginner only)

python export_onnx.py --checkpoint checkpoints_spatial_info_curriculum/best.pt --preset beginner --no-dynamic-spatial

Fixed to 9×9 input. Browser will reject other sizes with clear error message.

Comparison Modes (Optional)

To verify old modes are worse:

# Area-based shaped reward (over-rewards zero-floods)
python train.py --preset beginner --total-timesteps 1000000 --reward-mode shaped \
  --save-dir checkpoints_shaped

# Sparse terminal-only (very slow)
python train.py --preset beginner --total-timesteps 1000000 --reward-mode sparse \
  --save-dir checkpoints_sparse

Expected outcomes:

shaped: Plateaus at ~60 cells, 0 wins (area farming)
sparse: Very slow learning, low win rates
info + curriculum: Best performance, learns to solve

Limitations and Future Work

Receptive Field

Current RF is ~11×11 (5 conv layers, kernel 3, padding 1). This covers beginner 9×9 completely but only a local window on expert 30×16. Agent can’t “see” the whole board.

Workarounds implemented:

Global observation channels (mine density, frac hidden) broadcast global info
Transfer learning means agent starts with good local tactics

Future improvements:

Add 1-2 dilated conv layers (RF ~21×21 with minimal param increase)
Attention mechanism for global context
Recurrent policy (LSTM) to remember board state across steps

Huge Boards

Architecture supports arbitrary sizes (fully convolutional), but agent trained on ≤30×16 may play poorly on 50×50 or 100×100:

Tactics don’t generalize beyond training distribution
RF coverage becomes tiny fraction of board
Mine density and opening strategy differ

Path forward: Train on 50×50+ with size curriculum, or use recurrent/transformer policy.

Browser Compatibility

inference.js must match Python env.py observation encoding exactly. Mismatches (e.g. wrong channel order, missing global features) break inference silently (wrong actions, not crash).

Testing: Compare Python env._obs() output to JS encodeState() output for same board state (print first 20 values of each channel).

James Berkeley Larsen