Every month, Jane Street launches a new puzzle. This month, they unveiled something called “Arch Madness”, which is an absolutely beautiful little game. After showing their puzzle to my friends David P and James S, we spent many hours late into the night figuring it out. But these clever friends of mine didn’t stop there! David P went ahead and vibe-coded a little tool to help solve the puzzles.
I have always loved little daily puzzle games, and I saw the potential to turn David’s tool into a beautiful quick puzzle game. So, after an afternoon date with Cursor, I am launching my daily puzzle game, check it out here. Please let me know what you think! And if you like it, try to make puzzles of your own with the explorer tool. If you make a particularly good one and send me the json, it may just be featured!
]]>As promised two weeks ago, I’ve started my attempt to formulate Minesweeper in the language of reinforcement learning (RL). The mathematical backbone of RL is a Markov decision process (MDP), consisting of a state space $\mathcal{S}$, an action space $\mathcal{A}_s$ for each state $s \in \mathcal{S}$, transition probabilities $P_a(s,s’)$ for states $s,s’\in\mathcal{S}$ and action $a \in \mathcal{A}_s$, and the reward functions $R_a(s,s’)$. The goal of the MDP is to find the policy $\pi$ that determines the best action $a$ from any given state, i.e., \begin{equation} \pi^* = \arg\max_\pi \mathbb{E}_{s \sim \pi, a \sim \pi(s)} \left[ \sum_{t=0}^\infty \gamma^t R_a(s_t, a_t) \right] \end{equation} for some discount factor $\gamma \in [0,1]$.
Naïvely, the state of our Minesweeper game is the board configuration, an action would correspond to selecting/revealing cells, and the reward would be whether we have exploded or not. But things are going to get interesting… if our MDP knew the board configuration, the problem would be trivial! The optimal policy is to reveal all cells that aren’t mines. Somehow, we need to obscure the location of the mines from our MDP. Enter the partially observable Markov decision process (POMDP). Amazingly, in this sense the game of Minesweeper is more mathematically complicated than say Go or Chess where the players have perfect information. To reflect the partial observability of Minesweeper, we need to augment our MDP with an observation space $\mathcal{O}$ and base our policy and the corresponding actions on observations from $\mathcal{O}$. This simple distinction already rules out several RL algorithms, e.g. vanilla Q-learning.
To be continued…
As an aside, I love the way Andy Jones explains the policy gradient theorem.
First RL agent (no reward shaping). Model: PPO with CNN state encoder (3-channel obs: revealed, flagged, adj). Trained on Beginner 9×9, 500k steps (25:11, 331 step/s). Result: best=-0.141, mean_r=-0.182, wins=0. Exported to ONNX and wired into RLMS solver dropdown so you can watch it play—it’s bad on purpose, to motivate reward shaping. Next step: add light reward shaping (+0.02 per safe reveal) and retrain.
]]>Turns out, a new year’s resolution to blog every week is too ambitious. Maybe I can update that to once a month? Here goes a simple February blog post, coming back into the CompCath sandbox to see what I can put together.
I’m still on a bit of an unhealthy Minesweeper kick, so I channeled that energy this morning into a vibe-coded autosolver here. It is still very much a work in progress, and I couldn’t have done it without my loyal Cursor agent.
One of my primary motivations in having my own coded version of Minesweeper was my attempt to foray in the coming weeks into reinforcement learning. Back in the ACME Junior Core, we briefly experimented with RL in a Q-Learning lab, but I’d be lying if I said I fully internalized its beauty and simplicity.
Recently, I stumbled upon Rich Sutton’s Bitter Lesson, and it resonated with my intellect on a profound level. While I might go so far as to make this required reading for anyone living in the post-ChatGPT world, at the very least I hope the AI curious will take the time to give it a read. The TLDR: Rich is one of the pioneers of RL. The bitter lesson is that additional compute beats additional human thought. Instead of trying to simplify complex phenomena, to truly innovate in this day and age we need to just let the wonders of computational search and learning scale. That release–letting go of the relentless and futile search for understanding by the human mind and just embracing the inconceivable complexity–embodies the spirit of Computational Catharsis better than any other idea I’ve seen. Thanks Rich.
Maybe soon you will see my humble attempt at a reinforcement learning agent conquering the simple world of Minesweeper come live in your browser. Can it beat the best human solvers I could find? I’m skeptical.
With the arrival of the arbitrarily chosen day set aside by our society as the first of the year comes the ritualistic resolutions meant to change our behavior. One such resolution of mine was to revive Computational Catharsis as a sandbox for my thoughts. Here goes nothing.
Beyond that simple goal (coupled with a minimal habit of the James Clear variety) and a few others, I was thinking more abstractly about who I am, who I want to become, and what it takes to bridge the gap between the two. My mind kept gravitating towards the idea of a trajectory. The fundamental idea behind calculus is that dynamics of a system are best understood through a study of rates of change. If I want to predict or modify my own personal trajectory, perhaps I need to take a personal derivative of sorts: I propose that the derivative of who I am with respect to time is who I am becoming–the bridge to who I want to become. This state of being changes smoothly, hence the inevitable failure of overly ambitious goals.
I finished Charan Ranganath’s debut pop-science book, and was quite pleased with it. Beyond reinforcing my habit of regularly discounting my own recollections as “false memories”, Ranganath provided a medium for me to marvel at the remarkable computational abilities of our biological hardware. I found it similar in vibe and scientific rigor to Matthew Walker’s Why We Sleep, but maybe not as life changing for the simple reason that sleep studies inform behavior modification more naturally than memory studies. Embarrassingly, my biggest takeaway might be the brief introduction to the idea of cryptomnesia, something I’ve needed a word for since hearing the similarities between the openings of Leon Bridges’ Coming Home and Taylor Swift’s Lover.
Other curiosities I spent time with this past week:
]]>With November quickly fading away, I am thinking ahead to future iterations of the universe. Why not include a personal online blog/CV in such future iterations? Hence, the birth of Computational Catharsis.
]]>