I made a superhuman Generals.io agent with self-play RL [P]

Background

Hi everyone, I trained a self-play RL agent for Generals.io that reached superhuman-level and ranked #1 on the human 1v1 leaderboard. It began as my master's thesis where the goal was to beat a prior algorithm based agent.

We succeeded using behavior cloning, RL fine-tuning and reward shaping, but the agent was still consistently beaten by the top players. So I gave it a round two and fixed the largest bottlenecks:

• Reimplemented the whole pipeline in JAX (from NumPy/Torch)
• Used Vision Transformer instead of the CNN

Both are a result of the same idea: to invest in scaling rather than human priors and ad-hoc patches.

Guide and Resources

The blog is written as a guide for anyone building something similar - the dead ends, the decisions, and the intuitions and tricks I picked up along the way. It's all open source, including the fast JAX simulator - handy on its own if you want an imperfect-information RTS env to play with.

• Guide: https://kam.mff.cuni.cz/~straka/blog/generals.html
• Simulator (JAX): https://github.com/strakam/generals-bots
• Agent: https://github.com/strakam/AverageJoe

I hope you find the blogpost entertaining! Feedback and questions welcome 🤗.