Under-explored Game Genres for Reinforcement Learning Research
Go, Chess, DOTA and Starcraft 2 were all at various points in times held up as north stars for AI research.
Go and Chess famous for their long sequence calculations. Dota and Starcraft infamous for their ADD combination of 300 APM and macro strategy.
Turns out they’re all solved with a Reinforcement Learning pipeline and it seems like there doesn’t exist a game that Reinforcement Learning can’t solve.
I’d like to make the case that there a couple of under-explored game genres for Reinforcement Learning that would test entirely different skills from Go, Chess, Dota and Starcraft.
Roguelikes
A rogue-like is generally an adventure game with loads of variety and randomness where death is permanent. The game becomes very challenging as a result since a single complete run can take 2 to 100 hours and each run will involve spells, units and effects you’ve never seen before.
Games like Ancient Domain of Mystery have a large number of systems ranging from stealth, crafting, combat, history, agriculture and expert players are really good at figuring out how to manage the game’s randomness.
The way I would benchmark roguelikes is to compare the total training time to victory to the average amount of time it takes a human to beat the game. So your pipeline can’t 100 hours of training time.
Diplomacy games
Diplomacy is easily my favorite board game, I played it with 7 friends of mine over a 6 month period online. We would take a week to plot each next action so had ample time to host secret meetings where we all diverted and manipulated each other while tactfully making sure that we weren’t losing trust from the right strategic partners.
Negotiation in the Reinforcement Learning research community is often treated in an abstract manner where bots just send each other actions but don’t actually communicate and try to deceive each other.
If someone build a Reinforcement Learning bot that can email and consistently beat top human players at Diplomacy I would be very impressed.
Role Playing Games
Role Playing games are often long adventure games that have you talking to various people in the world and making decisions. Most involve some form of combat that’s heavily influenced by the story decisions you make. My favorite is Divinity Original Sin 2 which is the closest video game to Dungeons and Dragons that I’ve ever played.
From a Reinforcement Learning standpoint we could train with several different goals in mind — finish the game as fast as possible, finish the game with the virtuous ending, explore 100% of the map.
People have been talking about decision making systems for decades but all we’ve gotten from that talk is dashboards. Role-playing games are the closest simulation we have to a real life environment so an agent that can succeed consistently in such an environment would qualify as a decision making system.
Economic Simulators
Economic simulators are games that involve interacting with a stock market. Off World trading company has you building a colony on Mars for which you need various primary resources. The goal is complicated by having 7 other players competing for the same resources that want to be the first to own most of Mars.
You need to do things like make sure you don’t rely on a rare resource for your early expansion and not to dump your soon to be rare resources on the stock market.
I would suggest for all the people claiming to trade stocks with Reinforcement Learning to first test their game on an economic simulator where they could be wrong vs historical data where it’s impossible for them to be wrong.
Next steps
You probably play different games than me so if you have any genre suggestions you don’t often see mentioned in Reinforcement Learning benchmarks, please let me know!
