Build your own RL environments w/ Unity ML agents

Why Unity

A couple of months back I started working on a strategy video game based off the Lebanese Civil War. I experimented with a bunch of engines including building my own crappy one and the one I ended up having the most affection towards was Unity.

Why build a simulation? Why not use a benchmark?

At NeurIPS 2016 I had a brief discussion with a prominent Reinforcement Learning researcher where I made the case that new state of the art performance on Atari games was uninteresting and essentially over-fitting massive models on simple games.

What is Unity ML agents

In short Unity ML agents is a Unity plugin that lets you hook up your game agents (e.g: enemy AI) to a Reinforcement Learning server which is responsible for 3 things

  1. Training the RL model
  2. Inputting actions back to the game from the model
  1. Set a reward function

Unity ML agents at a glance

  1. Curriculum learning: Where you can chain more complex environments together after an agent excels at a more basic task
  2. Proximal Policy Optimization & Soft Actor Critic RL algorithms: examples of stable RL algorithms that’ll ensure your agents won’t be too “jerky”. You can download any algorithm you like from Github and integrate it in if you have specific needs
  1. Mesh: It’s a cube (this could be a humanoid instead)
  2. RigidBody and Collider: Physics constants associated with the agent
  3. Script: Looks like there a couple of scripts we need to check out

Code deep dive

There are 4 scripts associated with the soccer environment so let’s go over them 1 by 1.


Is responsible for holding all the data that helps instantiate an agent like its team, role and its physical properties like kicking power.


SoccerAcademy is a container for the simulation metadata so mainly holds references to the Brain objects which when attached to a Unity object augments it with the ability to do Reinforcement Learning and other misc stuff like gravitational constant, simulation speed and max number of steps.


The SoccerBallController is the logic attached to the ball. When the agents force the ball to cross a goal, the ball checks to see which of the 2 goals its in and increment the score of the right team.


SoccerFieldArea is a management layer that manages multiple simulations at the same time and updates a GUI so you can easily keep track of which team has the better strategy.

Configuring Reinforcement Learning algorithms

There’s one final loose end, how do you set hyper-parameters or configure your Reinforcement Learning algorithm?

Next steps

You can get pretty far using Unity ML agents for your research projects, video games or DYI robots. My recommendation here is that you do the same thing I did and click around on the various examples.

Robots will save us