The Robot Overlord Manual

Illustrated by Sarah Saroufim

After reading this book you should be a lot better prepared to start designing robots at home. I’m mostly focused on math and software because that’s what I know best.

This is a roughly 250 page textbook covering what are in my mind the most powerful ideas in Robotics and Machine Learning. My chapters have regularly made it to the front page of Hacker News and Google Search so subscribe and join the hundreds of thousands of readers that are looking to become Robot Overlords.

Table of Contents


How to Build Robots at Home — will give you a broad overview of the robotics at home problem including manufacturing, electronics and software.

The Robot Overlord Manual is primarily about simulating robots in software.

Deep Learning

Deep Learning is a programming technique where instead of describing how to go from inputs to outputs, you have labels from inputs to outputs and can generate the program automatically. Most applications so far have focused on image classification or text classification or generation but more recently Deep Learning has been used for robotic control problems.

  1. Can I use Deep Learning for that? A Type Theoretic Heuristic — This is a useful heuristic I’ve developed to figure out what’s possible and what isn’t with Deep Learning.

Reinforcement Learning

Reinforcement Learning is a technique to describe goals to robots and then have them execute them without telling them how to do it.

  1. How to Implement a Reinforcement Learning Library from Scratch — Reinforce.jl deep dive — How does Reinforcement Learning work and how does a real Reinforcement Learning library work?


Robots are physical objects moving in space and it’s challenging to describe them using just Newton’s laws. Different representations have different advantages so this section is about how we can improve physics engines to support more scalable Reinforcement Learning

  1. How to Turn physics into an Optimization Problem — Physics Engines are hard to build and slow but we can simplify them with a unique insight that’s very familiar to the Machine Learning community.

Software Design Patterns

Building software is hard so it’s important to constantly cultivate better tools and design patterns. Object Oriented Programming teaches lots of bad habits and often results in poor performance unless you’re an expert user.

Design patterns that I find a lot more exciting

  1. Writing Compositional Software with Multiple Dispatch — most design pattern books don’t talk about design patterns that are particularly well suited for scientific code. This interactive article in a Pluto.jl notebook will show you how multiple dispatch lets you write scientific code the way it should have always been written.


If there’s anything else you’d really like me to cover, please don’t hesitate to reach out to me either in the comments section below or on Twitter.

But if you’re hungry for more content.

  1. Machine Learning: The Great Stagnation — Risk free rewards, feedback loops, fake rigor and graduate student descent have created an environment where risk free rewards are now celebrated Machine Learning is now facing a great stagnation of ideas, the bureaucrats are running the asylum

Why should you read this?

The state of the textbook

When I was working on, I had to self teach myself a lot of unfamiliar concepts from math to physics to machine learning.

I learn best from textbooks but they have a few problems that I’ve always hated.

  1. Problems with no solutions: From homework to exercises left to the reader — you can’t validate your knowledge.

The Robot Overlord Manual

This is the book I wish I had when I was getting started. It’s a summary of all the mathematical and algorithmic ideas I believe are key to making robots mainstream.

The key features of the Robot Overlord Manual are

  1. Chapters can be read independently: If you ever feel stuck on some chapter read another one!

So I hope you enjoy reading this book as much as I enjoyed writing it — reach out to me if you have any feedback or just want to send me a note that I helped you in some way.

So go ahead, pick a random chapter and get started!

For more frequent updates on Robotics, Machine Learning and Math you can follow me on


Robots will save us

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store