Automatic Differentiation Step by Step

Mark Saroufim
14 min readNov 12, 2019

Automatic Differentiation lets you compute exact derivatives in constant time

Differentiation shows up everywhere from the backprop algorithm in deep neural networks to the equations of motion in physics and to pretty much any field that needs to quantify a rate of change.