# Clarification of the intuition behind backpropagation

I've been taking some time to try and understand the computations and mechanics of the machine learning algorithms I use in my day to day life.

Studying the backpropagation literature on the CS231n course, I want to make sure that I have understood the chain rule correctly before continuing my study.

Say I have the sigmoid function:

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

in this case, $x=w0x0+w1x1+w2$

We can write this function as a computational graph (Ignoring the coloured values for now): We can group the modularised nodes for computing the gradient of the sigmoid $w.r.t.$ its input into a single derivation:

$$\frac{d\sigma(x)}{d x}=(1 - \sigma(x))\sigma(x)$$

First, we perform forward propogation to obtain the outputs at each unit:

w = [2,-3,-3]
x = [-1, -2]

# Compute the forward pass
product = [w*x+w*x+w]
activation = 1 / 1 + math.exp(-product)


To calculate the gradient of the activation, we can use the above formula:

grad_product = (1 - activation) * activation


Where I feel I may be getting confused, or, at least less intuitive, is calculating the gradient for x and w:

grad_x = [w * activation + w * activation]
grad_w = [x * activation + x * activation + 1 * activation]


More concretely, I'm confused as to why we apply 1 * activation when calculating the gradient $w.r.t.$ w.

It may help the reader spot my theoretical difficulty if I try to reason the calculations of both x and w's gradients...

The gradient of each $x_i$ is given by the corresponding $w_i$ under the rule of multiplication: if $f(x,y) = f(xy)$ then $\frac{\partial f}{\partial x}=y$. Then, using the chain rule, we multiply these local gradients by the gradient of the successive node (for each path of $x$) to obtain its gradient w.r.t. the function output. This explains the computation for computing $\partial x$.

The gradient of $w_i$ is given in the exact same (inverse) way as explained above with the additional 1 * activation. I believe this additional expression is coming from $w_2$? The local gradient of an addition unit is always 1 for all inputs and the multiplication with activation is a result of chaining the gradient to the function's output?

I'm partially confident with my current understanding but would appreciate if someone could clarify my current intuition on the computations involved in computing gradients.

What you want to compute, is

$$\frac{\partial \sigma(\hat{x})}{\partial \vec{x}}=\left[\frac{\partial \sigma(\hat{x})}{\partial x_0},\frac{\partial \sigma(\hat{x})}{\partial x_1}\right]$$

and

$$\frac{\partial \sigma({\hat{x}})}{\partial \vec{w}}=\left[\frac{\partial \sigma(\hat{x})}{\partial w_0},\frac{\partial \sigma(\hat{x})}{\partial w_1},\frac{\partial \sigma(\hat{x})}{\partial w_2}\right]$$

knowing that $\hat{x}$ is in fact a function of those variables, as $\hat{x}=w_0x_0+w_1x_1+w_2$.

You can use the chain rule to compute this as:

$$\frac{\partial \sigma(\hat{x})}{\partial x_0}=\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}\frac{\partial \hat{x}}{\partial x_0}$$

You already know $\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}$

as its $$\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}=(1-\sigma(\hat{x}))\sigma(\hat{x})$$

and the second derivative is trivial (its just a polynomial! $\frac{\partial \hat{x}}{\partial x_0}=w_0$). You now only need to compute for the 5 partial derivatives. In short:

$$\frac{\partial \sigma(\hat{x})}{\partial \vec{x}}=\left[\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}w_0,\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}w_1\right]$$

$$\frac{\partial \sigma(\hat{\hat{x}})}{\partial \vec{w}}=\left[\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}x_0,\frac{\partial \sigma(x)}{\partial \hat{x}}x_1,\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}\right]$$

The best way to understand backpropagation for a programmer is in terms of the chain rule as a recursion.

Here's the chain rule. You have a nested function expression $y=f(g(x))$. First you look at it as two different functions: $$f(x)\\g(x)$$ When you do forward propagation it's nothing but this psudo code: $$t=g(x)\\ y=f(t)$$

Now, if you want to take a derivative, you apply a chain rule: $$y'=f(g(x))'=f'g'$$ where $$f'=df(t)/dt$$ and $$g'=dg(x)/dx$$ This is basically a recursion on a nested structure. If $g(x)=g(h(x))$, then you simply apply the chain rule again, and keep doing it until you reach the bottom, i.e. the input layer in case of NN.

Here's an example, a one neuron: $$a=sigmoid(Wx+b)$$ You have two functions here: $sigmoid(x)$ and $Wx+b$.

If you have two layers of neurons it's not much different: $$sigmoid(W_1*sigmoid(Wx+b)+b_1)$$ so you go backwards: $$z=Wx+b\\a_1=sigmoid(z)\\z_1=W_1*a_1+b_1\\a_2=sigmoid(z_1)$$