I've been taking some time to try and understand the computations and mechanics of the machine learning algorithms I use in my day to day life.

Studying the backpropagation literature on the CS231n course, I want to make sure that I have understood the chain rule correctly before continuing my study.

Say I have the sigmoid function:

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

in this case, $x=w0x0+w1x1+w2$

We can write this function as a computational graph (Ignoring the coloured values for now): enter image description here

We can group the modularised nodes for computing the gradient of the sigmoid $w.r.t.$ its input into a single derivation:

$$\frac{d\sigma(x)}{d x}=(1 - \sigma(x))\sigma(x)$$

First, we perform forward propogation to obtain the outputs at each unit:

w = [2,-3,-3] 
x = [-1, -2]

# Compute the forward pass 
product = [w[0]*x[0]+w[1]*x[1]+w[2]]
activation = 1 / 1 + math.exp(-product)

To calculate the gradient of the activation, we can use the above formula:

grad_product = (1 - activation) * activation 

Where I feel I may be getting confused, or, at least less intuitive, is calculating the gradient for x and w:

grad_x = [w[0] * activation + w[2] * activation]
grad_w = [x[0] * activation + x[1] * activation + 1 * activation]

More concretely, I'm confused as to why we apply 1 * activation when calculating the gradient $w.r.t.$ w.

It may help the reader spot my theoretical difficulty if I try to reason the calculations of both x and w's gradients...

The gradient of each $x_i$ is given by the corresponding $w_i$ under the rule of multiplication: if $f(x,y) = f(xy)$ then $\frac{\partial f}{\partial x}=y$. Then, using the chain rule, we multiply these local gradients by the gradient of the successive node (for each path of $x$) to obtain its gradient w.r.t. the function output. This explains the computation for computing $\partial x$.

The gradient of $w_i$ is given in the exact same (inverse) way as explained above with the additional 1 * activation. I believe this additional expression is coming from $w_2$? The local gradient of an addition unit is always 1 for all inputs and the multiplication with activation is a result of chaining the gradient to the function's output?

I'm partially confident with my current understanding but would appreciate if someone could clarify my current intuition on the computations involved in computing gradients.


2 Answers 2


What you want to compute, is

$$\frac{\partial \sigma(\hat{x})}{\partial \vec{x}}=\left[\frac{\partial \sigma(\hat{x})}{\partial x_0},\frac{\partial \sigma(\hat{x})}{\partial x_1}\right]$$


$$\frac{\partial \sigma({\hat{x}})}{\partial \vec{w}}=\left[\frac{\partial \sigma(\hat{x})}{\partial w_0},\frac{\partial \sigma(\hat{x})}{\partial w_1},\frac{\partial \sigma(\hat{x})}{\partial w_2}\right]$$

knowing that $\hat{x}$ is in fact a function of those variables, as $\hat{x}=w_0x_0+w_1x_1+w_2$.

You can use the chain rule to compute this as:

$$\frac{\partial \sigma(\hat{x})}{\partial x_0}=\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}\frac{\partial \hat{x}}{\partial x_0}$$

You already know $\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}$

as its $$\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}=(1-\sigma(\hat{x}))\sigma(\hat{x})$$

and the second derivative is trivial (its just a polynomial! $\frac{\partial \hat{x}}{\partial x_0}=w_0$). You now only need to compute for the 5 partial derivatives. In short:

$$\frac{\partial \sigma(\hat{x})}{\partial \vec{x}}=\left[\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}w_0,\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}w_1\right]$$

$$\frac{\partial \sigma(\hat{\hat{x}})}{\partial \vec{w}}=\left[\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}x_0,\frac{\partial \sigma(x)}{\partial \hat{x}}x_1,\frac{\partial \sigma(\hat{x})}{\partial \hat{x}}\right]$$


The best way to understand backpropagation for a programmer is in terms of the chain rule as a recursion.

Here's the chain rule. You have a nested function expression $y=f(g(x))$. First you look at it as two different functions: $$f(x)\\g(x)$$ When you do forward propagation it's nothing but this psudo code: $$t=g(x)\\ y=f(t)$$

Now, if you want to take a derivative, you apply a chain rule: $$y'=f(g(x))'=f'g'$$ where $$f'=df(t)/dt$$ and $$g'=dg(x)/dx$$ This is basically a recursion on a nested structure. If $g(x)=g(h(x))$, then you simply apply the chain rule again, and keep doing it until you reach the bottom, i.e. the input layer in case of NN.

Here's an example, a one neuron: $$a=sigmoid(Wx+b)$$ You have two functions here: $sigmoid(x)$ and $Wx+b$.

If you have two layers of neurons it's not much different: $$sigmoid(W_1*sigmoid(Wx+b)+b_1)$$ so you go backwards: $$z=Wx+b\\a_1=sigmoid(z)\\z_1=W_1*a_1+b_1\\a_2=sigmoid(z_1)$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.