In the 2-layer neural network with cross entropy loss for softmax hidden layer, why do I get 2 different answers when applying chain rule in 2 different ways? Please help me correct method#2; I am going crazy:

Given: $$ C = \sum_iy_ilog(\hat{y_i}) $$

$$ \hat{y} = softmax(z) $$ $$ z=xW+b $$

Method # 1 - Gives correct, texbook answer

Noticing $$ \frac{\partial C}{\partial b}=\frac{\partial C}{\partial{z}} \frac{\partial{z}}{\partial b} $$ and we know $$ \frac{\partial C}{\partial z} = y-\hat{y}, $$ $$ \frac{\partial z}{\partial b} = 1 $$ $$ \frac{\partial C}{\partial b}=y-\hat{y} $$

Method # 2 - Gives wrong answer, but why?: $$ \frac{\partial C}{\partial b}=\sum_{i} y_i\frac{\partial log(\hat{y_i})}{\partial b} $$ So then, $$ log(\hat{y}) = log(softmax(z))=log(\frac{exp(xW+b)}{\sum_k exp(x_kW_k+b)}) $$ $$=xW+b-log(\sum_k exp(x_kW_k+b)) $$ Therefore $$ \frac{\partial log(\hat{y})}{\partial b} = \frac{\partial}{\partial b}(xW+b)-\frac{\partial{log(\sum_k exp(x_kW_k+b))}}{\partial b} $$ $$ =1-\frac{1}{\sum_{l=1}^N exp(x_lW_l+b)}\frac{\sum_{k=1}^N\partial{exp(x_kW_k+b)}}{\partial b} $$ $$ = 1-\frac{1}{\sum_{l=1}^N exp(x_lW_l+b)}\sum_{k=1}^N\exp(x_kW_k+b) = 1-1 =0 $$ Obviously I messed it up in method 2, but cant figure what's wrong. I tried writing it out with all subscripts but still came to the same answer 0.


Your mistake is treating $\vec b$ as a scalar in the second method, when it is really a vector.

Let $i^*$ be the true label, then $y_{i^*} = 1$ and

$$\frac{\partial C}{\partial \vec b} = \frac{\partial\log(\hat y_{i^*})}{\partial b}$$

Now you had at some point the expression (i put in the subscripts for you)

$$\log(\hat y_i) = (x_i W_i +b_i) - \log \left( \sum_k \exp(x_kW_k+b_k) \right)$$

The derivative of the first term is 1 when $i = i^*$ and 0 otherwise, in other words, the derivative of the first term is $y$.

The derivative of the second term comes out to $\hat y$. This is because

$$\frac{\partial}{\partial b_i} \log \left( \sum_k \exp(\cdot)\right) = \frac{1}{\sum_k \exp(\cdot)} \frac{\partial}{\partial b_i} \exp(x_k W_k + b_k) = \frac{\exp(x_k W_k + b_k)}{\sum_k \exp(\cdot)} = \hat y_i$$

So the final expression is $y - \hat y$ like we wanted to show.

  • $\begingroup$ What a silly mistake on my part. Thanks for the answer. I can now sleep in peace. $\endgroup$ – Chetan Rawal Jul 17 '17 at 0:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.