# Differentiating cross entropy w.r.t the bias term

In the 2-layer neural network with cross entropy loss for softmax hidden layer, why do I get 2 different answers when applying chain rule in 2 different ways? Please help me correct method#2; I am going crazy:

Given: $$C = \sum_iy_ilog(\hat{y_i})$$

$$\hat{y} = softmax(z)$$ $$z=xW+b$$

Method # 1 - Gives correct, texbook answer

Noticing $$\frac{\partial C}{\partial b}=\frac{\partial C}{\partial{z}} \frac{\partial{z}}{\partial b}$$ and we know $$\frac{\partial C}{\partial z} = y-\hat{y},$$ $$\frac{\partial z}{\partial b} = 1$$ $$\frac{\partial C}{\partial b}=y-\hat{y}$$

Method # 2 - Gives wrong answer, but why?: $$\frac{\partial C}{\partial b}=\sum_{i} y_i\frac{\partial log(\hat{y_i})}{\partial b}$$ So then, $$log(\hat{y}) = log(softmax(z))=log(\frac{exp(xW+b)}{\sum_k exp(x_kW_k+b)})$$ $$=xW+b-log(\sum_k exp(x_kW_k+b))$$ Therefore $$\frac{\partial log(\hat{y})}{\partial b} = \frac{\partial}{\partial b}(xW+b)-\frac{\partial{log(\sum_k exp(x_kW_k+b))}}{\partial b}$$ $$=1-\frac{1}{\sum_{l=1}^N exp(x_lW_l+b)}\frac{\sum_{k=1}^N\partial{exp(x_kW_k+b)}}{\partial b}$$ $$= 1-\frac{1}{\sum_{l=1}^N exp(x_lW_l+b)}\sum_{k=1}^N\exp(x_kW_k+b) = 1-1 =0$$ Obviously I messed it up in method 2, but cant figure what's wrong. I tried writing it out with all subscripts but still came to the same answer 0.

Your mistake is treating $\vec b$ as a scalar in the second method, when it is really a vector.

Let $i^*$ be the true label, then $y_{i^*} = 1$ and

$$\frac{\partial C}{\partial \vec b} = \frac{\partial\log(\hat y_{i^*})}{\partial b}$$

Now you had at some point the expression (i put in the subscripts for you)

$$\log(\hat y_i) = (x_i W_i +b_i) - \log \left( \sum_k \exp(x_kW_k+b_k) \right)$$

The derivative of the first term is 1 when $i = i^*$ and 0 otherwise, in other words, the derivative of the first term is $y$.

The derivative of the second term comes out to $\hat y$. This is because

$$\frac{\partial}{\partial b_i} \log \left( \sum_k \exp(\cdot)\right) = \frac{1}{\sum_k \exp(\cdot)} \frac{\partial}{\partial b_i} \exp(x_k W_k + b_k) = \frac{\exp(x_k W_k + b_k)}{\sum_k \exp(\cdot)} = \hat y_i$$

So the final expression is $y - \hat y$ like we wanted to show.

• What a silly mistake on my part. Thanks for the answer. I can now sleep in peace. – Chetan Rawal Jul 17 '17 at 0:48