# Gradient of softmax with cross entropy loss

I'm working on implementing a simple deep model which uses cross-entropy loss, while using softmax to generate predictions.

More specifically, I am interested in obtaining the gradient of $$CE(softmax(\vec \beta), \vec x)$$ with $\beta = A^T \vec y$, such that $\beta_i = \vec a_i^T \vec y$ with respect to $\vec y$

By the chain rule, I get $$\frac{dCE}{dy} = \frac{dCE}{d softmax(\beta)} \frac{dsoftmax(\beta)}{d\beta} \odot \frac{d\beta}{d y}$$

Here, the elementwise product comes from the derivative of an elementwise function (chain rule applied to elementwise functions).

I calculate that $$\frac{dCE}{d \beta} = softmax(\beta) - \vec x$$

So far this makes sense (and I have managed to verify the above using various sources online). However, when I try to calculate $d\beta/dy$, I get $A^T$, which doesn't make sense, as the elementwise product isn't defined for matrices of differing dimensions (AKA my vector $softmax(\beta) - \vec x$ and matrix $A^T$)

Can anyone shed some light onto where I am going wrong?

• it should be something like $$\sum_i\frac{dCE}{d\beta_i}\frac{d\beta_i}{dy}$$ but i couldn't tell where goes wrong.. – dontloo Apr 12 '16 at 3:57
• So I managed to work out the partial in terms of an individual element... $\frac{dJ}{d(y)_i} = (a_k)_i - \frac{\sum_j (a_j)_i * \exp(a_j^T y)}{\sum_l \exp(a_l^T y)} = (a_k)_i - \sum_j (a_j)_i * \frac{\exp(a_j^T y)}{\sum_l \exp(a_l^T y)}$ (note, one assumption I am making is that my desired cross-entropy vector is one-hot with the kth element being the one element). I am unsure how to vectorize this notation. Any thoughts? – BigA Apr 12 '16 at 4:06

$y$ and $\beta$ are vectors, hence writing $M=\frac{d\beta}{dy}$ implies the result is a matrix $M$ with components $M_{ij}=\frac{d\beta_i}{dy_j}$, which happens to be equal to $A^T$. See Matrix Calculus for more identities.
• Sure, but then how does one make sense of the element-wise product? I am definitely understanding that the $d\beta/dy$ should result in a matrix, but I am confused as to how to understand $(softmax(\beta) - \vec x) \odot M$. Did I make a mistake somewhere else? – BigA Apr 12 '16 at 0:04