I'm working on implementing a simple deep model which uses cross-entropy loss, while using softmax to generate predictions.

More specifically, I am interested in obtaining the gradient of $$CE(softmax(\vec \beta), \vec x)$$ with $\beta = A^T \vec y$, such that $\beta_i = \vec a_i^T \vec y$ with respect to $\vec y$

By the chain rule, I get $$\frac{dCE}{dy} = \frac{dCE}{d softmax(\beta)} \frac{dsoftmax(\beta)}{d\beta} \odot \frac{d\beta}{d y}$$

Here, the elementwise product comes from the derivative of an elementwise function (chain rule applied to elementwise functions).

I calculate that $$\frac{dCE}{d \beta} = softmax(\beta) - \vec x$$

So far this makes sense (and I have managed to verify the above using various sources online). However, when I try to calculate $d\beta/dy$, I get $A^T$, which doesn't make sense, as the elementwise product isn't defined for matrices of differing dimensions (AKA my vector $softmax(\beta) - \vec x$ and matrix $A^T$)

Can anyone shed some light onto where I am going wrong?

  • $\begingroup$ it should be something like $$\sum_i\frac{dCE}{d\beta_i}\frac{d\beta_i}{dy}$$ but i couldn't tell where goes wrong.. $\endgroup$
    – dontloo
    Commented Apr 12, 2016 at 3:57
  • $\begingroup$ So I managed to work out the partial in terms of an individual element... $\frac{dJ}{d(y)_i} = (a_k)_i - \frac{\sum_j (a_j)_i * \exp(a_j^T y)}{\sum_l \exp(a_l^T y)} = (a_k)_i - \sum_j (a_j)_i * \frac{\exp(a_j^T y)}{\sum_l \exp(a_l^T y)}$ (note, one assumption I am making is that my desired cross-entropy vector is one-hot with the kth element being the one element). I am unsure how to vectorize this notation. Any thoughts? $\endgroup$
    – BigA
    Commented Apr 12, 2016 at 4:06

1 Answer 1


$y$ and $\beta$ are vectors, hence writing $M=\frac{d\beta}{dy}$ implies the result is a matrix $M$ with components $M_{ij}=\frac{d\beta_i}{dy_j}$, which happens to be equal to $A^T$. See Matrix Calculus for more identities.

  • $\begingroup$ Sure, but then how does one make sense of the element-wise product? I am definitely understanding that the $d\beta/dy$ should result in a matrix, but I am confused as to how to understand $(softmax(\beta) - \vec x) \odot M$. Did I make a mistake somewhere else? $\endgroup$
    – BigA
    Commented Apr 12, 2016 at 0:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.