I am aware that there is a similar post: Vectorization of Cross Entropy Loss

But looking it over for a couple hours, I still do not understand what they are trying to say.

I'm trying to find the update rules for using the cross entropy function in multiclass logistic regression, here for 10 classes.

enter image description here

I'm confused what this is even implying - is this the cost for the entire dataset or just the cost for a single example? I'm not sure if y and yhat are scalars and the summation goes through a single example or that y and yhat are vectors and that the i indexing means it's going through the entire set.

Assuming that it's going through the whole training set, how do we derive it w.r.t all the thetas? how to vectorize it, and to come up with a general update rule for all weights?

I would be so grateful for an answer - I've tried with pen and paper for a long time and haven't gotten it yet.

Thanks, A


This is not over the entire the training set. It's for a single example, $i$ represents the class index. So, if you have $n$ classes, $\mathbf y$ and $\mathbf{ \hat y}$ are $n\times 1$ real vectors. $\mathbf y$ is the ground truth, so it'll be a binary vector with the index corresponding to the correct class for the sample being $1$ and others being $0$.

To find the gradients, you can assume one sample only, and then sum the gradients for each training sample. So, for simplicity, do the differentiation using this formulation, i.e. $\frac{\partial L}{\partial \theta}$, where $y=\sigma( \theta^Tx+b)$.

  • $\begingroup$ thank you so much - that's a huge help . Is the σ in y=σ(θTx+b) representing the softmax or sigmoid function? And follow up - so you're finding the derivative of the entire theta vector (and treating the vector like a scalar, sort of?). Also, did you mean yhat instead of y in y=σ(θTx+b)? Thank you so much . $\endgroup$ – DownstairsPanda Nov 25 '20 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.