3
$\begingroup$

So, I made a bidirectional LSTM model for sentiment classification. Model's job was to predict ratings of movies(1-5 stars) based on the movie review.

While training the model I first used categorical cross entropy loss function. I trained the model for 10+ hours on CPU for about 45 epochs. While training every epoch showed model accuracy to be 0.5098(same for every epoch).

Then I changed the loss function to binary cross entropy and it seemed to be work fine while training. So, I want to know what exactly is the difference between these two?

$\endgroup$

marked as duplicate by Sycorax, Michael Chernick, kjetil b halvorsen, gung Jul 25 '18 at 14:19

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Binary cross entropy is for binary classification but categorical cross entropy is for multi class classification , but both works for binary classification , for categorical cross entropy you need to change data to to_categorical . $\endgroup$ – ᴀʀᴍᴀɴ Jul 17 '18 at 11:06
3
$\begingroup$

I would like to expand on ARMAN's answer:

Not getting into formulas the biggest difference would be that categorical crossentropy is based on the assumption that only 1 class is correct out of all possible ones (so output should be something like [0,0,0,1,0] if the rating is 4) while binary_crossentropy works on each individual output separately implying that each case can belong to multiple classes (for instance if predicting what items a customer will get it is possible that they will buy multiple ones; i.e. output like [0,1,0,1,0] is a valid one if you are using binary_crossentropy). As ARMAN pointed out if you only have 2 classes a 2 output categorical_crossentropy is equivalent to 1 output binary_crossentropy one.

In your specific case you should be using categorical_crossentropy since each review has exactly 1 rating. Binary_crossentropy gives you better scores but the outputs are not evaluated correctly. I would also recommend trying to use MSE loss since your data is ordinal (4 stars are closer to 5 than 1)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.