1
$\begingroup$

I am trying to perform text classification using the deep recurrent neural network. My network is incurring a huge loss of 94%, 80% and sometimes 100% with certain accuracy. It is surprising that with 64% validation accuracy the incurred loss is 96%. I want to comprehend that whether the incurred loss has direct relation to accuracy or accuracy is being calculated on correctly acquired data. I am using the categorical crossentroy function to estimate the loss. I am aware that the loss is a distance from true values and loss function basically calculate the error rate. But how this can be related to accuracy. For instance both the accuracy and loss is high than what does this depicts?

Text Classification using different neural networks

$\endgroup$
4
  • 1
    $\begingroup$ Crossentropy loss isn't a percent, so what are you doing to get a percent? $\endgroup$
    – Dave
    Nov 10, 2020 at 14:59
  • $\begingroup$ I am multiplying the acquired real value say 0.938 with 100. $\endgroup$ Nov 10, 2020 at 15:35
  • $\begingroup$ So if you achieved a cross-entropy loss of 1.15, you'd write 115%? $\endgroup$
    – Sycorax
    Nov 10, 2020 at 15:47
  • $\begingroup$ @Sycorax thanks for pointing out the mistake. This is what making me confuse and compelled me post this question and try to understand what relation we have between loss and accuracy. In a scenario where we both accuracy and loss is almost high, how do we interpret this? $\endgroup$ Nov 10, 2020 at 16:06

1 Answer 1

1
$\begingroup$

A high cross entropy and a high accuracy are not necessarily inconsistent results.

Here's a simple example. Suppose you have a problem with 10 classes, and a data set with 1 sample of each class. In 9 of the samples, the correct class has prediction $0.85$, but in one class it has prediction $0.001$. The cross-entropy loss in this case is almost $0.84$. This is because very incorrect predictions are penalized much more steeply than slightly incorrect predictions.

On the other hand, the accuracy of the model is 90%, because in 9 out of 10 cases, the correct class has the largest predicted probability.


First, you've written cross-entropy as a percent. This doesn't make much sense, because cross-entropy is not a part of a whole.

Also, you write

I am aware that the loss is a distance from true values and loss function basically calculate the error rate.

This is incorrect. The error rate of a model is the proportion of samples classified incorrectly. We can write $1 = \text{accuracy} + \text{error rate}$ because these names describe complementary concepts. Cross entropy measures "how wrong" a prediction is (larger values mean the prediction was more wrong), but this measurement of wrongness doesn't involve classification at all. Instead, cross-entropy penalizes predictions which are far from their correct values.

$\endgroup$
2
  • $\begingroup$ your answer is much appreciated but i have question what would be the maximum value of cross-entropy loss which depicts the error rate and what error rate is alarming what rate could be ignored? $\endgroup$ Nov 11, 2020 at 6:51
  • $\begingroup$ There is not an exact relationship between cross-entropy and error rate. You can have a low error rate and either a high or a low cross-entropy value, and you can manipulate the cross-entropy value to be arbitrarily large if you wish. Whether or not an error rate is "alarming" depends on the costs of errors: does a person who is perfectly healthy take a course of antibiotics (low risk) or does that person lose a healthy limb? $\endgroup$
    – Sycorax
    Nov 11, 2020 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.