0
$\begingroup$

I am trying to classify a particular image.

My labels are PatternA and NotPatternA.

NotPatternA is any image that I have not classified as PatternA. There are very few PatternAs compared to NotPatternA. In my data I have 500 PatternA and 6000 NotPatternA.

I construct my training and testing sets, by choosing 400 from PatternA at random for training and the remaining 100 I put into Testing.

For NotPatternA I randomly draw a sample of 500 images and again split it 400/100 into training and testing.

After 10 epochs I start to get very low loss and high accuracy (near 100%). BUT testing gives me very high loss (2+). However, testing accuracy is also good (near 80%).

Why might my testing loss be so high?

$\endgroup$
1
$\begingroup$

Overfitting. Try incorporating cross-validation into your training schedule. Most ML libraries provide a way of cross-validating your model during the training process so that you can stop training once your model begins to overfit.

Edit: The reason why you are observing "high accuracy" in your test data set is because of the label imbalance. Your model is overfitted to the training set, but when it sees the new test data it can just guess NotPatterA and still be correct 90% of the time (this is also why I put "high accuracy" in quotes - in fact, 80% accuracy is not a good result given your label ratio).

$\endgroup$
  • $\begingroup$ BUT my training set and my testing set both have an equal number of the 2 labels... Surely 80% accuracy is therefore good? $\endgroup$ – ManInMoon Sep 9 '19 at 10:40
  • 1
    $\begingroup$ You need to think carefully about the definition of the metric "accuracy". Accuracy is the ratio of correct predictions to total number of predictions. In both your data sets, there are approximately 90% of samples with label NotPatternA. Therefore if I were to create an "algorithm" which predicted NotPatternA 100% of the time, I would achieve 90% accuracy on both test and train datasets. $\endgroup$ – timchap Sep 9 '19 at 10:48
  • 1
    $\begingroup$ No, I don't think so. I have specifically chosen to have 50% of PatternA and 50% of NotPatternA in both training and testing. $\endgroup$ – ManInMoon Sep 9 '19 at 10:50
  • $\begingroup$ Oh, you're quite right. I misread that part of your question. $\endgroup$ – timchap Sep 9 '19 at 10:51
  • $\begingroup$ (My suggestion to use cross-validation stands though) $\endgroup$ – timchap Sep 9 '19 at 10:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.