1
$\begingroup$

I've heard different definitions of overfitting. One of them is "if the training error continues to decrease but testing error begins to increase at some point, it is overfitting". The other, "if the training mAP is way higher than testing mAP it is overfitting". I've even seen a definition saying "test score < train score is overfitting". I'm assuming score refers to accuracy.

Which one of them actually applies specifically to object detection?

I encountered a situation where both my train mAP and test mAP increases (test mAP - very slightly, sometimes fluctuate a little but no downward trend). But the difference between the two can increase by quite a bit i.e. train mAP is increasing at a faster rate than test mAP is. I've sometimes seen a big difference of around 20%.

Should I train for more epochs until I begin to see a downward trend in test mAP? Will there always be such a trend? Could I adjust any parameters such as learning rate so that I can see this trend faster?

*note: mAP values are averaged across a 5-fold CV

Sample of my mAP values (mAP vs iterations): mAP vs iterations

$\endgroup$
3
$\begingroup$

The common definition from Wikipedia is:

In statistics and machine learning, one of the most common tasks is to fit a "model" to a set of training data, so as to be able to make reliable predictions on general untrained data. In overfitting, a statistical model describes random error or noise instead of the underlying relationship.

So, overfitting is just sensitivity to a random noise. The good video about what it is: https://www.youtube.com/watch?v=u73PU6Qwl1I.

When talking about object detection it's best to conclude about overfitting from the number (portion) of misdetected examples, MSE is also popular as well. But mean average precision or let's say F2-score are ok too - it's just the curve itself might look kind of different.


About the plot, I'd refer you to DeepLearning book for more info. There is a technique called early stopping meant to prevent overfitting. So you should worry only when validation error is starting to increase and your mAP is around value 56. Early-stoping recommendations are applicable to your case.

So, you're not overfitting much yet.

A better approach is to save some intermediate weights now and then, so you could roll-back to the state where cv-error is lowest.

enter image description here

The real curve might look like this (especially when you're using cross-validation):

enter image description here

Source: http://www.byclb.com/TR/Tutorials/neural_networks/ch10_1.htm


If you decrease learning rate (bigger step) - it could make learning faster, but depending on your data your model might skip local/global minima, so it might take even more time. The overall recommendation is to try different rates like 0.1, 0.01, 0.001.

Actually increasing learning rate (smaller step) might help your model to find more optimal values. If you use SGD, you can make the process quite faster by applying momentum.

Hyper-parameters you might also try to change to make it faster

  • less neurons in hidden layer (less width), more hidden layers (more depth)
  • for CNN - reduce size of receptive field
  • for CNN - apply pooling

Keep in mind, that it all comes at a risk of loosing important details about input.

$\endgroup$
  • $\begingroup$ For "training map-estimation error" and "cross-validation set error", did you actually mean training mAP and cv mAP? I was talking about the mean average precision for training and cv. $\endgroup$ – User1915 Feb 23 '17 at 8:50
  • $\begingroup$ yes, sorry I wanted to say "MAP-evaluation", but was thinking about (unrelated) posteriori estimators at the time :) $\endgroup$ – dk14 Feb 23 '17 at 9:01
  • $\begingroup$ So it should be "training error decreases" instead of increases right? My mAP was increasing, meaning error is decreasing instead. I've taken Andrew Ng's coursera course before, but I felt his examples weren't directly related to mine. I don't have MSE values output so my only way of trying to determine overfitting is from the mAP values. $\endgroup$ – User1915 Feb 23 '17 at 9:05
  • $\begingroup$ thanks for the insights! Right now since I'm not overfitting, I could just stop there, albeit at the expense of getting a slightly higher mAP for more epochs? I'm very tempted to stop because it takes quite a long time to train. $\endgroup$ – User1915 Feb 23 '17 at 9:48
  • $\begingroup$ If your score looks good - then yes. I'd check some manual samples for being completely misclassified to estimate how important that error is for your practical case. $\endgroup$ – dk14 Feb 23 '17 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.