Let's say you design a CNN model and it starts overfitting at e.g 70% accuracy during training. Does that mean that 70% is the best accuracy that I can get out of that model design even if I add regularization? Or in other words, can regularization improve the performance of a model?

  • $\begingroup$ Are you talking about validation or training performance? Are you really referring to accuracy or at loss? $\endgroup$
    – Michael M
    Jun 30 '20 at 17:57
  • $\begingroup$ Welcome to CV. There are methods that look for an inflection point in validation error as an indication of parameters for a final model. If error on validation, which it has never seen, go up, then it is starting to overfit. This is an older method, but it works. Personally, after the parameters have been dialed in, I like to use all the training and validation data to make the final model, and then run it against the test (second holdout) to see what plausible performance looks like. $\endgroup$ Jun 30 '20 at 18:24
  • $\begingroup$ during the training after a few epochs, I see that the training accuracy is much higher than the validation accuracy, i.e overfitting my model to the training data. My model is a cnn without any regularization (dropout or any other). My question is if I add dropout, will I be able to achieve more accuracy (than 70% here), or this is the best that I can get out of this model?, and in order to get a better performance, I will have to change the model architecture for example?!? $\endgroup$
    – Sohrab
    Jun 30 '20 at 18:39
  • $\begingroup$ Validation accuracy around 70%, training accuracy around 80%. $\endgroup$
    – Sohrab
    Jun 30 '20 at 18:57
  • $\begingroup$ no, there is phenomenon known as double descent where model have to first overfit and after overfitting the performance gets better $\endgroup$
    – rep_ho
    Jun 30 '20 at 21:28

There is no way to tell without knowing more about your situation. Yes, you may have hit a plateau.

  • Or there may be hidden structure in your data that will only become apparent with (much) more data.
  • Or some of your predictors may need to be predicted themselves, and you may be able to do a better job at that.
  • Or you may not have included a hugely relevant predictor that can increase your validation accuracy to 90%, simply because you don't have the domain knowledge.

In general, it is very hard to say whether we have reached the end of the flagpole. See How to know that your machine learning problem is hopeless?


Actually your last question, does regularization improve model performance, is a bit off. regularization is a way to reduce overfitting, not to check if you hit a plateau yet.

As overfitting means your model try to fit even to noise in training data and not the real underlying pattern, the model will likely to perform bad on test data. That’s why regularization helps to improve performance, as it is adding a term to penalise models which are sensitive to noise.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.