Assume I have N data points and their number is very limited. The output range is very vast. I can do one of the fallowing things:

  1. Fit the problem to some regression algorithm (Tree regression, neural network etc ...)
  2. The problem can be divided also to 3 classes instead, like "bad, neutral, good".

The question is: which problem is more prone to over-fitting, if the data are limited in number ?


In any machine learning model, you're try to learn a set of rules which can be applied in the future to make predictions.

Overfitting happens when your rules are too specific to the data which you trained on:

  • When feature $X_1$ is equal to 100.456 then my target will be equal to 47.85.

A better model will have more general rules which work out of sample:

  • When feature $X_1$ is large, my target tends to be very large too.
  • When feature $X_2$ is less than 50, my target tends to be between 10 and 100.

Some regression algorithms are more prone to learning rules which are overfit to outliers. Ensemble algorithms are typically more robust in this respect.

Having a simpler target (3 classes) might make it easier for your setup to learn more general rules but I don't think it's possible to make any exact statement about classification problems being less prone to overfitting. You could still learn a rule like:

  • When feature $X_1$ is equal to 100.456 then my target will be class 'Good'.

The rule learned here is perhaps more likely to hold true out of sample since the output space is coarser. But I would still call it 'overfit'.

  • $\begingroup$ Thanks for answer, so it looks like a quite difficult topic with a lot space to talk about $\endgroup$ Sep 17 at 11:35

I suspect overfitting is likely to be more of a problem in the classification setting, because you have quantised the response variable and hence thrown away some information content of the dataset. Having more information in the data tends to make ML algorithms less susceptible to over-fitting, which is why adding more data usually helps.

This definitely seems to be true for finding the hyper-parameters of Gaussian processes, where over-fitting in model selection is a clear problem in classification 1, but seems less of a problem in a regression setting 2.

1 GC Cawley, NLC Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation, The Journal of Machine Learning Research 11, 2079-2107 (pdf)

2 Rekar O Mohammed, Gavin C Cawley, Over-fitting in model selection with Gaussian process regression, International Conference on Machine Learning and Data Mining in Pattern Recognition, Pages 192-205, 2017 (pdf)

BTW if the range of the response variable is vast, you might want to try some transformation (e.g. predict the logarithm of the response variable). Sometimes that is also useful if your application requires relative errors, rather than absolute.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.