37
$\begingroup$

I've understood the main concepts behind overfitting and underfitting, even though some reasons as to why they occur might not be as clear to me.

But what I am wondering is: isn't overfitting "better" than underfitting?

If we compare how well the model does on each dataset, we would get something like:

Overfitting: Training: good vs. Test: bad

Underfitting: Training: bad vs. Test: bad

If we have a look at how well each scenario does on the training and test data, it seems that for the overfitting scenario, the model does at least well for the training data.

The text in bold is my intuition that, when the model does badly on the training data, it will also do badly on the test data, which seems overall worse to me.

$\endgroup$
4
  • 17
    $\begingroup$ Why would you think it's any value at all to work on a training set and then still fail in testing/production? The former is only a stepping-stone to the latter. If anything, holding out value for training when testing fails would be confusing/fraudulent snake oil. $\endgroup$ Apr 29 at 0:50
  • 1
    $\begingroup$ overfitting is often confused with overparameterisation, which is not the same thing, however i think the concept is adequately explained in the question already as being to do with the fit on the training and test sets. $\endgroup$ Apr 29 at 7:56
  • $\begingroup$ An overfitted model do well on the dataset because it was designed (accidentally) to work well on the training set. $\endgroup$
    – Taladris
    Apr 30 at 7:38
63
$\begingroup$

Overfitting is likely to be worse than underfitting. The reason is that there is no real upper limit to the degradation of generalisation performance that can result from over-fitting, whereas there is for underfitting.

Consider a non-linear regression model, such as a neural network or polynomial model. Assume we have standardised the response variable. A maximally underfitted solution might completely ignore the training set and have a constant output regardless of the input variables. In this case the expected mean squared error on test data will be approximately the variance of the response variable in the training set.

Now consider an over-fitted model that exactly interpolates the training data. To do so, this may require large excursions from the true conditional mean of the data generating process between points in the training set, for example the spurious peak at about x = -5. If the first three training points were closer together on the x-axis, the peak would be likely to be even higher. As a result, the test error for such points can be arbitrarily large, and hence the expected MSE on test data can similarly be arbitrarily large.

source

Source: https://en.wikipedia.org/wiki/Overfitting (it is actually a polynomial model in this case, but see below for an MLP example)

Edit: As @Accumulation suggests, here is an example where the extent of overfitting is much greater (10 randomly selected data points from a linear model with Gaussian noise, fitted by a 10th order polynomial fitted to the utmost degree). Happily the random number generator gave some points that were not very well spaced out first time!

enter image description here

It is worth making a distinction between "overfitting" and "overparameterisation". Overparameterisation means you have used a model class that is more flexible than necessary to represent the underlying structure of the data, which normally implies a larger number of parameters. "Overfitting" means that you have optimised the parameters of a model in a way that gives a better "fit" to the training sample (i.e. a better value of the training criterion), but to the detriment of generalisation performance. You can have an over-parameterised model that does not overfit the data. Unfortunately the two terms are often used interchangeably, perhaps because in earlier times the only real control of overfitting was achieved by limiting the number of parameters in the model (e.g. feature selection for linear regression models). However regularisation (c.f. ridge regression) decouples overparameterisation from overfitting, but our use of the terminology has not reliably adapted to that change (even though ridge regression is almost as old as I am!).

Here is an example that was actually generated using an (overparameterised) MLP

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – Sycorax
    Apr 30 at 19:09
39
$\begingroup$

Roughly, overfitting is fitting the model to noise, while underfitting is not fitting a model to the signal. In your prediction with overfitting you'll reproduce the noise, the underfitting will show the mean, at best. It's like choosing between diarrhea and constipation. I'd probably choose the latter, so I'd go with underfitting, i.e. the mean.

$\endgroup$
5
  • 11
    $\begingroup$ Fitting noise vs. not fitting data!!! Super insightful! +1 $\endgroup$
    – Haitao Du
    Apr 29 at 6:11
  • 5
    $\begingroup$ Something tells me you don't like diarrhea? $\endgroup$ Apr 29 at 9:23
  • 5
    $\begingroup$ Created an account to upvote this concise summation and easy to grasp :) $\endgroup$ Apr 30 at 5:45
  • $\begingroup$ Insightful analogy! $\endgroup$ Apr 30 at 18:49
  • $\begingroup$ It seems to me that erring on the side of underfitting is also going to usually be more practical in real world applications. $\endgroup$
    – JimmyJames
    Apr 30 at 21:33
17
$\begingroup$

The question of what is good and what is bad depends on the problem, the question and the circumstances.

Some thoughts: The most impressive proof of how underfitting is not generally rejected so much is the universal use of simple linear regression. Problems in social sciences, life sciences, psychology etc. are almost never linear and if they are curved, they are almost never curved following a quadratic term. Nevertheless, some excellent research in these fields depends heavily on the idea, that one could draw a straight line as the representation of any relation.

A straight line is not prone to overfitting, very prone to underfitting. But that can give it credibility. "There is a connection because I can draw a reasonable straight line" is far more convincing then "There is a connection because I can draw splines" - because you can almost always overfit with splines.

People even judge the outcome of a linear regression by in-sample-R² without any form of Cross validation - would you believe a neural net of arbitrary size without cross validation or testing set anywhere near an OLS regression with small p and large R²?

$\endgroup$
8
$\begingroup$

I liked the question and the key concept to answer it is Bias–variance tradeoff. Both underfitted model and overfitted model have some valid use case.

For example my answer here

Is an overfitted model necessarily useless?

$\endgroup$
1
  • $\begingroup$ Initially, that was my starting point, Where I thought: It's better to end up with an overfitted model because it intuitively seems easier to decrease overfitting than to increase what the model learns. But given the answers, I've received that doesn't seem to be necessarily the case. Generally, I was referring to the case of neural networks though. $\endgroup$
    – LeLuc
    Apr 30 at 8:53
8
$\begingroup$

The "better" one is the one that performs better on the test set.

When you're running it in production, that's (ideally) going to be similar to the test set. It's not going to be data you've seen before, so the training set performance doesn't matter so much.

Although if they're roughly equal, underfitting is generally better.

  • Underfitting means you learnt simple approximations of patterns that actually exists in both the training and test data (and production data, hopefully). But overfitting means you learnt some patterns that only exists in the training data and not in the test data (and presumably not in the production data either). That could make your model behave in less consistent ways.
  • Underfitting should be the simpler model, and if you have 2 models that perform the same, you pick the simpler one. You don't want unnecessary complexity, which increases the likelihood of unexpected behaviour and can also be more resource intensive.
  • A simpler model can also be more interpretable, which can help to show that the way the model works makes sense and to understand the data better to guide business decisions in a broader context. Although interpretability is about a lot more than just over- or underfitting. If that's important to you, it would affect which type of model you use and various parts of how you train it.
$\endgroup$
3
  • 2
    $\begingroup$ Generally yes. I'm a corporate data scientist and the best model for me is often the one that will perform the best on data that will be generated in the future. That's why I prefer underfitting, even if the overfitted model still looks good in CV. $\endgroup$ Apr 29 at 15:12
  • 3
    $\begingroup$ Underfit models are also often easier to interpret, draw insight from, and build credibility - which in some business contexts is the bulk of the value the model generates! $\endgroup$
    – Neal
    Apr 29 at 22:19
  • $\begingroup$ @Neal That's very true. $\endgroup$ Apr 30 at 13:02
3
$\begingroup$

Parsimony usually a desirable feature in a machine learning model - this relates to the idea of Occam's Razor, that among models that produce equally good (or bad) results, you should generally prefer the simpler one. This is explicitly performed in certain model selection approaches such as the Minimum Description Length, which describes the "cost" of a model as related both to how well the model fits the data, and the complexity of the model. For two models that both perform poorly on a dataset, the simpler model is preferred. Overfit models typically require extra parameters that add "cost" to a model with no discernable benefit, so you're usually better off with an underfit model that yields similar error.

In a way, an overfit model is like a student who memorizes the sequence of answers on a multiple choice practice test, applies that same sequence to the real test, and fails. The underfit model, on the other hand, is the student who simply picks "C" for every answer, and does as poorly on the real test as they did on the practice. The overfit student is "surprised" by their poor performance, and did a lot of work for absolutely no benefit. The underfit student knows their strategy won't work well, but can implement that strategy very easily. Both students fail the test, but at least one knew ahead of time and didn't waste a lot of effort for nothing.

$\endgroup$
3
$\begingroup$

The answers so far are reasonable in the context of linear regression. But in a deep-learning context we usually train to the point of overfitting (if we have the resources to); then we go back and use the model saved most recently before that. I.e. underfitting just means "not there yet, carry on".

You also have to consider that the metric being used to measure the over- vs. under-fitting may not be the ideal one. As one example I've trained finance-trading algorithms with MSE, because it is quick to evaluate. But the real measure of how good the model is would be a backtest on the data, under trading conditions. Sometimes the under-fitted or over-fitted model does better than the one that minimized MSE.

In recent NLG (natural language generation) models, we've also been deliberately erring on the side of over-fitting, after blind-testing found users said they were better. This is the metric thing again, but I think it also ties in with the memorizing aspect that Haitao Du mentioned in his linked answer. So high bias, low variance ("underfitted") can mean it spouts nonsense, while low bias, high variance can mean it sounds just like the training data. Just like a human would.

To sum up, it depends.

$\endgroup$
2
  • 1
    $\begingroup$ I don't see that there's a fundamental difference between what under- / overfitting mean in deep learning vs linear regression. What's different is just that deep learning models take longer training to overfit, even when there are plenty of parameters available (which would make a linear model overfit immediately). $\endgroup$ Apr 30 at 7:15
  • $\begingroup$ The NLG example isn't necessarily specific to deep learning either. All that's happening here is, you overfit the model so it learns some noise in addition to the data and it turns out that humans actually find that the noise makes it sound more natural. Much the same thing could happen in non-deep-learned applications too. $\endgroup$ Apr 30 at 7:16
3
$\begingroup$

Imagine you're an ML algorithm that is supposed to do well on a test. You know the syllabus and hence, you have a set of example problems (with solutions) which you can practice from. Now, you have three options:

  1. Overfit: Memorize solutions of practice problems without actually understanding how you arrived at the solution. This will lead to failure on the test.

  2. Underfit: Solve the simplest problem with few transferable skills. You would solve some problems on test but won't be enough to cross the passing threshold score. However, there is room for improvement.

  3. Somewhere between Overfit and Underfit: You practice problems which appeared on previous years' test. Since such problems are more representative of the problems you will be seeing in your test tomorrow, there's a higher probability that you will do well. Which is what you want! Which is exactly the performance a machine aims for!

$\endgroup$
5
  • 1
    $\begingroup$ Regarding point 3: Current systems seem to require us to pass through an overfitting regime to reach ideal generalization. (i.e. exactly being able to interpolate the training data is the worst case scenario, and you'll overfit less with a higher capacity model) $\endgroup$ Apr 28 at 19:51
  • 5
    $\begingroup$ It makes sense to take an analogy to exams, but how you set it up seems off to me. “Overfit” should be, you learn the answers from the examples and previous year by heart, without actually understanding them. “Underfit” should be, you learn enough understanding to answer only the simplest questions, but at least that skill is transferrable to the question that actually show up in the new exam. So you won't get a good grade because you can't do difficult questions, but you may still (barely) pass. $\endgroup$ Apr 29 at 12:10
  • $\begingroup$ @leftaroundabout updated the post. Does this help? $\endgroup$
    – user529295
    Apr 30 at 12:25
  • 2
    $\begingroup$ Good analogy. I'm very curious how can I perform regularization on my exam-scoring capability :D $\endgroup$ Apr 30 at 18:49
  • $\begingroup$ @Nuclear03020704 How about dropout: Use only a (random) limited set of the transferrable skills you learned to solve any example exam question. $\endgroup$ Apr 30 at 19:17
3
$\begingroup$

Probably not your situation, but you can use machine learning as a memory. Suppose you have some inputs where you can train on the entire domain of inputs. That is, there is no possible input that wasn't previously known and used for training. That way, you can always generate the correct output. If the concept you wish to train on can be expressed in this way, then over training is memorising, and in limited situations, can be useful. Otherwise, as the other comments have stated, overfitting means you are biased to the training data and will do poorly on anything not in the training set. The goal is for generalisation, for the model to understand the concept, not memorise. The syllabus answer above explains it well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.