Is overfitting "better" than underfitting?

Question

I've understood the main concepts behind overfitting and underfitting, even though some reasons as to why they occur might not be as clear to me.

But what I am wondering is: isn't overfitting "better" than underfitting?

If we compare how well the model does on each dataset, we would get something like:

Overfitting: Training: good vs. Test: bad

Underfitting: Training: bad vs. Test: bad

If we have a look at how well each scenario does on the training and test data, it seems that for the overfitting scenario, the model does at least well for the training data.

The text in bold is my intuition that, when the model does badly on the training data, it will also do badly on the test data, which seems overall worse to me.

Answer 1

Overfitting is likely to be worse than underfitting. The reason is that there is no real upper limit to the degradation of generalisation performance that can result from over-fitting, whereas there is for underfitting.

Consider a non-linear regression model, such as a neural network or polynomial model. Assume we have standardised the response variable. A maximally underfitted solution might completely ignore the training set and have a constant output regardless of the input variables. In this case the expected mean squared error on test data will be approximately the variance of the response variable in the training set.

Now consider an over-fitted model that exactly interpolates the training data. To do so, this may require large excursions from the true conditional mean of the data generating process between points in the training set, for example the spurious peak at about x = -5. If the first three training points were closer together on the x-axis, the peak would be likely to be even higher. As a result, the test error for such points can be arbitrarily large, and hence the expected MSE on test data can similarly be arbitrarily large.

Source: https://en.wikipedia.org/wiki/Overfitting (it is actually a polynomial model in this case, but see below for an MLP example)

Edit: As @Accumulation suggests, here is an example where the extent of overfitting is much greater (10 randomly selected data points from a linear model with Gaussian noise, fitted by a 10th order polynomial fitted to the utmost degree). Happily the random number generator gave some points that were not very well spaced out first time!

It is worth making a distinction between "overfitting" and "overparameterisation". Overparameterisation means you have used a model class that is more flexible than necessary to represent the underlying structure of the data, which normally implies a larger number of parameters. "Overfitting" means that you have optimised the parameters of a model in a way that gives a better "fit" to the training sample (i.e. a better value of the training criterion), but to the detriment of generalisation performance. You can have an over-parameterised model that does not overfit the data. Unfortunately the two terms are often used interchangeably, perhaps because in earlier times the only real control of overfitting was achieved by limiting the number of parameters in the model (e.g. feature selection for linear regression models). However regularisation (c.f. ridge regression) decouples overparameterisation from overfitting, but our use of the terminology has not reliably adapted to that change (even though ridge regression is almost as old as I am!).

Here is an example that was actually generated using an (overparameterised) MLP

Answer 2

Roughly, overfitting is fitting the model to noise, while underfitting is not fitting a model to the signal. In your prediction with overfitting you'll reproduce the noise, the underfitting will show the mean, at best. It's like choosing between diarrhea and constipation. I'd probably choose the latter, so I'd go with underfitting, i.e. the mean.

Answer 3

The question of what is good and what is bad depends on the problem, the question and the circumstances.

Some thoughts: The most impressive proof of how underfitting is not generally rejected so much is the universal use of simple linear regression. Problems in social sciences, life sciences, psychology etc. are almost never linear and if they are curved, they are almost never curved following a quadratic term. Nevertheless, some excellent research in these fields depends heavily on the idea, that one could draw a straight line as the representation of any relation.

A straight line is not prone to overfitting, very prone to underfitting. But that can give it credibility. "There is a connection because I can draw a reasonable straight line" is far more convincing then "There is a connection because I can draw splines" - because you can almost always overfit with splines.

People even judge the outcome of a linear regression by in-sample-R² without any form of Cross validation - would you believe a neural net of arbitrary size without cross validation or testing set anywhere near an OLS regression with small p and large R²?

Answer 4

The "better" one is the one that performs better on the test set.

When you're running it in production, that's (ideally) going to be similar to the test set. It's not going to be data you've seen before, so the training set performance doesn't matter so much.

Although if they're roughly equal, underfitting is generally better.

• Underfitting means you learnt simple approximations of patterns that actually exists in both the training and test data (and production data, hopefully). But overfitting means you learnt some patterns that only exists in the training data and not in the test data (and presumably not in the production data either). That could make your model behave in less consistent ways.
• Underfitting should be the simpler model, and if you have 2 models that perform the same, you pick the simpler one. You don't want unnecessary complexity, which increases the likelihood of unexpected behaviour and can also be more resource intensive.
• A simpler model can also be more interpretable, which can help to show that the way the model works makes sense and to understand the data better to guide business decisions in a broader context. Although interpretability is about a lot more than just over- or underfitting. If that's important to you, it would affect which type of model you use and various parts of how you train it.

Answer 5

I liked the question and the key concept to answer it is Bias–variance tradeoff. Both underfitted model and overfitted model have some valid use case.

For example my answer here

Is an overfitted model necessarily useless?

Answer 6

Parsimony usually a desirable feature in a machine learning model - this relates to the idea of Occam's Razor, that among models that produce equally good (or bad) results, you should generally prefer the simpler one. This is explicitly performed in certain model selection approaches such as the Minimum Description Length, which describes the "cost" of a model as related both to how well the model fits the data, and the complexity of the model. For two models that both perform poorly on a dataset, the simpler model is preferred. Overfit models typically require extra parameters that add "cost" to a model with no discernable benefit, so you're usually better off with an underfit model that yields similar error.

In a way, an overfit model is like a student who memorizes the sequence of answers on a multiple choice practice test, applies that same sequence to the real test, and fails. The underfit model, on the other hand, is the student who simply picks "C" for every answer, and does as poorly on the real test as they did on the practice. The overfit student is "surprised" by their poor performance, and did a lot of work for absolutely no benefit. The underfit student knows their strategy won't work well, but can implement that strategy very easily. Both students fail the test, but at least one knew ahead of time and didn't waste a lot of effort for nothing.

Answer 7

The answers so far are reasonable in the context of linear regression. But in a deep-learning context we usually train to the point of overfitting (if we have the resources to); then we go back and use the model saved most recently before that. I.e. underfitting just means "not there yet, carry on".

You also have to consider that the metric being used to measure the over- vs. under-fitting may not be the ideal one. As one example I've trained finance-trading algorithms with MSE, because it is quick to evaluate. But the real measure of how good the model is would be a backtest on the data, under trading conditions. Sometimes the under-fitted or over-fitted model does better than the one that minimized MSE.

In recent NLG (natural language generation) models, we've also been deliberately erring on the side of over-fitting, after blind-testing found users said they were better. This is the metric thing again, but I think it also ties in with the memorizing aspect that Haitao Du mentioned in his linked answer. So high bias, low variance ("underfitted") can mean it spouts nonsense, while low bias, high variance can mean it sounds just like the training data. Just like a human would.

To sum up, it depends.

Answer 8

Imagine you're an ML algorithm that is supposed to do well on a test. You know the syllabus and hence, you have a set of example problems (with solutions) which you can practice from. Now, you have three options:

1. Overfit: Memorize solutions of practice problems without actually understanding how you arrived at the solution. This will lead to failure on the test.

2. Underfit: Solve the simplest problem with few transferable skills. You would solve some problems on test but won't be enough to cross the passing threshold score. However, there is room for improvement.

3. Somewhere between Overfit and Underfit: You practice problems which appeared on previous years' test. Since such problems are more representative of the problems you will be seeing in your test tomorrow, there's a higher probability that you will do well. Which is what you want! Which is exactly the performance a machine aims for!

Answer 9

Probably not your situation, but you can use machine learning as a memory. Suppose you have some inputs where you can train on the entire domain of inputs. That is, there is no possible input that wasn't previously known and used for training. That way, you can always generate the correct output. If the concept you wish to train on can be expressed in this way, then over training is memorising, and in limited situations, can be useful. Otherwise, as the other comments have stated, overfitting means you are biased to the training data and will do poorly on anything not in the training set. The goal is for generalisation, for the model to understand the concept, not memorise. The syllabus answer above explains it well.