5

WhenI try to research overfitting and underfitting, the most common algorithm and explanation I see revolves around polynomial regression. Why is this so? Is it just because it can be easily visualised like the graphs here:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html

Why is polynomial regression deemed suitable to give an insight into these overfitting/underfitting concepts? Are there any other similar algorithms that could be used?

2

3 Answers 3

11

Easy visualization is a huge point in favor of using polynomial regression for illustration. (Note that both "illustration" and "demonstration", etymologically, have to do with showing pictures!) It also helps that the degree of the polynomial controls the amount of overfitting, and that polynomial regression allows looking at bona fide nonlinearity in the relationship (although splines are a better way of dealing with this).

An alternative that is also commonly employed is to simply add spurious predictors. Your true data generating process might be y=b1x1+b2x2+ϵ, so a model y=β1x1 is underfitted, y=β1x1+β2x2 is fitted correctly, and y=β1x1+β2x2+β3x3 is overfitted - but it's hard to visualize the increased variability of fits from the overfitted model.

2
  • 7
    One somewhat boring point is: most people already know what polynomials are, since they are standard educational material in (I suspect) all high school level education throughout the world. Commented May 7, 2021 at 18:01
  • @MatthewDrury: good point! "Boring" is good, because if we understand the tools intuitively, then we can focus on the parts that are less intuitive, like over-/underfitting. Commented May 8, 2021 at 8:04
6

A more colloquial answer to this would be that the polynomial regression, the higher the order, has more flexibility to learn but also the tendency to memorize every data point in the training set, but with unneccessary and unpredictable wriggly waves, that makes forecasting nearly impossible as algorithms can get stuck in one area and extrapolation is quite problematic. Thus polynomial regression memorizes data wich is equal to overfitting, and does not learn to deal with new data, it only catches a certain situation. And as Stephan highlighted, the visualization helps a lot here.

Update just to make it clear: Overfit just means a model that learns its data very well, which can be very well shown with a polynomial. Actually we wouldn't need a train/test split to see it, because our mind should say that we cant explain something so perfectly. Look how the polynom tries to catch every data point in both pics. Just to make it more clear due to the comments below my post, I added another picture:

ML Overfitting: enter image description here

Picture by Jannis Seemann Udemy

  • The overfitting polynom is the red line.
  • Grey crosses are train data
  • Blue crosses are test data
  • Green line is a simpler model, (e.g. a power function) that doesnt catch the data exactly, but learns the general trend or a picture of a model situation.

You can see that the red polynom learnt the data very well, but behaves bad on the test data. The simpler model, not catches the whole train data but behaves well overall. If you are aware of this behavior you wont need a visual train/test split to be sure if your model overfits, if you choose to a high polynom.

Polynom behavior in general

This picture shows a general comparison of a overfitted model with a high order polynom and a polynom of a much lower order. You can already see, that the model with the higher order overfits, although you havent seen the test data.

To feel the power of a simple overfit, you can do pictures like that yourself: Use a simple scatterplot. Got to Microsoft excel, click on fitting the scatter plot with a linear trend. Than change to polynomial und increase polynomials to the highest order. The polynomial gives you the freedom to fit the data near perfectly, but that squiggly line does not normally fit on the test data, its totally useless (blue one). The red line although a much lower fit, may be extrapolated better and shows "maybe" the truth, that there is not much to expect, Your model will be bad, which is part of antoher discussion, but not overfitted. enter image description here Image by researchgate https://www.researchgate.net/profile/Scott-James-5/publication/255205930/figure/fig1/AS:670698729123859@1536918446824/The-green-squares-represent-the-original-data-set-are-fit-using-two-conceptual-models-a.png

6
  • Your picture example is not clear. What is the difference between square/round, red/blue? Commented May 8, 2021 at 3:34
  • You are right, it is not exactly fitting. The blue line focuses on the green dots, and shows the overfitting process of learning data. You can see how the high grade polynom moves through every dot, showing how data gets learned, but with unneccessary squiggly lines at the cost. The blue dots may be from a different scatter plot than the green, but thats not the point: you can see how the first three blue dots are "in line" with the green dots up to the 4th point. And it shows that fitting a line with a much lesser polynomial can result in a way badder R² although it makes more sense Commented May 8, 2021 at 6:51
  • I guess I mean I know what you are trying to show, but it isn't very clear. Like which is the training, which is the test? Maybe the colors could match? Commented May 8, 2021 at 15:57
  • I updated my answer. But you do not need a train/test split to say or see if a model is overfitted. Although i highlighted it again with a picture. You can just see that the model makes squiggly lines to exactly hit every data point. That is learning your data instead of identifying a general trend. But the new pic should capture it better. Commented May 8, 2021 at 18:43
  • I was using terms loosely. I know what you're trying to show, but for a reader who doesn't, I just think the explanation isn't clear (i.e. presumably the test is green squares, but that's not stated) Commented May 8, 2021 at 22:22
2

This is a less statsy answer, and more of a practical/human-oriented answer. While all the other answers are correct in their own manner. I think it is more to do with your audience. Let me explain:

If you're learning about overfitting then you're going to be pretty new to ML/Stats/Data Science. Therefore, not everyone taking said class, reading said blog post or watching said video is going to understand more complex models like GBMs, SVMs, GMMs and so on...

However, if you are learning ML you should sure as hell will understand the concept of linear regression (or line of best fit) and one would hope you'd understand polynomials (otherwise maybe you should be taking high school maths classes not trying to learn ML just yet).

Finally, you can create and recover the generative model precisely. This, in a learning phase, helps people fully understand what is going on and allow them to easy replicate results and play with the models themselves.


NB: Maybe the real question is "Why aren't more ML concepts introduced as polynomial regression examples, since they are so intuitive, expressive and accessible by everyone with a high school mathematics education?".

The honest answer to this is probably: "Because GBMs, SVMs, GMMs etc are freaking cool!".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.