When doing regression, for example, two hyper parameters to choose are often the capacity of the function (eg. the largest exponent of a polynomial), and the amount of regularisation. What I'm confused about, is why not just choose a low capacity function, and then ignore any regularisation? In that way, it will not overfit. If I have a high capacity function together with regularisation, isn't that just the same as having a low capacity function and no regularisation?


I recently made a little in browser app that you can use to play with these ideas: Scatterplot Smoothers (*).

Here's some data I made up, with a low degree polynomial fit

Quadratic Fit

It's clear that the quadratic polynomial is just not flexible enough to give a good fit to the data. We have regions of very high bias, between $0.6$ and $0.85$ all the data is below the fit, and after $0.85$ all the data is above the curve.

To rid ourselves of bias, we can increase the degree of the curve to three, but the problem remains, the cubic curve is still too rigid

Cubic Fit

So we continue to increase the degree, but now we incur the opposite problem

Ten Degree Fit

This curve tracks the data too closely, and has a tendency to fly off in directions not so well borne out by general patterns in the data. This is where regularization comes in. With the same degree curve (ten) and some well chosen regularization

Degree Ten Regularizaton

We get a really nice fit!

It's worth a little focus on one aspect of well chosen above. When you are fitting polynomials to data you have a discrete set of choices for degree. If a degree three curve is underfit and a degree four curve is overfit, you have nowhere to go in the middle. Regularization solves this problem, as it gives you a continuous range of complexity parameters to play with.

how do you claim "We get a really nice fit!". For me they all look the same, namely, inconclusive. Which rational are you using to decide what is a nice and a bad fit?

Fair point.

The assumption I'm making here is that a well fit model should have no discernable pattern in the residuals. Now, I'm not plotting the residuals, so you have to do a little bit of work when looking at the pictures, but you should be able to use your imagination.

In the first picture, with the quadratic curve fit to the data, I can see the following pattern in the residuals

  • From 0.0 to 0.3 they are about evenly placed above and below the curve.
  • From 0.3 to about 0.55 all the data points are above the curve.
  • From 0.55 to about 0.85 all the data points are below the curve.
  • From 0.85 on, they are all above the curve again.

I'd refer to these behaviours as local bias, there are regions where the curve is not well approximating the conditional mean of the data.

Compare this to the last fit, with the cubic spline. I can't pick out any regions by eye where the fit does not look like it's running precisely through the center of mass of the data points. This is generally (though imprecisely) what I mean by a good fit.

Final Note: Take all this as illustration. In practice, I do not recommend using polynomial basis expansions for any degree higher than $2$. Thier problems are well discussed elsewhere, but, for example:

  • Their behaviour at the boundaries of your data can be very chaotic, even with regularization.
  • They are not local in any sense. Changing your data in one place can significantly affect the fit in a very different place.

I instead, in a situation like you describe, reccomend using natural cubic splines along with regularization, which give the best compromise between flexibility and stability. You can see for yourself by fitting some splines in the app.

Natural Cubic Spline

(*) I believe this only works in chrome and firefox due to my use of some modern javascript features (and overall lazyness to fix it in safari and ie). The source code is here, if you are interested.

  • 3
    $\begingroup$ Thanks, and your browser tool is awesome -- I love little interactive demos like that! $\endgroup$ – Karnivaurus Jul 31 '16 at 17:04
  • $\begingroup$ @Karnivaurus Thanks, I'm glad I could help. The tool was fun to build, I like writing javascript : ) $\endgroup$ – Matthew Drury Aug 1 '16 at 3:50
  • 3
    $\begingroup$ +6. Good job writing this tool! You will get a bounty from me once the thread is old enough to put a bounty on it. $\endgroup$ – amoeba Aug 1 '16 at 21:55
  • 4
    $\begingroup$ +1 This is a really good answer. One way to show the instability of the high degree polynomial fit would be to plot the high-order regression with one data point removed for each point, and to contrast that with the RCS solution. $\endgroup$ – Sycorax Aug 2 '16 at 0:53
  • 1
    $\begingroup$ @MatthewDrury "restricted cubic splines" -- sorry about that. $\endgroup$ – Sycorax Aug 5 '16 at 18:41

No, it isn't the same. Compare, for example, a second-order polynomial without regularization to a fourth-order polynomial with it. The latter can posit big coefficients for the third and fourth powers so long as this seems to increase predictive accuracy, according to whatever procedure is used to choose the penalty size for the regularization procedure (probably cross-validation). This shows that one of the benefits of regularization is that it allows you to automatically adjust model complexity to strike a balance between overfitting and underfitting.

  • $\begingroup$ But if you add regularisation to a fourth order polynomial, this prevents it using the full extent of its expressiveness. So with enough regularisation, the expressiveness will be reduced to the point where it is as expressive as a second order polynomial. No? $\endgroup$ – Karnivaurus Jul 31 '16 at 15:36
  • $\begingroup$ Perhaps if you fixed your penalty size in advance, but what's the sense in that? The penalty size should be chosen based on the data. $\endgroup$ – Kodiologist Jul 31 '16 at 15:54

For polynomials even small changes in coefficients can make a difference for the higher exponents.

$L_2$ regularization ( least squares ) usually encourages many small coefficients but none exactly 0 and therefore the higher order monomials are able to make a difference.


All the answers are great and I have similar simulations with Matt to give you another example to show why complex model with regularization is usually better than simple model.

I made a analogy to have intuitive explanation.

  • Case 1 you only have a high school student with limited knowledge (a simple model without regularization)
  • Case 2 you have a graduate student but restrict him/her to only use high school knowledge to solve problems. (complex model with regularization)

If two persons are solving the same problem, usually the graduate students would work better solution, because the experience and insights about the knowledge.

Figure 1 is showing 4 fittings to the same data. 4 fittings are line, parabola, 3rd order model and 5th order model. You can observe the 5th order model may have overfitting problem.

enter image description here

On the other hand, in the second experiment, we will use 5th order model with different level of regularization. Compare last one with the second order model. (two models are highlighted) you will find the last one is similar (roughly have the same model complexity) to parabola, but slightly more flexible to the data well.

enter image description here

  • 1
    $\begingroup$ "roughly have the same model complexity" ... that's visually the "obvious" comparison, is there a mathematical way of measuring it? $\endgroup$ – Silverfish Aug 5 '16 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.