When doing regression, for example, two hyper parameters to choose are often the capacity of the function (eg. the largest exponent of a polynomial), and the amount of regularisation. What I'm confused about, is why not just choose a low capacity function, and then ignore any regularisation? In that way, it will not overfit. If I have a high capacity function together with regularisation, isn't that just the same as having a low capacity function and no regularisation?
$\begingroup$
$\endgroup$
asked Jul 31, 2016 at 14:36
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
10
I recently made a little in browser app that you can use to play with these ideas: Scatterplot Smoothers (*).
Here's some data I made up, with a low degree polynomial fit
It's clear that the quadratic polynomial is just not flexible enough to give a good fit to the data. We have regions of very high bias, between $0.6$ and $0.85$ all the data is below the fit, and after $0.85$ all the data is above the curve.
To rid ourselves of bias, we can increase the degree of the curve to three, but the problem remains, the cubic curve is still too rigid
So we continue to increase the degree, but now we incur the opposite problem
This curve tracks the data too closely, and has a tendency to fly off in directions not so well borne out by general patterns in the data. This is where regularization comes in. With the same degree curve (ten) and some well chosen regularization
We get a really nice fit!
It's worth a little focus on one aspect of well chosen above. When you are fitting polynomials to data you have a discrete set of choices for degree. If a degree three curve is underfit and a degree four curve is overfit, you have nowhere to go in the middle. Regularization solves this problem, as it gives you a continuous range of complexity parameters to play with.
how do you claim "We get a really nice fit!". For me they all look the same, namely, inconclusive. Which rational are you using to decide what is a nice and a bad fit?
Fair point.
The assumption I'm making here is that a well fit model should have no discernable pattern in the residuals. Now, I'm not plotting the residuals, so you have to do a little bit of work when looking at the pictures, but you should be able to use your imagination.
In the first picture, with the quadratic curve fit to the data, I can see the following pattern in the residuals
- From 0.0 to 0.3 they are about evenly placed above and below the curve.
- From 0.3 to about 0.55 all the data points are above the curve.
- From 0.55 to about 0.85 all the data points are below the curve.
- From 0.85 on, they are all above the curve again.
I'd refer to these behaviours as local bias, there are regions where the curve is not well approximating the conditional mean of the data.
Compare this to the last fit, with the cubic spline. I can't pick out any regions by eye where the fit does not look like it's running precisely through the center of mass of the data points. This is generally (though imprecisely) what I mean by a good fit.
Final Note: Take all this as illustration. In practice, I do not recommend using polynomial basis expansions for any degree higher than $2$. Their problems are well discussed elsewhere, but, for example:
- Their behaviour at the boundaries of your data can be very chaotic, even with regularization.
- They are not local in any sense. Changing your data in one place can significantly affect the fit in a very different place.
Instead, in a situation like you describe, I recommend using natural cubic splines along with regularization, which give the best compromise between flexibility and stability. You can see for yourself by fitting some splines in the app.
(*) I believe this only works in Chrome and Firefox due to my use of some modern javascript features (and overall laziness to fix it in Safari and IE). The source code is here, if you are interested.
answered Jul 31, 2016 at 16:34
-
5$\begingroup$ Thanks, and your browser tool is awesome -- I love little interactive demos like that! $\endgroup$ Jul 31, 2016 at 17:04
-
3$\begingroup$ +6. Good job writing this tool! You will get a bounty from me once the thread is old enough to put a bounty on it. $\endgroup$– amoebaAug 1, 2016 at 21:55
-
4$\begingroup$ +1 This is a really good answer. One way to show the instability of the high degree polynomial fit would be to plot the high-order regression with one data point removed for each point, and to contrast that with the RCS solution. $\endgroup$– Sycorax ♦Aug 2, 2016 at 0:53
-
1$\begingroup$ @MatthewDrury "restricted cubic splines" -- sorry about that. $\endgroup$– Sycorax ♦Aug 5, 2016 at 18:41
-
1
$\begingroup$
$\endgroup$
2
No, it isn't the same. Compare, for example, a second-order polynomial without regularization to a fourth-order polynomial with it. The latter can posit big coefficients for the third and fourth powers so long as this seems to increase predictive accuracy, according to whatever procedure is used to choose the penalty size for the regularization procedure (probably cross-validation). This shows that one of the benefits of regularization is that it allows you to automatically adjust model complexity to strike a balance between overfitting and underfitting.
answered Jul 31, 2016 at 15:04
-
$\begingroup$ But if you add regularisation to a fourth order polynomial, this prevents it using the full extent of its expressiveness. So with enough regularisation, the expressiveness will be reduced to the point where it is as expressive as a second order polynomial. No? $\endgroup$ Jul 31, 2016 at 15:36
-
1$\begingroup$ Perhaps if you fixed your penalty size in advance, but what's the sense in that? The penalty size should be chosen based on the data. $\endgroup$ Jul 31, 2016 at 15:54
$\begingroup$
$\endgroup$
For polynomials even small changes in coefficients can make a difference for the higher exponents.
$L_2$ regularization ( least squares ) usually encourages many small coefficients but none exactly 0 and therefore the higher order monomials are able to make a difference.
$\begingroup$
$\endgroup$
1
All the answers are great and I have similar simulations with Matt to give you another example to show why complex model with regularization is usually better than simple model.
I made a analogy to have intuitive explanation.
- Case 1 you only have a high school student with limited knowledge (a simple model without regularization)
- Case 2 you have a graduate student but restrict him/her to only use high school knowledge to solve problems. (complex model with regularization)
If two persons are solving the same problem, usually the graduate students would work better solution, because the experience and insights about the knowledge.
Figure 1 is showing 4 fittings to the same data. 4 fittings are line, parabola, 3rd order model and 5th order model. You can observe the 5th order model may have overfitting problem.
On the other hand, in the second experiment, we will use 5th order model with different level of regularization. Compare last one with the second order model. (two models are highlighted) you will find the last one is similar (roughly have the same model complexity) to parabola, but slightly more flexible to the data well.
answered Aug 5, 2016 at 13:52
-
1$\begingroup$ "roughly have the same model complexity" ... that's visually the "obvious" comparison, is there a mathematical way of measuring it? $\endgroup$ Aug 5, 2016 at 14:24
$\begingroup$
$\endgroup$
Model Complexity (model flexibility) is about representing the structures hidden in the data. To take an example of polynomial curve fitting, a higher-order polynomial (say, parabola/quadratic) provides more flexibility to represent the hidden structures compared to a lower-order one (say, line/linear) if there is indeed a hidden parabolic structure (that we found using EDA).
So, where does Regularization come in?
The observations/outcomes of a random experiment are noisy (we assume gaussian noise as a good approximation). When we use a higher-order polynomial, the higher the polynomial, the more the training points that lie exactly on the fitted curve. But, this results in poor generalization and the test set results are disappointing.
When we examine the coefficients of the higher order polynomials, they carry very high values. What has happened is that even though the model is flexible, it has tuned itself to the gaussian noise, so much so that the fitted curve oscillates rapidly near the ends of intervals between data points. So, during testing, a slight off-x results in a big off-y.
Regularization helps in keeping these coefficients at lower values, hence, the curve is smooth. We now have less training points on the curve, more training error, but less test error, means, better generalization (less Overfitting).
The choice between higher order polynomial and regularization is not that of excluding one for the other, but that of striking a balance between how higher the polynomial can be without losing too much on generalization.
When we talk about order of polynomial and regularization in the context of generalization/less-overfitting, there is a third lever that can reduce the overfitting caused by a higher-order polynomial. This lever is 'size of data'. More data helps in accommodating a higher-order polynomial.
References: 1 Pattern Recognition and Machine Learning - Christopher Bishop
