Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I am learning a statistical model, which includes a very large amount of parameters, which results in the risk of over-fitting. If I first learn the model parameters from the data, and then simply remove some of the parameters according to whichever criterion, would I be reducing the chance for overfitting?

On the one hand, less parameters - less overfitting is supposed to be true.

On the other hand one could claim that once the multi-parameter model was already fit, the parameters themselves were already learned incorrectly - and so reducing the number of parameters now does not help.

I should note that I have been led to believe that the former is true, though I'm not sure why.

share|improve this question
1  
Are you talking about estimating a model with $k$ parameters, but only using $m<k$ to predict, without changing the estimated model? Or are you talking about reducing the number of parameters in the estimation? – Dimitriy V. Masterov May 24 '12 at 18:39

3 Answers

up vote 4 down vote accepted

While removing parameters of the model and the relearning the weights will reduce overfitting (albeit at the potential cost of underfitting the data) simply removing the parameters after learning without any retraining will have highly unpredictable, and most likely detrimental, effects.

As for the question of "why not?", I think the question of "why?" is more appropriate. That is, your working hypothesis is that removing parameters will improve test error in an overfit model. I don't see any reason why you would expect this to be true.

share|improve this answer
+1 You will want to re-learn the remaining parameters after removing some of them. Think of your current solution as an extremum point on a manifold. If you reduce the dimensionality of the manifold, why should the same point remain an extremum? – Ansari May 24 '12 at 21:36
@anonymous_4322 removing variables can help prediction not the fit. Get a greta looking fit can have detrimental consequences in prediction. – Michael Chernick May 24 '12 at 22:23
@MichaelChernick yes, it can help the prediction, but I see no reason to expect it will, at least not in general. Even by restricting the model class I can't see a way in which one could accurately predict the effects of dropping variables after training. – anonymous_4322 May 24 '12 at 22:31
Your training is just model fitting. As I showed in the fourth degree polynomial example a line can look like a higher order polynomial if you let it wiggle through all the points. That is not a good way to fit or in your terminology to train the classifier. The terms after the linear term don't belong. Eliminating all three makes the fit less than perfect or you might view it as making the classification imperfect, having a positive error rate. But a classifier perfect on the data produces an incorrect model which will do much worse on a new test set than the simpler linear model in t. – Michael Chernick May 24 '12 at 22:46
up vote 1 down vote

Overfitting does not always hurt prediction. In one type of overfitting the variables are highly correlated. So they could be almost functionally related perhaps in a linear fashion. Suppose that X1=2X2+5X3 exactly without error. Then you could use any two of the variables in the model and get exactly the same result. The equations can look very different and still both fit and predict well. With another type of overfitting, it can lead to a poorer model than one with fewer parameters (when the problem is not collinearity) but rather the inclusion of too many variables results in fitting the noise as well as the signal. For example suppose we have a response that is a simple linear function of time observed with random mean zero noise. To fit the line we are given 5 pairs (t, f(t)) at distinct times t, where the function f(t) is "truly" f(t) =at+b and is observed with an additive noise component e(t). It we fit a straight line we may get a good (but not perfect) linear fit to these five points. The reason it is not perfect is because of the noise. So the 5 points do not all fall on a single straight line. However if we take a fourth degree polynomial of the form f(t) =a1 +a2 t +a3 t4$^2$ +a4 t$^3$ + a5 t$^4$ we can take the five values of f(t) on the left-hand side of the equation and plug in the five corresponding values of t on the righthand side we will have 5 linear equations in five unknowns that leads to a unique set of values for a1, a2, a3, a4, a5. So by overfitting we can get a wiggly polynomial to fit the data perfectly. But this function will not interpolate, extrapolate or predict well. Variable subset selection procedures will take out variables that may be highly correlated with other variables in the model and also remove variables that really have no relationship to the response. So yes they can reduce or eliminated both types of overfitting problems. when you drop variables from the model you do not keep the original parameter estimate that came about with the other variables included. You will refit the new model that contains fewer parameters and the coefficients should change. So there is no dilemma.

share|improve this answer
1  
This is semantics, but I would argue that overfitting always hurts prediction, by definition. In your first example with X1=2X2+5X3, I would argue that you have not actually overfit. – Michael McGowan May 24 '12 at 20:13
Also, it appears that your answer refers to least-squares regression models, whereas those are only a subset of possibly modeling techniques and the question was agnostic to modeling technique. – Michael McGowan May 24 '12 at 20:14
@MichaelMcGowan I wouldn't argue that point with you. i generally distinguish multicollinearity from overfitting the way you do. I got caught up in explaining what happens when too many variables are in the model and combined them under the umbrella of overfitting. That might have made it easier to explain to the OP. But I didn't like doing it. Apart from the semantics we do agree that mutlicollinearity does not necessarily hurt prediction but overfitting does. – Michael Chernick May 24 '12 at 20:24
@MichaelMcGowan I also agree to be generic I should have said refit the model rather than say refit by least squares. But these software packages that do variable selection using methods such as forward selection, backward selection and stepwise selection do it in the context of least squares linear regression fitting. I am going to edit my answer. – Michael Chernick May 24 '12 at 20:29
up vote 0 down vote

This leads you to regularization or alternatively to cross-validation.

share|improve this answer
3  
Thanks for the reply. I'm aware of regularization as a method of reducing the risk of overfitting, but how does that answer my question? To clarify - I'm not looking for a better method to learn my parameters, but actually am just interested if removing some parameters after the model has been learned can reduce overfitting, and the theoretical justification behind the answer. – dan12345 May 24 '12 at 13:20
2  
What exactly is your answer? Can you explain why you are posting these links? – Macro May 24 '12 at 13:23
Then I don't get it: why to reduce a fitted model but not reduce a model before fitting? – danas.zuokas May 25 '12 at 6:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.