4
$\begingroup$

I have a response variable(y) and 20 independent variables (Xs). I want to select several Xs in the linear regression, but I'm not sure how many variables should be selected. To select the best number of variables, I use the sum of the squared residuals (Res) in the 10-fold cross-validation given N selected variables (N=2~20). The process is repeated 1000 times given each N. My idea is that Res should firstly decrease as more variable could explain y better and then it should increase as too many variables should lead to over-fitting. To my surprise, Res decrease continually as N increase(see the Figure). I don't know how to explain it. Is it mean that all 20 variables contribute to y, or over-fitting happened?

P.S.: there are about 600 data points. The Res is calculated as the sum of the square of the difference between observed y and predicted y in each 10-fold cross-validation.

enter image description here

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Do you average out the Res from each cross-validation run? Do you actually calculate Res from the residuals of the hold-out samples? $\endgroup$
    – mpiktas
    Commented May 2, 2012 at 10:36
  • $\begingroup$ Try running MC simulation where you control the data to see whether this behaviour is general. $\endgroup$
    – mpiktas
    Commented May 2, 2012 at 10:37
  • $\begingroup$ mpiktas: I sumed the square of residuals of the hold-out samples in each fold. Then, I divide the sum by the number of data points,i.e.,600. $\endgroup$
    – friendpine
    Commented May 3, 2012 at 2:41
  • $\begingroup$ While it would not change the picture much, would it not make more sense to divide by the number of points in the hold-out sample? The benefit of more variables seem to flatten out to the right... what happens if you do the same plot for the rest of the variables? $\endgroup$
    – Erik
    Commented May 3, 2012 at 15:00

2 Answers 2

Reset to default
3
$\begingroup$

That's hard to say for certain. I see two possibilites:

  1. you calculate your residuals incorrectly, either using all the data or (worst case) just the 90% you used to train your model
  2. all the variables deliver some information. 20 variables with 600 data points is in a range that linear regression can handle, at least if those are binary variables or numeric data.
  3. most of the variables deliver information and you end up lucky with the few that don't. If you don't overfit (unlikely with the data/variables ratio), you have an about even chance for the residuals of the 10% test data to decrease even if there's no real relationship.

Some suggestions:

  • Use some sort of penalized regression with inbuilt feature selection (i.e. lasso, elastic net) and compare the results
  • Look at the distribution of the sum of the residuals in each of the 10 folds of the cross validation. Does it always decrease? Or just in 7 of 10 cases?
  • Made certain that you correct the sum of the correct residuals
  • Look critically at how you select what feature to include. Make certain that you always just the information from the 90% of the data that is your training set in that fold of the cross validation scheme
  • Generating a number of variables containing just random Gaussian noise might help to see what happens. You will know for certain that these should not help predicting anything.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Erik: I have double check my code and the prediction result in the 10-fold cross-validation. There is no problem with the calculation of the residuals. $\endgroup$
    – friendpine
    Commented May 3, 2012 at 2:43
  • $\begingroup$ Erik: There are median correlation(PCC=0.5-0.7) between the Xs. Therefore, I don't think all the variables should contribute to the Y. I suspect there are over-fitting.But I don't know why this happens. Does over-fitting also happen in the Cross-validation?? $\endgroup$
    – friendpine
    Commented May 3, 2012 at 2:49
3
$\begingroup$

You could try a conservative selection method, such as bic. if bic favours the full model then it is highly unlikely that you have overfitted in cross validation. To do a stepwise or backwards style bic ( which is fast) you set the pvalue significance level to $Pr(\chi^2_1>\log[n])$. So if you fit the full model and all your t statistics are greater than $\sqrt{\log[n]}$ then it is likely that all 20 variables are important. For your case this is roughly $|T|>2.5$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.