Over-fitting in the cross-validation I have a response variable(y) and 20 independent variables (Xs). I want to select several Xs in the linear regression, but I'm not sure how many variables should be selected. To select the best number of variables, I use the sum of the squared residuals (Res) in the 10-fold cross-validation given N selected variables (N=2~20). The process is repeated 1000 times given each N. My idea is that Res should firstly decrease as more variable could explain y better and then it should increase as too many variables should lead to over-fitting. To my surprise, Res decrease continually as N increase(see the Figure). I don't know how to explain it.  Is it mean that all 20 variables contribute to y, or over-fitting happened?
P.S.: there are about 600 data points. The Res is calculated as the sum of the square of the difference between observed y and predicted y in each 10-fold cross-validation.

 A: That's hard to say for certain. I see two possibilites:


*

*you calculate your residuals incorrectly, either using all the data or (worst case) just the 90% you used to train your model

*all the variables deliver some information. 20 variables with 600 data points is in a range that linear regression can handle, at least if those are binary variables or numeric data.

*most of the variables deliver information and you end up lucky with the few that don't. If you don't overfit (unlikely with the data/variables ratio), you have an about even chance for the residuals of the 10% test data to decrease even if there's no real relationship.


Some suggestions:


*

*Use some sort of penalized regression with inbuilt feature selection (i.e. lasso, elastic net) and compare the results

*Look at the distribution of the sum of the residuals in each of the 10 folds of the cross validation. Does it always decrease? Or just in 7 of 10 cases? 

*Made certain that you correct the sum of the correct residuals

*Look critically at how you select what feature to include. Make certain that you always just the information from the 90% of the data that is your training set in that fold of the cross validation scheme

*Generating a number of variables containing just random Gaussian noise might help to see what happens. You will know for certain that these should not help predicting anything.

A: You could try a conservative selection method, such as bic.  if bic favours the full model then it is highly unlikely that you have overfitted in cross validation.  To do a stepwise or backwards style bic ( which is fast) you set the pvalue significance level to $Pr(\chi^2_1>\log[n])$.  So if you fit the full model and all your t statistics are greater than $\sqrt{\log[n]}$ then it is likely that all 20 variables are important.  For your case this is roughly $|T|>2.5$.
