# In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting?

I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1.

My original modelling dataset has 100,000 observations - which I have divided into two samples - a training sample with 80,000 observations and a test sample of 20,000 observations. The samples were created randomly, maintaining the same proportion of 1 to 0 in both the samples (using the SURVEYSELECT procedure in SAS and $y$ as STRATA).

Let us assume that the percentage of observations with the value 1 for $y$ in both the samples is 10%.

I use the LOGISTIC procedure in SAS on the training sample to arrive at variables $x_1$ - $x_5$ which constitute my final model. The p-values associated with the Wald Chi Square are all <0.0001, which indicate that they are significant at the 99.99% confidence level.

However, when I run the LOGISTIC procedure on the test sample, using only $x_1$ - $x_5$ as independent variables, one of them say $x_4$ is no longer significant in the regression outputs - that is, the associated p-value is 0.6.

Does this mean my sampling is not proper? Or the model which I obtain from the training sample 'overfits' the data? Or both?

Should I be worried about this and ensure that the final variables I choose are significant in both the samples or this is not an issue in general?

-
How did you arrive at the variables $x_1$ to $x_5$. I'm guessing it was through some stepwise or all subsets method. –  Peter Flom Jan 17 '13 at 11:34
@PeterFlom Yes, primarily STEPWISE but I also use FORWARD and BACKWARD selection to check which variables are chosen. I also look for multicollinearity using VIF (PROC REG) and generally ensure that the final variables have a value less than 2 (3 at max). –  Mozan Sykol Jan 17 '13 at 11:48
How many variables did you have to begin with? If it was a thousand or so, I would not be surprised that even the nominal 0.01% is not that significant. –  StasK Jan 18 '13 at 12:49

You haven't overfit your model, what you've done is demonstrate (again) that stepwise, forward and backward methods don't work well for this type of task. (Although it was good that you used a training and test set, this let you see that these methods can find things that aren't there).

Model selection is a big topic and has often been discussed, both here and elsewhere. I would generally advise against any automatic variable selection scheme, but if you must use one, I suggest LASSO or LAR. Since you are using SAS, you can find both of these in GLMSELECT. Although this is intended for models that can be fit with PROC GLM, I have have good results using it for logistic models and then testing the resulting models further in LOGISTIC.

-
Thanks, but let's say that I choose the variable using 'judgment' rather than any selection method but only looking at results in the training sample. Later I find that it does not work in the test sample. What is the (statistical) conclusion, other than the fact that my judgment might be bad? –  Mozan Sykol Jan 17 '13 at 13:46
Personally, I am not a fan of deleting important variables from the model just because they are not significant. You can learn from negative results; also, some variables are important as covariates. –  Peter Flom Jan 17 '13 at 19:30
I am accepting this answer as the better of the two - but it still does not answer my question really. What I was looking for is not how to choose my model variables, but once chosen (for argument's sake, let's say given by God) and I am in a situation as I described - then how do I guess what has gone wrong, assuming something has gone wrong, and start correcting my mistakes? In my specific case, I just dropped the offending variables without loss of discriminatory or predictive power. –  Mozan Sykol Jan 23 '13 at 8:18