In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting? I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1.
My original modelling dataset has 100,000 observations - which I have divided into two samples - a training sample with 80,000 observations and a test sample of 20,000 observations. The samples were created randomly, maintaining the same proportion of 1 to 0 in both the samples (using the SURVEYSELECT procedure in SAS and $y$ as STRATA).
Let us assume that the percentage of observations with the value 1 for $y$ in both the samples is 10%.
I use the LOGISTIC procedure in SAS on the training sample to arrive at variables $x_1$ - $x_5$ which constitute my final model. The p-values associated with the Wald Chi Square are all <0.0001, which indicate that they are significant at the 99.99% confidence level.
However, when I run the LOGISTIC procedure on the test sample, using only $x_1$ - $x_5$ as independent variables, one of them say $x_4$ is no longer significant in the regression outputs - that is, the associated p-value is 0.6.
Does this mean my sampling is not proper? Or the model which I obtain from the training sample 'overfits' the data? Or both?
Should I be worried about this and ensure that the final variables I choose are significant in both the samples or this is not an issue in general?
 A: You haven't overfit your model, what you've done is demonstrate (again) that stepwise, forward and backward methods don't work well for this type of task. (Although it was good that you used a training and test set, this let you see that these methods can find things that aren't there).
Model selection is a big topic and has often been discussed, both here and elsewhere. I would generally advise against any automatic variable selection scheme, but if you must use one, I suggest LASSO or LAR. Since you are using SAS, you can find both of these in GLMSELECT. Although this is intended for models that can be fit with PROC GLM, I have have good results using it for logistic models and then testing the resulting models further in LOGISTIC. 
A: It can mean too many things, it may mean your data miss critical observations, it may mean that the problem you are facing is not as solvable as you thought  (e.g. trying to fit a nonlinear determinstic model to some iid data stream), and yes, sure, for the case you demonstrated, it can also mean your model is over-, or more properlly speaking, ill-fitted. 
Black-box modelling is always hard, ill-fitting or overfitting is a common outcome when one try to blindly fit some randomly-chosen, standard model structures to data when they have basically no idea of the phyiscs of the practical system. For many practical problems one has to has some decent grasp of the underlining physics of the system one try to model to develop a proper model, there is not a "default" method or standardized routine for modelling and no short-cuts, and thats why experienced modellers/data miners are all very highly rewarded in industry.
Edited: As for lasso method, personally I dont believe lasso method is that useful, if the problem can be solved by lasso, it certainly can be solved by many other regression methods, althrough at various degrees, the OP's problem (e.g. with 20000 randomly picked test samples) is not likely to be the case.
