I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1.
My original modelling dataset has 100,000 observations - which I have divided into two samples - a training sample with 80,000 observations and a test sample of 20,000 observations. The samples were created randomly, maintaining the same proportion of 1 to 0 in both the samples (using the SURVEYSELECT procedure in SAS and $y$ as STRATA).
Let us assume that the percentage of observations with the value 1 for $y$ in both the samples is 10%.
I use the LOGISTIC procedure in SAS on the training sample to arrive at variables $x_1$ - $x_5$ which constitute my final model. The p-values associated with the Wald Chi Square are all <0.0001, which indicate that they are significant at the 99.99% confidence level.
However, when I run the LOGISTIC procedure on the test sample, using only $x_1$ - $x_5$ as independent variables, one of them say $x_4$ is no longer significant in the regression outputs - that is, the associated p-value is 0.6.
Does this mean my sampling is not proper? Or the model which I obtain from the training sample 'overfits' the data? Or both?
Should I be worried about this and ensure that the final variables I choose are significant in both the samples or this is not an issue in general?