Binary Logit: A statistically insignificant predictor become significant after exclude some observation

Question

I am currently trying to explore the determinants of management turnover using binary logit.

In robustness check, I have added more control variable (Industry) to see whether the same results hold. Surprisingly, almost all effects stay the same (economically and statistically), but a statistically insignificant predictor become significant. But, the sample size is reduced by almost 20% (From 1000 to ~800) due to missing industry info in some observations. I then run a regression with the same sample size of ~800 and leaving the newly included control variables. The result is the same with the robustness test. I suspect that this could be due to reduction in observations. Could you guide me on the explanation or provide a link on why would reducing sample sizes make the previously insignificant predictor significant?

Thanks a lot in advance.

Answer 1

There are two pretty plausible explanation to my mind (and both might play a role at the same time):

1. Statistical significance is a random variable, sampling a different set of observations/subsetting differently/changing something about a model will lead to a different realization for this random variable. This is very often the explanation for "I changed something, something else is now significant/something is no longer significant". Once this is clear, it is also easily understood, why statistical significance - particularly with p-values in the 0.005 to 0.05 (and similarly non-significant p-values just over 0.05) should really not be overinterpreted (an never as "indicating truth").
2. It is possible that the observations for which some predictor is available are systematically different from those, for which they are not available. If that is the case listwise deletion (omitting the records with missing covariates) is simply an invalid analysis method, unless you want to specifically say something only for the type of observations, for which the information is available. More appropriate analysis approaches might rely on an appropriate imputation of the missing data.