What p-value threshold should I use for many-variable probit? I have a probit with 7,000 observations and 120 dummy independent variables, of which Stata omits 31 because of low n and 2 because of collinearity. I also have a set of five dummies to make a time variable, plus three continuous variables. One of my variables of interest comes out with p=.000 -- success! But a few others have p-values of .022, .033... It's easiest to just go with .05 as my threshold, but a colleague suggested I should go with a lower threshold because I have so many variables. Is she right, and if so, how do I figure out what threshold to use?
 A: The purpose of adjusting the significance level is to make sure that the Type I error, i.e. the probability of rejecting the null hypothesis when it is true, does not exceed a given level, usually the 0.05 threshold.
The simplest method to achieve this is through the use of the so-called Bonferroni correction. Suppose then that your goal is to control the type I error at 0.05. What this essentially says is that, if you have the p-values of a number of tests, $P_1,P_2,\ldots,P_n$, then you reject each null hypothesis if and only if
$$P_i \leq \frac{0.05}{n}$$
That is, you divide the significance level by the number of individual tests you look at. I bet this is the reason your colleague suggested a lower probability. It is worth mentioning that the correction is applicable no matter whether the tests are indpependent or not-and in your setting they are definitely not.  
The simplicity of this procedure is undoubtedly an asset but it does not come without drawbacks. The most important one is that the procedure might end up being highly conservative. Put simply, the achieved type I error might be much lower than the 0.05 you were intending. 
For this reason, I would advise you to use a global test instead, a Likelihood Ratio Test in the case of a Generalized Linear Model. This test will ensure that you have a sharp 0.05 significance level and more importantly, since it takes into account the covariance structure of the model, it will also be more powerful, i.e. it will reject the null hypothesis with higher probability when it is false.
I have no idea how to implement an LRT in Stata but perhaps you can look that up. You can probably carry it out using the Deviance Statistic too.
Hope this helps.
