Choosing the Significance Level in Multiple Regression

Question

In this paper on choosing the level of significance, the authors makes an argument for a more nuanced selection of the significance level than simply adhering to the standard 0.05. They argue that the "sample size, power of the test, and expected losses from Type I and II errors" can aid in setting the significance level for a given hypothesis test.

Their arguments seem persuasive to me. In particular, as noted often on this site, since the $p$-value drops as the sample size increases, there seems to be a logical case for setting higher significance levels as the sample size gets larger. (See the example on p. 4 of the article.)

The article provides a reference (Leamer, E. 1978, Specification Searches: Ad Hoc Inference with Nonexperimental Data, Wiley, New York) which contains tables of the critical values of the $F$ Test as a function of sample size, number of parameters, and number of restrictions. A comparison with the choice $\alpha=0.05$ shows the value of a methodological selection of the significance level. These tables however, as far as I can tell, apply to the whole regression, and not the individual regressors.

Hence my question: is there methodology, based on sample size, power, etc., for selecting the significance levels of the regressors in a model?

The concrete problem I am trying to solve is the following. I have a large number of regressors (ca. $7$) and a relatively small sample size (ca. $25$). I would like to leave all my regressors in the model for theoretical reasons. Some of them are highly significant $p \ll 0.001$, others are totally insignificant $p > 0.9$, but most are somewhere in the middle $0.1 < p < 0.2$. I would like to understand if there is a systematic way to go about setting my significance levels based on the setup of my model (sample size, degrees of freedom etc.), instead of simply adopting the conventional thresholds of $0.01$, $0.05$, and $0.1$. References are highly appreciated.

EDIT.

Power analysis was alluded to in the comments; this is definitely along the lines of what I was envisioning. For instance, using the solve_power functionality of statsmodels, if I set power=0.8, I can solve for the significance alpha as follows:

import statsmodels as sm

p_analysis = sm.stats.TTestIndPower()

p_analysis.solve_power(effect_size=2.0,
                       alpha=None,
                       nobs1=7,
                       power=0.8,
                       ratio=(25-7)/7)

 # output: 0.00158

In this case, I have two arbitrary parameters, power and effect_size. I chose $2.0$ for the latter according to Wikipedia's article on Cohen's rule of thumb; I could imagine that I might pick my three thresholds via effect sizes: "huge" ($2.0$), "very large" ($1.2$), "large" ($0.8$). These correspond to $\alpha$'s of $0.0015$, $0.078$, and $0.34$ respectively. But perhaps all am I doing here is trading one rule of thumb ($p < 0.05$) for another...also, as @Dave points out, by making power large, I can make alpha arbitrarily small...

Feedback on this attempt would be appreciated!

Answer 1

I would advance that statistical level thresholds are ultimately really arbitrary and driven more by the requirements of the relevant Journal publishers, peer-reviewers, model validators. Thus, in academia or business often the choice is not subject to the model developer but to the relevant peer-reviewing audience that will validate the model. Within this community some require alpha p-value thresholds of =< 0.10, others =< 0.05. And, that is just the accepted practices. There is not that much reasoning behind it.

If you get a p-value of 0.11, the peer-reviewing community will typically jump all over it for no truly valid reason. A colleague of mine stated to me "is a 89% bet that materially different fro a 90% bet?"

What is probably far more important than statistical threshold cut-offs is the underlying logic and explanatory power of an exogenous variable. In my own modeling effort, I often encounter variables that get selected based on statistical significance threshold criteria, but enter the model with the wrong coefficient signs. Thus, logic and directional explanatory power considerations should always play a preponderant role in variable selection over just considering statistical significance alone.

With 7 variables and a very small sample size of 25 observations, you have very few degrees of freedom left (18). You are hell bent on keeping all 7. I would relax your mandate. I would build a sequence of models with the best first variable, than 2, 3, etc. At each step I would observe how much the Adjusted R Square increase. I would also test your model in-Sample and Out-of-Sample and observe how well your model predicts.

By including all 7 variables you may run into several problems:

1. Well, the more variable you add, often the more you dilute the statistical significance of any of the included variables. A model with 3 variables may disclose very strong statistical significance for all 3 variables. When you add the next 4 variables, you may observe that the statistical significance of the first 3 has dropped markedly. That happens very often;

2. Overfitting. When adding variables your Adjusted R Square typically always increases. Even AIC, BIC scores will improve suggesting that everything is all right. But, it is not necessarily the case. When you conduct some Out-of-Sample testing, your simpler model with fewer variables may perform a lot better in prediction. And, that is what really matters.

3. Multicollinearity or near-multicollinearity. True multicollinear variables are rare. They should have a correlation of 0.90 or above to be multicollinear. In such a case, it be silly to include both within a model since they pretty much impart the same information. But, even variables that have lower correlations than 0.90 can be truly superfluous to a model and impart more Noise than Signal after conducting out-of-Sample testing on such a model.

As a side note, Roberto Pedace, PhD, professor in the Department of Economics, Scripps College, within his excellent book "Econometrics for Dummies" states that it is perfectly ok to have a combination of statistically significant and not-statistically significant variable within an econometrics model, as long as the underlying logic, directional sign, explanatory power of the variables make sense. I fully agree with him. But, I have yet to meet a peer-reviewer with such a nuanced and enlightened approach to variable statistical significance.