In this paper on choosing the level of significance, the authors makes an argument for a more nuanced selection of the significance level than simply adhering to the standard 0.05. They argue that the "sample size, power of the test, and expected losses from Type I and II errors" can aid in setting the significance level for a given hypothesis test.
Their arguments seem persuasive to me. In particular, as noted often on this site, since the $p$-value drops as the sample size increases, there seems to be a logical case for setting higher significance levels as the sample size gets larger. (See the example on p. 4 of the article.)
The article provides a reference (Leamer, E. 1978, Specification Searches: Ad Hoc Inference with Nonexperimental Data, Wiley, New York) which contains tables of the critical values of the $F$ Test as a function of sample size, number of parameters, and number of restrictions. A comparison with the choice $\alpha=0.05$ shows the value of a methodological selection of the significance level. These tables however, as far as I can tell, apply to the whole regression, and not the individual regressors.
Hence my question: is there methodology, based on sample size, power, etc., for selecting the significance levels of the regressors in a model?
The concrete problem I am trying to solve is the following. I have a large number of regressors (ca. $7$) and a relatively small sample size (ca. $25$). I would like to leave all my regressors in the model for theoretical reasons. Some of them are highly significant $p \ll 0.001$, others are totally insignificant $p > 0.9$, but most are somewhere in the middle $0.1 < p < 0.2$. I would like to understand if there is a systematic way to go about setting my significance levels based on the setup of my model (sample size, degrees of freedom etc.), instead of simply adopting the conventional thresholds of $0.01$, $0.05$, and $0.1$. References are highly appreciated.
Power analysis was alluded to in the comments; this is definitely along the lines of what I was envisioning. For instance, using the
solve_power functionality of
statsmodels, if I set
power=0.8, I can solve for the significance
alpha as follows:
import statsmodels as sm p_analysis = sm.stats.TTestIndPower() p_analysis.solve_power(effect_size=2.0, alpha=None, nobs1=7, power=0.8, ratio=(25-7)/7) # output: 0.00158
In this case, I have two arbitrary parameters,
effect_size. I chose $2.0$ for the latter according to Wikipedia's article on Cohen's rule of thumb; I could imagine that I might pick my three thresholds via effect sizes: "huge" ($2.0$), "very large" ($1.2$), "large" ($0.8$). These correspond to $\alpha$'s of $0.0015$, $0.078$, and $0.34$ respectively. But perhaps all am I doing here is trading one rule of thumb ($p < 0.05$) for another...also, as @Dave points out, by making
power large, I can make
alpha arbitrarily small...
Feedback on this attempt would be appreciated!