0
$\begingroup$

In this paper on choosing the level of significance, the authors makes an argument for a more nuanced selection of the significance level than simply adhering to the standard 0.05. They argue that the "sample size, power of the test, and expected losses from Type I and II errors" can aid in setting the significance level for a given hypothesis test.

Their arguments seem persuasive to me. In particular, as noted often on this site, since the $p$-value drops as the sample size increases, there seems to be a logical case for setting higher significance levels as the sample size gets larger. (See the example on p. 4 of the article.)

The article provides a reference (Leamer, E. 1978, Specification Searches: Ad Hoc Inference with Nonexperimental Data, Wiley, New York) which contains tables of the critical values of the $F$ Test as a function of sample size, number of parameters, and number of restrictions. A comparison with the choice $\alpha=0.05$ shows the value of a methodological selection of the significance level. These tables however, as far as I can tell, apply to the whole regression, and not the individual regressors.

Hence my question: is there methodology, based on sample size, power, etc., for selecting the significance levels of the regressors in a model?

The concrete problem I am trying to solve is the following. I have a large number of regressors (ca. $7$) and a relatively small sample size (ca. $25$). I would like to leave all my regressors in the model for theoretical reasons. Some of them are highly significant $p \ll 0.001$, others are totally insignificant $p > 0.9$, but most are somewhere in the middle $0.1 < p < 0.2$. I would like to understand if there is a systematic way to go about setting my significance levels based on the setup of my model (sample size, degrees of freedom etc.), instead of simply adopting the conventional thresholds of $0.01$, $0.05$, and $0.1$. References are highly appreciated.

EDIT.

Power analysis was alluded to in the comments; this is definitely along the lines of what I was envisioning. For instance, using the solve_power functionality of statsmodels, if I set power=0.8, I can solve for the significance alpha as follows:

import statsmodels as sm

p_analysis = sm.stats.TTestIndPower()

p_analysis.solve_power(effect_size=2.0,
                       alpha=None,
                       nobs1=7,
                       power=0.8,
                       ratio=(25-7)/7)

 # output: 0.00158

In this case, I have two arbitrary parameters, power and effect_size. I chose $2.0$ for the latter according to Wikipedia's article on Cohen's rule of thumb; I could imagine that I might pick my three thresholds via effect sizes: "huge" ($2.0$), "very large" ($1.2$), "large" ($0.8$). These correspond to $\alpha$'s of $0.0015$, $0.078$, and $0.34$ respectively. But perhaps all am I doing here is trading one rule of thumb ($p < 0.05$) for another...also, as @Dave points out, by making power large, I can make alpha arbitrarily small...

Feedback on this attempt would be appreciated!

$\endgroup$
6
  • 1
    $\begingroup$ Re "the p-value drops as the sample size increases:" not for any particular sequence of samples, and not when the null hypothesis is a better model of the data generation process than any of the null hypotheses. $\endgroup$
    – whuber
    Mar 17, 2021 at 21:02
  • 1
    $\begingroup$ Remember that you can get perfect power by setting the significance level to $\alpha=1$. In other words, you can have $100\%$ power if you always reject, but you probably don't want to reject under all circumstances. As you increase $\alpha$ to improve your power, you sacrifice your ability to screen for differences that are really there, in the extreme always flagging a difference as significant. $\endgroup$
    – Dave
    Mar 17, 2021 at 21:03
  • $\begingroup$ After seeing your edit, what do you aim to do with your model? Do you have some kind of binary decision to make based on significance? $\endgroup$
    – Dave
    Mar 17, 2021 at 21:13
  • $\begingroup$ @Dave thanks for your comment--I quite like the idea of power analysis (which I admittedly don't know much about), and was wondering if one could solve for $\alpha$ with a method like this: <statsmodels.org/stable/generated/…> $\endgroup$
    – Anthony
    Mar 17, 2021 at 21:23
  • $\begingroup$ @Dave with regards to the purpose of the model: it's a simple econometric test of the fit of production functions to data. The model itself (judging by the $F$ test) seems to be a good fit, but I would like to be able to say more about the individual regressors. $\endgroup$
    – Anthony
    Mar 17, 2021 at 21:26

1 Answer 1

1
$\begingroup$

I would advance that statistical level thresholds are ultimately really arbitrary and driven more by the requirements of the relevant Journal publishers, peer-reviewers, model validators. Thus, in academia or business often the choice is not subject to the model developer but to the relevant peer-reviewing audience that will validate the model. Within this community some require alpha p-value thresholds of =< 0.10, others =< 0.05. And, that is just the accepted practices. There is not that much reasoning behind it.

If you get a p-value of 0.11, the peer-reviewing community will typically jump all over it for no truly valid reason. A colleague of mine stated to me "is a 89% bet that materially different fro a 90% bet?"

What is probably far more important than statistical threshold cut-offs is the underlying logic and explanatory power of an exogenous variable. In my own modeling effort, I often encounter variables that get selected based on statistical significance threshold criteria, but enter the model with the wrong coefficient signs. Thus, logic and directional explanatory power considerations should always play a preponderant role in variable selection over just considering statistical significance alone.

With 7 variables and a very small sample size of 25 observations, you have very few degrees of freedom left (18). You are hell bent on keeping all 7. I would relax your mandate. I would build a sequence of models with the best first variable, than 2, 3, etc. At each step I would observe how much the Adjusted R Square increase. I would also test your model in-Sample and Out-of-Sample and observe how well your model predicts.

By including all 7 variables you may run into several problems:

  1. Well, the more variable you add, often the more you dilute the statistical significance of any of the included variables. A model with 3 variables may disclose very strong statistical significance for all 3 variables. When you add the next 4 variables, you may observe that the statistical significance of the first 3 has dropped markedly. That happens very often;

  2. Overfitting. When adding variables your Adjusted R Square typically always increases. Even AIC, BIC scores will improve suggesting that everything is all right. But, it is not necessarily the case. When you conduct some Out-of-Sample testing, your simpler model with fewer variables may perform a lot better in prediction. And, that is what really matters.

  3. Multicollinearity or near-multicollinearity. True multicollinear variables are rare. They should have a correlation of 0.90 or above to be multicollinear. In such a case, it be silly to include both within a model since they pretty much impart the same information. But, even variables that have lower correlations than 0.90 can be truly superfluous to a model and impart more Noise than Signal after conducting out-of-Sample testing on such a model.

As a side note, Roberto Pedace, PhD, professor in the Department of Economics, Scripps College, within his excellent book "Econometrics for Dummies" states that it is perfectly ok to have a combination of statistically significant and not-statistically significant variable within an econometrics model, as long as the underlying logic, directional sign, explanatory power of the variables make sense. I fully agree with him. But, I have yet to meet a peer-reviewer with such a nuanced and enlightened approach to variable statistical significance.

$\endgroup$
3
  • $\begingroup$ Thank you for the thoughtful answer. "Is a 89% bet that materially different fro a 90% bet?" is precisely what bothered my about having to choose significance levels, and my thought was to at least find a methodology to be consistent about it. $\endgroup$
    – Anthony
    Mar 17, 2021 at 23:28
  • 1
    $\begingroup$ One aspect of the borderline cases that I find interesting is the possibility to get flipped around when it comes to rejecting/not. If you get $p=0.101$, “Well it’s so close to $0.100$ that we should reject,” yet if you get a lower p-value in $0.099$, “ Golly, that’s too close for me to really want to reject!” $\endgroup$
    – Dave
    Mar 17, 2021 at 23:52
  • 1
    $\begingroup$ As you both indicated, that stuff is so arbitrary and discretionary. And, if you attempt to overlay some sort of rigorous methodology to arrive at a selection decision you just make it even more dogmatic. I think that's why people often go with whatever their model peer-reviewer will demand (typically =< 0.10 or 0.05). Maybe what is more important is to figure out what is the consequence of making a Type II vs. Type I error. I think in your model you have no consequences. But, imagine you are dealing with a vaccine clinical trial. There statistical significance becomes more pressing. $\endgroup$
    – Sympa
    Mar 19, 2021 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.