3
$\begingroup$

In statistical software like MINITAB and SAS, the default alpha value (critical p-value) is set as 0.15. I would like to know (1) if there is any statistic basis to set it as 0.15 and (2) if this is a common accepted value in statistics.

I have found that in this question, the critical p-value of 0.15 in R is actually based on AIC. So is AIC the basis to set 0.15 as default value?

Thanks in advance for your help!

$\endgroup$
7
$\begingroup$

The basis is cargo cult science. "$p$-values" produced from stepwise regression model building do not have the typical meaning of "probability of observing the estimate/test statistic assuming the null hypothesis is true," but rather "probability of observing the estimate/test statistic based on a series of unstated conditionals that are almost certainly predicated on some number of variables not included in the presented model."

Use stepwise regression if you want to appear to be performing meaningful statistical analysis, while providing results (estimates, "noise" variables, missing "real" variables, $p$-values, $R^{2}$, etc.) that are very likely to be biased.

$\endgroup$
3
  • 1
    $\begingroup$ If you don't care about the accuracy of p values for quantifying type I error, and your data set and full model are both medium-sized and well-behaved, you may be able to get a decent predictive model out of a stepwise regression, but I wouldn't trust it any more than that: Murtaugh, Paul A. 2009. “Performance of Several Variable-Selection Methods Applied to Real Ecological Data.” Ecology Letters 12 (10): 1061–68. doi:10.1111/j.1461-0248.2009.01361.x. $\endgroup$
    – Ben Bolker
    Oct 9 '14 at 15:58
  • 3
    $\begingroup$ I am currently working with a land-use regression on data my colleagues and I collected here in Portland. I am getting substantively better predictions using theory and cross-validation to construct my LUR model than my colleague's standard stepwise-model building approach. I would rather have a meaningful, translatable and predictive model than a "decently predictive" model. :) $\endgroup$
    – Alexis
    Oct 9 '14 at 16:31
  • $\begingroup$ I agree. Just giving the counterpoint. $\endgroup$
    – Ben Bolker
    Oct 9 '14 at 17:10
6
$\begingroup$

Stepwise variable selection causes serious bias when the $\alpha$ level for retaining variables is below 1.0 (and especially if it is below about 0.3). You are not addressing this bias in the $\beta$s. You need to incorporate penalization if using stepwise selection, or use the bootstrap to describe the disaster.

$\endgroup$
2
  • $\begingroup$ Thanks! Could you elaborate more about the penalization? $\endgroup$ Oct 10 '14 at 13:06
  • $\begingroup$ Look up L1 and L2 norm penalized estimation (lasso and ridge regression). Ridge regression = quadratic penalty. $\endgroup$ Oct 10 '14 at 16:24
5
$\begingroup$

In statistical software like MINITAB and SAS, the default alpha value (critical p-value) is set as 0.15. I would like to know (1) if there is any statistic basis to set it as 0.15

Not really. To my vague recollection (I have no reference to offer), it was originally just presented as a value that seemed to work fairly well, because 0.05 was too strict.

I have found that in this question, the critical p-value of 0.15 in R is actually based on AIC. So is AIC the basis to set 0.15 as default value?

I think the suggestion preceded wide awareness of the AIC (but it's possible that the idea to use it originally came via the AIC, it's nearly old enough). So I think it's more coincidental (though that "a value that seems to work fairly well" would come up close to AIC is hardly surprising).

and (2) if this is a common accepted value in statistics.

It was fairly widely accepted once (many decades ago). The whole idea (no matter what cutoff value is used) is now well understood to be pretty bad.

If I dig up any more precise information, I'll come back and add it.

$\endgroup$
3
$\begingroup$

See p195 of "Clinical Prediction Models" by EW Steyerberg. To select a candidate predictor based on AIC, the chi-squared value has to exceed twice its df. The P-value for a 1 df chi-squared that justifies the model change is 0.157 (which is where I think the 0.15 comes from). However the p value for a 2 df model (eg if you wanted to see if a nominal variable with 3 categories should be added) is 0.135. Other words the 0.15 criterion only applies to 1df changes. Stepwise selection of models using AIC adapts the critical p-value in an implicit way and using higher p-values allows inclusion of predictors with weaker effects, although the best results use model averaging. The AIC is also asymptotically equivalent to cross-validation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.