I want to use stepwise regression to reduce the number of variables. My dependent variable is a dummy variable (Fraud=1, None fraud=0) and I have 25 predictive variables. How can I do this?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
Do not use step-wise regression.
Because step-wise regression almost certainly will insure biased results. All statistics produced through step-wise model building have a nested chain of invisible/unstated "conditional on excluding X" and/or "conditional on including X" statements built into them with the result that:
- p-values are biased
- variances are biased
- parameter estimates are biased
- Coefficients of determination are biased
- false predictors are likely to be included
- true predictors are likely to be excluded
What to use instead of step-wise regression
Use substantive theory to guide which predictor variables to include in your model, and report non-significant findings. If needed you can table only significant results in the main text of an article or report, and include the full model output in an appendix. But step-wise regression is more or less a good way to get consistently unreliable model results.
Some references on the topic
Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66:411–421.
Flom, P. L. and Cassell, D. L. (2007). Stopping stepwise: Why stepwise and similar selection methods are bad, and what you should use.
Henderson, D. A. and Denison, D. R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64:251–257.
Huberty, C. J. (1989). Problems with stepwise methods—better alternatives. Advances in Social Science Methodology, 1:43–70.
Hurvich, C. M. and Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44(3):214–217.
Malek, M. H. and Coburn, D. E. B. J. W. (2007). On the inappropriateness of stepwise regression analysis for model building and testing. European Journal of Applied Physiology, 101(2):263–264.
McIntyre, S. H., Montgomery, D. B., Srinivasan, V., and Weitz, B. A. (1983). Evaluating the statistical significance of models developed by stepwise regression. Journal of Marketing Research, 20(1):1–11.
Pope, P. T. and Webster, J. T. (1972). The use of an $F$-statistic in stepwise regression procedures. Technometrics, 14(2):327–340.
Rencher, A. C. and Pun, F. C. (1980). Inflation of $R^{2}$ in best subset regression. Technometrics, 22(1):49–53.
Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282.
Sribney, B., Harrell, F., and Conroy, R. (2011). Problems with stepwise regression.
Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.
Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4):525–534.
Whittingham, M., Stephens, P., Bradbury, R., and Freckleton, R. (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75(5):1182–1189.
Wilkinson, L. (1979). Tests of significance in stepwise regression. Psychological Bulletin, 86(1):168–174.
-
2$\begingroup$ this is all true, but is it duplicating material that's already elsewhere on CV? $\endgroup$ Commented Aug 24, 2018 at 20:03
-
1$\begingroup$ It is me selecting from answers I have made elsewhere on CV, and incorporating new material. (There's a bunch of variations of questions about step-wise that don't quite feel like duplicate questions to me, although, in my opinion, they share very common answers.) $\endgroup$– AlexisCommented Aug 24, 2018 at 20:04
-
1$\begingroup$ What about using lasso and cross validation ? Apologies but your answer is clear on what not to dot - but unclear on what to do instead! $\endgroup$ Commented Aug 25, 2018 at 9:56
-
$\begingroup$ @XavierBourretSicotte You should probably read the section of my answer titled "What to use instead of step-wise regression" $\endgroup$– AlexisCommented Aug 25, 2018 at 17:07
$\begingroup$
$\endgroup$
4
As @ChrisUmphlett suggests, you can do this by stepwise reduction of a logistic model fit. However, depending on what you're trying to use this for, I would strongly encourage you to read some of the criticisms of stepwise regression on CV first.
There are certain very narrow contexts in which stepwise regression works adequately (e.g. simplifying an existing model for clinical use, or "lazy" picking of a predictive model in cases with a well-powered study, a relatively small number of predictors, and interest only in prediction rather than inference (Murtaugh 2009 Ecology Letters 12:1061-1068)), but there is almost always a better solution.
probably a better solution: penalized logistic regression
set.seed(1001)
X <- matrix(rnorm(25000),nrow=1000,ncol=25)
y <- rbinom(1000,size=1,prob=0.5)
library(glmnet)
ss2 <- glmnet(X,y,family="binomial")
You do have to read and understand a bit more to use penalized regression approaches: e.g. see this quick intro by Trevor Hastie.
still want to do stepwise logistic regression?
dd <- data.frame(y,X)
m1 <- glm(y~.,dd,family=binomial)
ss <- MASS::stepAIC(m1)
answered Aug 24, 2018 at 20:02
-
1$\begingroup$ Not sure I buy the "predictive" argument, since step-wise is about equally likely to include false predictors as true predictors, and about equally likely to exclude false and true predictors. There's a reason why step-wise isn't a cross-validation technique. $\endgroup$– AlexisCommented Aug 24, 2018 at 22:01
-
3$\begingroup$ The argument (which is pretty weak, but Murtaugh makes it) is that in the best-case scenario where you actually have a clean separation between large and small effects (and weakly correlated predictors, etc etc), almost any procedure -- including stepwise regression -- will work OK for building predictive models. $\endgroup$ Commented Aug 24, 2018 at 22:02
-
$\begingroup$ Huh.. I'm not getting that from Murtagh... more of an 'equally poorly' or 'equally meh'... notably with "If there is no "correct" model, there can be no best method of model building," he is more positive about stepwise (and other) model selection algorithms than some of the folks he (and I) have cited. (And thank you!) $\endgroup$– AlexisCommented Aug 24, 2018 at 22:11
-
1$\begingroup$ +1 Regularisation/penalisation is the way to go. I would also add that aside
glmnet, Firth's Bias-Reduced Logistic Regression as provided bylogistfis a very good alternative as it guards from complete separation issue naturally and still provides $p$-values (in case they are needed). $\endgroup$ Commented Aug 25, 2018 at 12:06