I am well aware of the problems of stepwise/forward/backward selection in regression models. There are numerous cases of researchers denouncing the methods and pointing to better alternatives. I was curious if there are any stories that exist where a statistical analysis:

  • has used stepwise regression;
  • made some important conclusions based on the final model
  • the conclusion was wrong, resulting in negative consequences for the individual, their research, or their organisation

My thought on this if stepwise methods are bad, then there should be consequences in the "real world" for using them.

  • 2
    $\begingroup$ If you don't find any such stories, it might be because stepwise regression is mostly used in basic research (or so I perceive). Basic researchers don't usually get in trouble for being wrong, so long as they didn't fake the data or something. $\endgroup$ Aug 19 '16 at 18:21
  • 3
    $\begingroup$ It's used a lot in the industry and in the class room. In research the authors probably would not disclose that they used it. In the industry the main two reasons are that a) those who are doing it were not trained in research, e.g. have undergraduate degrees or b) graduated decades ago. $\endgroup$
    – Aksakal
    Nov 23 '16 at 20:25
  • $\begingroup$ @Aksakal Not learning to begin with but getting a sheep skin anyway is the problem, not elapsed time. Exemplis gratis, me. I took one stats course circa 1971, and first used stats in a publication circa 2006. $\endgroup$
    – Carl
    Nov 23 '16 at 21:06
  • $\begingroup$ Related: Under torture, the data may yield false confessions. Examples? $\endgroup$ Jan 15 '18 at 18:00

There is more than one question being asked. The most narrow one is asking for an example of when stepwise regression has caused harm because it was perfomed stepwise. This is of course true, but can only be established unequivocally when the data used for stepwise regression is also published, and someone reanalyses it and publishes a peer reviewed correction with a published primary authors' retraction. To make accusations in any other context risks legal action, and, if we use a different data set, we could suspect that a mistake was made, but "statistics is never proving anything" and we would not be able to establish that a mistake was made; "beyond a reasonable doubt".

As a point of fact, one frequently gets different results depending on whether one does stepwise elimination or stepwise buildup of a regression equation, which suggest to us that neither approach is sufficiently correct to recommend its usage. Clearly, something else is going on, and that brings us to a broader question, also asked above, but in bullet form, amounting to "What are the problems with stepwise regression, anyhow? That is the more useful question to answer and has the added benefit that I will not have a law suit filed against me for answering it.

Doing it right for stepwise MLR, means using 1) physically correct units (see below), and 2) appropriate variable transformation for best correlations and error distribution type (for homoscedasticity and physicality), and 3) using all permutations of variable combinations, not step-wise, all of them, and 4) if one performs exhaustive regression diagnostics then one avoids missing high VIF (collinearity) variable combinations that would otherwise be misleading, then the reward is better regression.

As promised for #1 above, we next explore the correct units for a physical system. Since good results from regression are contingent upon the correct treatment of variables, we need to be mindful of the usual dimensions of physical units and balance our equations appropriately. Also, for biological applications, an awareness and accounting for the dimensionality of allometric scaling is needed.

Please read this example of physical investigation of a biologic system for how to extend the balancing of units to biology. In that paper, steps 1) through 4) above were followed and a best formula was found using extensive regression analysis namely, $GFR=k∗W^{1/4}V^{2/3}$, where $GFR$ is glomerular filtration rate, a marker of catabolism, where the units are understood using fractal geometry such that $W$, weight was a four dimensional fractal geometric construct, and V, volume, was called a Euclidean, or three dimensional variable. Then $1=\frac{1}{4} \frac{4}{3}+\frac{2}{3}$. So that the formula is dimensionally consistent with metabolism. That is not an easy statement to grasp. Consider that 1) It is generally unappreciated (unknown) that $GFR$ is a marker of metabolism. 2) Fractal geometry is only infrequently taught and the physical interpretation of the formula presented is difficult to grasp even for someone who has mathematical training.

  • 3
    $\begingroup$ This seems to describe a problem with regression in general, rather than stepwise regression specifically. $\endgroup$ Nov 23 '16 at 20:32
  • 2
    $\begingroup$ Yes, these are aspects of regression to consider in general. If I understand correctly where the question is coming from, though, it's motivated by stepwise regression often being denounced in favour of using the likes of LASSO, which wouldn't address the concerns you give here. $\endgroup$ Nov 23 '16 at 21:03
  • 5
    $\begingroup$ I appreciate your frankness and your good will in this matter, Carl. I will not deny that voting has its problems. The only effective way I know of changing the voting on a post is to change the answer--either to improve it technically, expand on it, or to communicate the ideas differently--and even then there's no guarantee it will get the desired response (or even any response at all!). Sometimes, respectful efforts made to understand the downvoters will elicit information that helps everyone appreciate (and upvote) such efforts at improving a post. $\endgroup$
    – whuber
    Nov 23 '16 at 23:30
  • 4
    $\begingroup$ @Carl I'd think that if you're getting regular downvotes the first thing to do is to consider how you might improve your posts (and often you have comments under them that suggest improvements). Speaking for myself, even where I disagree with a commenter, it turns out that they often raise issues that lead to a better answer anyway. I will say that I regularly notice issues with your answers that would nearly move me to downvote it myself. Where I have time to do so, I try to leave a comment. $\endgroup$
    – Glen_b
    Nov 23 '16 at 23:45
  • 4
    $\begingroup$ Note that many of the problems of stepwise regression -- such as issues with estimates biased away from 0, standard errors biased toward 0, nominal type I error rates much lower than actual and a variety of other problems are still present with all-subsets -- indeed, it's an issue with almost any form of optimizing (chapter 4 of Frank Harrell's Regression modeling strategies is a useful reference). Shrinkage/regularization can mitigate some of these issues (especially the tendency of selection to bias estimates outward) and out-of-sample assessment is an important tool for many of them. $\endgroup$
    – Glen_b
    Nov 24 '16 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.