9
$\begingroup$

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses meant to be tested on new data). Their paper is very interesting but, since the drop1() function belongs to stepwise regression approaches (which are badly looked upon), I was wondering whether this procedure could truly be recommended even in this case?
My guess is that hypotheses generated that way will be shaky, at best. But sometimes, shaky hypotheses may perhaps be acceptable starting points to deepen your investigations on a phenomenon, especially when this phenomenon has seldom been investigated and/or data are extremely costly or difficult to collect.

EDIT: I have deleted the second part of my question because I asked it here.
EDIT 2 (Contextual clarification): Ultimately, my goal is to identify the most influencial variables controlling the biological phenomenon I'm studying. All the variables whose effects I want to explore are theoretically important to explain my response variable (they have already been selected based on my knowledge of the study system), but my sample size is simply too small to include them all as covariates in a single "inferential" model.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
6
$\begingroup$

In exploratory analysis you have much more latitude for how you generate the hypotheses of interest, since there is no biasing of tests due to optimisation processes. (Of course, for this to apply you should ensure that when you undertake confirmatory analysis on the hypotheses, you use different data.) Nevertheless, that does not mean that there is no difference in optimality of different kinds of processes that can be used to identify hypotheses of possible interest. So while you can, in theory, "do whatever you want", you probably shouldn't.

Generally speaking, in exploratory analysis you will still want to identify hypotheses that have some evidentiary basis, so that you don't waste time testing lots of false hypotheses in the confirmatory phase. For this reason, it is often useful to have regard to the same types of statistical/evidentiary issues that will arise in confirmatory testing, though for a different reason. The main deficiency of stepwise methods in selection of variables is that it can travel through idiosyncratic paths that miss sets of explanatory variables with high evidence of a relationship to the response variable. This is why comparisons like the all-possible-models method are considered preferable to stepwise methods.

Assuming you have sufficient computational power to do so, I would recommend you conduct exploratory analysis by computing the goodness-of-fit statistics for all possible models and then examining those models that yield high levels of fit relative to the number of model parameters. This method is more likely to identify models with true hypotheses, and unlike the stepwise procedure, it is more systematic and will not miss important models. Since this is exploratory analysis, you should also allow yourself to be guided by exogenous concerns about what hypotheses/models are "interesting" in the context of your field, what are the costs of collecting data, etc., but you can use the all-possible-models method to augment this. This latter method will give more systematic statistical information on your exploratory data than stepwise methods.

Finally, you say that hypotheses generated by the stepwise method are "shaky". It is okay for hypotheses generated in the exploratory phase to be shaky, because the whole point is that you are only generating tentative hypotheses for later testing and confirmation. Indeed, arguably all hypotheses generated in the exploratory phase are and ought to be "shaky". The reason to prefer all-possible-models over stepwise methods is that it more systematically identifies hypotheses supported in the explanatory phase, which makes it a bit less likely that you will run down rabbit-holes in the confirmatory phase pursuing false hypotheses.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ While all seems fine here, there is a bit of a trade-off between all subsets and stepwise. All subsets will generate more "false positives" than stepwise because all subsets considers $2^p$ models while stepwise considers approximately $\frac{p^2}{2}$. With more false positives as a starting point, chances are some of them will not get caught in the confirmatory stage. For $p=10$, if we try $2^{10}$ false positives at 5% significance level in a confirmatory stage, approximately 51 will slip through. If we try only $\frac{10^2}{2}$, approximately 2-3 will slip through. $\endgroup$
    – Richard Hardy
    Commented May 8, 2023 at 17:13
  • 1
    $\begingroup$ The particular example above has only false positives and no true positives, but the general idea goes through in the case with one or more true positives, too. A somewhat related working paper that I like is P.R.Hansen's "A Winnerís Curse for Econometric Models". $\endgroup$
    – Richard Hardy
    Commented May 8, 2023 at 17:16
3
$\begingroup$

Exploration means basically, that you can do whatever you want. If there are, among your too many variables, some which are actually true predictors you can hope for the drop functions to find them. You can then gather nuew data to infer, whether these are really actual predictors.

However, gathering new data comes with a cost (or you would have done that before). So the actual question is: Are you ready to gather more data based on nothing more then a stepwise regression approach?

That is not so much a mathematical/statistical question but depends on how costly gathering new data is and how much a positive result would be worth for your further research/career etc.

So, basically, see if you got any better options then stepwise. If not, perform stepwise to reduce the number of candidate predictors. If the result looks really promising, consider to sample new data on those predictors to do inferential statistics on.

Improve this answer
$\endgroup$
9
  • 4
    $\begingroup$ Stepwise regression is not reliable for any purpose. If you want to use it, use the bootstrap to expose its unreliability and to put limits on what you can actually learn from it. There are examples in rms of bootstrapping variable importance measures to get confidence intervals on the importance ranking of competing predictors, and also demonstrations of the volatility of which variables are selected. $\endgroup$
    – Frank Harrell
    Commented Apr 26, 2022 at 11:57
  • 4
    $\begingroup$ +1 @Frank This isn't really a question about stepwise regression, so it's unfortunate that terminology is being used to discuss hypothetically including variables or not in one's exploratory examination of data. It's hard to see how one could object to such a process, for otherwise science could never even get started. $\endgroup$
    – whuber ♦
    Commented Apr 26, 2022 at 14:00
  • 5
    $\begingroup$ I see; a more appropriate answer would emphasize living within the restrictions imposed by the sample size. In my experience this is best done using data reduction (unsupervised learning) as a first step. This also gets around some problems with collinearity. $\endgroup$
    – Frank Harrell
    Commented Apr 26, 2022 at 15:23
  • 4
    $\begingroup$ Methods that are unreliable for formal analysis are unreliable for informal analysis. The probability that stepwise regression chooses the correct variables is virtually zero. $\endgroup$
    – Frank Harrell
    Commented May 2, 2022 at 16:28
  • 1
    $\begingroup$ I need to be convinced. One of the settings where stepwise regression fails is when potential predictors are not separable (collinearities are strong). When that happens the result will be somewhat of a random draw of variables, and a more stable and interpretable answer would arise if data reduction were done before stepwise regression (if it's even still needed after the unsupervised learning step). $\endgroup$
    – Frank Harrell
    Commented May 9, 2023 at 11:25
0
$\begingroup$

I think the assertion that stepwise selection is always unreliable is a bit too strong.

I think stepwise selection procedure can probably be amended to remove some of its weaknesses. One example might be to use a different sample from the same population in each step so that you are not inadvertently overfitting the model, and are only picking predictors or features that result in a model that generalises well enough for a purpose.

As with anything, when we are performing regression analysis, I do not think the notion that there is a "correct" model, or that there are "correct" or "incorrect" variables to choose is a useful one. I generally only care about a model's predictive performance, and its ability to describe the associations between the predictive and dependent variables. In fact, in many cases, the number of models that may produce an acceptable "fit" to the data and provide good predictive performance is probably very large, and it might be useful to think rather in terms of a set of models that are acceptable, and a set of models that are not, with the goal of a feature selection / model selection exercise being to identify a model that is in the set of acceptable ones.

But just my thought as a practitioner, rather than as a "scientist".

PS - I think the LASSO method, or other methods that use penalties and shrink coefficients, might well be slightly better founded, but I am not convinced they solve all of the issues with step-wise in any case.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In my case, I am ultimately interested in inference and not in prediction. So although I agree that there are probably no single "correct" model, my understanding is that variables that are good to predict my response are not necessarily the same than the ones controlling/explaining the biological phenomenon I'm studying, aren't they? $\endgroup$
    – Fanfoué
    Commented Jun 17, 2022 at 9:20
  • $\begingroup$ @Fanfoué If you are looking to determine the variable that control / explain the biological phenomenon, then a model selection procedure might only be a starting point to determine which variables "might" control the response. However, a regression model does not differentiate between causal factors and correlated factors. I think that is the domain of subject matter expertise. In other words, it is up to the expert to choose a set of potential predictors that potentially cause, rather than those that are only correlated to the phenomenon being investigated. $\endgroup$
    – mkone
    Commented Jun 17, 2022 at 11:10
  • 1
    $\begingroup$ Let me reassure you, that's what I usually do :) I edited my question to clarify this point. $\endgroup$
    – Fanfoué
    Commented Jun 17, 2022 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.