6
$\begingroup$

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses meant to be tested on new data). Their paper is very interesting but, since the drop1() function belongs to stepwise regression approaches (which are badly looked upon), I was wondering whether this procedure could truly be recommended even in this case?
My guess is that hypotheses generated that way will be shaky, at best. But sometimes, shaky hypotheses may perhaps be acceptable starting points to deepen your investigations on a phenomenon, especially when this phenomenon has seldom been investigated and/or data are extremely costly or difficult to collect.

EDIT: I have deleted the second part of my question because I asked it here.
EDIT 2 (Contextual clarification): Ultimately, my goal is to identify the most influencial variables controlling the biological phenomenon I'm studying. All the variables whose effects I want to explore are theoretically important to explain my response variable (they have already been selected based on my knowledge of the study system), but my sample size is simply too small to include them all as covariates in a single "inferential" model.

$\endgroup$

3 Answers 3

3
$\begingroup$

Exploration means basically, that you can do whatever you want. If there are, among your too many variables, some which are actually true predictors you can hope for the drop functions to find them. You can then gather nuew data to infer, whether these are really actual predictors.

However, gathering new data comes with a cost (or you would have done that before). So the actual question is: Are you ready to gather more data based on nothing more then a stepwise regression approach?

That is not so much a mathematical/statistical question but depends on how costly gathering new data is and how much a positive result would be worth for your further research/career etc.

So, basically, see if you got any better options then stepwise. If not, perform stepwise to reduce the number of candidate predictors. If the result looks really promising, consider to sample new data on those predictors to do inferential statistics on.

$\endgroup$
7
  • 2
    $\begingroup$ Stepwise regression is not reliable for any purpose. If you want to use it, use the bootstrap to expose its unreliability and to put limits on what you can actually learn from it. There are examples in rms of bootstrapping variable importance measures to get confidence intervals on the importance ranking of competing predictors, and also demonstrations of the volatility of which variables are selected. $\endgroup$ Apr 26 at 11:57
  • 2
    $\begingroup$ +1 @Frank This isn't really a question about stepwise regression, so it's unfortunate that terminology is being used to discuss hypothetically including variables or not in one's exploratory examination of data. It's hard to see how one could object to such a process, for otherwise science could never even get started. $\endgroup$
    – whuber
    Apr 26 at 14:00
  • 4
    $\begingroup$ I see; a more appropriate answer would emphasize living within the restrictions imposed by the sample size. In my experience this is best done using data reduction (unsupervised learning) as a first step. This also gets around some problems with collinearity. $\endgroup$ Apr 26 at 15:23
  • $\begingroup$ Thanks Bernhard, @FrankHarrell and @whuber for these answer/comments that seem, however, a bit contradictory. Globally, would you reject/disregard an explicitly exploratory paper that would use the drop1 function to suggest potentially "important" variables for further scrutiny (with a discussion emphasizing caution regarding the method but also discussing the theoretical support - or lack of it - behind these variable suggestions)? Also, any chance to have an answer/opinion regarding the value of an overfitted model for hypotheses generation? Or should I ask a separate question? $\endgroup$
    – Fanfoué
    May 2 at 10:27
  • 3
    $\begingroup$ Methods that are unreliable for formal analysis are unreliable for informal analysis. The probability that stepwise regression chooses the correct variables is virtually zero. $\endgroup$ May 2 at 16:28
2
$\begingroup$

In exploratory analysis you have much more latitude for how you generate the hypotheses of interest, since there is no biasing of tests due to optimisation processes. (Of course, for this to apply you should ensure that when you undertake confirmatory analysis on the hypotheses, you use different data.) Nevertheless, that does not mean that there is no difference in optimality of different kinds of processes that can be used to identify hypotheses of possible interest. So while you can, in theory, "do whatever you want", you probably shouldn't.

Generally speaking, in exploratory analysis you will still want to identify hypotheses that have some evidentiary basis, so that you don't waste time testing lots of false hypotheses in the confirmatory phase. For this reason, it is often useful to have regard to the same types of statistical/evidentiary issues that will arise in confirmatory testing, though for a different reason. The main deficiency of stepwise methods in selection of variables is that it can travel through idiosyncratic paths that miss sets of explanatory variables with high evidence of a relationship to the response variable. This is why comparisons like the all-possible-models method are considered preferable to stepwise methods.

Assuming you have sufficient computational power to do so, I would recommend you conduct exploratory analysis by computing the goodness-of-fit statistics for all possible models and then examining those models that should high levels of fit relative to the number of model parameters. This method is more likely to identify models with true hypotheses, and unlike the stepwise procedure, it is more systematic and will not miss important models. Since this is exploratory analysis, you should also allow yourself to be guided by exogenous concerns about what hypotheses/models are "interesting" in the context of your field, what are the costs of collecting data, etc., but you can use the all-possible-models method to augment this. This latter method will give more systematic statistical information on your exploratory data than stepwise methods.

Finally, you say that hypotheses generated by the stepwise method are "shaky". It is okay for hypotheses generated in the exploratory phase to be shaky, because the whole point is that you are only generating tentative hypotheses for later testing and confirmation. Indeed, arguably all hypotheses generated in the exploratory phase are and ought to be "shaky". The reason to prefer all-possible-models over stepwise methods is that it more systematically identifies hypotheses supported in the explanatory phase.

$\endgroup$
0
$\begingroup$

I think the assertion that stepwise selection is always unreliable is a bit too strong.

I think stepwise selection procedure can probably be amended to remove some of its weaknesses. One example might be to use a different sample from the same population in each step so that you are not inadvertently overfitting the model, and are only picking predictors or features that result in a model that generalises well enough for a purpose.

As with anything, when we are performing regression analysis, I do not think the notion that there is a "correct" model, or that there are "correct" or "incorrect" variables to choose is a useful one. I generally only care about a model's predictive performance, and its ability to describe the associations between the predictive and dependent variables. In fact, in many cases, the number of models that may produce an acceptable "fit" to the data and provide good predictive performance is probably very large, and it might be useful to think rather in terms of a set of models that are acceptable, and a set of models that are not, with the goal of a feature selection / model selection exercise being to identify a model that is in the set of acceptable ones.

But just my thought as a practitioner, rather than as a "scientist".

PS - I think the LASSO method, or other methods that use penalties and shrink coefficients, might well be slightly better founded, but I am not convinced they solve all of the issues with step-wise in any case.

$\endgroup$
3
  • $\begingroup$ In my case, I am ultimately interested in inference and not in prediction. So although I agree that there are probably no single "correct" model, my understanding is that variables that are good to predict my response are not necessarily the same than the ones controlling/explaining the biological phenomenon I'm studying, aren't they? $\endgroup$
    – Fanfoué
    Jun 17 at 9:20
  • $\begingroup$ @Fanfoué If you are looking to determine the variable that control / explain the biological phenomenon, then a model selection procedure might only be a starting point to determine which variables "might" control the response. However, a regression model does not differentiate between causal factors and correlated factors. I think that is the domain of subject matter expertise. In other words, it is up to the expert to choose a set of potential predictors that potentially cause, rather than those that are only correlated to the phenomenon being investigated. $\endgroup$
    – mkone
    Jun 17 at 11:10
  • 1
    $\begingroup$ Let me reassure you, that's what I usually do :) I edited my question to clarify this point. $\endgroup$
    – Fanfoué
    Jun 17 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.