1
$\begingroup$

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$).

I would like to perform forward stepwise regression on the data using an appropriate function in R, such as stepAIC from the MASS package.

Does the order in which the data appears in the data frame matter or will we get the same model each time the algorithm is run. By this I mean that I run forward stepwise regression and I get a result. Suppose I shuffle the order of the columns in the data frame and rerun the forward stepwise selection. Will I get a different result with the shuffled data frame?

I have experimented with small data sets and I get the same result when I shuffle the order of the variables, however I am unsure whether this will occur with very large data sets which can take several hours to run on HPC.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ The variables are chosen by various criteria, depending on the form of stepwise regression you use, but none of those criteria depend on the order of the variables. If you're going to select this method, then it will pay to review how it works so you can make wise choices and understand its output. Although this is good general advice for using any statistical procedure, it's particularly appropriate for stepwise regression, as you might have surmised from Dr. Harrell's somewhat acerbic reply. $\endgroup$
    – whuber ♦
    Commented Apr 28, 2021 at 19:15

1 Answer 1

Reset to default
2
$\begingroup$

The results will be equally invalid no matter how your sort the columns. See https://www.stata.com/support/faqs/statistics/stepwise-regression-problems

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the response, and I agree with you that there are issues with stepwise regression as mentioned in the link provided, however will the final model returned from the stepwise procedure always be the same regardless of the order of the variables in the columns? $\endgroup$
    – NM_
    Commented Apr 28, 2021 at 19:11
  • 1
    $\begingroup$ Why use an invalid statistical procedure? Have you bootstrapped the entire process to show how volatile the feature selection procedure really is? $\endgroup$
    – Frank Harrell
    Commented Apr 28, 2021 at 19:13
  • 2
    $\begingroup$ I am just a student working on a small project and I am performing this procedure because this is what I have been asked to do. Unfortunately I don't have a choice in the technique applied, however I will definitely bring up the concerns of stepwise regression with my supervisor. $\endgroup$
    – NM_
    Commented Apr 28, 2021 at 19:19
  • 2
    $\begingroup$ Please do bring up the serious concern, and add an eye-opening simulation of the variable selection process. Here is an example, with code: Section 4.3 of hbiostat.org/doc/rms.pdf . You'll see several other examples in the course notes where the bootstrap was used to try to replicate the variable selection process on a real dataset. $\endgroup$
    – Frank Harrell
    Commented Apr 28, 2021 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.