5
$\begingroup$

I have used the MATLAB regression learner application to do some stepwise regression with a 10-fold cross validation for feature selection. But now I want to code it myself and I'm confused about the algorithm!

So I divide my data into 10-folds and then train my model using 9-folds and test in on 1 remaining fold. I do it for 10 times. This gives me the performance of 10 linear regression models with different feature subsets. The regular way is to take the average of the 10 performance metrics (RMSE, MSE, R2,...) and decide which model performs the best.

But in my case the model is the same and the features are different. So I'm confused how to choose the best subset. Should I just compare the 10 performance metrics and choose the subset with the highest performance?

$\endgroup$
3
  • $\begingroup$ Do you do ten stepwise variable selection methods and wind up with ten different sets of features? $\endgroup$
    – Dave
    Commented Mar 21, 2023 at 17:34
  • $\begingroup$ yes and I guess that's wrong, right? $\endgroup$
    – Azarang
    Commented Mar 21, 2023 at 17:41
  • $\begingroup$ I can use CV to find cut-off for F in and out for stepwise regression not feature selection. I can use CV for feature selection comparing stepwise and relief for example right? $\endgroup$
    – Azarang
    Commented Mar 21, 2023 at 17:47

3 Answers 3

11
$\begingroup$

Running a single cross-validation loop yields an estimate of the out of sample predictive error associated with your modeling procedure, nothing more. You have 10 different models because stepwise selection is unstable, as @Dave explains. There is no reason to believe that any of your 10 models is 'right', but the mean of the cross-validation prediction error gives you an estimate of how large the prediction error will be in the future. At this point, you would run your procedure over the full dataset and use that as the final model. In general, I would advise against this, but that would be the protocol.

If you want to use cross-validation to determine the $F$-value to use as a cutoff for your modeling procedure, you need to do more. In that case, you would use a nested cross-validation scheme. In the outer loop, you would partition the data into $k$ folds and set one aside. Then you would perform another cross-validation loop on the remaining folds. In the inner loop, you would use some means to search over possible $F$-values; for instance, you could use a grid search over a series of possible $F$ cutoffs. For each possible $F$, there would be a average out of sample predictive accuracy score. You would take the cutoff that performed best and use it on the entire (nested) dataset to get a model. That model would be used to make predictions on the top level set that had been set aside, and from that you would get an estimate of the out of sample performance of a model that is selected in this manner. Then you would set the second fold aside, and perform the inner loop cross-validation and $F$ cutoff selection again, etc. After having done all this $k$ times, you could average those and get an average estimate of the out of sample performance of models selected in this manner. After that, you can repeat the search procedure that you had used in the inner loop on the outer loop alone (i.e., there wouldn't be an inner loop this time). That will give the model slightly more data to work with to select your final cutoff. Finally, you would fit your intended model using that cutoff on the whole dataset to get your final model, and you would have an estimate of how well a model of that type, selected in that manner, will perform out of sample. In short, the larger protocol is this:

  1. Run nested cross-validation, selecting a cutoff on the inner loop and then using it in the outer loop, to get an estimate of out of sample performance.
  2. Run cross-validation to select the cutoff to be used for the final model.
  3. Fit your model to the full dataset using the cutoff selected.

To get more detail, try reading: Nested cross validation for model selection and Training on the full dataset after cross-validation?. Again, I wouldn't recommend you use stepwise selection, even in this case because the parameters will still be biased (and the constituent hypothesis tests will still be garbage), but the out of sample estimate of the predictive performance of a model fitted in this manner should be OK.

$\endgroup$
0
8
$\begingroup$

Welcome to the instability of feature selection. This is totally predictable behavior and one of the reasons why stepwise regression is less of a panacea than it first seems to be. Sure, you select some variables that work well on the training data, and by limiting the variable count to just those that influence the outcome the most, you seem to restrict the opportunity for overfitting, right?

Unfortunately, you put yourself at risk of the variable selection overfitting to the training data. As you can see from your cross validation, just because a set of variables works on one sample does not assure it of working on another. That is, the feature selection is unstable, and with the selected features bouncing all over the place as you make changes to the data (which will be the case when you go predict on new data), there is justifiable doubt that the variables selected based on the training data will be the right variables for making predictions on new data.

If you want to use your model just to predict, then you might be better off bootstrapping the entire dataset, fitting a stepwise model to the bootstrap sample, applying that model to the entire data set, and seeing by how much the performance (on some metric of interest, say MSE or MAE) differs. This is related to the procedure I discuss here. If that is an acceptable amount, you have evidence that the overall stepwise procedure is effective, which can be the case for stepwise regression in pure prediction problems.

If you want to use the stepwise regression to select variables on which you do inferences like p-values or confidence intervals, all of these downstream inferences are distorted by the stepwise selection. While this link mentions Stata software, the theory does not care if you use Stata, MATLAB, Python, R, SAS, or any other software, and the previous sentence relates to points 2, 3, 4, and 7. Briefly, by doing the stepwise regression and then calculating statistics as if you have not, you are performing dishonest calculations that fail to account for the variable selection process.

$\endgroup$
5
  • 1
    $\begingroup$ there is no reason to think that the variables selected based on the training data are the right variables for making predictions on new data. This is way too harsh. If not, then there is no reason to believe anything based on training data, and we can trash all of statistics and machine learning... $\endgroup$ Commented Mar 22, 2023 at 9:33
  • $\begingroup$ @RichardHardy The claim is not that training means nothing to new data. The claim is that instability during training casts doubt, with good reason, on what follows. This is not specific to feature selection. If I did any kind of repeated testing (cross validation, bootstrap) and got results that were all over the place, I would be skeptical about how the modeling would perform moving forward. $\endgroup$
    – Dave
    Commented Mar 22, 2023 at 13:26
  • 2
    $\begingroup$ To me, there is a big difference between casts doubt and there is no reason to think, and I cannot help but interpret your statement the way I outlined in my first comment. As currently phrased, I find it more misleading than useful. $\endgroup$ Commented Mar 22, 2023 at 13:28
  • $\begingroup$ @RichardHardy I have made an edit that I hope softens the tone. $\endgroup$
    – Dave
    Commented Mar 22, 2023 at 13:33
  • 1
    $\begingroup$ I think it is phrased better now. Also, instable --> unstable. $\endgroup$ Commented Mar 22, 2023 at 14:08
6
$\begingroup$

Training 10 models and picking the best one based on the test set performance metrics is "cheating" - your performance metrics are no longer an unbiased measure of your overall model training procedure, since your model training procedure now uses the test data to select the model! A test set should only be used to evaluate a model, never to train or select it.

If you want one single set of features and one model, you can run your model training procedure on the entire dataset. You will not have a direct unbiased measure of performance (since you have no held-out test data), but the cross-validation performance should be a good approximation. You would generally expect a model trained on the full data to perform slightly better than CV suggests, as it is trained on more data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.