1
$\begingroup$

I would like to figure out which of 24 predictors serve best for predicting a continous outcome. I have a data set of 252 people, but only 150 people with full observations (without missings).

I've read a lot about different methods, but I feel completely overwhelmed.

As this is a regression problem, forward/backward/stepwise selection would be an opportunity, but maybe also a form of penalized regression?

I want to use R and first tried it with stepAIC from the MASS package, but I've also read that stepwise regression is not a very robust choice. And as I could only use full data then, I am worried if my model might suffer from overfitting...

Do you have any suggestions which method could fit best here, what package/functions might be helpful or tips for further reading?

Thanks!

Improve this question
$\endgroup$
1
  • $\begingroup$ I have occasionally used the "leave-one-out" technique, where the regression is first made with all predictors and then each predictor is iteratively removed from the regression one-by-one to determine the impact on fit statistics such as RMSE and R-squared. This can sometimes be helpful in weeding our the less useful predictors. $\endgroup$
    – James Phillips
    Commented Dec 20, 2019 at 13:13

2 Answers 2

Reset to default
1
$\begingroup$

As with any similar problem keep the following things in mind:

1. Overfitting:

This is a problem solved through validation (e.g. split train-test, cross-validation, etc.) and not necessarily in the model or through feature selection. So if you have a robust validation strategy in mind, overfitting should not be a problem.

2. Model selection:

You have a prediction problem to solve and ask yourself how best to solve it. First start with the proper model selection, maybe linear regression isn't the best tool for the job? If you have a lot of predictors and a lot of partially missing data maybe a randomForest regressor or ensemble method is much better suited?

Random Forest isn't harder to fit in R than a linear model and copes much, much better with missing data and non-linearity in your model. It also solves you a lot of headache in feature selection because it can cope better with irrelevant features or collinearity.

3. Feature Selection:

Finally you come to feature selection, should it still be necessary (see point 2). Instead of trying any of the stepwise, forwards, backwards models of determining predictors for linear regression start with an Explorative Data Analysis EDA.

What predictors are actually correlated with the outcome (and therefore relevant)? Do predictors correlate with each other, do they maybe have an underlying factor structure that needs to be handled?

With the amount of predictors you mention I would recommend to do a PCA or factor analysis first and then use the resulting components or factors in your predictive model instead of the raw predictors.

Improve this answer
$\endgroup$
0
$\begingroup$

You could use the variable importance plot if you are going to use the randomForest package.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.