Best model for data-based predictor selection (Regression, R) I would like to figure out which of 24 predictors serve best for predicting a continous outcome. I have a data set of 252 people, but only 150 people with full observations (without missings).
I've read a lot about different methods, but I feel completely overwhelmed.
As this is a regression problem, forward/backward/stepwise selection would be an opportunity, but maybe also a form of penalized regression?
I want to use R and first tried it with stepAIC from the MASS package, but I've also read that stepwise regression is not a very robust choice. And as I could only use full data then, I am worried if my model might suffer from overfitting...
Do you have any suggestions which method could fit best here, what package/functions might be helpful or tips for further reading?
Thanks!
 A: As with any similar problem keep the following things in mind:
1. Overfitting:
This is a problem solved through validation (e.g. split train-test, cross-validation, etc.) and not necessarily in the model or through feature selection. So if you have a robust validation strategy in mind, overfitting should not be a problem.
2. Model selection:
You have a prediction problem to solve and ask yourself how best to solve it. First start with the proper model selection, maybe linear regression isn't the best tool for the job? If you have a lot of predictors and a lot of partially missing data maybe a randomForest regressor or ensemble method is much better suited?
Random Forest isn't harder to fit in R than a linear model and copes much, much better with missing data and non-linearity in your model. It also solves you a lot of headache in feature selection because it can cope better with irrelevant features or collinearity.
3. Feature Selection:
Finally you come to feature selection, should it still be necessary (see point 2). Instead of trying any of the stepwise, forwards, backwards models of determining predictors for linear regression start with an Explorative Data Analysis EDA.
What predictors are actually correlated with the outcome (and therefore relevant)? Do predictors correlate with each other, do they maybe have an underlying factor structure that needs to be handled?
With the amount of predictors you mention I would recommend to do a PCA or factor analysis first and then use the resulting components or factors in your predictive model instead of the raw predictors.
A: You could use the variable importance plot if you are going to use the randomForest package.
