What is best way to choose a criterion for linear regression model performance? Should the best model refer to RMSE, adjusted R-squared or AIC values?
What about when variable selection methods such as stepwise, backward or forward elimination are applied?
RMSE and R-squared are both measures of goodness of fit, used to assess the quality of a model, and are directly related.
On the other hand, AIC is an information criterion used to compare a group of models (i.e., unless you can compare the AIC value of a model with others, it has not much use). According to wiki:
Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models
The guidelines above do not change if you use a feature-selection procedure or not. Nonetheless, adjusted R-squared is usually used to compare models on feature selecion, check this link for a brief intro on this.
In general, using adjusted R-squared will lead you to models with better predictive power, while AIC will allow you to select the simplest, yet efficient model. Again, from wiki:
AIC deals with the trade-off between the goodness of fit of the model and the simplicity of the model. In other words, AIC deals with both the risk of overfitting and the risk of underfitting.
