I have a data set with 181 observations. I have 9 predictors and I have developed different regression models using ordinary linear regression and stepwise linear regression. Now I'm trying to decide which model to choose. Here's summary of the models that I want to choose from:
RMSE% is RMSE divided by mean of actual data expressed as a percentage. and any column labeled CV means that I used 80% of my data to train the model and 20% to test and the last three columns with CV is obtained using the testing set. Those columns without CV uses all the data.
So I have two questions here:
- based on the fact that I am using linear regression and have only 181 observations, should I use training and testing set or should I just use R2 and RMSE?
- according to R2 and RMSE, model 2 is better. But is it a good model as the RMSE% is 14? isn't it too high? does it mean that I'm overfitting?