What metrics can I use to compare the predictions of two linear regression models on an independent data set other than RMSE?

Question

Model A was fit using 213 predictors, Model B with 252 predictors, on a data set with n=431. I test both on an independent data set (n=121) and am currently using RMSE to determine which model performs better.

I am looking for more measures to compare the predictions of the two models. I looked up mean absolute deviation, but am not sure what exactly what that tells me or if it is useful for a comparison of predictions between both models. Is it the variation of the predictions? Can I use r-squared (r2) or do I need an adjusted r2 since both models have a different number of predictors? I'm not sure how I would calculate the adjusted r2 for the test set because n=121 and p>n for both models. Am i misinterpreting the adjusted r2 calculation?

I have also created a plot of actual vs predicted values. Any suggestions are appreciated.

Answer 1

Mean squared error and mean absolute error are common. Then there are some that arise from time series forecasting but might make sense for you. The easiest to understand is mean absolute percentage error (MAPE), despite its problems. That page also mentions four alternatives to MAPE:

1. Mean Absolute Scaled Error (MASE)

2. Symmetric Mean Absolute Percentage Error (sMAPE)

3. Mean Directional Accuracy (MDA)

4. Mean Arctangent Absolute Percentage Error (MAAPE)

Regarding $R^2$ and adjusted $R^2$, there is a natural way to use the idea of $R^2$ for out-of-sample data. Regular $R^2$ compares the MSE of your model to the MSE of a model that guesses the mean of $y$ every time, regardless of what the features are.

$$ R^2 = 1 - \dfrac{ \sum_{i=1}^n \big(y_i - \hat y_i\big)^2 }{ \sum_{i=1}^n \big(y_i - \bar y\big)^2 } $$

Apply this same idea to out-of-sample truth values $y_i$ and predicted values $\hat y_i$, making sure to still use a denominator model that always guesses the same $\bar y$ that you calculated on the in-sample data.

$$ R^2_{out} = 1 - \dfrac{ \sum_{i=1}^n \big(y_i - \hat y_i\big)^2 }{ \sum_{i=1}^n \big(y_i - \bar y_{in}\big)^2 } $$

Adjusted $R^2$ on test data makes less sense, since the reason for having the penalty for a high parameter count in adjusted $R^2$ is to catch if the model only fits the data used to train the model, rather than being able to generalize. Checking the ability to generalize, rather than just fitting the training data, is exactly what an out-of-sample metric checks.

Answer 2

If you are mainly interested in metric for prediction in linear regression. Two of the most common metrices are Mean Square Error(MSE) and Mean Absolute Error (MAE). (Note: RMSE is square root of MSE)

The main difference between the two is they penalize difference between predicted value and actual value (i.e. residual) differently. Recall MSE is $\frac{1}{n} \sum_{i=1}^n (\hat{y}_i-y_i)^2$, where $\hat{y}_i$ is the predicted value, $y_i$ is the actual value and $n$ is the sample size. This indicates the residual $(\hat{y}_i-y_i)$ is penalized quadratically. On the other hand, MAE is defined as $\frac{1}{n} \sum_{i=1}^n |\hat{y}_i-y_i|$, which indicates the residual is penalized linearly. So the deciding factor is do you want to penalized more on larger residuals/ differences? If so, use MSE. If not, use MAE.

$R^2$ and adjusted-$R^2$ (from lm output) are used for model selection and they have nothing to do with predictive power of the model. Therefore, they are not useful based on your title and description.