1
$\begingroup$

Model A was fit using 213 predictors, Model B with 252 predictors, on a data set with n=431. I test both on an independent data set (n=121) and am currently using RMSE to determine which model performs better.

I am looking for more measures to compare the predictions of the two models. I looked up mean absolute deviation, but am not sure what exactly what that tells me or if it is useful for a comparison of predictions between both models. Is it the variation of the predictions? Can I use r-squared (r2) or do I need an adjusted r2 since both models have a different number of predictors? I'm not sure how I would calculate the adjusted r2 for the test set because n=121 and p>n for both models. Am i misinterpreting the adjusted r2 calculation?

I have also created a plot of actual vs predicted values. Any suggestions are appreciated.

$\endgroup$

2 Answers 2

1
$\begingroup$

Mean squared error and mean absolute error are common. Then there are some that arise from time series forecasting but might make sense for you. The easiest to understand is mean absolute percentage error (MAPE), despite its problems. That page also mentions four alternatives to MAPE:

  1. Mean Absolute Scaled Error (MASE)

  2. Symmetric Mean Absolute Percentage Error (sMAPE)

  3. Mean Directional Accuracy (MDA)

  4. Mean Arctangent Absolute Percentage Error (MAAPE)

Regarding $R^2$ and adjusted $R^2$, there is a natural way to use the idea of $R^2$ for out-of-sample data. Regular $R^2$ compares the MSE of your model to the MSE of a model that guesses the mean of $y$ every time, regardless of what the features are.

$$ R^2 = 1 - \dfrac{ \sum_{i=1}^n \big(y_i - \hat y_i\big)^2 }{ \sum_{i=1}^n \big(y_i - \bar y\big)^2 } $$

Apply this same idea to out-of-sample truth values $y_i$ and predicted values $\hat y_i$, making sure to still use a denominator model that always guesses the same $\bar y$ that you calculated on the in-sample data.

$$ R^2_{out} = 1 - \dfrac{ \sum_{i=1}^n \big(y_i - \hat y_i\big)^2 }{ \sum_{i=1}^n \big(y_i - \bar y_{in}\big)^2 } $$

Adjusted $R^2$ on test data makes less sense, since the reason for having the penalty for a high parameter count in adjusted $R^2$ is to catch if the model only fits the data used to train the model, rather than being able to generalize. Checking the ability to generalize, rather than just fitting the training data, is exactly what an out-of-sample metric checks.

$\endgroup$
1
$\begingroup$

If you are mainly interested in metric for prediction in linear regression. Two of the most common metrices are Mean Square Error(MSE) and Mean Absolute Error (MAE). (Note: RMSE is square root of MSE)

The main difference between the two is they penalize difference between predicted value and actual value (i.e. residual) differently. Recall MSE is $\frac{1}{n} \sum_{i=1}^n (\hat{y}_i-y_i)^2$, where $\hat{y}_i$ is the predicted value, $y_i$ is the actual value and $n$ is the sample size. This indicates the residual $(\hat{y}_i-y_i)$ is penalized quadratically. On the other hand, MAE is defined as $\frac{1}{n} \sum_{i=1}^n |\hat{y}_i-y_i|$, which indicates the residual is penalized linearly. So the deciding factor is do you want to penalized more on larger residuals/ differences? If so, use MSE. If not, use MAE.

$R^2$ and adjusted-$R^2$ (from lm output) are used for model selection and they have nothing to do with predictive power of the model. Therefore, they are not useful based on your title and description.

$\endgroup$
3
  • $\begingroup$ Welcome to Cross Validated! This was a good answer until the final paragraph. Since $R^2$ and adjusted $R^2$ are functions of the $MSE$, could you please elaborate on why you think they have nothing to do with the predictive power of the model? $\endgroup$
    – Dave
    Mar 24 at 18:58
  • $\begingroup$ Thanks for the comment. I should clarify that $R^2$ and adjusted $R^2$ only use information from training set (dataset with n= 431 in the question). Since OP is interested in the metric for testing set (dataset with n=121), $R^2$ and adjusted $R^2$ (from lm output) would not reflect anything from testing set and thus they are not applicable. $\endgroup$ Mar 24 at 19:07
  • $\begingroup$ There is a very reasonable way to use the idea of $R^2$ when you look at out-of-sample data, as my answer discusses. I do say, however, that an out-of-sample adjusted $R^2$ does not make sense. $\endgroup$
    – Dave
    Mar 24 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.