1
$\begingroup$
  1. Is the adjusted R square good for the interpretation of machine learning regression (SVM, Random Forest, Elastic Net, etc) model prediction accuracy?
  2. How to interpret machine learning regression prediction accuracy model with adjusted R square?
$\endgroup$
2
  • $\begingroup$ “Accuracy” tends to be used in classification problems rather than regression problems. (That it happens to be problematic, even for balanced classes, is a separate issue.) Do you mean a “regression” problem or a “classification” problem? $\endgroup$
    – Dave
    Feb 25, 2021 at 3:40
  • $\begingroup$ @Dave regression problem. $\endgroup$
    – vasili111
    Feb 25, 2021 at 4:15

1 Answer 1

1
$\begingroup$

An adjusted $R^2$ is problematic in practice for many machine learning models, since the degrees of freedom range from complicated to calculate to totally unclear. Combine that with the fact that many models are fit after performing preliminary steps like hyperparameter tuning, and calculating the degrees of freedom turns into a real mess.

The degrees of freedom matter to the adjusted $R^2$ because the standard adjusted $R^2$ from OLS linear regression uses a comparison of the residual variance of your model and the residual variance of a model that predicts the mean of $y$ every time, and adjusted $R^2$ considers unbiased estimators of those respective variances whose calculations require use of the degrees of freedom (that's the $n-p-1$ in the link).

If you want an $R^2$-style measure of model performance that penalizes the model for having many parameters that put it at risk of overfitting, you might want to calculate an out-of-sample $R^2$-style metric. As I discuss here, several calculations can be defended as out-of-sample $R^2$-style calculations, though the one that makes the most sense to me that I discuss here uses the following formula.

$$ 1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y_{\text{training}} \right)^2 }\right) $$

If you can get a handle on the degrees of freedom, however, something like adjusted $R^2$ makes perfect sense as a comparison of variances that are estimated with unbiased estimators.

Being explicit:

  1. If you get a handle on the degrees of freedom, I say this makes perfect sense. (However, do not underestimate the difficulty of getting a handle on the degrees of freedom.)

  2. I would interpret such a statistic as a comparison of unbiased estimates of error variances.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.