3
$\begingroup$

It is common in regression to see $R^2$ formulated as follows: $$R^2\equiv 1 - {SS_{\rm err}\over SS_{\rm tot}},$$ where $SS_\text{err}=\sum_i (y_i - f_i)^2$ and $SS_\text{tot}=\sum_i (y_i-\bar{y})^2$. This exact formulation seems to make the most sense if the goal of the regression is indeed to minimize the residual sum of squares. However, sometimes other, more exotic loss functions are more appropriate for a variety of reasons, and some predictive modeling algorithms might be tried to minimize this loss function.

I would like to define my $R^2$-like measure in the following way: $$R^2= 1 - \frac{\sum_il(y_i, f_i)}{\sum_il(y_i, \bar{y})},$$

where $l(y_i, f_i)$ denotes the "loss" occurred in some sense if $f_i$ is predicted but the actual answer is $y_i$. In the case of OLS regression, $l(y_i, f_i) = (y_i - f_i)^2$, but this allows for other predictive models based on least absolute deviations or even more exotic functions based upon the problem domain.

Is this valid to do? Is anything gained with this more general formulation of $R^2$? Is it still valid to call it $R^2$? Does this add any useful interpretations, or is it more likely to just be confusing? Should the traditional $R^2$ be reported as well?

$\endgroup$
3
  • 1
    $\begingroup$ It seems that you have a mathematical generalization. Therefore if you want to name it a generalized R square how could we argue against it. Now r square has the interpretation of percentage variance explained by the model. What does you generalized R square represent analogous to this interpretation? Is it something like a percentage loss explained?? I am having a hard time making such an analogy. $\endgroup$ Oct 1, 2012 at 21:42
  • $\begingroup$ Well whether this has a meaningful analogous interpretation is part of what I'm trying to figure out. For normal $R^2$, I prefer the interpretation that it measures the percentage reduction in "error" (mean-squared error) relative to the naive model of always predicting the average. For well-behaved loss functions (e.g. absolute deviations), I can convince myself that my generalized $R^2$ can be interpreted as the percentage reduction in loss relative to the naive model. I haven't yet convinced myself this makes sense if, for instance, I allow the loss function to be negative. $\endgroup$ Oct 2, 2012 at 13:36
  • $\begingroup$ Looks you have an example of Pseudo-R-squared $\endgroup$ Mar 26, 2023 at 4:30

1 Answer 1

1
$\begingroup$

For non-negative loss functions, this makes perfect sense to me. In nice situations, the usual $R^2$ you give can be interpreted as the proportion of variance in $y$ that is explained by the regression model. This gets lost for more complicated models or estimation techniques, but we still have an interpretation of that formula as how our model performs in terms of square loss compared to a model that predicts the mean of $y$ every time. For a model that aims to predict the conditional mean, what better baseline than a model that always predicts the overall mean, $\bar y?$

A measure that compares model performance to the performance of a baseline model makes sense to me. I would be comfortable applying that to classification accuracy (setting aside the issues with classification accuracy), and UCLA does the same, even if it takes some algebra to show the two to be equal. Also referring to UCLA, McFadden's $R^2$ uses this idea with the binomial log-likelihood ("log loss" or "crossentropy loss" in some circles). Somers's D applies this to the area under the ROC curve (where $AUC = 0.5$ is regarded as the performance of a baseline model). As a final example, quantile regression seems to have a $D^2$ measure that applies this idea to pinball loss.

If you look at the definition of pinball loss, $L_{\tau}$ below, for quantile regression, it looks pretty "exotic" to me, even if going through what each part means leads to a reasonable interpretation.

$$ l_{\tau}(y_i, \hat y_i) = \begin{cases} \tau\vert y_i - \hat y_i\vert, & y_i - \hat y_i \ge 0 \\ (1 - \tau)\vert y_i - \hat y_i\vert, & y_i - \hat y_i < 0 \end{cases}\\L_{\tau}(y, \hat y) = \sum_{i=1}^n l_{\tau}(y_i, \hat y_i) $$

Overall, not only is this a reasonable idea. It seems that there is considerable precedent in the literature for using this exact idea!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.