Can I define an $R^2$-like measure in this way when predicting with exotic loss functions?

Question

It is common in regression to see $R^2$ formulated as follows: $$R^2\equiv 1 - {SS_{\rm err}\over SS_{\rm tot}},$$ where $SS_\text{err}=\sum_i (y_i - f_i)^2$ and $SS_\text{tot}=\sum_i (y_i-\bar{y})^2$. This exact formulation seems to make the most sense if the goal of the regression is indeed to minimize the residual sum of squares. However, sometimes other, more exotic loss functions are more appropriate for a variety of reasons, and some predictive modeling algorithms might be tried to minimize this loss function.

I would like to define my $R^2$-like measure in the following way: $$R^2= 1 - \frac{\sum_il(y_i, f_i)}{\sum_il(y_i, \bar{y})},$$

where $l(y_i, f_i)$ denotes the "loss" occurred in some sense if $f_i$ is predicted but the actual answer is $y_i$. In the case of OLS regression, $l(y_i, f_i) = (y_i - f_i)^2$, but this allows for other predictive models based on least absolute deviations or even more exotic functions based upon the problem domain.

Is this valid to do? Is anything gained with this more general formulation of $R^2$? Is it still valid to call it $R^2$? Does this add any useful interpretations, or is it more likely to just be confusing? Should the traditional $R^2$ be reported as well?

Answer 1

For non-negative loss functions, this makes perfect sense to me. In nice situations, the usual $R^2$ you give can be interpreted as the proportion of variance in $y$ that is explained by the regression model. This gets lost for more complicated models or estimation techniques, but we still have an interpretation of that formula as how our model performs in terms of square loss compared to a model that predicts the mean of $y$ every time. For a model that aims to predict the conditional mean, what better baseline than a model that always predicts the overall mean, $\bar y?$

A measure that compares model performance to the performance of a baseline model makes sense to me. I would be comfortable applying that to classification accuracy (setting aside the issues with classification accuracy), and UCLA does the same, even if it takes some algebra to show the two to be equal. Also referring to UCLA, McFadden's $R^2$ uses this idea with the binomial log-likelihood ("log loss" or "crossentropy loss" in some circles). Somers's D applies this to the area under the ROC curve (where $AUC = 0.5$ is regarded as the performance of a baseline model). As a final example, quantile regression seems to have a $D^2$ measure that applies this idea to pinball loss.

If you look at the definition of pinball loss, $L_{\tau}$ below, for quantile regression, it looks pretty "exotic" to me, even if going through what each part means leads to a reasonable interpretation.

$$ l_{\tau}(y_i, \hat y_i) = \begin{cases} \tau\vert y_i - \hat y_i\vert, & y_i - \hat y_i \ge 0 \\ (1 - \tau)\vert y_i - \hat y_i\vert, & y_i - \hat y_i < 0 \end{cases}\\L_{\tau}(y, \hat y) = \sum_{i=1}^n l_{\tau}(y_i, \hat y_i) $$

Overall, not only is this a reasonable idea. It seems that there is considerable precedent in the literature for using this exact idea!