Revision d9afd1cd-c340-49fa-91b2-b6a8bcc565cd

As [Nick Cox pointed out][1], Pearson correlation between true and predicted values has an equivalence with the $R^2$ of classical linear regression. 

The trouble I see is that multiple expressions are equivalent to this definition of $R^2$ in classical linear modeling. For instance, in a classical linear model (with an intercept), the following are equivalent in the sense that their calculations all yield the same value.

1. $\left[\text{corr}\left(y, \hat y\right)\right]^2$

2. The proportion of total variance that is explained by the regression model

3. The percent reduction in square loss comparing the model with a benchmark that always predicts the overall mean, $\bar y$, that is, $\left[1-\left(\dfrac{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\hat y_i
\right)^2
}{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\bar y
\right)^2
}\right)\right]
$. This is a special case of the $R^*$ defined in equation (32) of [Gneiting and Resin (2023)][2], which they assert as a universal coefficient of determination.

If the regression only has one feature $x$ along with an intercept ("simple linear regression"), then a fourth calculation equivalent to the previous three is $\left[\text{corr}\left(x,  y\right)\right]^2$.

"Proportion of variance explained" is a bit tricky in all but the simplest of models (OLS linear regression evaluated in-sample), because those are the only times when a [decomposition of the total sum of squares][3] yielding that interpretation is guaranteed to hold. However, calculating the squared correlation between true $y$ and predicted $\hat y$ is easy enough, and calculating $\left[1-\left(\dfrac{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\hat y_i
\right)^2
}{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\bar y
\right)^2
}\right)\right]
$ looks worse than it is, and we have a computer to do the heavy lifting, anyway.

However, these are not equivalent in all situations. That nasty fraction has the interpretation of comparing the square loss of your model with that of a naïve benchmark that always predicts the conditional mean to be the marginal/pooled/overall mean. I find it reasonable to want the data science team to be better at predicting than a middle school student would do by running `AVERAGE(A:A)`, to use some Excel terminology, and going with that value every time. That calculation flags such situations with values below zero, indicating quite poor performance.

The Pearson correlation can miss some ways in which the true and predicted values differ. For instance, $y=(1, 2, 3)$ has a perfect Pearson correlation with $\hat y = (11, 21, 31)$. The predictions, however, are awful.

$$
\left[1-\left(\dfrac{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\hat y_i
\right)^2
}{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\bar y
\right)^2
}\right)\right]
=-621.5
$$ 

The predictions, at least in terms of square loss, are much worse than predicting $\bar y = 2$ every time. I give additional examples of this [here][4] and [here][5].

Consequently, a high value of $\left[\text{corr}\left(y, \hat y\right)\right]^2$ does not tell us quite as much as we would like. Such a calculation does not detect certain deviations the predictions can have from the true values. 

Mean squared error, on the other hand, detects all deviations between true and predicted values. If the prediction is not *exactly* the same as its corresponding true value, there is a penalty. However, mean squared error does not make an explicit comparison with any kind of benchmark. One most certainly can say something like "The current state of the art achieves a mean squared error of 11, so our mean squared error of 3 represents a considerable improvement," but I have seen plenty of work where all that gets reported are their results without any comparison with the performance of a benchmark or competitor, not even a naïve benchmark like, "Predict the overall mean every time." 

What Pearson correlation gives you that is not as explicit in the mean squared error, however, is a sense of how well the model is able to distinguish between distinct values. For instance, if $y=(1, 2, 3)$ and $\hat y = (11, 21, 31)$, the predictions are flawed, but the model has good ability to distinguish between distinct observed values. Pearson correlation can be seen as a measure of pure predictive discrimination, similar to how area under the receiver operator characteristic curve gives a sense of how well the two categories of a "classification" problem are distinguished from each other. Thus, Pearson correlation is not totally worthless. As you pointed out, despite the flaws of Pearson correlation, it is effective at determining if the model predicts high values when the true values are high and low values when the true values are low (ditto for Spearman correlation).

Your comment, `So my question is whether using both MSE as a metric for precise values and correlation as a metric for the overall “shape” of the output data gives any advantage over using only one or the other`, is getting at the right idea, it seems.

Addressing a few more pieces from the question:

> I assume that a good correlation value does not necessarily mean that the model is accurate

Exactly. Correlation will miss a number of ways in which predictions are incorrect, such as the example with $y=(1, 2, 3)$ and $\hat y = (11, 21, 31)$.

> does a bad correlation value mean that the model is definitely performing poorly?

This gets tricky, because [there is not a universal sense of what constitutes good and bad performance][6] that can be thought of like grades in school where a $90\%$ is an **A** that makes us happy and a $50\%$ is an **F** that makes us sad.$^{\dagger}$ I would say that the way to evaluate if a model is performing poorly is to compare it with the performance of a competitor or if the predictions are good enough to help you solve your overall task (the latter of which might be quite difficult to assess).


$^{\dagger}$Even that is not true, though. I remember one examination in graduate school where the class average was something like a $34\%$, so when I got my test back and saw my $41\%$, I thought, "Awesome, I got an **A**!"

**REFERENCE**

[Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.][2]


  [1]: https://stats.stackexchange.com/a/647743/247274
  [2]: https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-17/issue-2/Regression-diagnostics-meets-forecast-evaluation--conditional-calibration-reliability-diagrams/10.1214/23-EJS2180.full
  [3]: https://stats.stackexchange.com/questions/551915/interpreting-nonlinear-regression-r2
  [4]: https://datascience.stackexchange.com/a/114457/73930
  [5]: https://stats.stackexchange.com/questions/492247/r2-on-out-sample-data-set/584562#584562
  [6]: https://stats.stackexchange.com/a/414350/247274