8
$\begingroup$

A paper I saw used the Pearson correlation together with MSE to measure the performance of a machine learning model. After doing some research, I have seen that using Pearson correlations to evaluate a model can work if the exact values do not need to be precisely approximated but rather guessing correctly whether the output should be low/high or in between.

So my question is whether using both MSE as a metric for precise values and correlation as a metric for the overall “shape” of the output data gives any advantage over using only one or the other.

Also for correlation metrics. I assume that a good correlation value does not necessarily mean that the model is accurate, but does a bad correlation value mean that the model is definitely performing poorly?

I would also be grateful if anyone could tell me if there are any papers or books that discuss the use of correlation as a performance metric for regression models, as I could only find the standard metrics such as MSE etc in the papers or books.

$\endgroup$
1
  • 2
    $\begingroup$ Could you please give a citation for the paper? $\endgroup$
    – Dave
    Commented May 22 at 9:46

2 Answers 2

9
$\begingroup$

I take it that correlation here is the correlation between observed and predicted outcome or response values. If so, then using such a correlation is just a variant on using the coefficient of determination $R^2$, as is standard for reporting regression models and as can be calculated for any model that yields predicted values on the same scale as the original data.

That is, $R^2$ is essentially the square of the correlation between observed and predicted outcomes, although statistical people vary in how much emphasis they put on that. (There is much small print that others might add there, and they're welcome to do that.) A side comment is that here lies small scope for mischief over what is reported. So to naive readers $R^2$ of say 0.04 might look like disaster, while a correlation of 0.2 might seem less disastrous, but the results are naturally one and the same.

Watch out for many popular analogues of or alternatives to $R^2$ that, although sometimes helpful, are not squared correlations. (Labels like pseudo- may be visible.)

how to calculate R-squared in glm? is one of several threads that serves to signal one possibility as well as a range of views in this territory.

That said, there are many more reservations here.

  1. Technically, if your fitting procedure does not maximize $R^2$ as an explicit goal or as a side-effect of some other criterion (e.g. maximum likelihood), then there has to be a flag that you're evaluating a model by a criterion not used in fitting, which is going to be somewhere between acceptable or pragmatic and dubious or irrelevant. (Imagine evaluating sports people by how much they smile or their hair style rather than how well they did. Such irrelevance may be entertaining at best, but it is a side-issue.)

  2. A common objection to $R^2$, or at least a standard comment, is that it needs substantive interpretation. (Naturally all figures of merit need such interpretation.) Thus an $R^2$ that isn't very close to 1 in some fields signals experimental incompetence. In others an $R^2$ very close to 1 may signal outrageous over-fitting, fraud, or a very silly question. Further, in many fields $R^2$ being close to 0 doesn't rule out a model being interesting or useful. So simple predictors such as age or gender don't usually get you far in predicting academic performance, but it can still be of concern whether age or gender has a discernible effect. This last situation seems common in social science or medical applications.

  3. Correlation doesn't measure agreement, just as predicting temperature by the temperature multiplied by a positive constant gives you a perfect correlation, but that is not a helpful prediction. In short, bias can also be a problem.

  4. No single measure of model performance can capture all that is important about a model's virtues and limitations.

In general, and more positively, I often want to couple a scaled measure like $R^2$ with a scaled measure like mean square error MSE (or, greatly preferable, its root RMSE). It can be very helpful to get measures on the same scale as the original outcome. It is quite common that mischievous researchers cite a measure that makes their model look good and fail to cite any other. If people's heights are predicted to an RMSE of 0.1 mm or of 10 cm, then in either case I know that a model is useless, although for different reasons. Researchers should have a good feeling for the units of measurement used in their field.

$\endgroup$
7
$\begingroup$

As Nick Cox pointed out, Pearson correlation between true and predicted values has an equivalence with the $R^2$ of classical linear regression.

The trouble I see is that multiple expressions are equivalent to this definition of $R^2$ in classical linear modeling. For instance, in a classical linear model (with an intercept), the following are equivalent in the sense that their calculations all yield the same value.

  1. $\left[\text{corr}\left(y, \hat y\right)\right]^2$

  2. The proportion of total variance that is explained by the regression model

  3. The percent reduction in square loss comparing the model with a benchmark that always predicts the overall mean, $\bar y$, that is, $\left[1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right)\right] $. This is a special case of the $R^*$ defined in equation (32) of Gneiting and Resin (2023), which they assert as a universal coefficient of determination.

If the regression only has one feature $x$ along with an intercept ("simple linear regression"), then a fourth calculation equivalent to the previous three is $\left[\text{corr}\left(x, y\right)\right]^2$.

"Proportion of variance explained" is a bit tricky in all but the simplest of models (OLS linear regression evaluated in-sample), because those are the only times when a decomposition of the total sum of squares yielding that interpretation is guaranteed to hold. However, calculating the squared correlation between true $y$ and predicted $\hat y$ is easy enough, and calculating $\left[1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right)\right] $ looks worse than it is, and we have a computer to do the heavy lifting, anyway.

However, these are not equivalent in all situations. That nasty fraction has the interpretation of comparing the square loss of your model with that of a naïve benchmark that always predicts the conditional mean to be the marginal/pooled/overall mean. I find it reasonable to want the data science team to be better at predicting than a middle school student would do by running AVERAGE(A:A), to use some Excel terminology, and going with that value every time. That calculation flags such situations with values below zero, indicating quite poor performance.

The Pearson correlation can miss some ways in which the true and predicted values differ. For instance, $y=(1, 2, 3)$ has a perfect Pearson correlation with $\hat y = (11, 21, 31)$. The predictions, however, are awful.

$$ \left[1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right)\right] =-621.5 $$

The predictions, at least in terms of square loss, are much worse than predicting $\bar y = 2$ every time. I give additional examples of this here and here.

Consequently, a high value of $\left[\text{corr}\left(y, \hat y\right)\right]^2$ does not tell us quite as much as we would like. Such a calculation does not detect certain deviations the predictions can have from the true values.

Mean squared error, on the other hand, detects all deviations between true and predicted values. If the prediction is not exactly the same as its corresponding true value, there is a penalty. However, mean squared error does not make an explicit comparison with any kind of benchmark. One most certainly can say something like "The current state of the art achieves a mean squared error of 11, so our mean squared error of 3 represents a considerable improvement," but I have seen plenty of work where all that gets reported are their results without any comparison with the performance of a benchmark or competitor, not even a naïve benchmark like, "Predict the overall mean every time."

What Pearson correlation gives you that is not as explicit in the mean squared error, however, is a sense of how well the model is able to distinguish between distinct values. For instance, if $y=(1, 2, 3)$ and $\hat y = (11, 21, 31)$, the predictions are flawed, but the model has good ability to distinguish between distinct observed values. Pearson correlation can be seen as a measure of pure predictive discrimination, similar to how area under the receiver operator characteristic curve gives a sense of how well the two categories of a "classification" problem are distinguished from each other. Thus, Pearson correlation is not totally worthless. As you pointed out, despite the flaws of Pearson correlation, it is effective at determining if the model predicts high values when the true values are high and low values when the true values are low (ditto for Spearman correlation).

Your comment, So my question is whether using both MSE as a metric for precise values and correlation as a metric for the overall “shape” of the output data gives any advantage over using only one or the other, is getting at the right idea, it seems.

Addressing a few more pieces from the question:

I assume that a good correlation value does not necessarily mean that the model is accurate

Exactly. Correlation will miss a number of ways in which predictions are incorrect, such as the example with $y=(1, 2, 3)$ and $\hat y = (11, 21, 31)$.

does a bad correlation value mean that the model is definitely performing poorly?

This gets tricky, because there is not a universal sense of what constitutes good and bad performance that can be thought of like grades in school where a $90\%$ is an A that makes us happy and a $50\%$ is an F that makes us sad.$^{\dagger}$ I would say that the way to evaluate if a model is performing poorly is to compare it with the performance of a competitor or if the predictions are good enough to help you solve your overall task (the latter of which might be quite difficult to assess).

$^{\dagger}$Even that is not true, though. I remember one examination in graduate school where the class average was something like a $34\%$, so when I got my test back and saw my $41\%$, I thought, "Awesome, I got an A!"

REFERENCE

Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.

$\endgroup$
3
  • $\begingroup$ (+1) Very nice answer. I am optimistic that our stances are complementary. I would want to add a little emphasis that squaring errors is if not problematic, than at least arbitrary, for many kinds of models. But so is correlation -- which is based on squared deviations too. $\endgroup$
    – Nick Cox
    Commented May 22 at 18:45
  • $\begingroup$ I knew when reading this question we could expect a response from @Dave ;) (+1) $\endgroup$ Commented May 23 at 6:57
  • $\begingroup$ I can foresee wanting outside CV and indeed SE (a) to use and cite the paper by Gneiting and Resin (b) to acknowledge @Dave for the reference. If you're willing to tell me your full name, my profile should allow working out an email address for me. If you prefer to keep your anonymity, then naturally you should do nothing. $\endgroup$
    – Nick Cox
    Commented May 23 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.