MSE as a proxy to Pearson's Correlation in Regression Problems

Question

TL;DR (too long, didn't read):

I'm working on a time-series prediction problem, which I formulate as a Regression problem using Deep Learning (keras). I want to optimize for the Pearson correlation between my prediction and the true labels. I'm confused by the fact that using MSE as a proxy actually leads to better results (in terms of the correlation) than using Pearson as a loss function directly. Is it considered bad practice to use correlation metrics as loss functions for deep learning? If so, why?

Longer version:

I have a time-series prediction task: I observe values for $T$ consecutive time-steps and need to predict the value at time-step $T+1$. Since the values are usually in $[-200,200]$, I'm treating this as a regression problem, which I'm solving using Deep Learning (keras).

My question is regarding the choice of loss & metrics.

My data has true labels mostly around $[-10,10]$ with some extreme values. Many of the extreme values are erroneous and I don't want to shift my learning to focus on getting them right. In other words, I want to be able to catch the general trend (correctly classify period of positive versus negative values), and I can "live with" with predicting 100 instead of 200, for example.

For this reason, I think my evaluation metric should be the Pearson correlation between the predicted and true values.

Now, for the loss function: Ideally, if I want to optimize for high Pearson correlation, it would make sense to use that as the loss function, right? I've tested a simple architecture that is my "baseline model" twice: Once with using Pearson (as calculated on a mini-batch) directly as my loss function, and once with using the common MSE as a proxy. In both cases I track both MSE and Pearson for different epochs and I do "early stopping" based on a validation set.

My results:

• MSE as a loss: MSE 160, Pearson 0.7
• Pearson as a loss: MSE 250, Pearson 0.6

I understand the higher MSE for the Pearson loss being the result of the fact that optimizing for correlation has no scale, so all the prediction can be "off" by a factor in a way that increases the MSE. But how come using MSE as a proxy actually does better in terms of the Pearson correlation itself? Is there any optimization-related reason as to why Pearson correlation shouldn't be used as a loss function? Indeed, I see that in practice it's hardly used, but I would like to understand the reason behind this.

Answer 1

This is a good question and unfortunately unanswered for a long time, it seems that there was a partial answer given just a couple months after you asked this question here that basically just argues that correlation is useful when the outputs are very noisy and perhaps MSE otherwise. I think first of all we should look at the formulas for both.

$$MSE(y,\hat{y}) = \frac{1}{n} \sum_{i=1}^n(y_i - \hat{y_i})^2$$ $$R(y, \hat{y}) = \frac{\sum_{i=1}^n (y_i - \bar{y})(\hat{y_i} - \hat{\bar{y}})} {\sqrt{\sum ^n _{i=1}(y_i - \bar{y})^2} \sqrt{\sum ^n _{i=1}(\hat{y_i} - \hat{\bar{y}})^2}} $$

Some few things to note, in the case of linear regression we know that $\hat{\bar{y}} = \bar{y}$ because of unbiasedness of the regressor, so the model will simplify a little bit, but in general we can't make this assumption about ML algorithms. Perhaps more broadly it is interesting to think of the scatter plot in $\mathbb{R^2}$ of $ \{ y_i, \hat{y_i}\} $ correlation tells us how strong the linear relationship is between the two in this plot, and MSE tells us how far from the diagonal each point is. Looking at the counter examples on the wikipedia page you can see there are many relationships between the two that won't be represented.

I think generally correlation is tells similar things as $R^2$ but with directionality, so correlation is somewhat more descriptive in that case. In another interpretation, $R^2$ doesn't rely on the linearity assumption and merely tells us the percentage of variation in $y$ that's explained by our model. In other words, it compares the model's prediction to the naive prediction of guessing the mean for every point. The formula for $R^2$ is:

$$R^2(y,\hat{y}) = 1 - \frac{\sum_{i=1}^n (y_i-\hat{y})^2}{\sum_{i=1}^n (y_i-\bar{y})^2}$$

---

So how does $R$ compare to $R^2$? Well it turns out that $R$ is more immune to scaling up of one of the inputs this has to do with the fact that $R^2$ is homogenous of degree 0 only in both inputs, where $R$ is homogenous of degree 0 in either input. It's a little less clear what this might imply in terms of machine learning, but it might mean that the model class of $\hat{y}$ can be a bit more flexible under correlation. This said, under some additional assumptions, however, the two measures are equal, and you can read more about it here: http://www.win-vector.com/blog/2011/11/correlation-and-r-squared/.

Finally a last important thing to note is that $R$ and $R^2$ do not measure goodness of fit around the $y=\hat{y}$ line. It is possible (although odd) to have a predictor be linearly shifted away from the $y=\hat{y}$ line with an $R^2$ of one, but the predictions would still be "bad". In this case, MSE would be more informative in finding the better predictor than $R^2$ but perhaps this is more of a pathological case than an issue with using $R$ and $R^2$ as metrics.