Use Pearson's correlation coefficient as optimization objective in machine learning

Question

In machine learning (for regression problems), I often see mean-squared-error (MSE) or mean-absolute-error (MAE) being used as the error function to minimize (plus the regularization term). I am wondering if there are situations where using correlation coefficient would be more appropriate? if such situation exists, then:

1. Under what situations is correlation coefficient a better metric compared to MSE/MAE ?
2. In these situations, is MSE/MAE still a good proxy cost function to use?
3. Is maximizing correlation coefficient directly possible? Is this a stable objective function to use?

I couldn't find cases where correlation coefficient is used directly as the objective function in optimization. I would appreciate if people can point me to information in this area.

Answer 1

Maximizing correlation is useful when the output is highly noisy. In other words, the relationship between inputs and outputs is very weak. In such case, minimizing MSE will tend to make the output close to zero so that the predication error is the same as the variance of the training output.

Directly using correlation as objective function is possible for gradient descent approach (simply change it to minimizing minus correlation). However, I do not know how to optimize it with SGD approach, because the cost function and the gradient involves outputs of all training samples.

Another way to maximize correlation is to minimize MSE with constraining the output variance to be the same as training output variance. However, the constraint also involves all outputs thus there is no way (in my opinion) to take advantage of SGD optimizer.

EDIT: In case the top layer of the neural network is a linear output layer, we can minimize MSE and then adjust the weights and bias in the linear layer to maximize the correlation. The adjustment can be done similarly to CCA (https://en.wikipedia.org/wiki/Canonical_analysis).

Answer 2

We use Pearson´s correlation in our research and it works well. In our case it is quite stable. Since it is a translation and scale invariant measure it is only useful if you want to predict shape, not precise values. Hence, it is useful if you don't know if your target is in the solution space of your model and you are only interested in the shape. On the contrary, MSE reduces the averaged distance between the prediction and the targets, so it tries to fit the data as much as possible. This is probably the reason why MSE is more widely used, because you are usually interested in predicting precise values. If you minimize the MSE, then the correlation will increase.

Answer 3

I've done research in the field of Content Based Image Retrieval, where the goal was to have an embedding that is correlated to some similarity measure. So in this situation you don't care for the distance between embeddings to have an particular value (matching some arbitrarily scaled similarity distance measure). You just want them correlated.

In some of the experiments pearson correlation loss (keras) was used as the cost function. I don't recall having any training difficulties with it (using Adam optimizer).

Although it was applied batch-wise (and not on all outputs), the model had improved correlation ("real" correlation over the entire test set) compared to logcosh cost function.