6
$\begingroup$

Recently a project I've been a part of has involved training neural networks so that we maximize the Pearson correlation between actual and predicted values. So this came to my mind: why don't we change the mathematical workings of, say, gradient descent so that instead of minimizing RMSE, we maximize $r$? If we can make the network predict with a high correlation, all we have to do is chain a linear function to the predictions and we have good prediction.

$\endgroup$
7
$\begingroup$

Because that would be a completely different objective altogether. Note that unlike MSE, Pearson correlation is maximal iff there is a linear relationship between both variables. This means that

  • The network would "think" it has correctly learned its inputs if its output is roughly proportional to the dependent variable samples, rather than equal (or similar). Therefore predicting $Y$ or $2Y$ or $-Y$ (etc.) would be equivalent. This is generally undesirable, since we would like our network to give prediction similar to its inputs, rather than proportionally to said inputs.

  • There would not be a global minimum to the optimisation problem thus posed. Any proportional constant as set above would give an optimal solution. This is undesirable from a numerical point of view and would lead to instability.

$\endgroup$
  • $\begingroup$ It's worth noting that we don't even maximize $R^2$ for linear regression, where it might actually make sense to do so. $\endgroup$ – shadowtalker Mar 10 '15 at 16:03
  • $\begingroup$ But if we let the network converge to $r=1-\epsilon$, couldn't we just normalize the output linearly and have great predictions? $\endgroup$ – Simon Kuang Mar 10 '15 at 18:03
  • 2
    $\begingroup$ 1) Assuming the network would converge and 2) You'll find (write it down) that if you constrain everything to have mean 0 and variance 1 (which is equivalent to what you call normalizing linearly), maximizing correlation and minimizing mse is strictly equivalent. $\endgroup$ – Youloush Mar 11 '15 at 10:17
0
$\begingroup$

It's more like a comment, though I can't comment yet.

I'd suspect that minimizing RMSE respective to a normalized input is (roughly?) equivalent to maximizing the Pearson correlation coefficient, but the latter is more computationally expensive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.