Does it make sense to get the correlation between your predicted value (y_hat) and your true value (y) as a metric for overfitting for your training data? My logic is that if you have a predicted value that is following the behaviour of your true value so closely for the training set, that might be an indication that our model could potentially be over fitting and it won't perform as well with our test data. What do you think?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ The squared correlation between the predicted and true value is identical to $R^2$ in linear regression. $\endgroup$– logisticCommented Jul 22, 2019 at 12:22
-
$\begingroup$ Thanks, I know that, but R2 won't tell you much about overfitting or underfitting, it will just explain how much of your variance is explained by your model, whereas correlation coefficient will tell you the relationship between them and how similar they behave, which I think it is more useful to know if you are overfitting or now $\endgroup$– Felipe ArayaCommented Jul 22, 2019 at 12:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Let us assume:
y_true = [1,2,3,4,5]
y_pred = [1,2,3,4,5]
y_pred2 = [3,4,5,6,7]
y_pred3 = [2,6,9,10,15]
where, y_pred, y_pred2 and y_pred3 are predictions from 3 different models. If you calculate pearson correlation for them the answers would be:
corr_ypred = 1
corr_ypred2 = 1
corr_ypred3 = 0.9827
where, corr_ypred, corr_ypred2 and corr_ypred3 are correlation of y_true with y_pred, y_pred2 and y_pred3 respectively. Now, as you can see y_pred is clearly an overfitted model showing correlation of 1. y_pred2 and y_pred3 are, however, not overfitted but still show 1 and almost 1 correlation respectively. That is why it is not really possible to differentiate overfitted models with correlation.
-
$\begingroup$ Thank you! That was a good explanation $\endgroup$ Commented Jul 22, 2019 at 12:56