7
$\begingroup$

I have a dataset containing three variables. I have this confusion, low mse means higher correlation isn't it? Now when I calculate the mse for variable two and three where the third variable is my ground truth, I get higher mse than the one I get by comparing variable one and variable three (ground truth).

However, when I calculate the correlation using corr function in matlab, I get higher correlation between variable two and three around .78. On the other hand, the correlation between variable one and three is .76. Why is it so?

Let me tell you I have different number of variable1-variable3 and variable2-variable3 set. I meant to say sometimes the values of the variables are missing. So altogether, I have 1280 sets of variable1-variable 3 pairs and 400 of variable2-variable 3 pairs. As I have said I have higher mse for the variable1-variable 3 set than variable2-variable3 set calculating for the number of pair points 400 and 1280 respectively.

I am a bit confused. Any insights?

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you explicit what you mean by "I calculate the mse for variable two and three where the third variable is my ground truth"? $\endgroup$
    – gui11aume
    Aug 10, 2012 at 7:28
  • $\begingroup$ I mean I calculated the mean squared error. The first two variables are the predictions by my two predictors, while the third variable is my target or ground truth $\endgroup$
    – user31820
    Aug 10, 2012 at 10:52

2 Answers 2

Reset to default
9
$\begingroup$

I suggest we look at the definitions.

$$ MSE = \frac{1}{n-2}\sum_{i=1}^n (Y_i -\hat{Y}_i)^2. $$

Again, by definition

$$ \hat{Y}_i = r\frac{S_y}{S_x}(X_i - \bar{X}) + \bar{Y}, $$

so

$$SSE = \sum_{i=1}^n \left( (Y_i - \bar{Y}) - r\frac{S_y}{S_x}(X_i - \bar{X}) \right) ^2 = (n-1)S_y^2 + (n-1)r^2S_y^2 - 2(n-1)r^2S_y^2,$$

so that

$$ MSE = \frac{n-1}{n-2}S_y^2 \left(1-r^2\right).$$

MSE depends on $r^2$, and as you mentioned, the lower the correlation, the higher MSE, but as you can see it depends as well on the variance of the response variable $S_y^2$. Actually, this is expected, because MSE has a unit (the square of the unit of $Y$) and the coefficient of correltion does not. For that reason you cannot usually compare MSE of different regressions models (for the same reason you cannot compare km2 and kg2).

Also note that multiplying the values of $Y$ by 10 would not change the coefficient of correlation between $Y$ and $X$ (what I think you call your ground truth), but it would multiply MSE by 100.

In short, you can use MSE to compare different predictors (the name of "ground truths" in regression models) to model the same response variable, but you cannot use MSE to compare different response variables modeled by the same predictor.

Update: after looking at your data directly (thanks for sending) I computed the following measures with complete cases (the ones where all three variables are observed which is only 163 out of 5,374,625):

$$MSE(Y_1|Y_3) = 0.00275; cor(Y1,Y3) = 0.85 \\ MSE(Y_2|Y_3) = 0.0208; cor(Y2, Y3) = 0.61$$

On the other hand, when taking only the pairwise complete cases needed for the computation I obtained the following:

$$MSE(Y1|Y3) = 0.00323; cor(Y2, Y3) = 0.766 \; (n=464) \\ MSE(Y2|Y3) = 0.0117; cor(Y2, Y3) = 0.785 \; (n=1280)$$

Notice how the correlation changes when taken on different datasets (and the order actually changes). Even though I believe that the upper part of my answer is correct, @whuber is right in his comment that you cannot draw any conclusion if your measures are taken on different datasets.

Improve this answer
$\endgroup$
9
  • $\begingroup$ This is nice, but it's not clear that it's relevant to the question. (It's difficult to determine what the question is about, but the fundamental issue seems unrelated to regression: the OP is comparing "predictions" to "ground truth" using an RMSE. We can conceive of these data as a set of triples $(Y_1,Y_2,Y_3)$, but where substantial numbers of the $Y_2$ values (and perhaps some of the $Y_1$ values) are missing. The RMSEs are suitably standardized $L^2$ norms of the vector differences $\mathbf{Y_1-Y_3}$ and $\mathbf{Y_2-Y_3}$.) $\endgroup$
    – whuber ♦
    Aug 10, 2012 at 15:37
  • $\begingroup$ @whuber Although the OP doesn't mention regression as his means of prediction it is not a big stretch to assume that is what he is doing and he is trying to relate MSE to r. So I think gui11aume's use of regression to compute the relationship between r and MSE makes the important points that should relieve the OPs confusion even if some other form of prediction is on his mind. $\endgroup$
    – Michael R. Chernick
    Aug 10, 2012 at 20:43
  • $\begingroup$ @Michael I suspect the really important point in this question concerns the effects of missing values: see the third paragraph in the question. $\endgroup$
    – whuber ♦
    Aug 10, 2012 at 20:56
  • $\begingroup$ @whuber I think the point was well made that you can't compare MSE from different models. In this case the models differ by choice of covariate. This is further complicated by the fact that for one set of covariates there are 1280 cases and for the other 400 because so many are thrown out due to a missing covariate. Bias could be introduced by the missingness and there are huge differences in sample size. Comparing MSE between models is meaningless. $\endgroup$
    – Michael R. Chernick
    Aug 10, 2012 at 21:04
  • 1
    $\begingroup$ @whuber, yeah I have some Y1 and Y2 values missing. But when calculating the MSE, correlation, I have only used those values where both Y1,Y3 and Y2,Y3 pairs are available. Lets say I have 100 Y1,Y3 pairs available with both Y1 and Y3 available and 200 Y2,Y3 pairs available with both Y2 and Y3 available $\endgroup$
    – user31820
    Aug 11, 2012 at 13:26
1
$\begingroup$

Correlation: The degree or extent to which variables are linearly related is called the correlation among variables.

MSE: If $T$ be an estimator of $\theta$. $MSE(T)$ is a measure of the spread of the values of $T$ around $\theta$

These two are different so don't be confused

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ yeah that's true but if the MSE is high it means the variables are less correlated isn't it? So there is some kind of relationship between correlation and MSE $\endgroup$
    – user31820
    Aug 11, 2012 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.