0
$\begingroup$

I am testing my model by 2 different experiments:

  1. No test set: I just use cross-validation on the training set.

  2. I take a subset of the dataset and use it as a test set (I use the same subset in the training data as well).

Now what happens is that I get a high correlation coefficient on the first experiment and higher RMSE.

But I get lower correlation coefficient on the second one but lower RMSE.

I am not sure how should I evaluate these results. Can I say that

a. I get a lower correlation coefficient on the second experiment because I am using a smaller dataset?

b. RMSE was smaller in the second case because our model was able to explain a smaller subset of the dataset better?

Am I on the right track?

$\endgroup$
1
$\begingroup$

In terms of a. the correlation is automatically standardised for sample size. No correlation is even explained as being bigger or larger because of a certain sample size.

In terms of b. what RMSE you are referring to is a little ambiguous. But the fraction of explained variance is the square of the correlation, so that explanation sounds at least muddled: RMSE is a measure of unexplained variation, which is a failure, rather than a success.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer. So you mean I can't even compare two experiments as better/worse by looking at the correlation coefficient? (since it is a metric just depends on the sample size) $\endgroup$ – user1048858 Jun 7 '13 at 15:46
  • $\begingroup$ I didn't say that; I just tried to answer your questions. You have not given criteria for "better", but in most statistical contexts a stronger correlation is usually considered better than a weaker one. But saying that correlation is adjusted for sample size doesn't mean that it depends on sample size. The whole point of the calculation is to make the sample size irrelevant. If you want general statistical advice on how to think about your experiments, you need to tell us more about them. $\endgroup$ – Nick Cox Jun 7 '13 at 15:52
  • $\begingroup$ Sorry, I think I couldn't explain the problem well. So let's say that I have two regression models one with Correlation Coefficient of 0.9 and RMSE (root mean squared error) of 0.55 and the other has Correlation Coefficient of 0.2 and RMSE of 0.45. By just looking at these metrics, can I decide which model is better then the other? $\endgroup$ – user1048858 Jun 7 '13 at 16:18
  • $\begingroup$ If that's the only information, the model with correlation 0.9 is probably better. But there must be some other difference to explain why correlation and RMSE don't match as expected. A glance at scatter plots should be enough to give you a clear answer and tell whether correlation and regression are the best thing to do, which is more important than any of these details. $\endgroup$ – Nick Cox Jun 7 '13 at 16:24
  • 1
    $\begingroup$ If correlation and regression are relatively new to you, work through the chapters in Freedman, Pisani, Purves, Statistics, New York: W.W. Norton, any edition; or any similar text. $\endgroup$ – Nick Cox Jun 7 '13 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.