2
$\begingroup$

I am doing regression on a numeric dataset. This dataset is combined from 2 sources (one is from real-life data, one is from synthetic data). When I run cross-validation on this combined data, I get a correlation coefficient of 0.9. But when I train the model on the combined data, but test it only on real-life data, I get a correlation coefficient of 0.2.

I know that correlation coefficient gives the quality of least squares fitting, so in that case why would my model performs well on combined data but performs bad on real-life data (even though I trained my model on it)? I read some book sections on correlation coefficient, but intuitively I can't comment on why I am getting this result.

$\endgroup$
3
  • $\begingroup$ This look like a duplicate of stats.stackexchange.com/questions/61173/… In general, it doesn't surprise me that real data are more problematic than synthetic data. $\endgroup$
    – Nick Cox
    Jun 10, 2013 at 17:24
  • $\begingroup$ What is the number of observations in the synthetic data, and what is the number of observations in real-life data? Is the former bigger than the latter? $\endgroup$ Jun 10, 2013 at 19:06
  • $\begingroup$ Yes the real data is %10 of the synthetic data. Is it why I get a low correlation when I use the subset of the data? My main confusion is: since I also use the real data in the training phase and get good correlation, I am surprised to get such a low correlation when I use a subset of this data. Is the main reason the size of the test set? $\endgroup$ Jun 10, 2013 at 20:03

1 Answer 1

2
$\begingroup$

My guess based on your commentary:

Once the synthetic data is much bigger than the real-life data, the correlation coefficient is being weighted mainly for the synthetic data.

Your real life-data are probably outliers in comparison with synthetic data values.

This is the reason you are getting low r² and probably high RMSE on validation phase.
Putting in another way: it was almost like fitting a model to a dataset A, and try to validate it with a dataset B with very different features from A.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.