I am testing my model by 2 different experiments:
No test set: I just use cross-validation on the training set.
I take a subset of the dataset and use it as a test set (I use the same subset in the training data as well).
Now what happens is that I get a high correlation coefficient on the first experiment and higher RMSE.
But I get lower correlation coefficient on the second one but lower RMSE.
I am not sure how should I evaluate these results. Can I say that
a. I get a lower correlation coefficient on the second experiment because I am using a smaller dataset?
b. RMSE was smaller in the second case because our model was able to explain a smaller subset of the dataset better?
Am I on the right track?