I am using repeated 10-fold cross-validation to estimate how well a regression model will generalize. The cross-validation is repeated a number of times to estimate the variance in the procedure. The evaluation measure of interest is the Pearson's correlation coefficient. Now there are two possible ways to calculate this;
Case 1. The correlation is calculated after each cross-validation run (over the whole data set as each example appears only once as a test case in each cross-validation run) and the resulting coefficients will be aggregated across the repeats.
Case 2. The predications for each example is aggregated (using mean or median) across the repeats. This gives one prediction vector for the whole data set which can be correlated with the original values.
Is one way better or worse than the other?
How does the interpretation of the resulting correlation coefficient change for the two cases?
I found one related post but not much other information.
EDIT
Calculating the correlation over the whole data set:
Case 1. For a single cross-validation run each example is predicted once. Thus, all the predictions in one cross-validation run can be collected in a single vector (of length equal to the number of examples in the whole data set). The correlation between those predicted values and the actual values is then calculated.
Case 2. The predictions from the cross-validation runs are aggregated resulting in a single vector (again with length equal to the number of examples in the whole data set). The correlation between those aggregate predictions and the actual values is calculated.
