I am using repeated 10-fold cross-validation to estimate how well a regression model will generalize. The cross-validation is repeated a number of times to estimate the variance in the procedure. The evaluation measure of interest is the Pearson's correlation coefficient. Now there are two possible ways to calculate this;

Case 1. The correlation is calculated after each cross-validation run (over the whole data set as each example appears only once as a test case in each cross-validation run) and the resulting coefficients will be aggregated across the repeats.

Case 2. The predications for each example is aggregated (using mean or median) across the repeats. This gives one prediction vector for the whole data set which can be correlated with the original values.

Is one way better or worse than the other?

How does the interpretation of the resulting correlation coefficient change for the two cases?

I found one related post but not much other information.


Calculating the correlation over the whole data set:

Case 1. For a single cross-validation run each example is predicted once. Thus, all the predictions in one cross-validation run can be collected in a single vector (of length equal to the number of examples in the whole data set). The correlation between those predicted values and the actual values is then calculated.

Case 2. The predictions from the cross-validation runs are aggregated resulting in a single vector (again with length equal to the number of examples in the whole data set). The correlation between those aggregate predictions and the actual values is calculated.

  • $\begingroup$ I don't understand option 1 (specifically, the "over the whole dataset" part). Could you elaborate? Also, what kind of cross validation are you using? $\endgroup$
    – user20160
    Aug 3, 2017 at 9:54
  • $\begingroup$ @user20160 I have edited the question to explain the "over the whole data set" part. $\endgroup$
    – Krrr
    Aug 3, 2017 at 10:25
  • 1
    $\begingroup$ @kap kdd.org/exploration_files/v12-1-p49-forman-sigkdd.pdf discusses a related problem. $\endgroup$
    – Ami Tavory
    Aug 3, 2017 at 11:40
  • 1
    $\begingroup$ Option 1 seems reasonable to me. If in doubt, you can test on simulated data where you know the answer. Option 2 would give misleading results. By averaging predictions across folds, you would no longer be testing the same model, but something akin to an ensembled version of it (like bagging, but using subsampling instead of the bootstrap). $\endgroup$
    – user20160
    Aug 3, 2017 at 13:38
  • $\begingroup$ @AmiTavory +1 for the interesting article. However, it does not address the repeated cross-validation setup. $\endgroup$
    – Krrr
    Aug 3, 2017 at 15:42


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.