I was thinking about cross-validation and how it is the most appropriate way to do it...
Let's take the case of binary logistic regression where the goal is to calculate the AUC.
Make the partition of the data using k folds. What is the correct way to get the cross-validated AUC :
1) Train the model using k-1 folds and predict on the kth fold. Calculate the AUC and repeat until all folds served as test set. This will give at the end k AUC values, which we average to get the cross-validated AUC.
2) Train the model using k-1 folds and predict on the kth fold. Save the predictions. Repeat until all folds served as test set. This will give a vector of predictions, one for each subject in the dataset. Calculate the AUC using this vector of predictions and the vector of observed responses.
My intuition and idea of cross-validation suggest that 2) is the correct one...