1
$\begingroup$

In cross validation, we repeat training the model with resampled training data and measure average of the errors from the different resampled training samples. So cross validation is basically a frequentist method. It measures generalization error of the training "procedure" rather than that of the finally trained model itself. The former considers averaging over virtual "training" data while the later uses only actually observed data to "train" the model (of course, the distribution of "future" data should be considered to estimate the generalization error in any case). This is an usual contrasting aspect of frequentist and Bayesian. Efton and Hastie also point this in page 218 of "Computer Age Statistical Inference".

However, there are literatures on "Bayesian cross validation" where the procedure of cross validation is exactly same but the prediction is just based on the Bayesian posterior obtained by resampled training data (see, e.g., this). So, "Bayesian cross validation" still estimates the generalization error in the sense of frequentist. It looks like a contradiction to the terminology "Bayesian". Is this my misunderstanding on "Bayesian cross validation" or cross validation in general? In what sense is "Bayesian cross vadlidation" Bayesian?

$\endgroup$

1 Answer 1

0
$\begingroup$

Most Bayesian applications I've seen in this setting do not use resampled data for parameter estimation but only for accuracy assessment of models developed using regular Bayesian whole-sample methods. Regarding the use of resampling or leave-out-one methods in Bayesian modeling, this is indeed a sampling-based procedure and not a parameter space-only procedure. To understand, first consider a pure Bayesian assessment as shown in http://hbiostat.org/rmsc/titanic.html#bayesian-analysis where you'll see a table that has the highest posterior density uncertainty interval for C (concordance probability = AUROC in the binary Y case here) of [0.861, 0.872]. This is a very narrow interval. How do we interpret it? It assesses the uncertainty in model performance on this sample of X and Y, where the only unreliability comes from unreliability in estimating regression coefficients $\beta$. This is useful but does not tell us about the likely future performance of the model on a new set of data X. To be able to do that, Bayesian model performance measures use sampling-based ideas instead, the most popular being a leave-out-one method because of the speed with which it can be approximated from using all the posterior draws, say from the Stan system.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.