# Bayesian interpretation of the repeated sampling principle

My question is philosophical rather than practical, and I will try to explain it through an example:

Consider a Kaggle competition. All these contests have a similar structure:

1. A "train dataset" is given which is composed by a "target" variable and (usually many) covariates. A "test dataset" is given as well, which does not include the target variable.
2. Using the "train dataset", competitors estimate a model. Using the "test dataset", they provide a prediction for the unknown target variable.
3. Prediction's goodness is based on some loss function (Mean Squared Error, AUC index, etc), and a winner is awarded.

From a frequentist perspective, this setting is absolutely legitimate. There is an unknown "true" relationship between the "target variable" and the "covariates". For istance, let $y$ be the continuous target variable and $\hat{y}$ the prediction, then the MSE

$$MSE(y, \hat{y}) = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2,$$ consistently estimates the discrepancy between predictions and real data. Either implicitly or explicitly, we are making use of the principle of repeated sampling, since the discrepancy is assessed on the test dataset.

Now, let's go Bayesian. I still want to partecipate to Kaggle, but I reject the principle of repeated sampling. Conditionally to the "train dataset", together with my prior beliefs, I provide a posterior prediction. But how should I interpret the MSE score? Tools like cross-validation are meaningless?

If one consider the Bayesian prediction as it were a frequentist estimator, the problem is avoided. Instead, can I be truly Bayesian and still give a meaning to the test set?

• You might want to elaborate on how 'There is an unknown "true" relationship between the "target variable" and the "covariates"' relates to this 'principle of repeated sampling'. At present it's not clear what the latter amounts to, and what it's got to do with the former. – conjugateprior Feb 1 '18 at 4:57
• Is it possible you've mixed up the assumption of fixed quantity of interest which is not a random variable with the representation of your uncertainty about its value, which is? – conjugateprior Feb 1 '18 at 5:02