My question is philosophical rather than practical, and I will try to explain it through an example:

Consider a Kaggle competition. All these contests have a similar structure:

  1. A "train dataset" is given which is composed by a "target" variable and (usually many) covariates. A "test dataset" is given as well, which does not include the target variable.
  2. Using the "train dataset", competitors estimate a model. Using the "test dataset", they provide a prediction for the unknown target variable.
  3. Prediction's goodness is based on some loss function (Mean Squared Error, AUC index, etc), and a winner is awarded.

From a frequentist perspective, this setting is absolutely legitimate. There is an unknown "true" relationship between the "target variable" and the "covariates". For istance, let $y$ be the continuous target variable and $\hat{y}$ the prediction, then the MSE

$$MSE(y, \hat{y}) = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2,$$ consistently estimates the discrepancy between predictions and real data. Either implicitly or explicitly, we are making use of the principle of repeated sampling, since the discrepancy is assessed on the test dataset.

Now, let's go Bayesian. I still want to partecipate to Kaggle, but I reject the principle of repeated sampling. Conditionally to the "train dataset", together with my prior beliefs, I provide a posterior prediction. But how should I interpret the MSE score? Tools like cross-validation are meaningless?

If one consider the Bayesian prediction as it were a frequentist estimator, the problem is avoided. Instead, can I be truly Bayesian and still give a meaning to the test set?

  • $\begingroup$ You might want to elaborate on how 'There is an unknown "true" relationship between the "target variable" and the "covariates"' relates to this 'principle of repeated sampling'. At present it's not clear what the latter amounts to, and what it's got to do with the former. $\endgroup$ – conjugateprior Feb 1 '18 at 4:57
  • $\begingroup$ Is it possible you've mixed up the assumption of fixed quantity of interest which is not a random variable with the representation of your uncertainty about its value, which is? $\endgroup$ – conjugateprior Feb 1 '18 at 5:02

Regardless of whether you are using frequentist methods or Bayesian methods (or any other methodology), this is a problem where you are asked to give point-based predictions of an unobserved vector of "response" variables in the test set. The MSE is merely a distance measure, measuring the error in your prediction. It is a deterministic function that compares actual values to their (point-based) predictions. There is nothing in this set-up that philosophically favours one theory of statistics or the other. All that this set-up is doing is formulating a prediction problem, and a measure for determining how close your prediction was to the actual values being predicted.

I'm not sure what you mean when you say that this (explicitly or implicitly) makes use of the "principle of repeated sampling". The mere fact that it is possible to formulate a distance measure comparing a set of predicted values with their actual values has nothing to do with "sampling" or probability. Discrepancy is assessed on the test dataset because that is what you are asked to predict in the prediction problem.

From a Bayesian perspective, you can take your prior belief, and the training dataset and you use these to obtain a posterior belief about model parameters, etc. You then apply this to the test dataset to obtain predictive distributions for the unknown values of interest, and make point predictions of these (by taking the mean of the predictive distribution). Once you know the values you were trying to predict, you can calculate the MSE score, and this represents a measure of your prediction error. You could regard this as the measure of "loss" in your analysis. The methodology of Bayesian statistics is a method used to make rational inferences of unknowns based on observed data. Like any method of inference, you do not expect to be able to exactly predict values; you expect some errors in your prediction, and it is useful to be able to measure the degree to which prediction error occurs.

  • $\begingroup$ +1 for noting that MSE is operating as a loss function. $\endgroup$ – conjugateprior Feb 1 '18 at 4:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.