When may I make a Bayesian interpretation of the posterior?

Question

I believe one major advantage of Bayesian inference is the intuitiveness of interpretation. This is my primary interest. However, it's not completely clear to me when it's OK to make such an interpretation.

I make the potentially false assumption that fitting a probability model in the frequentist way is virtually the same as fitting the same model with a flat prior in a Bayesian way. Please nuance or correct that as interest number one (1).

And my main interest (2) is if my assumption is true (and if a flat prior happens to be the best prior I could possibly come up with), does the posterior of the model fitted in a frequentist way, but sampled say with an MCMC sampler, allow for me to make a Bayesian type of interpretation? e.g., the probability that an individual described by some particular configuration of $X$ will have an income ($Y$) greater than $100K is 76%.

I've read that what makes an analysis Bayesian is that it involves prior information. Is that really the essence? I've also read that you can't make such interpretations as my example about income from frequentist results. Does sampling from the posterior with MCMC methods move me away from frequentist methods sufficiently to make a Bayesian interpretation?

I greatly appreciate your direction. Thank you.

Answer 1

I make the potentially false assumption that fitting a probability model in the frequentist way is virtually the same as fitting the same model with a flat prior in a Bayesian way. Please nuance or correct that as interest number one (1).

If the flat prior contains the Maximum Likelihood Estimator (MLE), then the MAP (Maximum A Posteriori) and the MLE coincide. However, despite these values coincide, their interpretation is entirely different. Some people may argue that the MAP is not a Bayesian estimator, though.

And my main interest (2) is if my assumption is true (and if a flat prior happens to be the best prior I could possibly come up with), does the posterior of the model fitted in a frequentist way, but sampled say with an MCMC sampler, allow for me to make a Bayesian type of interpretation? e.g., the probability that an individual described by some particular configuration of X will have an income (Y) greater than $100K is 76%.

The sentence "the posterior of the model fitted in a frequentist way" makes no sense. You have to either estimate the parameters using a Bayesian approach or a frequentist approach.

Recall that the MLE is defined as the value of the parameter, that maximises the likelihood, this is

$$\hat{\theta} = \mathrm{argmax}_{\theta}\, {\mathcal L}(\theta;\text{Data}),$$

therefore no MCMC is involved here necessarily, while Bayesian inference is based on the posterior distribution

$$\pi(\theta\vert \text{Data}) \propto {\mathcal L}(\theta;\text{Data}) \pi(\theta).$$

Also, Bayesian estimators are not unique. MCMC is just a numerical technique to obtain samples from the posterior distribution and it has no philosophical implications. The same happens with the MLE, if doesn't matter if you calculate it through the Newton-Raphson method or a Simmulated Annealing, its interpretation is the same.