What exactly is a Bayesian model?

Can I call a model wherein Bayes' Theorem is used a "Bayesian model"? I am afraid such a definition might be too broad.

So what exactly is a Bayesian model?

• A Bayesian model is a statistical model made of the pair prior x likelihood = posterior x marginal. Bayes' theorem is somewhat secondary to the concept of a prior. – Xi'an Dec 14 '14 at 7:02

In essence, one where inference is based on using Bayes theorem to obtain a posterior distribution for a quantity or quantities of interest form some model (such as parameter values) based on some prior distribution for the relevant unknown parameters and the likelihood from the model.

i.e. from a distributional model of some form, $f(X_i|\mathbf{\theta})$, and a prior $p(\mathbf{\theta})$, someone might seek to obtain the posterior $p(\mathbf{\theta}|\mathbf{X})$.

A simple example of a Bayesian model is discussed in this question, and in the comments of this one - Bayesian linear regression, discussed in more detail in Wikipedia here. Searches turn up discussions of a number of Bayesian models here.

But there are other things one might try to do with a Bayesian analysis besides merely fit a model - see, for example, Bayesian decision theory.

• In linear regression, is $\theta$ here equal to the vector $[\beta_0, \beta_1, ..., \beta_n]$? If not, what is it? – BCLC Aug 2 '15 at 9:32
• @BCLC It would usually include $\sigma$ as well. – Glen_b -Reinstate Monica Aug 2 '15 at 9:37
• @BCLC You seem to be conflating frequentist and Bayesian inference. Bayesian inference focuses on whatever quantities you're interested in. If you're interested in parameters (e.g. inference about particular coefficients), the idea would be to seek posterior distributions [parameters|data]. If you're interested in the the mean function ($\mu_Y|X$), then you would seek a posterior distribution for that (which is of course a function of the (multivariate) distribution of $\mathbb{\beta}$). You might use OLS in your estimation, but the parameters of the posterior will be shifted by the prior ... – Glen_b -Reinstate Monica Aug 2 '15 at 10:48
• ... see the wikipedia page on Bayesian regression and some of the discussions here on CV – Glen_b -Reinstate Monica Aug 2 '15 at 10:49
• That calculation sometimes comes up (whether you called it $\sigma^2$ or $\phi$), for various reasons. My earlier comment is not in any way in conflict with that calculation. $\sigma$ (or equivalently $\sigma^2$ or $\phi$) is a parameter, and you have to deal with it along with the other parameters. However, while it would be rare that you know $\sigma$; for example if you are doing Gibbs sampling, the conditional would be relevant. If you just want inference on $\beta$, you'd integrate $\sigma$ (or $\sigma^2$ etc) out from $\theta|y$ rather than condition on $\sigma$. – Glen_b -Reinstate Monica Aug 5 '15 at 21:02

A Bayesian model is just a model that draws its inferences from the posterior distribution, i.e. utilizes a prior distribution and a likelihood which are related by Bayes' theorem.

Can I call a model wherein Bayes' Theorem is used a "Bayesian model"?

No

I am afraid such a definition might be too broad.

You are right. Bayes' theorem is a legitimate relation between marginal event probabilities and conditional probabilities. It holds regardless of your interpretation of probability.

So what exactly is a Bayesian model?

If you're using prior and posterior concepts anywhere in your exposition or interpretation, then you're likely to be using model Bayesian, but this is not the absolute rule, because these concepts are also used in non-Bayesian approaches.

In a broader sense though you must be subscribing to Bayesian interpretation of probability as a subjective belief. This little theorem of Bayes was extended and stretched by some people into this entire world view and even, shall I say, philosophy. If you belong to this camp then you are Bayesian. Bayes had no idea this would happen to his theorem. He'd be horrified, me thinks.

• This appears to be the first answer to introduce the important point made in its first line: the mere use of Bayes' Theorem does not make something a Bayesian model. I would like to encourage you to go further with this thought. You seem to back down where you say that "using prior and posterior concepts" makes a model Bayesian. Doesn't that simply amount to applying Bayes' Theorem again? If not, could you explain what you mean by "concepts" in this passage? After all, classical (non-Bayesian) statistics uses priors and posteriors to prove admissibility of many procedures. – whuber Dec 29 '14 at 15:23
• @whuber, it was more like a simple rule of thumb. Whenever I see "prior" in the paper it ends up being or claiming to be from Bayesian point of view. I'll clarify my point though. – Aksakal Dec 29 '14 at 15:26

A statistical model can be seen as a procedure/story describing how some data came to be. A Bayesian model is a statistical model where you use probability to represent all uncertainty within the model, both the uncertainty regarding the output but also the uncertainty regarding the input (aka parameters) to the model. The whole prior/posterior/Bayes theorem thing follows on this, but in my opinion, using probability for everything is what makes it Bayesian (and indeed a better word would perhaps just be something like probabilistic model).

That means that most other statistical models can be "cast into" a Bayesian model by modifying them to be using probability everywhere. This is especially true for models that rely on maximum likelihood, as maximum likelihood model fitting is a strict subset to Bayesian model fitting.

• MLE is used and was developed outside Bayesian model, so it's not very clear what you mean by it being "strict subset to Bayesian model fitting". – Aksakal Dec 29 '14 at 18:04
• From a Bayesian perspective the MLE is what you get when you assume flat priors, fit the model and use the most probable parameter configuration as a point estimate. Whether this is a special case of Bayesian "philosophy of statistics" I leave for others to discuss, but it is certainly a special case of Bayesian model fitting. – Rasmus Bååth Dec 29 '14 at 21:56
• The problem with this statement is that it leaves an impression that you need to be subscribed to some sort of Bayesian thinking in order to use MLE. – Aksakal Dec 29 '14 at 21:58
• I'm not sure what you mean. IMO you don't need to subscribe to Bayesian thinking when using Bayesian statistics more than you need to subscribe to matrix think when doing linear algebra or Gaussian thinking when using a normal distribution. I'm also not saying that MLE has to be interpreted as a subset of Bayesian model fitting (even though it falls pretty natural to me). – Rasmus Bååth Dec 29 '14 at 22:04

Your question is more on the semantic side: when can I call a model "Bayesian"?

Drawing conclusions from this excellent paper:

Fienberg, S. E. (2006). When did bayesian inference become "bayesian"? Bayesian Analysis, 1(1):1-40.

there are 2 answers:

• Your model is first Bayesian if it uses Bayes' rule (that's the "algorithm").
• More broadly, if you infer (hidden) causes from a generative model of your system, then you are Bayesian (that's the "function").

Surprisingly, the "Bayesian models" terminology that is used all over the field only settled down around the 60s. There are many things to learn about machine learning just by looking at its history!

• You seem to mention only one of the "two answers". Maybe write something about both? – Tim Dec 19 '14 at 8:59
• thanks for the note, I edited my answer to separate the 2 parts of my sentence. – meduz Dec 29 '14 at 15:02