Can I call a model wherein Bayes' Theorem is used a "Bayesian model"? I am afraid such a definition might be too broad.
So what exactly is a Bayesian model?
$\begingroup$
$\endgroup$
1
asked Dec 14, 2014 at 4:02
-
10$\begingroup$ A Bayesian model is a statistical model made of the pair prior x likelihood = posterior x marginal. Bayes' theorem is somewhat secondary to the concept of a prior. $\endgroup$– Xi'anCommented Dec 14, 2014 at 7:02
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
10
In essence, one where inference is based on using Bayes theorem to obtain a posterior distribution for a quantity or quantities of interest form some model (such as parameter values) based on some prior distribution for the relevant unknown parameters and the likelihood from the model.
i.e. from a distributional model of some form, $f(X_i|\mathbf{\theta})$, and a prior $p(\mathbf{\theta})$, someone might seek to obtain the posterior $p(\mathbf{\theta}|\mathbf{X})$.
A simple example of a Bayesian model is discussed in this question, and in the comments of this one - Bayesian linear regression, discussed in more detail in Wikipedia here. Searches turn up discussions of a number of Bayesian models here.
But there are other things one might try to do with a Bayesian analysis besides merely fit a model - see, for example, Bayesian decision theory.
-
$\begingroup$ In linear regression, is $\theta$ here equal to the vector $[\beta_0, \beta_1, ..., \beta_n]$? If not, what is it? $\endgroup$– BCLCCommented Aug 2, 2015 at 9:32
-
1$\begingroup$ @BCLC It would usually include $\sigma$ as well. $\endgroup$– Glen_bCommented Aug 2, 2015 at 9:37
-
1$\begingroup$ @BCLC You seem to be conflating frequentist and Bayesian inference. Bayesian inference focuses on whatever quantities you're interested in. If you're interested in parameters (e.g. inference about particular coefficients), the idea would be to seek posterior distributions [parameters|data]. If you're interested in the the mean function ($\mu_Y|X$), then you would seek a posterior distribution for that (which is of course a function of the (multivariate) distribution of $\mathbb{\beta}$). You might use OLS in your estimation, but the parameters of the posterior will be shifted by the prior ... $\endgroup$– Glen_bCommented Aug 2, 2015 at 10:48
-
1$\begingroup$ ... see the wikipedia page on Bayesian regression and some of the discussions here on CV $\endgroup$– Glen_bCommented Aug 2, 2015 at 10:49
-
1$\begingroup$ That calculation sometimes comes up (whether you called it $\sigma^2$ or $\phi$), for various reasons. My earlier comment is not in any way in conflict with that calculation. $\sigma$ (or equivalently $\sigma^2$ or $\phi$) is a parameter, and you have to deal with it along with the other parameters. However, while it would be rare that you know $\sigma$; for example if you are doing Gibbs sampling, the conditional would be relevant. If you just want inference on $\beta$, you'd integrate $\sigma$ (or $\sigma^2$ etc) out from $\theta|y$ rather than condition on $\sigma$. $\endgroup$– Glen_bCommented Aug 5, 2015 at 21:02
$\begingroup$
$\endgroup$
A Bayesian model is just a model that draws its inferences from the posterior distribution, i.e. utilizes a prior distribution and a likelihood which are related by Bayes' theorem.
$\begingroup$
$\endgroup$
2
Can I call a model wherein Bayes' Theorem is used a "Bayesian model"?
No
I am afraid such a definition might be too broad.
You are right. Bayes' theorem is a legitimate relation between marginal event probabilities and conditional probabilities. It holds regardless of your interpretation of probability.
So what exactly is a Bayesian model?
If you're using prior and posterior concepts anywhere in your exposition or interpretation, then you're likely to be using model Bayesian, but this is not the absolute rule, because these concepts are also used in non-Bayesian approaches.
In a broader sense though you must be subscribing to Bayesian interpretation of probability as a subjective belief. This little theorem of Bayes was extended and stretched by some people into this entire world view and even, shall I say, philosophy. If you belong to this camp then you are Bayesian. Bayes had no idea this would happen to his theorem. He'd be horrified, me thinks.
-
4$\begingroup$ This appears to be the first answer to introduce the important point made in its first line: the mere use of Bayes' Theorem does not make something a Bayesian model. I would like to encourage you to go further with this thought. You seem to back down where you say that "using prior and posterior concepts" makes a model Bayesian. Doesn't that simply amount to applying Bayes' Theorem again? If not, could you explain what you mean by "concepts" in this passage? After all, classical (non-Bayesian) statistics uses priors and posteriors to prove admissibility of many procedures. $\endgroup$– whuber ♦Commented Dec 29, 2014 at 15:23
-
$\begingroup$ @whuber, it was more like a simple rule of thumb. Whenever I see "prior" in the paper it ends up being or claiming to be from Bayesian point of view. I'll clarify my point though. $\endgroup$– AksakalCommented Dec 29, 2014 at 15:26
$\begingroup$
$\endgroup$
4
A statistical model can be seen as a procedure/story describing how some data came to be. A Bayesian model is a statistical model where you use probability to represent all uncertainty within the model, both the uncertainty regarding the output but also the uncertainty regarding the input (aka parameters) to the model. The whole prior/posterior/Bayes theorem thing follows on this, but in my opinion, using probability for everything is what makes it Bayesian (and indeed a better word would perhaps just be something like probabilistic model).
That means that most other statistical models can be "cast into" a Bayesian model by modifying them to be using probability everywhere. This is especially true for models that rely on maximum likelihood, as maximum likelihood model fitting is a strict subset to Bayesian model fitting.
-
$\begingroup$ MLE is used and was developed outside Bayesian model, so it's not very clear what you mean by it being "strict subset to Bayesian model fitting". $\endgroup$– AksakalCommented Dec 29, 2014 at 18:04
-
$\begingroup$ From a Bayesian perspective the MLE is what you get when you assume flat priors, fit the model and use the most probable parameter configuration as a point estimate. Whether this is a special case of Bayesian "philosophy of statistics" I leave for others to discuss, but it is certainly a special case of Bayesian model fitting. $\endgroup$ Commented Dec 29, 2014 at 21:56
-
$\begingroup$ The problem with this statement is that it leaves an impression that you need to be subscribed to some sort of Bayesian thinking in order to use MLE. $\endgroup$– AksakalCommented Dec 29, 2014 at 21:58
-
1$\begingroup$ I'm not sure what you mean. IMO you don't need to subscribe to Bayesian thinking when using Bayesian statistics more than you need to subscribe to matrix think when doing linear algebra or Gaussian thinking when using a normal distribution. I'm also not saying that MLE has to be interpreted as a subset of Bayesian model fitting (even though it falls pretty natural to me). $\endgroup$ Commented Dec 29, 2014 at 22:04
$\begingroup$
$\endgroup$
5
Your question is more on the semantic side: when can I call a model "Bayesian"?
Drawing conclusions from this excellent paper:
Fienberg, S. E. (2006). When did bayesian inference become "bayesian"? Bayesian Analysis, 1(1):1-40.
there are 2 answers:
- Your model is first Bayesian if it uses Bayes' rule (that's the "algorithm").
- More broadly, if you infer (hidden) causes from a generative model of your system, then you are Bayesian (that's the "function").
Surprisingly, the "Bayesian models" terminology that is used all over the field only settled down around the 60s. There are many things to learn about machine learning just by looking at its history!
-
$\begingroup$ You seem to mention only one of the "two answers". Maybe write something about both? $\endgroup$– TimCommented Dec 19, 2014 at 8:59
-
$\begingroup$ thanks for the note, I edited my answer to separate the 2 parts of my sentence. $\endgroup$– meduzCommented Dec 29, 2014 at 15:02
-
-
1$\begingroup$ thanks for noticing @prosti ! I have updated the link - after seven years since I answered this question, still of actuality... $\endgroup$– meduzCommented May 17, 2021 at 7:20
-
$\begingroup$ Link broke again - I updated your broken link with a link pointing to the DOI. $\endgroup$– StefanCommented Nov 11, 2023 at 20:39