4
$\begingroup$

I have the following interpretation of the Bayesian probability and inference (without referring to Measure Theory, I am still at the very beginning of learning it): Let's say we have five random variables $x_1,x_2,x_3,x_4,p$ given as in the following graphical model:

enter image description here

Let's say we have already observed $x_1,x_2,x_3$ and we want to make a Bayesian inference on the variable $x_4$.

We assume a prior probability density function $p(p)$ for the parameter $p$ and we assume that each $x_i$ is generated from the distribution $f(x_i|p)$. Then according to our assumptions about the prior distribution $p$ and the generative distribution $f$, we assume a sample space $\Omega$ which contains all possible tuples of $(x_1,x_2,x_3,x_4,p)$ and each single outcome $\omega$ has a probability value assigned to it, which corresponds to our degree of belief about this single outcome, according to the Bayesian interpretation of probability. Then we have the event set $\mathcal{F}$ as well, according to the definition of a probability space. This space and its probability measures entail the prior distribution $p(p)$ and the distributions $f(x_i|p)$ and the independence of $x_i$s given $p$ . I assume that one of these outcomes, $\omega$ is selected and all events in $\mathcal{F}$ which contain $\omega$ are assumed as "occured".

In practice, we cannot ever observe the value of the parameter $p$ in the selected outcome, $\omega$. But we are able to observe some of the $x_i$s which constitute our data, for example $D=(x_1,x_2,x_3)$. Then the broadest inference we can do about $x_4$ is to calculate the posterior distribution of $p$ given the data and then integrate over all possible single values of $p$, which might be generated the data $x_1,x_2,x_3$ and the target variable $x_4$ like in the following:

$$ P(x_4|x_1,x_2,x_3) = \int_{p} f(x_4|p) P(p|x_1,x_2,x_3) dp$$

This more or less constitutes my imagination of how the Bayesian probability and inference works. Is this view correct? If not, what is wrong about it?

$\endgroup$
  • 1
    $\begingroup$ I think you're right, but I assume your inference algorithm is for illustration. Belief propagation is much more efficient. $\endgroup$ – Neil G Aug 19 '14 at 11:53
  • $\begingroup$ 1) Yes, I try to show only how the things should work theoretically, there isn't any practical consideration here at all. I studied Bayesian statistics already to some degree and I am now trying to understand how the Bayesian viewpoint sees the world compared to the Frequentist one. One of the biggest differences, as far as I know, should be that we assign "degree of our beliefs" to each single outcome $\omega$ in Bayesian, where the frequentists see the probability measure as the frequency of occurrence for each $\omega$ if we would pick $\omega$s infinitely. $\endgroup$ – Ufuk Can Bicici Aug 19 '14 at 12:11
  • $\begingroup$ 2) The second big difference should be that in the frequentist setting the parameter $p$ is an unknown constant, not a random variable. $p$ is a random variable in the Bayesian setting clearly. I often interpret the graphical model in the question like that as well: First, $p$ is picked from the prior distribution $p(p)$. This picking is unknown to us. Then $x_1,x_2,x_3$ are generated from $f(x_i|p)$ using the picked $p$. We are able to observe this data and use it to make the prediction $P(x_4|x_3,x_2,x_1)$. What confuses me is that some resources say that the value of $p$ changes (cont.) $\endgroup$ – Ufuk Can Bicici Aug 19 '14 at 12:20
  • 1
    $\begingroup$ @UfukCanBiçici In the Bayesian setting, specific to your example, the parameter p is an unknown constant, it does not change. But of course you can have model where it changes too. $\endgroup$ – Mankka Aug 19 '14 at 19:38
  • 1
    $\begingroup$ There is no such thing as an unknown constant within the standard subjectivist Bayesian framework. Constancy is an epistemological state, not an ontological one. $\endgroup$ – Ben Mar 2 '18 at 20:55
2
$\begingroup$

Your mathematics are all correct, but it is useful to distinguish between parts of your analysis that are just standard model formation pertaining to the sampling mechanism (which would also occur in classical statistics) and parts that are specific to a Bayesian analysis.

The statements you have made about the allowable values of the variables, and the sampling density $f(x|p)$ are model assumptions pertaining to the sampling mechanism. They are nothing to do with the Bayesian method. Regardless of whether you are doing your analysis with Bayesian methods or classical methods, you will need to form a belief about the allowable parameter space $p \in \mathcal{P}$ and the sampling density $f (x_1, x_2, x_3, x_4 | p)$ that represents the sampling mechanism. There is no necessity in either statistical methodology to assume that the $x$ values are IID, but if you think that appropriately represents the sampling mechanism, then that assumption is fine.

The specification of the sampling mechanism gives you the likelihood function $L_\boldsymbol{x}(p)$. You can proceed from this point using Bayesian methods, or classical methods (MLEs, MOMs, classical hypothesis tests etc.). If you adopt the Bayesian approach then the only additional thing you need is a prior distribution on the parameter $p$ to represent your prior uncertainty about this parameter (or if you want to do robust Bayesian analysis, you might have a set of priors distributions instead).

By specifying a prior distribution for the parameter, you can derive the posterior and predictive distributions of interest. You are correct that the predictive distribution for $x_4$ is:

$$f(x_4| x_1, x_2, x_3) = \int \limits_\mathcal{P} f(x_4 | p) \pi (p| x_1, x_2, x_3) dp.$$

(Since you already have a parameter called $p$, I have used the common notation $\pi$ to denote the posterior density.) Remember, Bayesian inference works by representing uncertainty about the parameters as a probabilistic belief, and then proceeding naturally from this starting point to the posterior belief. Bayesian and classical methods both start with consideration of the sampling mechanism and the resulting sampling distributions, and neither necessitate any particular form.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.