Is my interpretation of Bayesian probability and inference correct?

Question

I have the following interpretation of the Bayesian probability and inference (without referring to Measure Theory, I am still at the very beginning of learning it): Let's say we have five random variables $x_1,x_2,x_3,x_4,p$ given as in the following graphical model:

Let's say we have already observed $x_1,x_2,x_3$ and we want to make a Bayesian inference on the variable $x_4$.

We assume a prior probability density function $p(p)$ for the parameter $p$ and we assume that each $x_i$ is generated from the distribution $f(x_i|p)$. Then according to our assumptions about the prior distribution $p$ and the generative distribution $f$, we assume a sample space $\Omega$ which contains all possible tuples of $(x_1,x_2,x_3,x_4,p)$ and each single outcome $\omega$ has a probability value assigned to it, which corresponds to our degree of belief about this single outcome, according to the Bayesian interpretation of probability. Then we have the event set $\mathcal{F}$ as well, according to the definition of a probability space. This space and its probability measures entail the prior distribution $p(p)$ and the distributions $f(x_i|p)$ and the independence of $x_i$s given $p$ . I assume that one of these outcomes, $\omega$ is selected and all events in $\mathcal{F}$ which contain $\omega$ are assumed as "occured".

In practice, we cannot ever observe the value of the parameter $p$ in the selected outcome, $\omega$. But we are able to observe some of the $x_i$s which constitute our data, for example $D=(x_1,x_2,x_3)$. Then the broadest inference we can do about $x_4$ is to calculate the posterior distribution of $p$ given the data and then integrate over all possible single values of $p$, which might be generated the data $x_1,x_2,x_3$ and the target variable $x_4$ like in the following:

$$ P(x_4|x_1,x_2,x_3) = \int_{p} f(x_4|p) P(p|x_1,x_2,x_3) dp$$

This more or less constitutes my imagination of how the Bayesian probability and inference works. Is this view correct? If not, what is wrong about it?

Answer 1

Your mathematics are all correct, but it is useful to distinguish between parts of your analysis that are just standard model formation pertaining to the sampling mechanism (which would also occur in classical statistics) and parts that are specific to a Bayesian analysis.

The statements you have made about the allowable values of the variables, and the sampling density $f(x|p)$ are model assumptions pertaining to the sampling mechanism. They are nothing to do with the Bayesian method. Regardless of whether you are doing your analysis with Bayesian methods or classical methods, you will need to form a belief about the allowable parameter space $p \in \mathcal{P}$ and the sampling density $f (x_1, x_2, x_3, x_4 | p)$ that represents the sampling mechanism. There is no necessity in either statistical methodology to assume that the $x$ values are IID, but if you think that appropriately represents the sampling mechanism, then that assumption is fine.

The specification of the sampling mechanism gives you the likelihood function $L_\boldsymbol{x}(p)$. You can proceed from this point using Bayesian methods, or classical methods (MLEs, MOMs, classical hypothesis tests etc.). If you adopt the Bayesian approach then the only additional thing you need is a prior distribution on the parameter $p$ to represent your prior uncertainty about this parameter (or if you want to do robust Bayesian analysis, you might have a set of priors distributions instead).

By specifying a prior distribution for the parameter, you can derive the posterior and predictive distributions of interest. You are correct that the predictive distribution for $x_4$ is:

$$f(x_4| x_1, x_2, x_3) = \int \limits_\mathcal{P} f(x_4 | p) \pi (p| x_1, x_2, x_3) dp.$$

(Since you already have a parameter called $p$, I have used the common notation $\pi$ to denote the posterior density.) Remember, Bayesian inference works by representing uncertainty about the parameters as a probabilistic belief, and then proceeding naturally from this starting point to the posterior belief. Bayesian and classical methods both start with consideration of the sampling mechanism and the resulting sampling distributions, and neither necessitate any particular form.