3
$\begingroup$

I need to learn both the VB and EM methods for Bayesian Networks. Before going into detail of both algorithms, which I am a bit aware of, I need to EXACTLY understand the basic motivations behind them. Different resources use the terms "inference, parameters, estimation, learning" so intermingled that I easily lose the track and find myself more confused.

I will try to explain the purpose of the algorithms in a comparative way, as far as I understand them for now (probably incorrectly). As the question, I am kindly asking you to correct me and to show me my thinking errors in my explanation below so I can correctly grasp the fundamentals.

So, we have a very general system of random variables. It is commonly said that there are no parameters in a Bayesian Inference task (only variables) and therefore we have two types of variables: Set of observed variables, $D$ and unobserved ones. The unobserved ones consists of the set of the latent variables (and/or observable but missing variables) $Z$ and the ones on which we want to make an estimation, inference, etc, $X$. The whole system consists of the random variable sets $D, X$ and $Z$.

So the general Bayesian Inference problem is $P(X|D) = \int P(X,Z|D)dZ = \int P(X|Z,D)P(Z|D)dZ$ , we condition on the given data and integrate out all "nuisance" variables.

As far as I understand, EM tries to make a "point" estimate to the posterior distribution $P(X|D)$ by finding (not exactly finding, by converging actually) $X^* = arg max_{X} P(X|D)$ in which $X^*$ is the point estimates of all variables $x$ which are $x \in X$. So we take these estimates as the "observed values of $X$" from now on and use in the following inference steps along with the $D$. For example if we have two disjoint $Z_1$ and $Z_2$ with $Z_1 \cup Z_2 = Z$ and we want to infer $Z_1$ it is now $P(Z_1|D,X=X^*) = \int P(Z_1,Z_2|D,X=X^*) dZ_2$

The Variational Bayes method treats $P(X|D) = \int P(X,Z|D)dZ = \int P(X|Z,D)P(Z|D)dZ$ by finding a tractable approximation to the posterior $P(X|D)$, say, $Q(X)$. This is not a point estimate, rather a simpler but complete distribution. We can use it for future inferences based on the data $D$. So again using $Z_1$ and $Z_2$, it is:

$P(Z_1|D) = \int\int P(Z_1,Z_2,X|D) dZ_2dX = \int\int P(Z_1,Z_2|X,D) P(X|D) dZ_2 dX = \int\int P(Z_1,Z_2|X,D) Q(X) dZ_2 dX$

for a new inference.

Now, are these really the basis for the both algorithms, did I understand their functions correctly? If not, please kindly show me the correct ones.

Thanks in advance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.