I need to learn both the VB and EM methods for Bayesian Networks. Before going into detail of both algorithms, which I am a bit aware of, I need to EXACTLY understand the basic motivations behind them. Different resources use the terms "inference, parameters, estimation, learning" so intermingled that I easily lose the track and find myself more confused.

I will try to explain the purpose of the algorithms in a comparative way, as far as I understand them for now (probably incorrectly). As the question, I am kindly asking you to correct me and to show me my thinking errors in my explanation below so I can correctly grasp the fundamentals.

So, we have a very general system of random variables. It is commonly said that there are no parameters in a Bayesian Inference task (only variables) and therefore we have two types of variables: Set of observed variables, $D$ and unobserved ones. The unobserved ones consists of the set of the latent variables (and/or observable but missing variables) $Z$ and the ones on which we want to make an estimation, inference, etc, $X$. The whole system consists of the random variable sets $D, X$ and $Z$.

So the general Bayesian Inference problem is $P(X|D) = \int P(X,Z|D)dZ = \int P(X|Z,D)P(Z|D)dZ$ , we condition on the given data and integrate out all "nuisance" variables.

As far as I understand, EM tries to make a "point" estimate to the posterior distribution $P(X|D)$ by finding (not exactly finding, by converging actually) $X^* = arg max_{X} P(X|D)$ in which $X^*$ is the point estimates of all variables $x$ which are $x \in X$. So we take these estimates as the "observed values of $X$" from now on and use in the following inference steps along with the $D$. For example if we have two disjoint $Z_1$ and $Z_2$ with $Z_1 \cup Z_2 = Z$ and we want to infer $Z_1$ it is now $P(Z_1|D,X=X^*) = \int P(Z_1,Z_2|D,X=X^*) dZ_2$

The Variational Bayes method treats $P(X|D) = \int P(X,Z|D)dZ = \int P(X|Z,D)P(Z|D)dZ$ by finding a tractable approximation to the posterior $P(X|D)$, say, $Q(X)$. This is not a point estimate, rather a simpler but complete distribution. We can use it for future inferences based on the data $D$. So again using $Z_1$ and $Z_2$, it is:

$P(Z_1|D) = \int\int P(Z_1,Z_2,X|D) dZ_2dX = \int\int P(Z_1,Z_2|X,D) P(X|D) dZ_2 dX = \int\int P(Z_1,Z_2|X,D) Q(X) dZ_2 dX$

for a new inference.

Now, are these really the basis for the both algorithms, did I understand their functions correctly? If not, please kindly show me the correct ones.

Thanks in advance


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.