1
$\begingroup$

In the expectation propagation for the generative aspect model, Minka uses Taylor series for the parameter estimation of the topics $p(w\mid a)$ eq 31.

I am a little confused in the last equation. He expresses the expectation of a function in terms of Taylor expansion as follows (eq 40), $Var(\lambda)$ is the covariance matrix of $\lambda$:

\begin{equation} \mathbb{E}\left[f(\boldsymbol\lambda)\right] \approx f(\mathbb{E}\left[\boldsymbol\lambda\right]) + \frac{1}{2} Tr\left(f''(\mathbb{E}\left[\boldsymbol\lambda\right]) Var(\boldsymbol\lambda)\right) \end{equation}

However, in another post I found the following derivation for multivariate Taylor expansion:

\begin{equation} \mathbb{E}[f(\lambda)] \approx f(\mathbb{E}\lambda) + \frac{1}{2} \sum_{i=1}^n H_f(\mathbb{E}\lambda)_{ii} Var(\lambda_i). \end{equation}

The only difference is that in the first approximation Minka gets the product of the hessian and the covariance matrix inside the trace operation. This involves the interaction terms $Cov(\lambda_i,\lambda_j)$. However, Michał Stolarczyk in the stats exchange post gets the trace of the diagonal of the hessian and the diagonal of the covariance matrix; for instance no interaction terms.

Using the interactions terms of the covariance matrix, I get the expression (eq 33) by Minka in his paper:

\begin{equation} S_{ia} = \frac{\sum_bp(w\mid b)^2m_{iab}}{(\sum_bp(w\mid b)m_{iab})^2}-1 \end{equation}

However, using Michał's expression directs me to the following expression:

\begin{equation} S_{ia} = \frac{\sum_bp(w\mid b)^2m_{iab}-\sum_bp(w\mid b)^2m_{iab}^2}{(\sum_bp(w\mid b)m_{iab})^2} \end{equation}

Minka's result uses the interaction terms and the one shown comes from the following expression

\begin{equation} (\sum_bp(w\mid b)m_{iab})^2=\sum_bp(w\mid b)^2m_{iab}^2 + \sum_{k\neq j}p(w\mid b=i)p(w\mid b=j)m_{iak}m_{iaj} \end{equation}

However, Michał's derivation makes sense to me. So, I am confused about the expression of multivariate Taylor expansion for the moments of functions of random variables. Which one is correct or when should I use either one?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.