In the expectation propagation for the generative aspect model, Minka uses Taylor series for the parameter estimation of the topics $p(w\mid a)$ eq 31.

I am a little confused in the last equation. He expresses the expectation of a function in terms of Taylor expansion as follows (eq 40), $Var(\lambda)$ is the covariance matrix of $\lambda$:

\begin{equation} \mathbb{E}\left[f(\boldsymbol\lambda)\right] \approx f(\mathbb{E}\left[\boldsymbol\lambda\right]) + \frac{1}{2} Tr\left(f''(\mathbb{E}\left[\boldsymbol\lambda\right]) Var(\boldsymbol\lambda)\right) \end{equation}

However, in another post I found the following derivation for multivariate Taylor expansion:

\begin{equation} \mathbb{E}[f(\lambda)] \approx f(\mathbb{E}\lambda) + \frac{1}{2} \sum_{i=1}^n H_f(\mathbb{E}\lambda)_{ii} Var(\lambda_i). \end{equation}

The only difference is that in the first approximation Minka gets the product of the hessian and the covariance matrix inside the trace operation. This involves the interaction terms $Cov(\lambda_i,\lambda_j)$. However, Michał Stolarczyk in the stats exchange post gets the trace of the diagonal of the hessian and the diagonal of the covariance matrix; for instance no interaction terms.

Using the interactions terms of the covariance matrix, I get the expression (eq 33) by Minka in his paper:

\begin{equation} S_{ia} = \frac{\sum_bp(w\mid b)^2m_{iab}}{(\sum_bp(w\mid b)m_{iab})^2}-1 \end{equation}

However, using Michał's expression directs me to the following expression:

\begin{equation} S_{ia} = \frac{\sum_bp(w\mid b)^2m_{iab}-\sum_bp(w\mid b)^2m_{iab}^2}{(\sum_bp(w\mid b)m_{iab})^2} \end{equation}

Minka's result uses the interaction terms and the one shown comes from the following expression

\begin{equation} (\sum_bp(w\mid b)m_{iab})^2=\sum_bp(w\mid b)^2m_{iab}^2 + \sum_{k\neq j}p(w\mid b=i)p(w\mid b=j)m_{iak}m_{iaj} \end{equation}

However, Michał's derivation makes sense to me. So, I am confused about the expression of multivariate Taylor expansion for the moments of functions of random variables. Which one is correct or when should I use either one?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.