help to reproduce this derivation

Question

I have been reading this (http://www.jting.net/pubs/2007/ting-ICRA2007.pdf) paper and attempting to derive the Variational EM update equations here. So, the model is given as follows:

$$ y_i \sim N(\beta x_i, \frac{w_i}{\sigma^2}) \\ \beta \sim N(\beta_0, \Sigma_0) \\ w_i \sim G(a, b) $$

So, here $y_i$ are the observations that depend on $x_i$ through the coefficients $\beta$. There are normal and gamma distributional assumptions on $\beta$ and $w_i$, which are the noise variances. The idea being every point is independent but not identically distributed.

Now, the joint model is written as:

$$ p(\beta) \prod_{i=1}^{N}P(y_i|x_i, w_i, \beta) \prod_{i=1}^{N} P(w_i) $$

Taking the log, we have:

$$ \log P(\beta) + \sum_{i=1}^{N} \log P(y_i|x_i, w_i, \beta) + \sum_{i=1}^{N} \log P(w_i) $$

Now, as in the paper, we use variational bayes and mean field approximation to do inference on $\beta$ and $W$. So, for the update on the distribution parameters for $\beta$, I need to take the expectation wrt to $Q(w)$. Doing that, I first expand the equations as follows. I am dropping the expression for $P(w_i)$ as it does not depend on $\beta$.

$$ \sum_{i=1}^{N} \bigg[\log (\frac{1}{\sigma} \sqrt{\frac{w_i}{2 \pi}}) - \frac{1}{2}(y_i -\beta^T x_i) \frac{w_i}{\sigma^2}(y_i - \beta^T x_i)\bigg] + \log \big[(2 \pi)^{-k/2} |\Sigma_0|^{-1/2}\big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big] $$

Here $k$ is the number of $\beta$ components. Now, dropping some constant terms wrt $\beta$, I have:

$$ \big[\frac{1}{2} \sum_{i=1}^{N} \log (w_i) - \frac{w_i}{\sigma^2}(y_i - \beta^T x_i)(y_i - \beta^T x_i)\big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big] $$

Now, in the paper the expectation for $\beta$ is given as:

$$ <\beta> = \Sigma_{\beta}\big(\Sigma_0^{-1}\beta_0 + \frac{1}{\sigma^2} \sum_{i=1}^{N} <w_i>y_ix_i \big) $$

Now, I am having a lot of trouble deriving this step from the expression that I got before. Any pointers would be greatly appreciated!

Answer 1

If you drop all the terms independent from $\beta$ in $$\big[\frac{1}{2} \sum_{i=1}^{N} \log (w_i) - \frac{w_i}{\sigma^2}(y_i - \beta^T x_i)(y_i - \beta^T x_i)\big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big]$$you get $$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} w_i \{(\beta^T x_i)^2 - 2y_i \beta^T x_i \} \big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big]$$and taking prior expectations in $\beta$ and in $w_i$ get you to$$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} <w_i> \{(\beta_0^T x_i)^2 +\text{tr}(x_ix_i^T\Sigma_0) - 2y_i \beta_0^T x_i \} \big] - \frac{\text{dim}(\beta)}{2} $$ or, up to an additive constant$$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} <w_i> \{x_i^T\Sigma_0 x_i +(y_i - \beta_0^T x_i)^2 \} \big] - \frac{\text{dim}(\beta)}{2} $$ The trace appears and disappears because of the following $$(\beta^Tx_i)^2=\underbrace{\beta^Tx_ix_i^T\beta}_{\substack{\text{scalar}\\\text{equal to}\\\text{transpose}}}=\underbrace{\text{tr}(\beta^Tx_ix_i^T\beta)}_{\substack{\text{trace of}\\\text{scalar}}}=\underbrace{\text{tr}(x_ix_i^T\beta\beta^T)}_{\substack{\text{invariance}\\\text{by}\\\text{shifting}}}$$ and$$\mathbb{E}[\text{tr}(x_ix_i^T\beta\beta^T)=\text{tr}(\mathbb{E}[x_ix_i^T\beta\beta^T])=\text{tr}(x_ix_i^T\mathbb{E}[\beta\beta^T])= \text{tr}(x_ix_i^T\{\beta_0\beta^T_0+\Sigma_0\}) $$