I have been reading this (http://www.jting.net/pubs/2007/ting-ICRA2007.pdf) paper and attempting to derive the Variational EM update equations here. So, the model is given as follows:

$$ y_i \sim N(\beta x_i, \frac{w_i}{\sigma^2}) \\ \beta \sim N(\beta_0, \Sigma_0) \\ w_i \sim G(a, b) $$

So, here $y_i$ are the observations that depend on $x_i$ through the coefficients $\beta$. There are normal and gamma distributional assumptions on $\beta$ and $w_i$, which are the noise variances. The idea being every point is independent but not identically distributed.

Now, the joint model is written as:

$$ p(\beta) \prod_{i=1}^{N}P(y_i|x_i, w_i, \beta) \prod_{i=1}^{N} P(w_i) $$

Taking the log, we have:

$$ \log P(\beta) + \sum_{i=1}^{N} \log P(y_i|x_i, w_i, \beta) + \sum_{i=1}^{N} \log P(w_i) $$

Now, as in the paper, we use variational bayes and mean field approximation to do inference on $\beta$ and $W$. So, for the update on the distribution parameters for $\beta$, I need to take the expectation wrt to $Q(w)$. Doing that, I first expand the equations as follows. I am dropping the expression for $P(w_i)$ as it does not depend on $\beta$.

$$ \sum_{i=1}^{N} \bigg[\log (\frac{1}{\sigma} \sqrt{\frac{w_i}{2 \pi}}) - \frac{1}{2}(y_i -\beta^T x_i) \frac{w_i}{\sigma^2}(y_i - \beta^T x_i)\bigg] + \log \big[(2 \pi)^{-k/2} |\Sigma_0|^{-1/2}\big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big] $$

Here $k$ is the number of $\beta$ components. Now, dropping some constant terms wrt $\beta$, I have:

$$ \big[\frac{1}{2} \sum_{i=1}^{N} \log (w_i) - \frac{w_i}{\sigma^2}(y_i - \beta^T x_i)(y_i - \beta^T x_i)\big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big] $$

Now, in the paper the expectation for $\beta$ is given as:

$$ <\beta> = \Sigma_{\beta}\big(\Sigma_0^{-1}\beta_0 + \frac{1}{\sigma^2} \sum_{i=1}^{N} <w_i>y_ix_i \big) $$

Now, I am having a lot of trouble deriving this step from the expression that I got before. Any pointers would be greatly appreciated!

  • 1
    $\begingroup$ The notations do not add up: $y_i$ is a scalar, $\beta$ is a vector so (a) $\beta x_i$ should be $\beta^Tx_i$ and (b) there is no reason for a transpose in the scala $(y_i - \beta x_i)^T$. $\endgroup$ – Xi'an Nov 6 '15 at 11:36
  • $\begingroup$ @Xi'an Oh yes, sorry that is a mistake on my part. I will clean this up now $\endgroup$ – Luca Nov 6 '15 at 11:37

If you drop all the terms independent from $\beta$ in $$\big[\frac{1}{2} \sum_{i=1}^{N} \log (w_i) - \frac{w_i}{\sigma^2}(y_i - \beta^T x_i)(y_i - \beta^T x_i)\big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big]$$you get $$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} w_i \{(\beta^T x_i)^2 - 2y_i \beta^T x_i \} \big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big]$$and taking prior expectations in $\beta$ and in $w_i$ get you to$$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} <w_i> \{(\beta_0^T x_i)^2 +\text{tr}(x_ix_i^T\Sigma_0) - 2y_i \beta_0^T x_i \} \big] - \frac{\text{dim}(\beta)}{2} $$ or, up to an additive constant$$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} <w_i> \{x_i^T\Sigma_0 x_i +(y_i - \beta_0^T x_i)^2 \} \big] - \frac{\text{dim}(\beta)}{2} $$ The trace appears and disappears because of the following $$(\beta^Tx_i)^2=\underbrace{\beta^Tx_ix_i^T\beta}_{\substack{\text{scalar}\\\text{equal to}\\\text{transpose}}}=\underbrace{\text{tr}(\beta^Tx_ix_i^T\beta)}_{\substack{\text{trace of}\\\text{scalar}}}=\underbrace{\text{tr}(x_ix_i^T\beta\beta^T)}_{\substack{\text{invariance}\\\text{by}\\\text{shifting}}}$$ and$$\mathbb{E}[\text{tr}(x_ix_i^T\beta\beta^T)=\text{tr}(\mathbb{E}[x_ix_i^T\beta\beta^T])=\text{tr}(x_ix_i^T\mathbb{E}[\beta\beta^T])= \text{tr}(x_ix_i^T\{\beta_0\beta^T_0+\Sigma_0\}) $$

  • $\begingroup$ Thanks for the answer. Can you elaborate on why you take the prior expectation in $\beta$ and $w_i$? I am having difficulty understanding that step. Also, this expression does seem very different from the one in the paper as well. Also, any chance you could point me to the identity that let you simplify the second term with the trace operator? $\endgroup$ – Luca Nov 6 '15 at 12:05
  • $\begingroup$ That is how I understood your question. If you only want the expectation in $w_i$ then it is$$-\frac{1}{2\sigma^2}\big[\sum_{i=1}^{N} <w_i >\{(\beta^T x_i)^2 - 2y_i \beta^T x_i \} \big] - \frac{1}{2} \big[(\beta - \beta_0)^T \Sigma_0^{-1}(\beta - \beta_0)\big]$$because it is a linear function of the $w_i$'s. $\endgroup$ – Xi'an Nov 6 '15 at 12:08
  • $\begingroup$ In this case I am looking at the update for the expectation of $\beta$ and the way I understood it from looking at VB, it is done by taking the expectation wrt to the other parameter distribution, which in my case is $W$ i.e. $Q(\beta, W) \approx Q(\beta) Q(W)$ and I am taking the expectation wrt to $Q(W)$ to get the posterior approximation for $Q(\beta)$ $\endgroup$ – Luca Nov 6 '15 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.