$$y_i \sim N(w_0 + w_1x_i, \sigma^2_j)$$ $$\mathbf{w} \sim N(0,\alpha^2 I) $$ Data is $D$, posterior distribution $p(\mathbf{w}|D)$ is approximated according to mean-field approximation $$p(\mathbf{w}|D) \approx \prod_{d=0}^1 q(w_d)= \prod_{d=0}^1 N(w_d | \mu_d, \sigma^2_d)$$

How to calculate the KL divergence of: $$ \text{KL} \left[q_{\lambda}(\mathbf{w}) | p(\mathbf{w})\right]$$

I know that analytical solution of KL for two univariate gaussians would be: $$-\frac{1}{2} + \log(\frac{\sigma_1^2}{\sigma_2^2}) + \frac{\sigma_2^2 + (\mu_1-\mu_2)^2}{2\sigma_1^2}$$ But not sure what to do here since $p(\mathbf{w}|D)$ itself has two components.


You can use the formula for KL divergence between two multivariate normal distributions: $$D_{KL}(\mathcal N_0\vert\vert N_1)=\frac{1}{2}\left(\text{tr}(\Sigma_1^{-1}\Sigma_0)+(\mu_1-\mu_0)^T\Sigma_1^{-1}(\mu_1-\mu_0)-k+\ln\left(\frac{\det\Sigma_1}{\det\Sigma_0}\right)\right)$$

In your case, $\mathcal N_0=\mathcal N \left(\mathbf{\mu_0}=\begin{bmatrix}\mu_0\\\mu_1\end{bmatrix}, \mathbf{\Sigma_0}=\begin{bmatrix}\sigma_0^2&0\\0&\sigma_1^2\end{bmatrix}\right)$, and $\mathcal N_1=\mathcal N( \mathbf{\mu_1}\mathbf = 0, \mathbf{\Sigma_1}=\alpha^2\mathbf I)$, and $k=2$. Note the differences between the bold and normal fonts, i.e. $\mathbf{\mu_0}\neq\mu_0$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.