Bayesian linear regression KL divergence

Question

$$y_i \sim N(w_0 + w_1x_i, \sigma^2_j)$$ $$\mathbf{w} \sim N(0,\alpha^2 I) $$ Data is $D$, posterior distribution $p(\mathbf{w}|D)$ is approximated according to mean-field approximation $$p(\mathbf{w}|D) \approx \prod_{d=0}^1 q(w_d)= \prod_{d=0}^1 N(w_d | \mu_d, \sigma^2_d)$$

How to calculate the KL divergence of: $$ \text{KL} \left[q_{\lambda}(\mathbf{w}) | p(\mathbf{w})\right]$$

I know that analytical solution of KL for two univariate gaussians would be: $$-\frac{1}{2} + \log(\frac{\sigma_1^2}{\sigma_2^2}) + \frac{\sigma_2^2 + (\mu_1-\mu_2)^2}{2\sigma_1^2}$$ But not sure what to do here since $p(\mathbf{w}|D)$ itself has two components.

Answer 1

You can use the formula for KL divergence between two multivariate normal distributions: $$D_{KL}(\mathcal N_0\vert\vert N_1)=\frac{1}{2}\left(\text{tr}(\Sigma_1^{-1}\Sigma_0)+(\mu_1-\mu_0)^T\Sigma_1^{-1}(\mu_1-\mu_0)-k+\ln\left(\frac{\det\Sigma_1}{\det\Sigma_0}\right)\right)$$

In your case, $\mathcal N_0=\mathcal N \left(\mathbf{\mu_0}=\begin{bmatrix}\mu_0\\\mu_1\end{bmatrix}, \mathbf{\Sigma_0}=\begin{bmatrix}\sigma_0^2&0\\0&\sigma_1^2\end{bmatrix}\right)$, and $\mathcal N_1=\mathcal N( \mathbf{\mu_1}\mathbf = 0, \mathbf{\Sigma_1}=\alpha^2\mathbf I)$, and $k=2$. Note the differences between the bold and normal fonts, i.e. $\mathbf{\mu_0}\neq\mu_0$.