# Bayesian Linear Regression as a Gaussian Process

I am stuck in showing that Bayesian linear regression can be viewed as a Gaussian Process. Can anybody show me how to get from step (4) to step (5)?

$$\tilde f_i= f(\mathbf{x}^{(i)}) := \mathbf{w}^T\mathbf{x}^{(i)}+b, \quad \mathbf{w}\sim\mathcal{N}(0,\sigma_w^2\mathbb{I}), \; b\sim\mathcal{N}(0,\sigma_b^2)$$

\begin{align} \text{cov}(\tilde f_i,\tilde f_j) &= \mathbb{E}[\tilde f_i \tilde f_j] - \mathbb{E}[\tilde f_i]\mathbb{E}[\tilde f_j] \\ &= \mathbb{E}[\tilde f_i \tilde f_j] - 0\\ &=\mathbb{E}\left[(\mathbf{w}^T\mathbf{x}^{(i)}+b)^T(\mathbf{w}^T\mathbf{x}^{(j)}+b)\right] \tag{4}\\ &= \sigma_w^2 {\mathbf{x}^{(i)}}^T\mathbf{x}^{(j)} + \sigma_b^2 \tag{5} \\ &= k(\mathbf{x}^{(i)},\mathbf{x}^{(j)}). \end{align}

These calculations come from here : https://www.inf.ed.ac.uk/teaching/courses/mlpr/2019/notes/w5b_gaussian_process_kernels.pdf

\begin{align} \text{cov}(\tilde f_i,\tilde f_j) &=\mathbb{E}\left[(\mathbf{w}^T\mathbf{x}^{(i)}+b)^T(\mathbf{w}^T\mathbf{x}^{(j)}+b)\right] \\ &= \mathbb{E}\left[({\mathbf{x}^{(i)}}^T\mathbf{w}+b^T)(\mathbf{w}^T\mathbf{x}^{(j)}+b)\right] \\ &= \mathbb{E}\left[{\mathbf{x}^{(i)}}^T\mathbf{w}\mathbf{w}^T\mathbf{x}^{(j)}+b^Tb + {\mathbf{x}^{(i)}}^T\mathbf{w}b + b^T\mathbf{w}^T\mathbf{x}^{(j)}\right] \\ &= \mathbb{E}\left[{\mathbf{x}^{(i)}}^T\mathbf{w}\mathbf{w}^T\mathbf{x}^{(j)}+b^Tb\right] + \mathbb{E}\left[{\mathbf{x}^{(i)}}^T\mathbf{w}b\right] + \mathbb{E}\left[b^T\mathbf{w}^T\mathbf{x}^{(j)}\right] \\ &= \mathbb{E}\left[{\mathbf{x}^{(i)}}^T\mathbf{w}\mathbf{w}^T\mathbf{x}^{(j)}+b^2\right] + {\mathbf{x}^{(i)}}^T\mathbb{E}\left[\mathbf{w}b\right] + \mathbb{E}\left[b^T\mathbf{w}^T\right]\mathbf{x}^{(j)} \tag{1'} \\ &= \mathbb{E}\left[{\mathbf{x}^{(i)}}^T\mathbf{w}\mathbf{w}^T\mathbf{x}^{(j)}\right] + \mathbb{E}\left[b^2\right] + 0 + 0 \tag{2'}\\ &= {\mathbf{x}^{(i)}}^T\mathbb{E}\left[\mathbf{w}\mathbf{w}^T\right]\mathbf{x}^{(j)} + \sigma_b^2 \tag{3'}\\ &= {\mathbf{x}^{(i)}}^T \sigma_w^2 \, \mathbb{I}\mathbf{x}^{(j)} + \sigma_b^2 = \sigma_w^2 {\mathbf{x}^{(i)}}^T\mathbf{x}^{(j)} + \sigma_b^2 \end{align}
• Thank you very much! This is also what I came up with but why is $\mathbb{E}[b^2] = \sigma_b^2$ and $\mathbb{E}[w w^T ] = \sigma_w^2$? Given that $\sigma_w^2$ is the variance of the weights and $\sigma_b^2$ of the bias and $\mathbb{V}[X] = \mathbb{V}[X] - \mathbb{V}[X^2] - E[X]^2$ Edit: Nevermind, I just realized that $\mathbb{E}[X]^2$ has to be 0 in both cases. Thank you!
• It is because we made the assumption that $\mathbf{w}$ is normally distributed with zero mean and covariance matrix $\sigma_w^2\mathbb{I}$, and similarly $b$ is gaussian centered with $\sigma_b^2$ variance. Feb 1 at 17:09