# Details of Bayesian linear regression

In Bayesian Linear Regression, we have a data set with {$x_{i}$, $t_{i}$} where $x_{i}$ are input vectors and $t_{i}$ are their resulting observations. We want to find a vector $\bf{w}$ in order to maximize the posterior distribution $p({\bf{w}}|{\bf{t}})$ and hopefully make good prediction for future $x_{i}$'s.

In this Bayesian formalism, we multiply a prior $p({\bf{w}}|\alpha)$ by the likelihood function $p({\bf{t}}|{\bf{w}})$ to obtain $p({\bf{w}}|{\bf{t}})$ and then use this distribution as a prior and repeat the process again. However, I have a question regarding the details of this procedure.

For example, suppose we don't have any data points at first and rely exclusively on the prior, and as long as we can infer the hyperparameter $\alpha$, we are going to be able to do that. In the next step, we receive a data point $\{x_{1}$, $t_{1}\}$, and we calculate the likelihood $p({\bf{t_{1}}}|{\bf{w}})$ which involves a Gaussian with mean ${\bf{w}}^{T}\phi({x_{1}})$ and variance $\beta^{-1}$. Using a model such as $w_{0}+w_{1}x$, this amounts to this:

$$p({\bf{t}}|{\bf{w}}) \propto \exp{\{\beta(t_{1} - (w_{0}+w_{1}x_{1}))^{2}\}}$$

so if that is correct, we know everything except $w_{0}$ and $w_{1}$. Do I get a probability for each combination of $w_{0}$ and $w_{1}$? In any case, I think this is a function of $\bf{w}$ that we can multiply by the prior to obtain a posterior.

Did I understand anything wrong about this procedure of $\text{posterior}\propto\text{likelihood}\times\text{prior}$? If so, I would appreciate any corrections.

UPDATE: I added a picture from Bishop's book "Pattern Recognition and Machine Learning" in which this process is described. The first column is the likelihood, second column is the prior/posterior and third column describes the input vectors in blue and samples of $\bf{w}$ taken from the posterior in the equation $w_{0}+w_{1}x$.

Thanks

• Quick comment (I hope I haven't misunderstood your question): it is $\text{posterior}\propto\text{likelihood}\times\text{prior}$ and not equal to. There is a proportionality constant that should not be neglected and that ensures that the posterior is a probability density for any likelihood and prior. – tibL Mar 26 '13 at 10:40
• This sounds a question asking for re-assurance, and basically, yes you've got the interpretation correct. Bayes is similar to the "loss function" + "regularising penalty" framework of machine learning. In bayes you have log-likelihood and log-prior as the equivalents - but you also get accuracy measures as well as predictions. – probabilityislogic Mar 26 '13 at 12:57
• @tibL Yes, I meant proportional to the likelihood $\times$ prior. I already change that part in my question. – Robert Smith Mar 26 '13 at 17:03
• @probabilityislogic Yes, I'm looking for re-assurance and in the process, I'd like to get another detailed description to make sure I understand how it works. The part that I thought was a bit weird is that we have a likelihood with $\bf{w}$ as unknowns, so it's not possible to solve it (although we can plot it for every value of $\bf{w}$) but after multiplying likelihood by the prior, we obtain a probability $p({\bf{w}}|{\bf{t}})$ where is natural to ask for a given $\bf{w}$ and get a number. – Robert Smith Mar 26 '13 at 17:10

One interesting thing (and the reason it is 'a bit weird' and we can't solve for a unique $${\bf{w}}_{\text {ML}}$$) about the likelihood functions plotted in the left column of Fig. 3.7, is that they use fewer data points than the number of parameters in the linear model, i.e., $$M\lt N$$. So this is the 'opposite' situation compared to that of the usual 'tall', $$N\times M$$ with $$N>M$$, design matrices, $$\bf\Phi$$, the geometry of which is discussed in section 3.1.2. There we can solve for a unique least squares $${\bf{w}}_{\text {ML}}=({\bf\Phi}^{\text T}{\bf\Phi})^{-1}{\bf\Phi}^{\text T}{\bf t}$$ such that $${\bf\Phi w}_{\text {ML}}$$ is the orthogonal projection of $$\bf t$$ onto the $$M$$-dimensional hyperplane spanned by the columns of $$\bf\Phi$$ (assuming $$\text{rank}({\bf\Phi})=M$$). But with a wide($$N) $$\bf\Phi$$, there is a non-empty null-space, $$\text{Kern}$$, of dimension $$M-N$$ (assuming $$\text {rank}({\bf\Phi})=N$$).
For the likelihood functions of Fig. 3.7, $$M=2, N=1$$, $$\text{dim}(\text{Kern})=1$$. A single point does not uniquely define a line. (Here is an example of a situation where it is impossible to normalize the likelihood function so it can't be a density.) If we held off and plotted the likelihood functions based on 2 or more points, they would have unique maxima(assuming at least 2 didn't fall on top of each other). In either case, the posterior have unique $${\bf w}_\text{MAP}$$ as you pointed out.
• One question I have is why doesn’t the mode/mean of the posterior distribution march from that of the prior to the MLE as data points are added? It seems that it should given what Bishop says in sec 2.3.6 about eqs 2.141, 2.242, and the analogous eqs 3.50, 3.51 as $N$ goes from 0 to large. – Don Slowik Apr 29 '19 at 16:41