Details of Bayesian linear regression

Question

In Bayesian Linear Regression, we have a data set with {$x_{i}$, $t_{i}$} where $x_{i}$ are input vectors and $t_{i}$ are their resulting observations. We want to find a vector $\bf{w}$ in order to maximize the posterior distribution $p({\bf{w}}|{\bf{t}})$ and hopefully make good prediction for future $x_{i}$'s.

In this Bayesian formalism, we multiply a prior $p({\bf{w}}|\alpha)$ by the likelihood function $p({\bf{t}}|{\bf{w}})$ to obtain $p({\bf{w}}|{\bf{t}})$ and then use this distribution as a prior and repeat the process again. However, I have a question regarding the details of this procedure.

For example, suppose we don't have any data points at first and rely exclusively on the prior, and as long as we can infer the hyperparameter $\alpha$, we are going to be able to do that. In the next step, we receive a data point $\{x_{1}$, $t_{1}\}$, and we calculate the likelihood $p({\bf{t_{1}}}|{\bf{w}})$ which involves a Gaussian with mean ${\bf{w}}^{T}\phi({x_{1}})$ and variance $\beta^{-1}$. Using a model such as $w_{0}+w_{1}x$, this amounts to this:

$$p({\bf{t}}|{\bf{w}}) \propto \exp{\{\beta(t_{1} - (w_{0}+w_{1}x_{1}))^{2}\}}$$

so if that is correct, we know everything except $w_{0}$ and $w_{1}$. Do I get a probability for each combination of $w_{0}$ and $w_{1}$? In any case, I think this is a function of $\bf{w}$ that we can multiply by the prior to obtain a posterior.

Did I understand anything wrong about this procedure of $\text{posterior}\propto\text{likelihood}\times\text{prior}$? If so, I would appreciate any corrections.

UPDATE: I added a picture from Bishop's book "Pattern Recognition and Machine Learning" in which this process is described. The first column is the likelihood, second column is the prior/posterior and third column describes the input vectors in blue and samples of $\bf{w}$ taken from the posterior in the equation $w_{0}+w_{1}x$.

Thanks

Answer 1

One interesting thing (and the reason it is 'a bit weird' and we can't solve for a unique ${\bf{w}}_{\text {ML}}$) about the likelihood functions plotted in the left column of Fig. 3.7, is that they use fewer data points than the number of parameters in the linear model, i.e., $M\lt N$. So this is the 'opposite' situation compared to that of the usual 'tall', $N\times M$ with $N>M$, design matrices, $\bf\Phi$, the geometry of which is discussed in section 3.1.2. There we can solve for a unique least squares ${\bf{w}}_{\text {ML}}=({\bf\Phi}^{\text T}{\bf\Phi})^{-1}{\bf\Phi}^{\text T}$${\bf t}$ such that ${\bf\Phi w}_{\text {ML}}$ is the orthogonal projection of $\bf t$ onto the $M$-dimensional hyperplane spanned by the columns of $\bf\Phi$ (assuming $\text{rank}({\bf\Phi})=M$). But with a wide($N<M$) $\bf\Phi$, there is a non-empty null-space, $\text{Kern}$, of dimension $M-N$ (assuming $\text {rank}({\bf\Phi})=N$).

For the likelihood functions of Fig. 3.7, $M=2, N=1$, $\text{dim}(\text{Kern})=1$. A single point does not uniquely define a line. (Here is an example of a situation where it is impossible to normalize the likelihood function so it can't be a density.) If we held off and plotted the likelihood functions based on 2 or more points, they would have unique maxima(assuming at least 2 didn't fall on top of each other). In either case, the posterior have unique ${\bf w}_\text{MAP}$ as you pointed out.