This question is from Bishop's pattern recognition's chapter 3 about Bayesian linear regression. Chapter 3 starts with MLE and regularization to solve the overfitting problem.
But MLE and regularization are not enough to solve the overfitting problem, hence the book comes up with Bayesian linear regression methods and I find it confusing.
they define the ${\bf w}$ as a random variable $$p({\bf w}) = N({\bf w}|{\bf m}_0, {\bf S}_0) \qquad{(3.48)}$$ I think this ${\bf w } \ $is only to represent the belief of the prior(subjective) then they just defined it as mean ${\bf m_0}$ and ${\bf S_0}\ $ as variance.
$${\bf m}_N = {\bf S}_N(S_0^{-1}{\bf m}_0+\beta\Phi^T{\bf t}) \qquad{(3.50)}$$
$${\bf S}_N^{-1} = {\bf S}_0^{-1}+\beta\Phi^T\Phi \qquad{(3.51)}$$
and finally they come up with log likelihood function
$$\ln p({\bf w}|{\bf t}) = -\frac{\beta}{2}\sum_{n=1}^{N}\{t_n-{\bf w}^T\phi({\bf x}_n)\}^2 - \frac{\alpha}{2}{\bf w}^T{\bf w}+const \qquad{(3.55)}$$
I'm not really sure with the ${\bf t} \ $ terms in (3.55) equation . I think this ${\bf t} \ $ represent the target from some sample data.(if it's right is this sample data also selected as randomly?)
after they find the log likelihood function(3.55). they are doing "predictive distribution".
$$p(t|{\bf t}, \alpha, \beta) = \int p(t|{\bf w}, \beta)p({\bf w}|{\bf t}, \alpha, \beta)d{\bf w} \qquad{(3.57)}$$
also confused about what is t and ${\bf t} \ $ represent in the equation. and also what they are predicting?
this (3.57) equation mean and variance is founded as
$$p(t\;|\;{\bf t}, \alpha, \beta) = N(t\;|\;{\bf m}_N^T\phi({\bf x}), \sigma_N^2({\bf x})) \qquad{(3.58)}$$
$$\sigma_N^2({\bf x}) = \frac{1}{\beta} + \phi({\bf x})^T S_N\phi({\bf x}) \qquad{(3.59)}$$
this variance's first term represent the original noise's variance and they said this second term represent the uncertainty of the ${\bf w }\ $ how they know this part represent the uncertainty of the ${\bf w }\ $
and they said $( \sigma\_{N+1}^2({\bf x}) \le \sigma\_N^2({\bf x}) )$ and when N is bigger it becomes narrower(I don't understand how second term is going narrower when N is increase)
this picture shows when N grows. I guess this red line just represent the random variable and it changes as N. but what does green line represent for?
I'm trying to clarify the each terms. thank you
