noise-free Gaussian Process likelihood

Question

I am learning Gaussian Process reading GPML. I am a bit confused with understanding the Bayesian analysis.

Let consider the standard linear regression model with "Gaussian noise", i.e, $$ f(\textbf{x}) = \textbf{x}^T\textbf{w}, \qquad y = f(\textbf{x}) + \epsilon, \qquad \epsilon \sim \mathcal{N}(0,\sigma_n^2) $$ where $\textbf{w}, \textbf{x} \in \mathbb{R}^n$.

Note that $$ \epsilon = y - f(\textbf{x}) = y - \textbf{x}^T\textbf{w} \sim \mathcal{N}(0,\sigma_n^2) $$ and the p.d.f. of $\epsilon$ is $$ p_\epsilon(z) = \frac{1}{\sqrt{2\pi}\sigma_n}\exp\left(-\frac{z^2}{2\sigma_n^2}\right). $$ Therefore, given $\textbf{x}$ and $\textbf{w}$, the p.d.f. of $y$ would be $$ p_{y\mid \textbf{x}, \textbf{w}}(z) = \frac{1}{\sqrt{2\pi}\sigma_n}\exp\left(-\frac{(z-\textbf{x}^T\textbf{w})^2}{2\sigma_n^2}\right). $$ It then follows from the Bayes' rule that $$ p_{\textbf{w}\mid y, \textbf{x}}(w) = \frac{p_{y\mid \textbf{x},\textbf{w}}(z)\cdot p_{\textbf{w}}(w)}{p_{y\mid \textbf{x}}(z)}. $$

Question: In the noise-free setup, how one can derive the posterior distribution $p_{\textbf{w}\mid y, \textbf{x}}(w)$? It seems that in the noise-free setup, $y$ is completely determined by $\textbf{x}$ and $\textbf{w}$. Thus $$p_{y\mid \textbf{x},\textbf{w}}(z) = \delta_{\{z=\textbf{x}^T\textbf{w}\}}(z), $$ a point distribution. Then this gives $$ p_{\textbf{w}\mid y, \textbf{x}}(w) = \frac{\delta_{\{z=\textbf{x}^T\textbf{w}\}}(z)}{P_{\textbf{w}}(\textbf{x}^T\textbf{w}=z \mid \textbf{x})}p_{\textbf{w}}(w) ? $$ (actually, not 100% sure about this).

At this point, I don't see how the Bayesian approach is somehow useful in the noise-free scenario. Or Is Bayesian approach based on the assumption of the presence of the noise?

Answer 1

You're almost right. As you said $p(w | x, y) \propto p(y | x, w) p(w)$ and $p(y | x, w) = \delta(y - x^tw)$, i.e. $$p(w |x, y) \propto p(w)\delta(y - x^tw).$$ This means that your measure is concentrated on the values of $w$ which are consistent with the observation $(x, y)$, which will generally restrict $w$ to come from a hyperplane instead of the whole space, but the relative probabilities are the same as in your prior.

This makes sense. The only information your observation $(x, y)$ gives you is that $w$ has to be consistent with this very observation and Bayes' theorem reflects this information perfectly.