1
$\begingroup$

I am learning Gaussian Process reading GPML. I am a bit confused with understanding the Bayesian analysis.

Let consider the standard linear regression model with "Gaussian noise", i.e, $$ f(\textbf{x}) = \textbf{x}^T\textbf{w}, \qquad y = f(\textbf{x}) + \epsilon, \qquad \epsilon \sim \mathcal{N}(0,\sigma_n^2) $$ where $\textbf{w}, \textbf{x} \in \mathbb{R}^n$.

Note that $$ \epsilon = y - f(\textbf{x}) = y - \textbf{x}^T\textbf{w} \sim \mathcal{N}(0,\sigma_n^2) $$ and the p.d.f. of $\epsilon$ is $$ p_\epsilon(z) = \frac{1}{\sqrt{2\pi}\sigma_n}\exp\left(-\frac{z^2}{2\sigma_n^2}\right). $$ Therefore, given $\textbf{x}$ and $\textbf{w}$, the p.d.f. of $y$ would be $$ p_{y\mid \textbf{x}, \textbf{w}}(z) = \frac{1}{\sqrt{2\pi}\sigma_n}\exp\left(-\frac{(z-\textbf{x}^T\textbf{w})^2}{2\sigma_n^2}\right). $$ It then follows from the Bayes' rule that $$ p_{\textbf{w}\mid y, \textbf{x}}(w) = \frac{p_{y\mid \textbf{x},\textbf{w}}(z)\cdot p_{\textbf{w}}(w)}{p_{y\mid \textbf{x}}(z)}. $$

Question: In the noise-free setup, how one can derive the posterior distribution $p_{\textbf{w}\mid y, \textbf{x}}(w)$? It seems that in the noise-free setup, $y$ is completely determined by $\textbf{x}$ and $\textbf{w}$. Thus $$p_{y\mid \textbf{x},\textbf{w}}(z) = \delta_{\{z=\textbf{x}^T\textbf{w}\}}(z), $$ a point distribution. Then this gives $$ p_{\textbf{w}\mid y, \textbf{x}}(w) = \frac{\delta_{\{z=\textbf{x}^T\textbf{w}\}}(z)}{P_{\textbf{w}}(\textbf{x}^T\textbf{w}=z \mid \textbf{x})}p_{\textbf{w}}(w) ? $$ (actually, not 100% sure about this).

At this point, I don't see how the Bayesian approach is somehow useful in the noise-free scenario. Or Is Bayesian approach based on the assumption of the presence of the noise?

$\endgroup$
1
$\begingroup$

You're almost right. As you said $p(w | x, y) \propto p(y | x, w) p(w)$ and $p(y | x, w) = \delta(y - x^tw)$, i.e. $$p(w |x, y) \propto p(w)\delta(y - x^tw).$$ This means that your measure is concentrated on the values of $w$ which are consistent with the observation $(x, y)$, which will generally restrict $w$ to come from a hyperplane instead of the whole space, but the relative probabilities are the same as in your prior.

This makes sense. The only information your observation $(x, y)$ gives you is that $w$ has to be consistent with this very observation and Bayes' theorem reflects this information perfectly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.