4
$\begingroup$

I have the following question concerning Bayesian linear regression on my machine learning assignment:

Consider $f = w^Tx$, where $p(w) ∼ N(w | 0, Σ)$. Show that $p(f | x)$ is Gaussian.

I believe that $p(f|x) ∼ N(w^Tx, \sigma^2)$ for some variance $\sigma^2$ but I'm not sure how to show it. I'm also not sure why the prior was mentioned for this question since we're deriving an expression for the likelihood. If anyone could give me some tips to push me in the right direction it'd be appreciated (please don't solve the entire problem).

$\endgroup$
2
  • 6
    $\begingroup$ This is a classical probability result, the linear transform of a Gaussian being Gaussian with the mean linearly transformed and (hint) the variance quadratically transformed. $\endgroup$
    – Xi'an
    Jan 27, 2015 at 5:50
  • $\begingroup$ Didn't notice you are another man :-) You might want to see my answer on the same problem (apparently, asked by one of your peers): stats.stackexchange.com/a/135078/62549 $\endgroup$ Jan 27, 2015 at 14:59

2 Answers 2

0
$\begingroup$

$p(f|\mathbf{x})$ is actually the prior for Gaussian process not the likelihood. Gaussian process by definition is assuming a particular prior over functions (represented as random variable $f|\mathbf{x}$) not parameters ($\mathbf{w}$). When the model is $y = f(x) + \epsilon$ where usually $\epsilon$ is considered to the noise in observation $y$, the likelihood (of $f$) is $p(y | f)$, the prior (of $f$) is $p(f|\mathbf{x})$ and the posterior (of $f$) is $p(f|y)$. $f$ is called the latent/hiden variable meaning that we cannot observe its values. Therefore, we are usually not interested in $p(f|y)$. $y$ is the variable representing observation samples/training data.

By the assumption of linear regression model, $f$ is a linear function. Therefore, it can be expressed as $\mathbf{w}^T\mathbf{x}$. Now, we want to estimate parameters $\mathbf{w}$ using a Bayesian approach. The likelihood (of parameters) would be $p(y|\mathbf{w})$, the posterior (of parameters) is $p(\mathbf{w}|y)$ and the prior (of parameters) is $p(\mathbf{w})$. You are trying to be prove assuming $\mathbf{w}\sim\mathcal{N}(\mathbf{0}, \Sigma)$ and linearity of $f$ entails Gaussian process prior assumption.

You are proving when the Gaussian process assumption meets the linear regression assumption then the two priors (one over functions, the other over parameters) are equivalent. Mean is not necessarily zero though. That is to simplify algebra.

$\endgroup$
2
  • $\begingroup$ It doesn't answer the original question: how to find $\sigma^2$. $\endgroup$ Jan 27, 2015 at 9:03
  • $\begingroup$ I replied to this part: "I'm also not sure why the prior was mentioned for this question since we're deriving an expression for the likelihood." $\endgroup$
    – Leila
    Jan 27, 2015 at 13:56
0
$\begingroup$

$p(f|x)$ can not be $\mathcal{N}(w^T x, \sigma^2)$ since $w$ is a random variable and you don't condition on it. If you did, then, $p(f|x,w)$ would be singular: for given $w$ and $x$ we know exactly what the outcome $f$ will be (since there's no noise in $f$). It's like having $\sigma=0$, though, $\mathcal{N}(\mu, 0)$ is not well-defined. All randomness in $f$ comes from $w$.

As mentioned in the comments, $f = w^T x$ turns out to be Gaussian, as well. This means $f \sim \mathcal{N}(\mu, \sigma^2)$ where

$$ \mu = \mathbb{E} [f] = \mathbb{E} [w^T x] \\ \sigma^2 = \mathbb{E} [f^2] - \mathbb{E} [f]^2 = \mathbb{E} [w^T x w^T x] - \mu^2 $$

The most difficult to compute here is $\mathbb{E} [w^T x w^T x]$, but using algebraic manipulations (like, rearranging terms) you can do it.

$\endgroup$
2
  • $\begingroup$ Can you elaborate on what you meant by needing to condition on w? I thought $p(f|x)$ is also normally distributed with the mean $w^Tx$. I am also new to this stuff so not understanding the nuances $\endgroup$
    – Luca
    Jan 27, 2015 at 9:32
  • 1
    $\begingroup$ @Luca, $w$ is a r.v., so we can't use it as a distribution parameter. In order to do so, we need to fix one of its realizations, that is, condition on (concrete value of) $w$ (that means you will get function of $w$). But then $f$ becomes non-random, since all randomness in $f$ comes from $w$. $\endgroup$ Jan 27, 2015 at 9:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.