I am doing a problem on Bayesian regression but I'm having a lot of trouble with it. Here is the question:

  1. Consider $f=w^Tx$, $p(w)\sim N(w|0,\Sigma)$. Show that $p(f|x)$ is Gaussian.

  2. Find the mean and covariance of $f$.

  3. Consider $t=f+\epsilon$ where $\epsilon\sim N(0,0.001)$. What is $p(f|t,x)$?

I'm just really confused about the notation. Since we are doing Bayesian regression, shouldn't we really be trying to compute $p(w|f)$? Why are we conditioning on $x$, which doesn't have a prior on it? What does $p(f|x)$ and $p(f|t,x)$ even mean?

  • $\begingroup$ What did you try? $\endgroup$ Jan 26, 2015 at 19:28
  • $\begingroup$ I honestly don't even know where to begin since as I said I don't even understand the notation $\endgroup$
    – Aden Dong
    Jan 26, 2015 at 20:37

1 Answer 1

  1. Yes, in order to come up with an estimate of model parameters $w$ we need to find a (mode of a) distribution over $w$ given all the data we have: $p(w | \mathcal D)$. The data $\mathcal D$ can be various, but usually it's a set of observation pairs $\{ (\mathbf{x}_n, y_n) \}_{n=1}^N$ where $\mathbf x$ has the same meaning as $x$ in your formulas, and $y$ is the observed output (that is, $w^T \mathbf{x}$, probably, corrupted with some noise).

  2. Because we observed $x$ and want our model to take this into account. Observing data $\mathcal D$ changes posterior distribution for parameter $p(w|\mathcal D)$, and this is the actual learning: we infer a model using the data.

    It's okay that $x$ doesn't have a prior. You can think that there is an unknown distribution $p(x)$, but, luckily, you don't need it (because you always condition on what variable), and, thus, don't need to make any model assumptions about this distribution. Intuitively, we can say that if we condition on $x$, then $x$ has already happened, it doesn't matter how probable it was.

  3. Again, think of $x$ as of random variable with unknown distribution. If you condition on that r.v. then it becomes fixed, a parameter.

    Then you, probably, are interested in how to prove that $p(f|x)$ is Gaussian. My suggestion is to show that $f = w^T x$ is, essentially, a linear combination of 1D random variables $w_i$ with weights $x_i$ (yes, that's correct). I leave you to determine distribution of each of $w_i$ and show that their combination is Gaussian. Then you can estimate parameters of this distribution, using properties of expectation (linearity would be the most useful).

    Finally, the $p(f | t, x)$ is an interesting thing. Here $(x,t)$ is what I meant by $\mathcal D$ in [1]. As you can see, $f$ and $t$ are closely related, so one carries some information about the other. Thus, if you know $t$, then this limits "set" of possibilities (don't take this statement too seriously, it's wrong, but conveys some intuition) and vice versa. Here you will need Bayes' theorem in order to find this distribution.

  • $\begingroup$ Wonderful! I think I know where to start now. Would it be possible for you to elaborate a little on how p(f|x) is a linear combination of $w_i$ with weights $x_i$ relates to $p(w)=N(w|0,\Sigma)$? Thanks! $\endgroup$
    – Aden Dong
    Jan 26, 2015 at 23:03
  • 1
    $\begingroup$ @AdenDong, consider a 2-dimensional case, $f = w_1 x_1 + w_2 x_2$. We treat xs as fixed parameters and $(w_1, w_2)$ is a pair of dependent variables with joint probability $p(w_1, w_2) = p([w_1, w_2]) = \mathcal{N}((w_1, w_1) | 0, \Sigma)$. You now need to determine distribution of each $w_i$ (you might want learn more about multivariate normal distribution and marginalizations). Then you use properties of this distribution (how it behaves under addition and multiplication by a constant) to derive distribution over f. $\endgroup$ Jan 26, 2015 at 23:22
  • $\begingroup$ So essentially I have to compute $p(w_i)$ by marginalizing out all other variables (basically taking a bunch of integrals). Is my understanding correct? $\endgroup$
    – Aden Dong
    Jan 27, 2015 at 6:33
  • $\begingroup$ @AdenDong, yes, that's correct. Though, exact computation might be complicated, so you only need to know distribution family without concrete parameters. It's easier to estimate parameters the other way, using vectorized form of $w$. $\endgroup$ Jan 27, 2015 at 8:56
  • $\begingroup$ how should I use the vectorized form of w to find the mean and covariance of f? $\endgroup$
    – Aden Dong
    Jan 27, 2015 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.