1
$\begingroup$

Suppose I have a multivariate normal ${\bf{Y}}|{\bf{\theta}} \sim {\bf{MVN}}(X {\bf{\beta}}, \sigma^{2}H(\phi))$ where ${\bf{Y}}$ is a set of observations ${\bf{Y}} = \{y({\bf{s}}_{1}),y({\bf{s}}_{2}),... ,y({\bf{s}}_{n})\}$ and $H(\phi)$ is a covariance matrix $H(\phi) = \rho({\bf{s}}_{i} - {\bf{s}}_{j};\phi)$. Now I want to predict a value $y_{0}$, so as it is usual, the predictive posterior distribution is

$$p(y_{0}|{\bf{y}}, X, x_{0}) = \int p(y_{0}|{\bf{y}},{\bf{\theta}}, x_0)p({\bf{\theta}}|{\bf{y}}, X)d\theta$$

Usually, this is described as a product of $\text{Likelihood} \times \text{Posterior}$. However, in this case, the likelihood was initially expressed as a multivariate normal but I just want to predict the value $y({\bf{s}})$ at some point ${\bf{s}}_{0}$, so obviously I can't express a single point as a multivariate normal. What should I do?

As an alternative, I should be able to use a conditional distribution $y_{0} | \bf{Y}$ from a joint distribution $\left( y_{0}, {\bf{Y}} \right)$ in the following way:

$$\left( \begin{array}{ccc} Y_{1} \\ Y_{2} \end{array} \right) \sim N\left(\left( \begin{array}{ccc} \mu_{1} \\ \mu_{2} \end{array} \right), \left( \begin{array}{ccc} \Omega_{11} & \Omega_{12} \\ \Omega_{21} & \Omega_{22} \end{array} \right)\right)$$

The problem is that this in some way breaks with the idea of using $\text{Likelihood} \times \text{Posterior}$ for the predictive posterior distribution.

Thanks!

$\endgroup$
  • $\begingroup$ I am not sure that this question is answerable as written. I don't see how you can predict a single point when you didn't pick a point (or time frame) to start with. Moreover, since you have a multivariate normal, you don't even have a point, but rather a matrix. By taking expectations, you can update the mean using the equation you provided. If you just want to predict a single point, I think you have to articulate further what it is that you want to do, i.e. the marginal mean, the conditional mean, etc. $\endgroup$ – rocinante Sep 12 '14 at 18:20
  • 1
    $\begingroup$ I'm not sure what you mean that I didn't pick a a point to start with. I have a whole set of observations $\bf{Y}$. Maybe, the confusion arises from the fact that I want to obtain the predictive posterior distribution for $y_{0}({\bf{s}})$ at some point $s_{0}$. I will update the question to make it clear. $\endgroup$ – Robert Smith Sep 12 '14 at 18:53
0
$\begingroup$

It sounds like you are trying to do inference with a Gaussian process. The correct approach is to use the conditional distribution, as in the second part of your question. The reason that you cannot use the Likelihood $\times$ Posterior approach is because $\theta$ is an infinite-dimensional object, representing the value of the process at all locations ${\bf s}$.

$\endgroup$
  • $\begingroup$ Thank you for your answer. $\theta$ is an infinite-dimensional object? I have been using $\theta$ as parameters. Are yuu sure about that? By the way, I'm not necessarily using Gaussian processes. I know that the derivation is the same, but here the dimensions are finite, although I don't think there is a strong distinction when implementing Gaussian processes in practice. $\endgroup$ – Robert Smith Sep 13 '14 at 19:01
  • $\begingroup$ I just proposed a very simple explanation of why a conditional distribution is needed. If you notice a mistake, please let me know. $\endgroup$ – Robert Smith Sep 13 '14 at 19:38
0
$\begingroup$

I think I got a very simple answer.

I overlooked the fact that the predictive posterior distribution is derived with a "conditional likelihood":

\begin{array} pp(y_{0}|{\bf{y}}) &= \int p(y_{0}, \theta| {\bf{y}})d\theta \\ & = \int p(y_{0}|{\theta, \bf{y}})p(\theta| {\bf{y}})d\theta \end{array}

where I omitted ${\bf{X}}$ and $x_{0}$ for simplicity. The full predictive posterior distribution is what I initially wrote in my question:

$$p(y_{0}|{\bf{y}}, X, x_{0}) = \int p(y_{0}|{\bf{y}},{\bf{\theta}}, x_0)p({\bf{\theta}}|{\bf{y}}, X)d\theta$$

Only when $y_{0}$ is independent of the previous observations ${\bf{y}}$, it is possible to set $p(y_{0}|{\bf{y}},{\bf{\theta}}, x_0) = p(y_{0}|{\bf{\theta}}, x_0)$ to obtain the regular form $$\text{Predictive} = \text{Likelihood} \times \text{Posterior}$$

$$p(y_{0}|{\bf{y}}, X, x_{0}) = \int p(y_{0}|{\bf{\theta}}, x_0)p({\bf{\theta}}|{\bf{y}}, X)d\theta$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.