0
$\begingroup$

In the Probabilistic Principal Component Analysis (PPCA) paper (Tipping and Bishop, 1999), the model is given as a linear relationship as follows:

$\mathbf{t}=\mathbf{W}\mathbf{x}+\mu+\epsilon$ (1)

where $\mathbf{t}$ is the observed variable, $\mathbf{x}$ is the latent variable, $\epsilon$ is the Gaussian error, $\mu$ is bias term, and $\mathbf{W}$ is the parameter matrix. It is assumed that $\epsilon \sim \mathcal{N}(0,\sigma^2\mathbf{I})$, and $\mathbf{x}\sim\mathcal{N}(0,\mathbf{I})$.

I didn't understand how we ended up the following conditional distribution:

$\mathbf{t}|\mathbf{x}\sim\mathcal{N}(\mathbf{W}\mathbf{x}+\mu, \sigma^2\mathbf{I})$ (2)

Since the sum of two Gaussians has a mean $\mu_1 + \mu_2$ and variance $\sigma^2_1 + \sigma^2_2$, and $E[\mathbf{C}\mathbf{x}]=\mathbf{C}E[\mathbf{x}]$, $Var[\mathbf{C}\mathbf{x}]=\mathbf{C}Var[\mathbf{x}]\mathbf{C}^T$ where $\mathbf{C}$ is a constant matrix, I can see that $\mathbf{t}$ has a marginal distribution $\mathcal{N}(\mu, \mathbf{W}\mathbf{I}\mathbf{W}^T+\sigma^2\mathbf{I})$. However, I didn't understand why the conditional is written in the first place and $p(\mathbf{t})$ is obtained by marginalization in the paper.

Another question is that, in a latent variable model like this, how to obtain the marginal distribution $p(\mathbf{t})$ if the prior on $\mathbf{x}$ is not Gaussian. For instance, if we want to do sparse modeling, in which $\mathbf{x}$ is the sparse code, we place a Laplace prior on $\mathbf{x}$. Though I am not sure, I think $\mathbf{t}|\mathbf{x}$ would still be Gaussian since the error term is Gaussian, and $\mathbf{t}$ would have another unknown distribution without an analytical solution, therefore we would need to use approximation methods like Laplace's method, variational Bayes, or MCMC. Is this correct?

$\endgroup$
1
$\begingroup$

I didn't understand how we ended up the following conditional distribution

For the conditional distribution $p(t \mid x)$, we hold the latent variable at a fixed value $x$, and consider the distribution of $t$ under this condition. As you said, $t = Wx + \mu + \epsilon$. $W$ and $\mu$ are fixed parameters, and $p(t \mid x)$ is implicitly conditioned on them. So, to produce $t$, we add a fixed vector $Wx + \mu$ to a normally distributed random variable $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$. This means that, given $x$, $t$ will be normally distributed with mean $Wx + \mu$ and covariance matrix $\sigma^2 I$.

I didn't understand why the conditional is written in the first place and $p(t)$ is obtained by marginalization in the paper

Conditional distributions are a convenient way to think about latent variable models. We can think of the model as a way to generate the data. Suppose we have some fixed parameters ($W, \mu, \sigma^2$) and want to generate an observation. First, generate a latent value $x$ by sampling from the prior $p(x)$. Then, generate an observation by sampling from the conditional distribution $p(t \mid x)$, given the chosen $x$.

how to obtain the marginal distribution $p(t)$ if the prior on $x$ is not Gaussian

$p(t)$ can be obtained by marginalizing the joint distribution $p(t, x)$ over $x$:

$$p(t) = \int_\mathcal{X} p(t, x) dV$$

The joint distribution can be factored as $p(t, x) = p(t \mid x) p(x)$ so:

$$p(t) = \int_\mathcal{X} p(t \mid x) p(x) dV$$

The conditional distribution $p(t \mid x)$ is Gaussian as before (nothing has changed here). The prior $p(x)$ is whatever you've now declared it to be.

As before, note that the distributions above are implicitly conditioned on the parameters. To do inference, you'd need to work with the posterior distribution over parameters, or at least the likelihood function. If you can't compute the integrals in closed form, you'd need to work with an approximation or use sampling, as you mentioned.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.