1
$\begingroup$

From Statistical Inference of Casella, a conditional pdf is obtained through the joint pdf by the following equation,

(1) $$ p_a(x_a | x_b) = \frac {p(x_a, x_b)}{p_b(x_b)} $$

In section 2.1.3 of PRML, the author did not mention explicitly the equation, but I guess he using the same approach. The author obtain the conditional mean and covariant matrix by directly manipulating the quadratic form of the exponent of the joint pdf. I understand how he obtain the conditional mean and covariant matrix, but since he does not explain how to obtain the corresponding Gaussian pdf, I have no idea if that pdf is equal to (1).

This are the derived conditional mean and covariant matrix derived from the joint pdf, and other relevant vectors and matrices

The result conditional mean of $x_a$, given $x_b$ $$ \mu_{a|b} = \mu_a - \Lambda_{aa}^{-1} \Lambda_{ab} (x_b - \mu_b) \\ $$

The resulted conditional covariant matrix $$ \Sigma_{a|b} = \Lambda_{aa}^{-1} \\ $$

The random vector $$ x = \binom{x_a}{x_b} \\ $$

The joint mean $$ \mu = \binom{\mu_a}{\mu_b} \\ $$

The joint covariant matrix and its inverse $$ \Sigma = \begin{bmatrix} \Sigma_{aa} & \Sigma_{ab}\\ \Sigma_{ba} & \Sigma_{bb} \\ \end{bmatrix} \\ \Lambda = \Sigma^{-1} = \begin{bmatrix} \Lambda _{aa} & \Lambda _{ab}\\ \Lambda _{ba} & \Lambda _{bb} \\ \end{bmatrix} \\ $$

The quadratic form of the exponent of the joint pdf $$ (x - \mu)^T \Sigma (x - \mu) $$

(2) The general multivariate Gaussian pdf on a random vector $X \in R^D$ $$ N(x | \mu, \Sigma) = \frac{1}{(2\pi)^{0.5D}} \frac{1}{|\Sigma|^{0.5}} \exp\big(-0.5(x-\mu)^T \Sigma^{-1} (x-\mu) \big) $$

$\endgroup$
5
  • $\begingroup$ Can you write your question explicitly (i.e. mention the equations in PRML) and point out where you didn't understand? $\endgroup$
    – gunes
    Jan 29, 2021 at 10:27
  • $\begingroup$ There is no equation in the section I don't understand. As I said, the author only show how to obtain the conditional mean and covariant matrix. I want to verify using the parameters to obtain the equation (1), but I failed to do so. $\endgroup$
    – Hiep
    Jan 29, 2021 at 10:36
  • $\begingroup$ I still don't understand what you mean by "but he does not explain how to obtain the corresponding Gaussian pdf", w/o looking into the section you mentioned. It's better if you could elaborate your question and flow. $\endgroup$
    – gunes
    Jan 29, 2021 at 10:40
  • $\begingroup$ The author manipulated the quadratic form in the exponent of the joint pdf to find the mean and covariant matrix, but not from (1), and so I still find it difficult to verify if the parameters are actually of the conditional Gaussian distribution. Btw, I added relevant matrices and vectors. $\endgroup$
    – Hiep
    Jan 29, 2021 at 10:57
  • 1
    $\begingroup$ Does this answer your question? Interpretation of multivariate conditional gaussian function form? $\endgroup$
    – Peter O.
    Jan 31, 2021 at 4:07

1 Answer 1

0
$\begingroup$

The author (C. Bishop) intentionally doesn't use the classic relation (1) because it'd then be required to calculate the denominator by marginalising the joint PDF. This integral is quite cumbersome in may cases in bayesian literature and there are approximate ways of getting it. However, in the Gaussian case, we have a way of calculating it in an indirect way (i.e. not by marginalising the joint). The author simply exploits the form of the joint PDF to come up directly with the conditional PDF. The idea can be outlined as follows:

The conditional PDF is a function of $x_a$, and since $x_b$ is given, any expression containing $x_b$ can be regarded as a constant. Therefore, the denominator has no effect on the form of $x_a$'s conditional PDF.

Ex: If let's say $p(x_a,x_b)=g(x_b)e^{-x_a h(x_b)}$ (assume $x_a$ is not a vector for this particular case), then $p(x_a|x_b)$ is in the form of an exponential random variable's PDF because all other terms including $p(x_b)$ in the denominator don't change the form of the conditional ($Ce^{-\lambda x})$.

That's why the author manipulates the joint PDF and says that it's actually in the following form: $$\begin{align}p(x_a,x_b)&=C\exp(-0.5(x-\mu)^T\Sigma^{-1}(x-\mu))\\&=C\exp(-0.5(x_a-b)^TS^{-1}(x_a-b)+g(x_b))\end{align}$$

where $g(.)$ is a function. And, this form is Gaussian for $x_a$. So, the conditional PDF is Gaussian. We don't worry exactly what would the denominator be or what exactly $g(x_b)$ brings us since it'd all have to satisfy the normalization property of PDFs anyway.

Noting that the form is Gaussian, it all boils down to finding what $S$ and $b$ is which are conditional covariance and the mean. Of course, there are a lot of steps between the first and the second line here, and if you have questions about specific steps, you can ask them as follow-ups.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. I was ignorant a conditional pdf can be "any" function, rather than defined by the form (1). Regarding the last equation in your answer, the exponent has the same form as that in the general Gaussian pdf (2) (I included it in my question), but the constant outside of the exponent still depend on S, i.e. the square root of the determinant of S. Based on the form (2), I cannot convince myself the last equation is Gaussian with mean b and covariance matrix S. Could you clarify the point? $\endgroup$
    – Hiep
    Jan 31, 2021 at 4:07
  • 1
    $\begingroup$ The last equation is joint PDF, it's proportional to conditional PDF by a constant (in terms of terms other than $x_a$). The constant outside the exponent can depend on S. Since S is a constant, too, it's not a problem. $\endgroup$
    – gunes
    Jan 31, 2021 at 14:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.