I have a multi-dimensional Gaussian, and I want to find the distribution for the case that some variables are a linear function of the others.
For example, in the case of a two-dimensional Gaussian, say I want to find the probability distribution for the case where both variables have the same value $P(x_a=x_b)$. In particular, I'd like to do this for a Gaussian Mixture Model with the more general form $P(x_a|x_b=\alpha x_a+\beta$)
I've tried deriving this on my own using the standard procedure for deriving a conditional Gaussian ($P(x_a|x_b=const)$) (e.g. Bishop p.85 or completing the square).
But when I choose some sigma, mu, with alpha=1, beta=0, and I evaluate P_old(x=0,y=0) I get a different number than P_new(0). I've included the derivation below (I can add more steps -- it's all in LaTeX already).
- Can anyone see what I'm doing wrong?
- Or is there an alternative approach to this?
- Do I have to change anything for the case of a Mixture Model?
Let: $$ x=\left(\begin{array}{c} x_{a}\\ x_{b} \end{array}\right) $$ $$ \mu=\left(\begin{array}{c} \mu_{a}\\ \mu_{b} \end{array}\right) $$ $$ \Sigma=\left(\begin{array}{cc} \Sigma_{aa} & \Sigma_{ab}\\ \Sigma_{ba} & \Sigma_{bb} \end{array}\right) $$ $$ \begin{aligned}\Lambda & \equiv\Sigma^{-1}\\ & \equiv\left(\begin{array}{cc} \Lambda_{aa} & \Lambda_{ab}\\ \Lambda_{ba} & \Lambda_{bb} \end{array}\right) \end{aligned} $$
$$ \begin{aligned}y & \equiv-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)\\ & =-\frac{1}{2}[(x_{a}^{T}\Lambda_{aa}x_{a}-x_{a}^{T}\Lambda_{aa}\mu_{a}-\mu_{a}^{T}\Lambda_{aa}x_{a}+\mu_{a}^{T}\Lambda_{aa}\mu_{a})\\ & +(x_{b}^{T}\Lambda_{ba}x_{a}-x_{b}^{T}\Lambda_{ba}\mu_{a}-\mu_{b}^{T}\Lambda_{ba}x_{a}+\mu_{b}^{T}\Lambda_{ba}\mu_{a})\\ & +\left(x_{a}^{T}\Lambda_{ab}x_{b}-x_{a}^{T}\Lambda_{ab}\mu_{b}-\mu_{a}^{T}\Lambda_{ab}x_{b}+\mu_{a}^{T}\Lambda_{ab}\mu_{b}\right)\\ & +\left(x_{b}^{T}\Lambda_{bb}x_{b}-x_{b}^{T}\Lambda_{bb}\mu_{b}-\mu_{b}^{T}\Lambda_{bb}x_{b}+\mu_{b}^{T}\Lambda_{bb}\mu_{b}\right)] \end{aligned} $$
Now assume $$x_{b}=\alpha x_{a}+B$$ for some matrix $\alpha$ and vector $\beta$
$$\begin{aligned}y & =-\frac{1}{2}[(x_{a}^{T}\Lambda_{aa}x_{a}-x_{a}^{T}\Lambda_{aa}\mu_{a}-\mu_{a}^{T}\Lambda_{aa}x_{a}+\mu_{a}^{T}\Lambda_{aa}\mu_{a})\\ & +(\left(\alpha x_{a}+B\right)^{T}\Lambda_{ba}x_{a}-\left(\alpha x_{a}+B\right)^{T}\Lambda_{ba}\mu_{a}-\mu_{b}^{T}\Lambda_{ba}x_{a}+\mu_{b}^{T}\Lambda_{ba}\mu_{a})\\ & +\left(x_{a}^{T}\Lambda_{ab}\left(\alpha x_{a}+B\right)-x_{a}^{T}\Lambda_{ab}\mu_{b}-\mu_{a}^{T}\Lambda_{ab}\left(\alpha x_{a}+B\right)+\mu_{a}^{T}\Lambda_{ab}\mu_{b}\right)\\ & +\left((\alpha x_{a}+B)^{T}\Lambda_{bb}\left(\alpha x_{a}+B\right)-\left(\alpha x_{a}+B\right)^{T}\Lambda_{bb}\mu_{b}-\mu_{b}^{T}\Lambda_{bb}\left(\alpha x_{a}+B\right)+\mu_{b}^{T}\Lambda_{bb}\mu_{b}\right)] \end{aligned} $$
Collecting all constants and terms linear and quadratic in $x_a$: $y=-\frac{1}{2}\left(x_{a}^{t}y_{q}x_{a}+x_{a}^{t}y_{l}+y_{c}\right)$
$y_{q}=(\Lambda_{aa}+\alpha^{T}\Lambda_{ba}+\Lambda_{ab}\alpha+\alpha^{T}\Lambda_{bb}\alpha)$ $\begin{aligned}y_{l} & =2((\Lambda_{ab}(B-\mu_{b})-\Lambda_{aa}\mu_{a})+\alpha^{T}(\Lambda_{bb}(B-\mu_{b})-\Lambda_{ba}\mu_{a}))\end{aligned}$
From the standard form of a Gaussian, $\exp(-\frac{1}{2}(x-\mu)^{T}\sigma^{-1}(x-\mu))=\exp(-\frac{1}{2}[x^{T}\sigma^{-1}x-x^{T}\sigma^{-1}\mu-\mu^{T}\sigma^{-1}x+\mu^{T}\sigma^{-1}\mu])=\exp(-\frac{1}{2}[x^{T}\sigma^{-1}x-x^{T}2\sigma^{-1}\mu-\mu^{T}\sigma^{-1}\mu])$
I assert:
$$\sigma_{x_{b}=\alpha x_{a}+\beta}^{-1}=y_{q}=(\Lambda_{aa}+\alpha^{T}\Lambda_{ba}+\Lambda_{ab}\alpha+\alpha^{T}\Lambda_{bb}\alpha)$$ and
$$ \begin{aligned}y_{l} & =-2\sigma^{-1}\mu\\ \mu{}_{x_{b}=\alpha x_{a}+\beta} & =-\frac{1}{2}\sigma_{x_{b}=\alpha x_{a}+\beta}y_{l}\\ & =-(\Lambda_{aa}+\alpha^{T}\Lambda_{ba}+\Lambda_{ab}\alpha+\alpha^{T}\Lambda_{bb}\alpha)^{-1}((\Lambda_{ab}(B-\mu_{b})-\Lambda_{aa}\mu_{a})+\alpha^{T}(\Lambda_{bb}(B-\mu_{b})-\Lambda_{ba}\mu_{a})) \end{aligned} $$
