0
$\begingroup$

Whilst looking at the answer to this question: Gaussian Discriminant Analysis and sigmoid function

I was wondering how they got from $$log(\frac{\pi_1}{\pi_0}) -(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 $$

to

$$ log(\frac{\pi_1}{\pi_0}) + (\mu_1 - \mu_0)^T\Sigma^{-1}x + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1) $$

More specifically - I don't understand how they got this term: $$(\mu_1 - \mu_0)^T\Sigma^{-1}x$$

Any help would be much appreciated!

$\endgroup$

1 Answer 1

1
$\begingroup$

Recall from matrix rules that:

(1) the matrix multiplication operator is distributive with respect to addition, meaning $A(B+C)=AB+AC$

(2) the transpose operator respects addition, meaning $(A+B)^T=A^T+B^T$.

Starting with the leftmost term (neglect the $\frac{1}{2}$ for now): from (1) we get $$(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)=(x-\mu_0)^T\Sigma^{-1}x-(x-\mu_0)^T\Sigma^{-1}\mu_0$$

and from (2): $$(x-\mu_0)^T\Sigma^{-1}x=x^T\Sigma^{-1}x-\mu_0^T\Sigma^{-1}x$$

and so the leftmost term can be written as

$$\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)=\frac{1}{2}\left(x^T\Sigma^{-1}x-\mu_0^T\Sigma^{-1}x-x^T\Sigma^{-1}\mu_0+\mu_0^T\Sigma^{-1}\mu_0\right)$$

adding up both terms: $$-(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)/2+(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)/2= \frac{1}{2}\left( -x^T\Sigma^{-1}x+\mu_1^T\Sigma^{-1}x+x^T\Sigma^{-1}\mu_1-\mu_1^T\Sigma^{-1}\mu_1+x^T\Sigma^{-1}x-\mu_0^T\Sigma^{-1}x-x^T\Sigma^{-1}\mu_0+\mu_0^T\Sigma^{-1}\mu_0 \right)= \frac{1}{2}\left( \mu_1^T\Sigma^{-1}x-\mu_0^T\Sigma^{-1}x+x^T\Sigma^{-1}\mu_1-x^T\Sigma^{-1}\mu_0-\mu_1^T\Sigma^{-1}\mu_1+\mu_0^T\Sigma^{-1}\mu_0 \right)= \frac{1}{2}\left( (\mu_1-\mu_0)^T\Sigma^{-1}x+x^T\Sigma^{-1}(\mu_1-\mu_0)-\mu_1^T\Sigma^{-1}\mu_1+\mu_0^T\Sigma^{-1}\mu_0 \right) $$

Now, note three things: (a) the transpose of a scalar is a scalar itself; (b) $\left((\mu_1-\mu_0)^T\Sigma^{-1}x\right)^T=x^T\Sigma^{-1}(\mu_1-\mu_0)$ and (c) $x^T\Sigma^{-1}(\mu_1-\mu_0)$ is a scalar, so we can add up:

$$\frac{1}{2}\left( (\mu_1-\mu_0)^T\Sigma^{-1}x+x^T\Sigma^{-1}(\mu_1-\mu_0)-\mu_1^T\Sigma^{-1}\mu_1+\mu_0^T\Sigma^{-1}\mu_0 \right)=\frac{1}{2}\left(2(\mu_1-\mu_0)^T\Sigma^{-1}x -\mu_1^T\Sigma^{-1}\mu_1+\mu_0^T\Sigma^{-1}\mu_0 \right)$$ $$=(\mu_1-\mu_0)^T\Sigma^{-1}x+\frac{1}{2}\left(\mu_0^T\Sigma^{-1}\mu_0 -\mu_1^T\Sigma^{-1}\mu_1 \right)$$

$\blacksquare$.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks! I didn't realise $\left((\mu_1-\mu_0)^T\Sigma^{-1}x\right)^T=x^T\Sigma^{-1}(\mu_1-\mu_0) $ was a scalar! $\endgroup$
    – clostar
    Aug 16, 2021 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.