5
$\begingroup$

Can someone please explain how the third line becomes the fourth line? enter image description here

$\endgroup$
1
  • 2
    $\begingroup$ For vectors $u$ and $v$ and a square symmetric matrix $A$ you ask whether $y^\prime A y + u^\prime A u = (y-u)^\prime A (y-u) + 2u^\prime A y.$ This algebraic identify follows from the two properties listed at the left along with the symmetry of $A.$ Perhaps seeing the relation in this simpler notation will help you work through the steps. $\endgroup$
    – whuber
    Commented Feb 27, 2022 at 17:31

2 Answers 2

8
$\begingroup$

It is an issue of expanding and tidying up. You have for example

  • $(x-\mu_k)^T\Sigma^{-1}(x-\mu_k) = x^T\Sigma^{-1}x - \mu_k^T\Sigma^{-1}x-x^T\Sigma^{-1}\mu_k+\mu_k^T\Sigma^{-1}\mu_k$ and
  • $\mu_k^T\Sigma^{-1}x=x^T\Sigma^{-1}\mu_k$ and $\mu_k^T\Sigma^{-1}\mu_K=\mu_K^T\Sigma^{-1}\mu_k$ and
  • $(\mu_k+\mu_K)^T\Sigma^{-1}(\mu_k-\mu_K) = \mu_k^T\Sigma^{-1}\mu_k -\mu_K^T\Sigma^{-1}\mu_K$

so

$\log\left(\frac{\pi_k}{\pi_K}\right) -\frac12(x-\mu_k)^T\Sigma^{-1}(x-\mu_k) +\frac12(x-\mu_K)^T\Sigma^{-1}(x-\mu_K)$
$= \log\left(\frac{\pi_k}{\pi_K}\right) -\frac12 x^T\Sigma^{-1}x+ x^T\Sigma^{-1}\mu_k- \frac12\mu_k^T\Sigma^{-1}\mu_k +\frac12 x^T\Sigma^{-1}x- x^T\Sigma^{-1}\mu_K+ \frac12\mu_K^T\Sigma^{-1}\mu_K$
$= \log\left(\frac{\pi_k}{\pi_K}\right) - \frac12(\mu_k^T\Sigma^{-1}\mu_k - \mu_K^T\Sigma^{-1}\mu_K) + x^T\Sigma^{-1}\mu_k- x^T\Sigma^{-1}\mu_K$ $= \log\left(\frac{\pi_k}{\pi_K}\right) - \frac12\mu_k^T\Sigma^{-1}\mu_k -\mu_K^T\Sigma^{-1}\mu_K + x^T\Sigma^{-1}(\mu_k- \mu_K)$

$\endgroup$
7
$\begingroup$

Another way would be adding some terms to make the first multiplicands the same, and group (subtract the added terms outside of the parantheses as well): $$\begin{align}A&=-\frac{1}{2}(x-\mu_k-\overbrace{\mu_K}^{new})^T\Sigma^{-1}(x-\mu_k)+\frac{1}{2}(x-\mu_K-\overbrace{\mu_k}^{new})^T\Sigma^{-1}(x-\mu_K)\\&- \frac{1}{2}\mu_K^T\Sigma^{-1}(x-\mu_k)+\frac{1}{2}\mu_k^T\Sigma^{-1}(x-\mu_K)\rightarrow \text{subtract newly added terms}\\&=\frac{1}{2}(x-\mu_k-\mu_K)^T\Sigma^{-1}(\mu_k-\mu_K)+\frac{1}{2}x^T\Sigma^{-1}(\mu_k-\mu_K)\\&=x^T\Sigma^{-1}(\mu_k-\mu_K)-\frac{1}{2}(\mu_k+\mu_K)^T\Sigma^{-1}(\mu_k-\mu_K)\end{align}$$

Note that $x^T\Sigma^{-1}\mu=\mu^T\Sigma^{-1}x$ since inverse of cov. matrix is symmetric.

$\endgroup$
1
  • 5
    $\begingroup$ +1 for noting that symmetry is essential. $\endgroup$
    – whuber
    Commented Feb 27, 2022 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.