Linear Discriminant Analysis - deriving classifier expression for multivariate normal distribution

Question

From Elements of Statistical Learning chap 4 - https://web.stanford.edu/~hastie/Papers/ESLII.pdf

We have K classes and we are modeling the posterior probability : $P(G=k|X=x)=\frac{f_{k}(x)\pi_{k}}{\sum_{l=1}^{K}f_{l}(x)}$

We suppose a multivariate gaussian model for $f_{k}(x)$ that is to say

$f_{k}(x)=\frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma_{k}|^{\frac{1}{2}}}\exp^{-\frac{1}{2}(x-\mu_{k})^{T}\Sigma_{k}^{-1}(x-\mu_{k})}$.

We assume covariance matrix identical for all $k$, $\Sigma_{k}=\Sigma$. We derive the log-ratio of posterio probabilities,

$\log{\frac{P(G=k|X=x)}{P(G=l|X=x)}}=\log{\frac{f_{k}(x)}{f_{l}(x)}}+\log{\frac{\pi_{k}}{\pi_{l}}}$

The book finds $\log{\frac{f_{k}(x)}{f_{l}(x)}}$ to equal $-\frac{1}{2}(\mu_{k}+\mu_{l})^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})+x^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})$ which I could not properly derive. Would you have the hand calculation for this line ?

Answer 1

We have $$\log\frac{f_k(x)}{f_\ell(x)}= \log\frac{\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k}))}{\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell}))}$$ $$=\log(\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})))-\log(\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})))$$ $$=-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})+\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})$$ Note that $B(x,y):=\frac12x^T\Sigma^{-1}y$ is a symmetric bilinear form, therefore: $$\log\frac{f_k(x)}{f_\ell(x)}=-B(x-\mu_k,x-\mu_k)+B(x-\mu_\ell,x-\mu_\ell)$$ $$=-B(x,x)+B(x,\mu_k)+B(\mu_k,x)-B(\mu_k, \mu_k)+B(x,x)-B(x,\mu_\ell)-B(\mu_\ell,x)+B(\mu_\ell, \mu_\ell)$$ $$=2B(x,\mu_k)-2B(x,\mu_\ell)-B(\mu_k,\mu_k)+B(\mu_\ell,\mu_\ell)$$ $$=2B(x,\mu_k-\mu_\ell)+B(\mu_\ell+\mu_k,\mu_\ell-\mu_k)$$ $$=2B(x,\mu_k-\mu_\ell)-B(\mu_k+\mu_\ell,\mu_k-\mu_\ell)$$ $$=x^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})-\frac{1}{2}(\mu_{k}+\mu_{\ell})^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})$$