# Linear Discriminant Analysis - deriving classifier expression for multivariate normal distribution

From Elements of Statistical Learning chap 4 - https://web.stanford.edu/~hastie/Papers/ESLII.pdf

We have K classes and we are modeling the posterior probability : $$P(G=k|X=x)=\frac{f_{k}(x)\pi_{k}}{\sum_{l=1}^{K}f_{l}(x)}$$

We suppose a multivariate gaussian model for $$f_{k}(x)$$ that is to say

$$f_{k}(x)=\frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma_{k}|^{\frac{1}{2}}}\exp^{-\frac{1}{2}(x-\mu_{k})^{T}\Sigma_{k}^{-1}(x-\mu_{k})}$$.

We assume covariance matrix identical for all $$k$$, $$\Sigma_{k}=\Sigma$$. We derive the log-ratio of posterio probabilities,

$$\log{\frac{P(G=k|X=x)}{P(G=l|X=x)}}=\log{\frac{f_{k}(x)}{f_{l}(x)}}+\log{\frac{\pi_{k}}{\pi_{l}}}$$

The book finds $$\log{\frac{f_{k}(x)}{f_{l}(x)}}$$ to equal $$-\frac{1}{2}(\mu_{k}+\mu_{l})^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})+x^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})$$ which I could not properly derive. Would you have the hand calculation for this line ?

## 1 Answer

We have $$\log\frac{f_k(x)}{f_\ell(x)}= \log\frac{\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k}))}{\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell}))}$$ $$=\log(\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})))-\log(\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})))$$ $$=-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})+\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})$$ Note that $$B(x,y):=\frac12x^T\Sigma^{-1}y$$ is a symmetric bilinear form, therefore: $$\log\frac{f_k(x)}{f_\ell(x)}=-B(x-\mu_k,x-\mu_k)+B(x-\mu_\ell,x-\mu_\ell)$$ $$=-B(x,x)+B(x,\mu_k)+B(\mu_k,x)-B(\mu_k, \mu_k)+B(x,x)-B(x,\mu_\ell)-B(\mu_\ell,x)+B(\mu_\ell, \mu_\ell)$$ $$=2B(x,\mu_k)-2B(x,\mu_\ell)-B(\mu_k,\mu_k)+B(\mu_\ell,\mu_\ell)$$ $$=2B(x,\mu_k-\mu_\ell)+B(\mu_\ell+\mu_k,\mu_\ell-\mu_k)$$ $$=2B(x,\mu_k-\mu_\ell)-B(\mu_k+\mu_\ell,\mu_k-\mu_\ell)$$ $$=x^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})-\frac{1}{2}(\mu_{k}+\mu_{\ell})^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})$$

• thanks, why is it a symmetric bilinear form, precisely Apr 28, 2019 at 12:52
• A covariance matrix $\Sigma$ is symmetric, therefore also the inverse $\Sigma^{-1}$ and $B(x,y)=x^T\Sigma^{-1}y=(x^T\Sigma^{-1}y)^T=y^T{\Sigma^{-1}}^T{x^T}^T=y^T\Sigma^{-1}x=B(y,x)$. Apr 28, 2019 at 13:06
• how do you justify $x^{T}Σ^{−1}y=(x^{T}\Sigma^{−1}y)^{T}$ ? Apr 28, 2019 at 19:49
• It's a real number, i.e. a 1x1-matrix. So transposing doesn't do anything there. Apr 28, 2019 at 20:37