4
$\begingroup$

From Elements of Statistical Learning chap 4 - https://web.stanford.edu/~hastie/Papers/ESLII.pdf

We have K classes and we are modeling the posterior probability : $P(G=k|X=x)=\frac{f_{k}(x)\pi_{k}}{\sum_{l=1}^{K}f_{l}(x)}$

We suppose a multivariate gaussian model for $f_{k}(x)$ that is to say

$f_{k}(x)=\frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma_{k}|^{\frac{1}{2}}}\exp^{-\frac{1}{2}(x-\mu_{k})^{T}\Sigma_{k}^{-1}(x-\mu_{k})}$.

We assume covariance matrix identical for all $k$, $\Sigma_{k}=\Sigma$. We derive the log-ratio of posterio probabilities,

$\log{\frac{P(G=k|X=x)}{P(G=l|X=x)}}=\log{\frac{f_{k}(x)}{f_{l}(x)}}+\log{\frac{\pi_{k}}{\pi_{l}}}$

The book finds $\log{\frac{f_{k}(x)}{f_{l}(x)}}$ to equal $-\frac{1}{2}(\mu_{k}+\mu_{l})^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})+x^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})$ which I could not properly derive. Would you have the hand calculation for this line ?

$\endgroup$

1 Answer 1

2
$\begingroup$

We have $$\log\frac{f_k(x)}{f_\ell(x)}= \log\frac{\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k}))}{\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell}))}$$ $$=\log(\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})))-\log(\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})))$$ $$=-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})+\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})$$ Note that $B(x,y):=\frac12x^T\Sigma^{-1}y$ is a symmetric bilinear form, therefore: $$\log\frac{f_k(x)}{f_\ell(x)}=-B(x-\mu_k,x-\mu_k)+B(x-\mu_\ell,x-\mu_\ell)$$ $$=-B(x,x)+B(x,\mu_k)+B(\mu_k,x)-B(\mu_k, \mu_k)+B(x,x)-B(x,\mu_\ell)-B(\mu_\ell,x)+B(\mu_\ell, \mu_\ell)$$ $$=2B(x,\mu_k)-2B(x,\mu_\ell)-B(\mu_k,\mu_k)+B(\mu_\ell,\mu_\ell)$$ $$=2B(x,\mu_k-\mu_\ell)+B(\mu_\ell+\mu_k,\mu_\ell-\mu_k)$$ $$=2B(x,\mu_k-\mu_\ell)-B(\mu_k+\mu_\ell,\mu_k-\mu_\ell)$$ $$=x^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})-\frac{1}{2}(\mu_{k}+\mu_{\ell})^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})$$

$\endgroup$
4
  • $\begingroup$ thanks, why is it a symmetric bilinear form, precisely $\endgroup$
    – kiriloff
    Commented Apr 28, 2019 at 12:52
  • $\begingroup$ A covariance matrix $\Sigma$ is symmetric, therefore also the inverse $\Sigma^{-1}$ and $B(x,y)=x^T\Sigma^{-1}y=(x^T\Sigma^{-1}y)^T=y^T{\Sigma^{-1}}^T{x^T}^T=y^T\Sigma^{-1}x=B(y,x)$. $\endgroup$
    – Kornel
    Commented Apr 28, 2019 at 13:06
  • $\begingroup$ how do you justify $x^{T}Σ^{−1}y=(x^{T}\Sigma^{−1}y)^{T}$ ? $\endgroup$
    – kiriloff
    Commented Apr 28, 2019 at 19:49
  • 1
    $\begingroup$ It's a real number, i.e. a 1x1-matrix. So transposing doesn't do anything there. $\endgroup$
    – Kornel
    Commented Apr 28, 2019 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.