3
$\begingroup$

From Elements of Statistical Learning chap 4 - https://web.stanford.edu/~hastie/Papers/ESLII.pdf

We have K classes and we are modeling the posterior probability : $P(G=k|X=x)=\frac{f_{k}(x)\pi_{k}}{\sum_{l=1}^{K}f_{l}(x)}$

We suppose a multivariate gaussian model for $f_{k}(x)$ that is to say

$f_{k}(x)=\frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma_{k}|^{\frac{1}{2}}}\exp^{-\frac{1}{2}(x-\mu_{k})^{T}\Sigma_{k}^{-1}(x-\mu_{k})}$.

We assume covariance matrix identical for all $k$, $\Sigma_{k}=\Sigma$. We derive the log-ratio of posterio probabilities,

$\log{\frac{P(G=k|X=x)}{P(G=l|X=x)}}=\log{\frac{f_{k}(x)}{f_{l}(x)}}+\log{\frac{\pi_{k}}{\pi_{l}}}$

The book finds $\log{\frac{f_{k}(x)}{f_{l}(x)}}$ to equal $-\frac{1}{2}(\mu_{k}+\mu_{l})^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})+x^{T}\Sigma^{-1}(\mu_{k}-\mu_{l})$ which I could not properly derive. Would you have the hand calculation for this line ?

$\endgroup$
2
$\begingroup$

We have $$\log\frac{f_k(x)}{f_\ell(x)}= \log\frac{\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k}))}{\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell}))}$$ $$=\log(\exp(-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})))-\log(\exp(-\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})))$$ $$=-\frac{1}{2}(x-\mu_{k})^{T}\Sigma^{-1}(x-\mu_{k})+\frac{1}{2}(x-\mu_{\ell})^{T}\Sigma^{-1}(x-\mu_{\ell})$$ Note that $B(x,y):=\frac12x^T\Sigma^{-1}y$ is a symmetric bilinear form, therefore: $$\log\frac{f_k(x)}{f_\ell(x)}=-B(x-\mu_k,x-\mu_k)+B(x-\mu_\ell,x-\mu_\ell)$$ $$=-B(x,x)+B(x,\mu_k)+B(\mu_k,x)-B(\mu_k, \mu_k)+B(x,x)-B(x,\mu_\ell)-B(\mu_\ell,x)+B(\mu_\ell, \mu_\ell)$$ $$=2B(x,\mu_k)-2B(x,\mu_\ell)-B(\mu_k,\mu_k)+B(\mu_\ell,\mu_\ell)$$ $$=2B(x,\mu_k-\mu_\ell)+B(\mu_\ell+\mu_k,\mu_\ell-\mu_k)$$ $$=2B(x,\mu_k-\mu_\ell)-B(\mu_k+\mu_\ell,\mu_k-\mu_\ell)$$ $$=x^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})-\frac{1}{2}(\mu_{k}+\mu_{\ell})^{T}\Sigma^{-1}(\mu_{k}-\mu_{\ell})$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thanks, why is it a symmetric bilinear form, precisely $\endgroup$ – kiriloff Apr 28 '19 at 12:52
  • $\begingroup$ A covariance matrix $\Sigma$ is symmetric, therefore also the inverse $\Sigma^{-1}$ and $B(x,y)=x^T\Sigma^{-1}y=(x^T\Sigma^{-1}y)^T=y^T{\Sigma^{-1}}^T{x^T}^T=y^T\Sigma^{-1}x=B(y,x)$. $\endgroup$ – Kornel Apr 28 '19 at 13:06
  • $\begingroup$ how do you justify $x^{T}Σ^{−1}y=(x^{T}\Sigma^{−1}y)^{T}$ ? $\endgroup$ – kiriloff Apr 28 '19 at 19:49
  • 1
    $\begingroup$ It's a real number, i.e. a 1x1-matrix. So transposing doesn't do anything there. $\endgroup$ – Kornel Apr 28 '19 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.