13
$\begingroup$

I'm studying 'Introduction to Statistical Learning' by James, Witten, Hastie, Tibshirani.

In page 139, of their book, they began by introducing Bayes' Theorem $p_k(X)=P(Y=k|X=x) = \dfrac{\pi_kf_k(x)}{\sum_{l=1}^k \pi_l f_l(x)}$. $\pi$ is not mathematical constant, but denotes the prior probability. Nothing is strange in this equation.

The book claims that it wants to obtain an estimate for $f_k(x)$ that it can plug into the equation given above. To estimate $f_k(x)$, it assumes that is normal. In the one dimensional setting, $f_k(x)=\dfrac{1}{\sqrt{2\pi}\sigma}\exp(-\dfrac{1}{2\sigma^2}(x-\mu_k)^2)$, where $\mu_k$ and $\sigma^2_k$ are the mean and variance for the $k$th class. It is assumed that $\sigma^2_1 = \sigma^2_2 = \cdots = \sigma^2_K$. (I began to get confused from the last statement.)

Plugging $f_k$ into $p_x$, you have this quite messy equation (1) :

$$p_x(k)=\dfrac{\pi_k \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_k)^2)}{\sum_{l=1}^K \pi_l \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_l)^2)}. $$

Again, no surprises here since it is just substitution.


The Bayes' Classifier involves assigning an observation to the class for which equation (1) is the largest. Taking the log of Equation (1) and rearranging terms, it is not hard to show that this is equivalent to assigning the observation to the class for which the following is the largest:

$$\delta_k(x)=x \cdot \dfrac{\mu_k}{\sigma^2} - \dfrac{\mu_k^2}{2\sigma^2} + \log(\pi_k) $$

Question: I do not understand where this came from, and what it means. I tried doing the log of equation and it doesn't become this. Are we taking the derivative somewhere here, since this is the largest observation?

$\endgroup$
1

1 Answer 1

8
$\begingroup$

You can express equation (1) up to a proportionality constant,

$$p_x(k)\propto \pi_k \frac{1}{\sqrt{2\pi}\sigma}\exp \left(-\frac{1}{2\sigma^2}\left(x-\mu_k \right)^2 \right) $$

so if you then take logs

$$ \log p_x(k) \propto \log \pi_k - \log (\sqrt{2\pi} \sigma) -\frac{1}{2\sigma^2}\left(x-\mu_k \right)^2 $$

where $- \log (\sqrt{2\pi} \sigma)$ again goes into the proportionality constant because it doesn't depend on $k$. Then expand the squared term and you are there (note that expanding the bracket will give another term which will vanish into $\propto$).

$\endgroup$
2
  • $\begingroup$ I laughed when I read your answer. It IS that simple?! Brilliant! May I know what the quantity $\delta_k(x)$ means (in layman's terms)? My background in stats is poor, but I am able to follow the mathematics. $\endgroup$
    – cgo
    Apr 14, 2015 at 16:05
  • 2
    $\begingroup$ Yea, that's all there is to it. $\delta_k (x)$ is called the linear discriminant function. It's just a different way of writing the posterior probability that an observation belongs to class $k$ given its characteristic $x$. So picking the class with the highest posterior probability from Bayes rule is the same as picking the class with the highest value for the LDA. You can set $\delta_k(x) = \delta_l(x)$ to compute the Bayesian decision boundary (which gives a threshold for when an observation should be classified as $k$ or $l$). $\endgroup$
    – Andy
    Apr 14, 2015 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.