I'm currently on the Classification chapter in Introduction to Statistical Learning in which Linear Discriminant Analysis is discussed. In the chapter, using Bayes' Theorem, we can define the probability an observation belong to class k given an observation $X$ by

$P(Y=k|X=x) = \frac{\pi_kf_k(x)}{\sum{\pi_if_i(x)}}$, where $f_k(x)=P(X=x|Y=k)$

The author says to assume $f_k(x)$ is the normal distribution, so he makes it equal to the normal density function $N(\mu, \sigma)$. However, since the normal distribution is a continuous distribution, should $f_k(x)=P(X=x|Y=k)$ be $0$? Trying to find $P(X=x)$ for any continuous distribution is 0, so why isn't that the case here? Shouldn't we be using the the CDF of the normal distribution and finding $F(X=x)$, or maybe put some boundaries around $x$ and find the area under the curve there?


1 Answer 1


In the book, the statement (in section 4.4.1 Using Bayes' Theorem for Classification) has this footnote

Technically this definition is only correct if $x$ is a discrete random variable. If $x$ is continuous then $f_k(x)dx$ would correspond to the probability of $x$ falling in in a small region $dx$ around $x$

While the definition of $f_k(x)$ as $P(X=x|Y=k)$ is a bit handwavy, the resulting use of Bayes' Theorem is not, and it is correct that $$P(Y=k|X=x)= \frac{\pi_kf_k(x)}{\sum_l \pi_l f_l(x)}$$ Bayes' Theorem works for densities, not just for probabilities.

One way to make the argument rigorous is to work in terms of $\int_A f_k(x)\,dx$, where $A$ is a set containing $x$. This gives you $P(Y=k|X\in A)$, and as $A$ gets smaller, the probability will approach what's in the book. There are other ways that involve more maths.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.