# Bayes theorem in the context of generative classification models

In the Introduction to Statistical Learning p. 142 in chapter 4.4 on generative models for classification the formula $$P(Y = k|X = x) = \frac{π_k \cdot f_k (x)} {\sum_{l=1}^{K}π_l f_l(x)}$$ is given to determine the probability of class $$k$$ given the features $$x$$.

I understand that the Bayes theorem states $$P(A|B) = \frac{P(A)\cdot P(B|A)}{P(B)}$$. This implies $$P(k) = π_k$$, $$P(x|k) = f_k(x)$$, $$P(x) = \sum_{l=1}^{K}π_l f_l(x)$$.

I see that $$P(k)$$ is a prior probability, which can be found by looking at how often class $$k$$ occurs as opposed to the other class (calculating $$n_k/n$$). What is $$P(x)$$ though? How can I think about it in terms of a prior probability? How can there be a prior probability of $$x$$, if I can plug any kind of (unseen) data into $$x$$?

Note this question on why $$P(x) = \sum_{l=1}^{K}π_l f_l(x)$$.

• Are you sure that the features $X$ should be discrete here? Features are usually continuous with an associated probability density function $f_X(x)$. Jan 8, 2023 at 4:23
• As also pointed out by pglpm below, features can be anything, discrete, continuous or a mixed. Jan 9, 2023 at 15:34

The answer depends on the specific application. For example, the features/predictors $$X$$ (discrete, continuous, or mixed) could be results from a battery of clinical tests; say blood pressure, colesterol level, and so on. In this case $$p(X)$$ may represent the distribution (continuous/density, discrete/mass, or mixed) of such values in the population of interest, or our uncertainty about which particular values $$X$$ we'll observe in the next patient.

This distribution is important. Just as an example, consider the case where $$X$$ is binary with values $${+,-}$$, the predictand $$Y$$ is also binary with values $${0,1}$$, and we have the following conditional probabilities: \begin{aligned}p(0|+)&=0 & p(1|+)&=1,\\ p(0|-)&=1/2 & p(1|-)&=1/2\ . \end{aligned} The value $$X=\mathord{+}$$ is a perfect predictor, whereas $$X=\mathord{-}$$ is useless. Is the predictor $$X$$ good or not, overall?

The answer depends on $$p(X)$$. If values $$X=\mathord{-}$$ are rarely seen in the population, i.e. $$p(\mathord{+})/p(\mathord{-}) \gg 1$$, then $$X$$ is overall a good predictor, because we'll mostly encounter perfect-prediction situations with $$X=\mathord{+}$$. If instead $$p(\mathord{-})/p(\mathord{+}) \gg 1$$, then $$X$$ is overall a poor predictor: most of the times we'll be clueless about $$Y$$, except few lucky cases where we can be completely certain about it. This is also one of the reasons why the mutual information between $$X$$ and $$Y$$ depends on the probability $$p(X)$$.

The distribution $$p(X)$$ thus also affects our choice between two predictors (assuming we can't use them jointly for some reason). Imagine there's a second predictor $$Z$$ with values "a", "b" and these conditional probabilities: \begin{aligned}p(0|\mathrm{a})&=1/4 & p(1|\mathrm{a})&=3/4,\\ p(0|\mathrm{b})&=1/2 & p(1|\mathrm{b})&=1/2\ . \end{aligned}

Which to choose, $$X$$ or $$Z$$? At first $$X$$ would seem best, because $$X=\mathord{-}$$ and $$Z=\mathrm{b}$$ are equally useless, but $$X=\mathord{+}$$ predicts better than $$Z=\mathrm{a}$$.

Yet the answer again depends on $$p(X)$$ and $$p(Z)$$. If $$p(X=\mathord{+})$$ is very low (rare in the population), whereas $$p(Z=\mathrm{a})$$ is very high (frequent in the population), then on average we'll have better predictions by using $$Z$$.

In situations where you can arbitrarily set the value of $$X$$, then typically you don't use $$p(X)$$, which is formally just equal to 1 (you know which value you set). But also in such a context, you may be interested about some kind of future (or past but unknown) performance, and you may need to know how often any particular value $$X=x$$ will be set by you or whoever sets it. Then $$p(X)$$ enters again, and its assessment method differs wildly depending on the context.

I recommend MacKay's and Jaynes's books on these general matters, as well as a very illuminating paper by Lindley & Novick. You may also want to look up material about the base-rate fallacy.