1
$\begingroup$

In the Introduction to Statistical Learning p. 142 in chapter 4.4 on generative models for classification the formula $P(Y = k|X = x) = \frac{π_k \cdot f_k (x)} {\sum_{l=1}^{K}π_l f_l(x)}$ is given to determine the probability of class $k$ given the features $x$.

I understand that the Bayes theorem states $P(A|B) = \frac{P(A)\cdot P(B|A)}{P(B)} $. This implies $P(k) = π_k$, $P(x|k) = f_k(x)$, $P(x) = \sum_{l=1}^{K}π_l f_l(x)$.

I see that $P(k)$ is a prior probability, which can be found by looking at how often class $k$ occurs as opposed to the other class (calculating $n_k/n$). What is $P(x)$ though? How can I think about it in terms of a prior probability? How can there be a prior probability of $x$, if I can plug any kind of (unseen) data into $x$?

Note this question on why $P(x) = \sum_{l=1}^{K}π_l f_l(x)$.

$\endgroup$
2
  • $\begingroup$ Are you sure that the features $X$ should be discrete here? Features are usually continuous with an associated probability density function $f_X(x)$. $\endgroup$
    – mhdadk
    Jan 8, 2023 at 4:23
  • $\begingroup$ As also pointed out by pglpm below, features can be anything, discrete, continuous or a mixed. $\endgroup$ Jan 9, 2023 at 15:34

1 Answer 1

0
$\begingroup$

The answer depends on the specific application. For example, the features/predictors $X$ (discrete, continuous, or mixed) could be results from a battery of clinical tests; say blood pressure, colesterol level, and so on. In this case $p(X)$ may represent the distribution (continuous/density, discrete/mass, or mixed) of such values in the population of interest, or our uncertainty about which particular values $X$ we'll observe in the next patient.

This distribution is important. Just as an example, consider the case where $X$ is binary with values ${+,-}$, the predictand $Y$ is also binary with values ${0,1}$, and we have the following conditional probabilities: $$ \begin{aligned}p(0|+)&=0 & p(1|+)&=1,\\ p(0|-)&=1/2 & p(1|-)&=1/2\ . \end{aligned} $$ The value $X=\mathord{+}$ is a perfect predictor, whereas $X=\mathord{-}$ is useless. Is the predictor $X$ good or not, overall?

The answer depends on $p(X)$. If values $X=\mathord{-}$ are rarely seen in the population, i.e. $p(\mathord{+})/p(\mathord{-}) \gg 1$, then $X$ is overall a good predictor, because we'll mostly encounter perfect-prediction situations with $X=\mathord{+}$. If instead $p(\mathord{-})/p(\mathord{+}) \gg 1$, then $X$ is overall a poor predictor: most of the times we'll be clueless about $Y$, except few lucky cases where we can be completely certain about it. This is also one of the reasons why the mutual information between $X$ and $Y$ depends on the probability $p(X)$.

The distribution $p(X)$ thus also affects our choice between two predictors (assuming we can't use them jointly for some reason). Imagine there's a second predictor $Z$ with values "a", "b" and these conditional probabilities: $$ \begin{aligned}p(0|\mathrm{a})&=1/4 & p(1|\mathrm{a})&=3/4,\\ p(0|\mathrm{b})&=1/2 & p(1|\mathrm{b})&=1/2\ . \end{aligned} $$

Which to choose, $X$ or $Z$? At first $X$ would seem best, because $X=\mathord{-}$ and $Z=\mathrm{b}$ are equally useless, but $X=\mathord{+}$ predicts better than $Z=\mathrm{a}$.

Yet the answer again depends on $p(X)$ and $p(Z)$. If $p(X=\mathord{+})$ is very low (rare in the population), whereas $p(Z=\mathrm{a})$ is very high (frequent in the population), then on average we'll have better predictions by using $Z$.


In situations where you can arbitrarily set the value of $X$, then typically you don't use $p(X)$, which is formally just equal to 1 (you know which value you set). But also in such a context, you may be interested about some kind of future (or past but unknown) performance, and you may need to know how often any particular value $X=x$ will be set by you or whoever sets it. Then $p(X)$ enters again, and its assessment method differs wildly depending on the context.

I recommend MacKay's and Jaynes's books on these general matters, as well as a very illuminating paper by Lindley & Novick. You may also want to look up material about the base-rate fallacy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.