2
$\begingroup$

Most of classification models are based on searching for a class maximizing P(class | features). But why not the opposite, P(features | class)?

Then, the second question is, are there any situations where the latter is more appropriate?

$\endgroup$
5
  • $\begingroup$ Have a look at the Bayes Decision Rule. The rule describes exactly what you wrote here: that we should choose the class that maximizes P(class | features). Why use the Bayes decision rule? Because it minimizes the probability of error. $\endgroup$
    – mhdadk
    Aug 7, 2022 at 14:44
  • $\begingroup$ See also stats.stackexchange.com/q/118696/296197 $\endgroup$
    – mhdadk
    Aug 7, 2022 at 14:45
  • 1
    $\begingroup$ It's because in the canonical regression problem, we have the features and want to figure out what the class is, not the other way around! $\endgroup$ Aug 7, 2022 at 15:13
  • $\begingroup$ Why are you asking? Could you tell us more about how would you like to use it? As the answers below say, the obvious answer is that if you want to predict an unknown class from the features, then P(features | class) is not really useful. $\endgroup$
    – Tim
    Aug 8, 2022 at 7:54
  • $\begingroup$ We actually use the latter problem exactly to convert it to the former one. (LDA as an example). $\endgroup$
    – ttnphns
    Aug 8, 2022 at 10:08

3 Answers 3

1
$\begingroup$

We do classify with P(features | class). Recall from Bayes rule...

$$ P(y \vert x ) \propto P(x \vert y) P(y) $$

with equality if we divide by a normalizing constant. P(features | class) is a part of P(class | features)

$\endgroup$
0
$\begingroup$

The class is unknown. The features are known. It doesn’t make sense to use the unknown to predict the known. We neither have access to the unknown nor need to guess about the known.

For instance, it makes sense to use world events from this past weekend to predict if the stock market will go up or down tomorrow. It makes less sense to use tomorrow’s market movement to predict what happened over the weekend.

Also worth a mention is that, by Bayes’ theorem, P(features|class) is implicitly considered by P(class|features).

$\endgroup$
0
$\begingroup$

Interesting question, what is actually $P$(features $\mid$ class)? It is the conditional probability distribution of the feature vector, given a specific class outcome. Its expected value $E[P$(features $\mid$ class)$]$ is the average of this probability distribution.

The capital letter $P$ indicates that the distribution of the features is (multivariate) discrete. This means that each possible outcome of the feature vector has one associated probability with it. Also \begin{equation} \sum_{\vec{f} \; \in \; feature-space} P(\vec{f} \mid class) \; = \;1 \end{equation} The more features in the feature-space, the smaller will the probabilities $P$(features $\mid$ class) generally be.

In essence, the distribution of $P$(features $\mid$ class) is interesting to study as it represents the typicality of the various feature outcomes, for the given class. It therefore provides insight into the objects/cases that belong to this specific class. This is where $P$(features $\mid$ class) is of interest.

A final note relates to continuous features. These are real number measurements so Bayes classification rule instead makes use of the density: p(features | class). Density values are only interesting when compared relatively to each other. Or when they are integrated over an interval. So for continuous features p(features | class) is generally not studied.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.