I was sitting at a lecture on Naive Bayes, and the speaker, on a slide, said:

Given a feature $x = \begin{bmatrix} x_1, \ldots, x_n \end{bmatrix}^T$, the probability of the feature belong to class $c$ is given by,

$P(c|x) = P(x,c)/P(x) = P(x|c)P(c)/P(x)$

I was a bit surprised by this notation.

From Wikipedia, the Bayes rule only apply for either "events" or "random variables".


Is the feature (or data) $x$ considered to be event or random variable in this context?


Typically in machine learning (or statistics), we don't think of features/covariates as random variables. Or at least, we often don't care about the random nature of our features. We just want to model $P(y | X)$, where $y$ is the outcome of interest, and $X$ is our set of covariates. We care about the random nature of $y$ given $X$, but we assume $X$ is known so we don't bother with modeling the random nature of it.

Naive Bayes is a special case, though. Under Naive Bayes, we think of things in terms of a hierarchical process. At the very top is the outcome $y$. This is simply a binomial random variable (assuming binary outcomes). Then, our features follow a distribution conditional on the value of $y$. If we know the $P(y = 1)$ and $P(X | y = 0)$ and $P(X | y = 1)$, we can use Bayes Theorem to compute $P(y = 1 | X)$. The key point here is that using this approach, we need a model for $P(X | y)$ to make inference about $P(y | X)$.

So for this particular model, you do need to think about the distribution of $X$, in contrast with most other models that allow you to directly compute $P(y |X)$ and ignore the random aspect of $X$.


You mean using uppercase $P$? But then, if those were events they'd also use uppercase letters for the events. While people usually use uppercase $P$ for probabilities, not probability mass functions, same with uppercase letters for events, but there's nothing strict about it, as notation in math is flexible and can be ambiguous unless considering the context. Moreover, Bayes theorem is the same no matter if we are talking about probabilities, probability mass functions, or probability density functions.

  • $\begingroup$ I think this is missing the point of the OP's question: why do we talk about the distribution of the features when using the Naive Bayes estimator, but with other estimators, we typically only talk about the distribution of the response? $\endgroup$ – Cliff AB Mar 1 at 5:46
  • $\begingroup$ @CliffAB I'm not sure if that's the point, that's why I provided alternative answer. $\endgroup$ – Tim Mar 1 at 6:02
  • $\begingroup$ yeah, it's very possible I just saw the question I wanted to see $\endgroup$ – Cliff AB Mar 1 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.