14
$\begingroup$

To my (very basic) understanding, Naive Bayes estimates probabilities based on the class frequencies of each feature in the training data. But how does it calculate the frequency of continuous variables? And when doing prediction, how does it classify a new observation that may not have the same values of any observation in the training set? Does it use some sort of distance measure or find the 1NN?

$\endgroup$
10
$\begingroup$

There are many ways to perform naive Bayes classification (NBC). A common technique in NBC is to recode the feature (variable) values into quartiles, such that values less than the 25th percentile are assigned a 1, 25th to 50th a 2, 50th to 75th a 3 and greater than the 75th percentile a 4. Thus a single object will deposit one count in bin Q1, Q2, Q3, or Q4. Calculations are merely done on these categorical bins. Bin counts (probabilities) are then based on the number of samples whose variable values fall within a given bin. For example, if a set of objects have very high values for feature X1, then this will result in a lot of bin counts in the bin for Q4 of X1. On the other hand, if another set of objects has low values for feature X1, then those objects will deposit a lot of counts in the bin for Q1 of feature X1.

It's actually not a really clever calculation, it's rather a way of discretizing continuous values to discrete, and exploitation thereafter. The Gini index and information gain can be easily calculated after discretization to determine which features are the most informative, i.e., max(Gini).

Be advised, however, that there are many ways to perform NBC, and many are quite different from one another. So you just need to state which one you implemented in a talk or paper.

$\endgroup$
2
$\begingroup$

The heart of Naive Bayes is the heroic conditional assumption:

$$P(x \mid X, C) = P(x \mid C)$$

In no way must $x$ be discrete. For example, Gaussian Naive Bayes assumes each category $C$ has a different mean and variance: density $p(x \mid C = i) = \phi(\mu_i, \sigma^2_i)$.

There are different ways to estimate the parameters, but typically one might:

  • Use maximum likelihood with labelled data. (In the case of normal distribution, maximum likelihood estimates of the mean and variance are basically the sample mean and the sample variance.)
  • Something like EM Algorithm with unlabelled data.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.