51

I'm going to run through the whole Naive Bayes process from scratch, since it's not totally clear to me where you're getting hung up. We want to find the probability that a new example belongs to each class: $P(class|feature_1, feature_2,..., feature_n$). We then compute that probability for each class, and pick the most likely class. The problem is that we ...


44

The general term Naive Bayes refers the the strong independence assumptions in the model, rather than the particular distribution of each feature. A Naive Bayes model assumes that each of the features it uses are conditionally independent of one another given some class. More formally, if I want to calculate the probability of observing features $f_1$ ...


31

In general the naive Bayes classifier is not linear, but if the likelihood factors $p(x_i \mid c)$ are from exponential families, the naive Bayes classifier corresponds to a linear classifier in a particular feature space. Here is how to see this. You can write any naive Bayes classifier as* $$p(c = 1 \mid \mathbf{x}) = \sigma\left( \sum_i \log \frac{p(x_i ...


27

There is no single answer about which is the best classification method for a given dataset. Different kinds of classifiers should be always considered for a comparative study over a given dataset. Given the properties of the dataset, you might have some clues that may give preference to some methods. However, it would still be advisable to experiment with ...


26

Most Machine Learning problems are easy! See for example at John Langford's blog. What he's really saying is that ML makes problems easy, and this presents a problem for researchers in terms of whether they should try to apply methods to a wide range of simple problems or attack more difficult problems. However the by-product is that for many problems the ...


25

In $$ p(Y=C|\mathbf{x}) = \frac{p(\mathbf{x}|Y=C)p(Y=C)}{~\sum_{k=1}^{|C|}{}p(\mathbf{x}|Y=C_k)p(Y=C_k)} $$ both the denominator and the numerator can become very small, typically because the $p(x_i \vert C_k)$ can be close to 0 and we multiply many of them with each other. To prevent underflows, one can simply take the log of the numerator, but one needs ...


24

This paper seems to prove (I can't follow the math) that bayes is good not only when features are independent, but also when dependencies of features from each other are similar between features: In this paper, we propose a novel explanation on the superb classification performance of naive Bayes. We show that, essentially, the dependence distribution;...


21

In general, algorithms that exploit distances or similarities (e.g. in form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling, but ...


18

Accuracy vs F-measure First of all, when you use a metric you should know how to game it. Accuracy measures the ratio of correctly classified instances across all classes. That means, that if one class occurs more often than another, then the resulting accuracy is clearly dominated by the accuracy of the dominating class. In your case if one constructs a ...


17

You always need this 'fail-safe' probability. To see why consider the worst case where none of the words in the training sample appear in the test sentence. In this case, under your model we would conclude that the sentence is impossible but it clearly exists creating a contradiction. Another extreme example is the test sentence "Alex met Steve." where "...


17

Here is a list I found on http://www.dataschool.io/comparing-supervised-learning-algorithms/ indicating which classifier needs feature scaling: Full table: In k-means clustering you also need to normalize your input. In addition to considering whether the classifier exploits distances or similarities as Yell Bond mentioned, Stochastic Gradient Descent is ...


16

Unlike some classifiers, multi-class labeling is trivial with Naive Bayes. For each test example $i$, and each class $k$ you want to find: $$\arg \max_k P(\textrm{class}_k | \textrm{data}_i)$$ In other words, you compute the probability of each class label in the usual way, then pick the class with the largest probability.


16

Actually this is pretty simple: Bayes classifier chooses the class that has greatest a posteriori probability of occurrence (so called maximum a posteriori estimation). The 0-1 loss function penalizes misclassification, i.e. it assigns the smallest loss to the solution that has greatest number of correct classifications. So in both cases we are talking about ...


15

Let's say you've trained your Naive Bayes Classifier on 2 classes, "Ham" and "Spam" (i.e. it classifies emails). For the sake of simplicity, we'll assume prior probabilities to be 50/50. Now let's say you have an email $(w_1, w_2,...,w_n)$ which your classifier rates very highly as "Ham", say $$P(Ham|w_1,w_2,...w_n) = .90$$ and $$P(Spam|w_1,w_2,..w_n) = ....


14

I'd be cautious about over interpreting Google trends. Here's naive bayes (blue) vs. k-means (red). What does it mean? I can make up a story that common variation is due to machine learning classes that teach both naive bayes and k-means. But that's just an educated guess, not an answer. I really don't know. And unless we start surveying people who search ...


13

You can use any kind of predictor in a naive Bayes classifier, as long as you can specify a conditional probability $p(x|y)$ of the predictor value $x$ given the class $y$. Since naive Bayes assumes predictors are conditionally independent given the class, you can mix-and-match different likelihood models for each predictor according to any prior knowledge ...


11

In general, to train Naive Bayes for n-dimensional data, and k classes you need to estimate $P(x_i | c_j)$ for each $1 \leq i \leq n$, $1 \leq j \leq k$ . You can assume any probability distribution for any pair $(i,j)$ (although it's better to not assume discrete distribution for $P(x_i|c_{j_1})$ and continuous for $P(x_i | c_{j_2})$). You can have Gaussian ...


11

You will not break the algorithm by having a word which shows up in $100\%$ of messages. The forumlas you are using for the probability are wrong. For the two-word case, here is an example to show why. Suppose your words are $a$, $b$, and $x$ and that you have two messages to use to build the classifier. The first message is spam and reads a b. The second ...


10

There are many ways to perform naive Bayes classification (NBC). A common technique in NBC is to recode the feature (variable) values into quartiles, such that values less than the 25th percentile are assigned a 1, 25th to 50th a 2, 50th to 75th a 3 and greater than the 75th percentile a 4. Thus a single object will deposit one count in bin Q1, Q2, Q3, or ...


9

Sure you can use Naive Bayes. You just have to specify what form the conditional distribution will have. I can think of a few options: Binary distribution: Binarize your data using a threshold, and you revert to the problem that you were already solving. Parametric distribution: If there is some reasonable parametric distribution, e.g. Gaussian, you can ...


9

Another possibility are neural networks, if you use the cross-entropy as the cost functional with sigmoidal output units. That will provide you with the estimates you are looking for. Neural networks, as well as logistic regression, are discriminative classifiers, meaning that they attempt to maximize the conditional distribution on the training data. ...


9

Informally, to be 'Bayesian' about a model (Naive Bayes just names a class of discrete mixture models) is to use Bayes theorem to infer the values of its parameters or other quantities of interest. To be 'Frequentist' about the same model is, roughly, and among other things, to use the sampling distribution of estimators that depend on those quantities to ...


8

Naive Bayes generally uses a decision rule like $$ \text{argmax}_{C_i} P(C_i)P(D|C_i), $$ which comes from the fact we can write $$ P(C_i|D) = \frac{P(C_i)P(D|C_i)}{P(D)}. $$ and drop the denominator $P(D)$ since it does not depend on the class. However, since $P(D) << 1$ (i.e. there are many possible documents), neglecting it will cause the output of ...


8

As other answers correctly state, the reported probabilities from models such as logistic regression and naive Bayes are estimates of the class probability. If the model were true, the probability would indeed be the probability of a correct classification. However, it is quite important to understand that this could be misleading because the model is ...


8

Note that dat$X in your code is a numeric variable. The NaiveBayes implementation in klaR for numeric predictor variables calculates the mean and standard deviations of the predictor variable at each outcome level. Rather than dealing with standard deviations of 0, the klaR authors decided to throw an error. If you change dat$X to a factor, it will create ...


8

It is linear only if the class conditional variance matrices are the same for both classes. To see this write down the ration of the log posteriors and you'll only get a linear function out of it if the corresponding variances are the same. Otherwise it is quadratic.


8

Different from the nearest neighbor algorithm, the Naive Bayes algorithm is not a lazy method; A real learning takes place for Naive Bayes. The parameters that are learned in Naive Bayes are the prior probabilities of different classes, as well as the likelihood of different features for each class. In the test phase, these learned parameters are used to ...


7

Assigning all patterns to the negative class certainly is not a "wierd result". It could be that the Bayes optimal classifier always classifies all patterns as belonging to the majority class, in which case your classifier is doing exactly what it should do. If the density of patterns belonging to the positive class never exceeds the density of the ...


7

Having used Naive Bayesian Classifiers extensively in segmentation classification tools, my experience is consistent with published papers showing NBC to be comparable in accuracy to linear discriminant and CART/CHAID when all of the predictor variables are available. (By accuracy both "hit rate" in predicting the correct solution as the most likely one, ...


Only top voted, non community-wiki answers of a minimum length are eligible