37

In general the naive Bayes classifier is not linear, but if the likelihood factors $p(x_i \mid c)$ are from exponential families, the naive Bayes classifier corresponds to a linear classifier in a particular feature space. Here is how to see this. You can write any naive Bayes classifier as* $$p(c = 1 \mid \mathbf{x}) = \sigma\left( \sum_i \log \frac{p(x_i ...


35

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling, but ...


31

Here is a list I found on http://www.dataschool.io/comparing-supervised-learning-algorithms/ indicating which classifier needs feature scaling: Full table: In k-means clustering you also need to normalize your input. In addition to considering whether the classifier exploits distances or similarities as Yell Bond mentioned, Stochastic Gradient Descent is ...


30

In $$ p(Y=C|\mathbf{x}) = \frac{p(\mathbf{x}|Y=C)p(Y=C)}{~\sum_{k=1}^{|C|}{}p(\mathbf{x}|Y=C_k)p(Y=C_k)} $$ both the denominator and the numerator can become very small, typically because the $p(x_i \vert C_k)$ can be close to 0 and we multiply many of them with each other. To prevent underflows, one can simply take the log of the numerator, but one needs ...


27

There is no single answer about which is the best classification method for a given dataset. Different kinds of classifiers should be always considered for a comparative study over a given dataset. Given the properties of the dataset, you might have some clues that may give preference to some methods. However, it would still be advisable to experiment with ...


20

Let's say you've trained your Naive Bayes Classifier on 2 classes, "Ham" and "Spam" (i.e. it classifies emails). For the sake of simplicity, we'll assume prior probabilities to be 50/50. Now let's say you have an email $(w_1, w_2,...,w_n)$ which your classifier rates very highly as "Ham", say $$P(Ham|w_1,w_2,...w_n) = .90$$ and $$P(Spam|w_1,w_2,..w_n) = ....


19

Unlike some classifiers, multi-class labeling is trivial with Naive Bayes. For each test example $i$, and each class $k$ you want to find: $$\arg \max_k P(\textrm{class}_k | \textrm{data}_i)$$ In other words, you compute the probability of each class label in the usual way, then pick the class with the largest probability.


18

You always need this 'fail-safe' probability. To see why consider the worst case where none of the words in the training sample appear in the test sentence. In this case, under your model we would conclude that the sentence is impossible but it clearly exists creating a contradiction. Another extreme example is the test sentence "Alex met Steve." where "...


16

Actually this is pretty simple: Bayes classifier chooses the class that has greatest a posteriori probability of occurrence (so called maximum a posteriori estimation). The 0-1 loss function penalizes misclassification, i.e. it assigns the smallest loss to the solution that has greatest number of correct classifications. So in both cases we are talking about ...


15

You can use any kind of predictor in a naive Bayes classifier, as long as you can specify a conditional probability $p(x|y)$ of the predictor value $x$ given the class $y$. Since naive Bayes assumes predictors are conditionally independent given the class, you can mix-and-match different likelihood models for each predictor according to any prior knowledge ...


14

You will not break the algorithm by having a word which shows up in $100\%$ of messages. The forumlas you are using for the probability are wrong. For the two-word case, here is an example to show why. Suppose your words are $a$, $b$, and $x$ and that you have two messages to use to build the classifier. The first message is spam and reads a b. The second ...


14

I'd be cautious about over interpreting Google trends. Here's naive bayes (blue) vs. k-means (red). What does it mean? I can make up a story that common variation is due to machine learning classes that teach both naive bayes and k-means. But that's just an educated guess, not an answer. I really don't know. And unless we start surveying people who search ...


11

It is linear only if the class conditional variance matrices are the same for both classes. To see this write down the ration of the log posteriors and you'll only get a linear function out of it if the corresponding variances are the same. Otherwise it is quadratic.


11

Informally, to be 'Bayesian' about a model (Naive Bayes just names a class of discrete mixture models) is to use Bayes theorem to infer the values of its parameters or other quantities of interest. To be 'Frequentist' about the same model is, roughly, and among other things, to use the sampling distribution of estimators that depend on those quantities to ...


11

Different from the nearest neighbor algorithm, the Naive Bayes algorithm is not a lazy method; A real learning takes place for Naive Bayes. The parameters that are learned in Naive Bayes are the prior probabilities of different classes, as well as the likelihood of different features for each class. In the test phase, these learned parameters are used to ...


10

Another possibility are neural networks, if you use the cross-entropy as the cost functional with sigmoidal output units. That will provide you with the estimates you are looking for. Neural networks, as well as logistic regression, are discriminative classifiers, meaning that they attempt to maximize the conditional distribution on the training data. ...


9

I think that the problem is not that you are using the classification methods poorly, but rather that this data has little predictive power for the regions. First of all, two classes have little data. table(y) East Asia & Pacific 35 Europe & Central Asia 56 Latin America & ...


8

Adding to the excellent (but too short) answer by Yell Bond. Look at what happens with a linear regression model, we write it with only two predictors but the issue do not depend on that. $$ Y_i = \beta_0 + \beta_1 x_i + \beta_2 z_i + \epsilon_i $$ $i=1, \dots, n$. Now if you, say, center and scale the predictors to get $$ x_i^* = (x_i - \bar{x})/\...


7

I disagree with discretizing to get rid of collinearity. It doesn't get rid of it, it just pushes it under a rug where it can cause problems while being less visible. "Number of guards" seems like a mediating variable. There is a lot of recent work on mediators, much of it by MacKinnon and his colleagues. E.g. this book but he has also written articles and ...


7

Here is a nice paper that addresses some of the 'systemic' shortcomings of the Multinomial Naive Bayes (MNB) classifier. The idea is that you can boost the performance of MNB through some tweaks. And they do mention using (uniform) Dirichlet priors. Overall if you're interested in MNB and you haven't read this paper yet, I would strongly recommend to do so....


7

One basic technique is Naive Bayes, which is often used for spam filtering. Essentially, you look at the frequencies of words appearing in sentences that you have already judged to be spam, and also at the frequencies of those words appearing in sentences you have already judged to be not spam, and then use those frequencies to make a judgement about new ...


7

To create a good model, the model has to be built on training data which is of the same "structure" as the data the model will applied later on. This is the one boring assumption which underlies all classification models. So by using an balanced data set meanwhile the real world is not balanced, you have already introduced a bias. While there are cases ...


7

In the paper "Tackling the Poor Assumptions of Naive Bayes Text Classifiers" the authors deal with this problem, among others, which stem from the character of the naive bayes algorithm. Having highly skewed data leads to a bias in your weights, which causes the bad precision. Concretely for the problem of skew data, what they proposed what they call the ...


7

Typically one would use Laplace smoothing, essentially adding an artificial observation of every feature for every class. This is done to avoid the issue of having never observed a feature in one class causing a zero that propagates. This is also called a uniform prior. For a feature never seen ever in any training data, the "uniform prior" means ...


7

Disregarding those words is another way to handle it. It corresponds to averaging (integrate out) over all missing variables. So the result is different. How? Assuming the notation used here: $$ P(C^{*}|d) = \arg\max_{C} \frac{\prod_{i}p(t_{i}|C)P(C)}{P(d)} \propto \arg\max_{C} \prod_{i}p(t_{i}|C)P(C) $$ where $t_{i}$ are the tokens in the vocabulary and $d$...


7

This question is rather simple if you are familiar with Bayes estimators, since it is the directly conclusion of Bayes estimator. In the Bayesian approach, parameters are considered to be a quantity whose variation can be described by a probability distribution(or prior distribution). So, if we view the procedure of picking up as multinomial distribution, ...


7

There are two types of classification model, generative model and discriminative model. Naive Bayes is a generative model, and to train Naive Bayes, your training data should be generated by the true process, and future data will be generated by that process as well. Balancing the data isn't part of the true process, so you can't do that. On the other hand,...


7

The third option is right one. In general, it is true that: $$ \log(ab) = \log(a) + \log(b)$$ Plugging in the Naive Bayes equation, you get $$ \log(P(\text{class }_i| \textbf{ data})) \propto \log(P(\text{class}_i)) + \sum_j \log(P(\textrm{data}_j|\text{class}_i))$$ This value may be negative. If your all of your terms were actual probabilities, they'd be ...


7

I've found that there isn't an incredible benefit in using downsampling/upsampling when classes are moderately imbalanced (i.e., no worse than 100:1) in conjunction with a threshold invariant metric (like AUC). Sampling makes the biggest impact for metrics like F1-score and Accuracy, because the sampling artificially moves the threshold to be closer to what ...


Only top voted, non community-wiki answers of a minimum length are eligible