Skip to main content
39 votes
Accepted

What algorithms need feature scaling, beside from SVM?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. Graphical-model ...
yell's user avatar
  • 566
38 votes
Accepted

Example of how the log-sum-exp trick works in Naive Bayes

In $$ p(Y=C|\mathbf{x}) = \frac{p(\mathbf{x}|Y=C)p(Y=C)}{~\sum_{k=1}^{|C|}{}p(\mathbf{x}|Y=C_k)p(Y=C_k)} $$ both the denominator and the numerator can become very small, typically because the $p(...
Franck Dernoncourt's user avatar
38 votes

What algorithms need feature scaling, beside from SVM?

Here is a list I found on http://www.dataschool.io/comparing-supervised-learning-algorithms/ indicating which classifier needs feature scaling: Full table: In k-means clustering you also need to ...
Franck Dernoncourt's user avatar
20 votes
Accepted

Why is the naive bayes classifier optimal for 0-1 loss?

Actually this is pretty simple: Bayes classifier chooses the class that has greatest a posteriori probability of occurrence (so called maximum a posteriori estimation). The 0-1 loss function penalizes ...
Tim's user avatar
  • 139k
15 votes

Is Naive Bayes becoming more popular? Why?

I'd be cautious about over interpreting Google trends. Here's naive bayes (blue) vs. k-means (red). What does it mean? I can make up a story that common variation is due to machine learning classes ...
Matthew Gunn's user avatar
  • 22.5k
11 votes
Accepted

what is the “learning” that takes place in Naive Bayes?

Different from the nearest neighbor algorithm, the Naive Bayes algorithm is not a lazy method; A real learning takes place for Naive Bayes. The parameters that are learned in Naive Bayes are the prior ...
Hossein's user avatar
  • 3,484
10 votes

How does Naive Bayes work with continuous variables?

The heart of Naive Bayes is the heroic conditional assumption: $$P(x \mid X, C) = P(x \mid C)$$ In no way must $x$ be discrete. For example, Gaussian Naive Bayes assumes each category $C$ has a ...
Matthew Gunn's user avatar
  • 22.5k
10 votes

Bayes classifier?

The Bayes classifier is the one that classifies according to the most likely category given the predictor $x$, i.e., $$ \text{arg max}_j P(Y = j \mid X = x) . $$ Since these "true" probabilities are ...
dsaxton's user avatar
  • 12.2k
10 votes

Why does Naive Bayes work better when the number of features >> sample size compared to more sophisticated ML algorithms?

What the author is getting at is that Naive Bayes implicitly treats all features as being independent of one another, and therefore the sorts of curse-of-dimensionality problems which typically rear ...
jon_simon's user avatar
  • 2,049
9 votes
Accepted

Poor multiclass classification using Caret in R

I think that the problem is not that you are using the classification methods poorly, but rather that this data has little predictive power for the regions. First of all, two classes have little ...
G5W's user avatar
  • 2,630
9 votes
Accepted

Maximum likelihood in Naive Bayes classifier

Bayesian model is defined in terms of likelihood function (probability of observing the data given the parameters) and priors (assumed distributions for the estimated parameters). Naive Bayes ...
Tim's user avatar
  • 139k
8 votes

In Naive Bayes, why bother with Laplace smoothing when we have unknown words in the test set?

This question is rather simple if you are familiar with Bayes estimators, since it is the directly conclusion of Bayes estimator. In the Bayesian approach, parameters are considered to be a quantity ...
T. Jiang's user avatar
  • 157
8 votes
Accepted

Intuitive example for MLE, MAP and Naive Bayes classifier

You find a coin, and want to check what are the chances (probability) that if you flip it, it would land on the "Heads" side. You flip it 10 times (a sample of 10 observations) and count the number of ...
ihadanny's user avatar
  • 3,340
8 votes

What algorithms need feature scaling, beside from SVM?

Adding to the excellent (but too short) answer by Yell Bond. Look at what happens with a linear regression model, we write it with only two predictors but the issue do not depend on that. $$ Y_i = ...
kjetil b halvorsen's user avatar
8 votes
Accepted

The purpose of threshold in naive bayes algorithm

In short: The threshold is not a part of the Naive Bayes algorithm A Naive Bayes algorithm will be able to say for a certain sample, that the probability of it being of C1 is 60% and of C2 is 40%. ...
Lejafar's user avatar
  • 618
7 votes

How was this intergral derived from Bayes' Rule in David Heckerman's Bayesian Network paper?

It comes from the following: $$p(D \vert \xi) = \int_\Theta p(D, \theta | \xi)d\theta = \int_\Theta p(D \vert \theta, \xi) p(\theta \vert \xi) d\theta.$$ This isn't actually coming from Eqn. 1 (which ...
jld's user avatar
  • 20.4k
7 votes

The difference between the Bayes Classifier and The Naive Bayes Classifier?

Naive Bayes assumes conditional independence, $P(X|Y,Z)=P(X|Z)$, Whereas more general Bayes Nets (sometimes called Bayesian Belief Networks) will allow the user to specify which attributes are, in ...
ShainaR's user avatar
  • 183
7 votes

Naive Bays or Naive Bayes?

It is just a common spelling error, see this link.
Samuel's user avatar
  • 490
7 votes
Accepted

When to use Bernoulli Naive Bayes?

Bernoulli Naive Bayes is for binary features only. Similarly, multinomial naive Bayes treats features as event probabilities. Your example is given for nonbinary real-valued features $(x,y)$, which do ...
scherm's user avatar
  • 1,035
7 votes

Intuition for why LDA is a special case of naive Bayes

Here's my intuition: The LDA classifier assumes that across all classes, the $p$ predictors $\boldsymbol{X}_k$ (for $k=1, \dots,p$) all share some covariance matrix ${\boldsymbol \Sigma}$, but may ...
Wesley's user avatar
  • 580
6 votes

How is Naive Bayes a Linear Classifier?

I'd like add one additional point: the reason for some of the confusion rests on what it means to be performing "Naive Bayes classification". Under the broad topic of "Gaussian Discriminant Analysis ...
MrDrFenner's user avatar
6 votes

Hidden Markov Model and Naive Bayes similarity

Hidden Markov Model assumes a relationship between $y_n$ and $y_{n+1}$. For example say we are doing natural language processing, and $y_n$ denotes the $n$-th world in a sentence. If we know $y_n$ is ...
dontloo's user avatar
  • 16.5k
6 votes
Accepted

With the Naive Bayes classifier, why do we have to normalize the probabilities after calculating the probabilities of each hypothesis?

You do not have to normalize the probabilities if you only care about knowing which class ($\hat{y}$) your input ($\mathbf{x}=x_1, \dots, x_n$) most likely belongs to, since the maximum a posteriori (...
Franck Dernoncourt's user avatar
6 votes

In layman's terms, why is Naive Bayes the dominant algorithm used for text-classification?

I'm taking your word for Naive Bayes' popularity here as language processing isn't my specialty: One reason NB is useful is the bias–variance tradeoff. Spam/sentiment type data are often noisy and ...
einar's user avatar
  • 4,272
6 votes

Product of two normal distributions (for Bayes Rule) is not product of normal output variables?

The problem is that you’re abusing the notation. What do you exactly mean by $\mathcal{N}(\mu, \sigma^2)$ in here? Is it values of random variable, it’s probability density function, or maybe ...
Tim's user avatar
  • 139k
5 votes

Naive Bayes feature probabilities: should I double count words?

In other words, does the fact that one document mentioned viagra 3 times instead of once really not matter? It does matter. The Multinomial Naive Bayes model takes into account each occurrence of a ...
Franck Dernoncourt's user avatar
5 votes
Accepted

How to interpret the outcome tables of a naive Bayes classifier

Since you do not provide your data, I cannot demonstrate with that. Instead, I will illustrate the meaning of the tables using the iris dataset. ...
G5W's user avatar
  • 2,630
5 votes

Prove $P(A|B)=P(A\cap B)/P(B)$?

As already pointed out in the comment, the statement $$ {\displaystyle P(A|B):={\frac {P(A\cap B)}{P(B)}}} $$ is the very definition of the conditional probability. Check for example here on ...
user190080's user avatar
5 votes

Standardisation in Naive Bayes?

I assume you are using Gaussian, not Multinomial/Binomial Naive Bayes? For Gaussian Naive Bayes, the estimator learns the mean and standard deviation of each feature (per class). At prediction time ...
Jon Nordby's user avatar
  • 1,542

Only top scored, non community-wiki answers of a minimum length are eligible