# what is the “learning” that takes place in Naive Bayes?

As I recall algorithms like nearest neighbor don't build a model based on training data and then apply that model to test data. It just takes each new instance and compares it to all the data to find the closest one, etc.

What about Naive Bayes? It seems to be similar. For example neural networks learn parameters and then use the corresponding model on test data. But for Naive Bayes I don't see where the learning takes place. There are no learned parameters. It seems to again look at the entire dataset for each prediction. Can anyone comment on this?

Additionally, then what is the use of a training/test split. I can see that we would want a test set because we want it to be labeled but beyond that I don't see why we need test/train?

• "counting" (eg. to get prior distribution) is a type of "learning" Apr 7 '17 at 15:46

Different from the nearest neighbor algorithm, the Naive Bayes algorithm is not a lazy method; A real learning takes place for Naive Bayes. The parameters that are learned in Naive Bayes are the prior probabilities of different classes, as well as the likelihood of different features for each class. In the test phase, these learned parameters are used to estimate the probability of each class for the given sample.

In other words, in Naive Bayes, for each sample in the test set, the parameters determined during training are used to estimate the probability of that sample belonging to different classes. For example, $P(c|x)\propto P(c)P(x_1|c)P(x_2|c)...p(x_n|c)$ where $c$ is a class and $x$ is a test sample. All quantities $P(c)$ and $P(x_i|c)$ are parameters which are determined during training and are used during testing. This is similar to NN, but the kind of learning and the kind of applying the learned model is different.

As an example, take a look at the Naive Bayes implementation in nltk. See the train and prob_classify methods. In the train method, label_probdist and feature_probdist are computed, and in the prob_classify method, these parameters are used to estimate the probability of different class for a test sample. Just note that _label_probdist and _feature_probdist are respectively initialized to label_probdist and feature_probdist in the constructor.

About your second question (the final paragraph), even for the lazy methods such as the nearest neighbor method, we need to split data into train/test. This is because we want to evaluate the performance of the model obtained based on the training data on some samples that are not seen during training to obtain a reasonable measure of the model generalization.

• I don't see that the learning would take place in the training has anything to do with what happens on the testing. With NN the model which was learned is applied to the test but nothing special is applied to the test in Naive Bayes it seems. Further, unless I don't recall properly nearest neighbor does not use any information it used in the training to improve the result in the test
– matt
Apr 7 '17 at 15:29
• @matt I modified the answer to be more clear. Look at the second paragraph. Is it now clear? Apr 7 '17 at 16:20
• It seems to me that the entire process is recalculated in the test set. Is that wrong? Nothing is carried over from the training set to the test set, whereas in something like a neural network, the learned parameters are carried over
– matt
Apr 8 '17 at 1:15
• @matt No! The process is not repeated for the test set. The whole parameters are carried over from the training set to the test set. In this aspect, it is completely like NNs. Apr 8 '17 at 2:21
• thanks for the comments. Do you know of an implementation of this that shows that it learns the parameters and carries them over to the testing task? I downloaded e1071 and then tried to view naiveBayes but it just gives me a kind of wrapper like this:naiveBayes function (x, ...) UseMethod("naiveBayes") <environment: namespace:e1071>
– matt
Apr 8 '17 at 14:19

I did not see any parameters in the Naive Bayes Classifier.

I think we do not need to learn parameters like what we do in Neural networks, but we learn(or calculate) the prior probability from the training data. Then we apply the prior we calculated from training data to make predictions in test data.

• The estimated probabilities are parameters.
– Tim
Nov 2 '19 at 12:21