As I recall algorithms like nearest neighbor don't build a model based on training data and then apply that model to test data. It just takes each new instance and compares it to all the data to find the closest one, etc.

What about Naive Bayes? It seems to be similar. For example neural networks learn parameters and then use the corresponding model on test data. But for Naive Bayes I don't see where the learning takes place. There are no learned parameters. It seems to again look at the entire dataset for each prediction. Can anyone comment on this?

Additionally, then what is the use of a training/test split. I can see that we would want a test set because we want it to be labeled but beyond that I don't see why we need test/train?

  • $\begingroup$ "counting" (eg. to get prior distribution) is a type of "learning" $\endgroup$ – Haitao Du Apr 7 '17 at 15:46

Different from the nearest neighbor algorithm, the Naive Bayes algorithm is not a lazy method; A real learning takes place for Naive Bayes. The parameters that are learned in Naive Bayes are the prior probabilities of different classes, as well as the likelihood of different features for each class. In the test phase, these learned parameters are used to estimate the probability of each class for the given sample.

In other words, in Naive Bayes, for each sample in the test set, the parameters determined during training are used to estimate the probability of that sample belonging to different classes. For example, $P(c|x)\propto P(c)P(x_1|c)P(x_2|c)...p(x_n|c)$ where $c$ is a class and $x$ is a test sample. All quantities $P(c)$ and $P(x_i|c)$ are parameters which are determined during training and are used during testing. This is similar to NN, but the kind of learning and the kind of applying the learned model is different.

As an example, take a look at the Naive Bayes implementation in nltk. See the train and prob_classify methods. In the train method, label_probdist and feature_probdist are computed, and in the prob_classify method, these parameters are used to estimate the probability of different class for a test sample. Just note that _label_probdist and _feature_probdist are respectively initialized to label_probdist and feature_probdist in the constructor.

About your second question (the final paragraph), even for the lazy methods such as the nearest neighbor method, we need to split data into train/test. This is because we want to evaluate the performance of the model obtained based on the training data on some samples that are not seen during training to obtain a reasonable measure of the model generalization.

| cite | improve this answer | |
  • $\begingroup$ I don't see that the learning would take place in the training has anything to do with what happens on the testing. With NN the model which was learned is applied to the test but nothing special is applied to the test in Naive Bayes it seems. Further, unless I don't recall properly nearest neighbor does not use any information it used in the training to improve the result in the test $\endgroup$ – matt Apr 7 '17 at 15:29
  • 1
    $\begingroup$ @matt I modified the answer to be more clear. Look at the second paragraph. Is it now clear? $\endgroup$ – Hossein Apr 7 '17 at 16:20
  • $\begingroup$ It seems to me that the entire process is recalculated in the test set. Is that wrong? Nothing is carried over from the training set to the test set, whereas in something like a neural network, the learned parameters are carried over $\endgroup$ – matt Apr 8 '17 at 1:15
  • $\begingroup$ @matt No! The process is not repeated for the test set. The whole parameters are carried over from the training set to the test set. In this aspect, it is completely like NNs. $\endgroup$ – Hossein Apr 8 '17 at 2:21
  • $\begingroup$ thanks for the comments. Do you know of an implementation of this that shows that it learns the parameters and carries them over to the testing task? I downloaded e1071 and then tried to view naiveBayes but it just gives me a kind of wrapper like this:naiveBayes function (x, ...) UseMethod("naiveBayes") <environment: namespace:e1071> $\endgroup$ – matt Apr 8 '17 at 14:19

I did not see any parameters in the Naive Bayes Classifier.

I think we do not need to learn parameters like what we do in Neural networks, but we learn(or calculate) the prior probability from the training data. Then we apply the prior we calculated from training data to make predictions in test data.

| cite | improve this answer | |
  • $\begingroup$ The estimated probabilities are parameters. $\endgroup$ – Tim Nov 2 '19 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.