Overfit by removing misclassified objects? Actually this question may be simple for you, but I need to learn the correct answer.
If I remove misclassified instances from data set with Naive Bayes (it gives minimum FP rate) and then train logistic classifier on this filtered data set, will it overfit or not?
Thanks in advance.
 A: The following is not restricted to NB + LogRes
Overfitting = Loss of generalization.
When you train a model on dataset you generally assume that the data you use for training has a similar structure than the data the model is applied to later (the general assumption of predicting the future from the past). So if you remove parts of the data (namely the misclassified instances) and train a model on this reduced dataset, you effectively change the structure of the data in comparison to the test dataset (and hence violate this assumption). In this case the following can happen (when testing this model on an unreduced test-dataset):
In the best case nothing happens, e.g. of the following reasons:


*

*The missclassified instances represented only a tiny subspace of the dataspace (corresponds to a high accuracy achieved by the first model)

*The model classifies one part of the dataspace better and another one worse so that they even out.


In the worst case the quality decreases rapidly, because of overfitting / loss of generalization power. The model focuses too hard on the part of the dataspace of in first step correctly classified instances and hence is not able anymore to make even an approximate statement for the rest of the dataspace. 

I think what you are actually looking for is called Boosting, where one restricts the dataspace to the missclassified instances (i.e. doing the opposite of your strategy) to refine the model. The procedure tries to avoid overfitting by combining the different (subspace-)models afterwards, but nevertheless it is still an issue.
Here is an plain-text explanation of boosting with a illustrative graphic you might find helpful.
A: Naive Bayes and Logistic Regression (Classification) are both linear classifiers. If you remove all misclassified instances, then you will allow an infinite number of separators to have 0 training error. In the case of the logistic regression, this translate to your information matrix being singular (The information matrix must be inverted at each iteration of GLM). 
I don't know if that's what you mean by overfit.
