I'm using naive bayes classifier to classify between two groups of data. One group of the data is much larger than the other (above 4 times). I'm using the prior probability of each group in the classifier.
The problem is that the result I get has 0% true positive rate and 0% false positive rate. I got the same results when I set the prior to 0.5 and 0.5 .
How can I set my threshold to something better so I could get a more balanced results?
I had a similar problem when using Logistic Regression classifier. I solved it by subtracting the prior term from the bias.
When I use Fisher Linear Discriminant on this data, I get good results with the threshold set in the middle.
I assume there is some common solution to this problem, I just couldn't find it.
UPDATE: I've just noticed that I the classifier is overfitting. The performance on the training set is perfect (100% correct).
If I use equal groups, then the classifier starts classifying to the "small" group as well, but the performance is pretty bad (worse than FLD or LR).
UPDATE2: I think the problem was that I was using full covariance matrix. Running with diagonal covariance matrix gave me more "balanced" results.