9
$\begingroup$

I'm using naive bayes classifier to classify between two groups of data. One group of the data is much larger than the other (above 4 times). I'm using the prior probability of each group in the classifier.

The problem is that the result I get has 0% true positive rate and 0% false positive rate. I got the same results when I set the prior to 0.5 and 0.5 .

How can I set my threshold to something better so I could get a more balanced results?

I had a similar problem when using Logistic Regression classifier. I solved it by subtracting the prior term from the bias.

When I use Fisher Linear Discriminant on this data, I get good results with the threshold set in the middle.

I assume there is some common solution to this problem, I just couldn't find it.

UPDATE: I've just noticed that I the classifier is overfitting. The performance on the training set is perfect (100% correct).

If I use equal groups, then the classifier starts classifying to the "small" group as well, but the performance is pretty bad (worse than FLD or LR).

UPDATE2: I think the problem was that I was using full covariance matrix. Running with diagonal covariance matrix gave me more "balanced" results.

Improve this question
$\endgroup$
2
  • $\begingroup$ 0% true positive and 0% false positive? It must be putting everything in the other group, then... try setting your prior at 80% for the "positive" group. But first, check to see you aren't making a mistake somewhere in the code... $\endgroup$
    – jbowman
    Commented Dec 13, 2011 at 21:27
  • $\begingroup$ Maybe the prior is too big/small ? (some problems with Floating-Point Arithmetic ?) $\endgroup$
    – Dov
    Commented Dec 13, 2011 at 21:35

2 Answers 2

Reset to default
7
$\begingroup$

Assigning all patterns to the negative class certainly is not a "wierd result". It could be that the Bayes optimal classifier always classifies all patterns as belonging to the majority class, in which case your classifier is doing exactly what it should do. If the density of patterns belonging to the positive class never exceeds the density of the patterns belonging to the negative class, then the negative class is more likely regardless of the attribute values.

The thing to do in such circumstances is to consider the relative importance of false-positive and false-negative errors, it is rare in practice that the costs of the two different types of error are the same. So determine the loss for false positive and false negative errors and take these into account in setting the threshold probability (differing misclassification costs is equivalent to changing the prior probabilities, so this is easy to implement for naive Bayes). I would recommend tuning the priors to minimise the cross-validation estimate of the loss (incorporating your unequal misclassification costs).

If your misclassification costs are equal, and your training set priors representative of operational conditions, then assuming that your implementation is correct, it is possible that you already have the best NB classifier.

Improve this answer
$\endgroup$
4
  • $\begingroup$ check my update please. My classifier is overfitting. any way, if I use equal groups then the classifier starts behaving better, but still low performance.. changing the threshold (priors) doesn't affect the performance at all when one group is much larger. thanks $\endgroup$
    – Ran
    Commented Dec 14, 2011 at 21:03
  • 1
    $\begingroup$ @Ran I cannot get rid of the feeling that something is missing here.1. Can you please report the AUC of each classifier ? 2. How many positive / negative instances do you have per class ? 3. How did you validate the classifier ? which-k-fold-cv ? loo ? 3. Note that NB's nature is to create extreme probabilities p(class|x) close to 0 or 1, so one has to find the best decision-threshold, i.e. t so that p(class=1|x)>t => class 1, else class 0. Finding such a decision threshold is equivalent to adjusting the priors. $\endgroup$
    – steffen
    Commented Dec 15, 2011 at 7:26
  • $\begingroup$ @Ran changing the priors has to affect the performance ;). If small changes do not help, try some extreme. $\endgroup$
    – steffen
    Commented Dec 15, 2011 at 7:28
  • 2
    $\begingroup$ @Ran, it is difficult to determine what the problem is without having additional information about the size of the training set, nature of the attributes etc. However one thing comes to mind which is that NB has a problem if the conditional probabilities are ever zero, as if that happens, the output will be zero regardless of the values of any other attributes. Are the probabilities for the minority class always exactly zero? If that is the case, that may be the problem, in which case using the Laplace correction to estimate the conditional probabilities might help. $\endgroup$
    – Dikran Marsupial
    Commented Dec 15, 2011 at 17:49
0
$\begingroup$

Enlarge the smaller data group to fit the big group by calculation. It will stretch the smaller group's data, but it will allow a more equal calculation.

If you still get weird results like you currently do, check your whole implementation from start to hunt down a (probably simple) error.

Improve this answer
$\endgroup$
2
  • $\begingroup$ what do you mean by "Enlarge the smaller data... by calculation" ? can you further explain ? $\endgroup$
    – Dov
    Commented Dec 14, 2011 at 8:15
  • $\begingroup$ classifying everything as belonging to the negative class isn't a wierd result, sometimes that is the right thing to do because the density of negative patterns always exceeds the density of positive patterns everywhere. $\endgroup$
    – Dikran Marsupial
    Commented Dec 14, 2011 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.