I'm fitting a classifier with cross-entropy loss (i.e. Bernoulli likelihood). Some examples are very clearly associated with one class or the other, and despite some attempts at regularization, the classifier sometimes assigns probabilities $<10^{-9}$.

While the true probability might actually be this small in some cases, it's still causing some headaches. Among other things, the gradient is basically vertical in this region, and I'm running into numerical errors.

My question is this: in addition to regularizing the model's complexity, as I'm currently doing, is there anything wrong with also penalizing the model for making extreme predictions?

I'd also be interested in knowing if there are published papers discussing this approach, and what kinds of penalties would make the most sense. It seems like the most natural approach would be a beta-distributed prior on the outcome variable (equivalent to a prior belief that the classifier shouldn't return extreme values), but I'm very open to alternatives.


1 Answer 1


Here is a paper with an idea which might work for you (Tackling the Poor Assumptions of Naive Bayes Text Classifiers, Rennie et al.). In the paper they propose several modifications to the Naive Bayes algorithm in order to deal with some of its weaknesses.

One of them is the problem with skewed data. The idea (what they call complement naive bayes) is not to model each class directly but its complement. As they explain in the paper, it is the idea of doing one-vs-all (in multiclass problems) in order to get more even data sets.

Another one that might interest you, is to modify the weights in order to correct for assumptions that the data does not fulfill.

I have seen these ideas come up a couple of times (see for example here), in addition to do some regularization as you suggest) or perform stratified sampling to produce balanced training sets.

Hope that helps


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.