# regularized loss function

I'm fitting a classifier with cross-entropy loss (i.e. Bernoulli likelihood). Some examples are very clearly associated with one class or the other, and despite some attempts at regularization, the classifier sometimes assigns probabilities $<10^{-9}$.

While the true probability might actually be this small in some cases, it's still causing some headaches. Among other things, the gradient is basically vertical in this region, and I'm running into numerical errors.

My question is this: in addition to regularizing the model's complexity, as I'm currently doing, is there anything wrong with also penalizing the model for making extreme predictions?

I'd also be interested in knowing if there are published papers discussing this approach, and what kinds of penalties would make the most sense. It seems like the most natural approach would be a beta-distributed prior on the outcome variable (equivalent to a prior belief that the classifier shouldn't return extreme values), but I'm very open to alternatives.