How to compute KL-divergence when there are categories of zero counts?

Question

I have two very large discrete frequency distributions (about 4 million items), and each contains many items with counts of 0. I want to calculate the KL divergence between them and use the empirical probability as the probability estimate. However, this seems to result in a negative value, which if I understand correctly, should not be possible with KL divergence, and I think it's because there are counts of 0.

For instance, if I run the following R code, I get a negative:

library(philentropy)

mat <- matrix(
  rep(1:271, each = 2), # 271 or higher all yield negative values
  nrow = 2
)
mat[2, 1] <- 0
KL(mat, est.prob = "empirical")

Result: -7.181671e-08

If I add a very small fixed number to every count, though, I get a very small but positive result:

mat <- mat + 0.01
KL(mat, est.prob = "empirical")

Result: 0.0001433061

Essentially, I'm using Laplace smoothing (I think) to get rid of 0 counts. Is this statistically valid, though?

Answer 1

It is valid to do smoothing if you have good reason to believe the probability of any specific to occur is not actually zero and you just didn't have a large enough sample size to view it.

Besides for it many times being a good idea to use an additive smoothing approach the KL divergence cannot be less than zero.

The reason it came out zero is probably an implementation issue and not because the true calculation using the estimated probabilities gave a negative result. See here https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence for a discussion on the non-negativity of KL divergence.

The question is also why you want to calculate the KL divergence. Do you want to compare multiple distributions and see which is closes to some specific distribution? In this case, probably it's better for the package you are using to do smoothing and this shouldn't rank of the output KL divergences on each distribution.