I am trying to use KL divergence as the separation measure between the classes. I have the positive, negative samples for 2 distributions and want to adjust the algorithm parameters to get the best possible separation defined by the maximum (or minimum) of the measure.

Previously I was trying with Bhattacharyya distance, but the normal distribution does not reflect the nature of my data and the results were not the best. This is why I switched to KL divergence, created histograms from the data samples and the method worked pretty nicely. The KL formula:

$$D_{KL}(P||Q) = \sum_{i}P(i) \log\left(\frac{Q(i)}{P(i)}\right) $$

To make it the symmetric I am using it as:

$$ D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2} $$

But if I want to measure the separation I could (theoretically) use

\begin{equation} \label{eq1} D_{KL}(P,Q) = \sum_{i}^NP(i) \left|\log\left(\frac{Q(i)}{P(i)}\right)\right|, \text{where} \;\ \forall i \in [0,N], Q(i) > 0 \, \text{and} \, P(i) > 0 \end{equation}

to get always positive addend for each $i$ to get the measure of disparity between distributions.

I would not need the property $d(x,y)=d(y,x)$, just would want to concentrate on avoiding negative addends. How would it look like from your perspective? Does it make sense?

  • $\begingroup$ Have you tried to prove the triangle inequality for your proposed solution to it becoming a metric? $\endgroup$
    – Jan
    Oct 11, 2018 at 10:13
  • $\begingroup$ I don't really need to make it a metric. As I will only compare the results on the same data, no absolute difference will be needed, only relative $\endgroup$
    – user223260
    Oct 11, 2018 at 10:15

1 Answer 1


Ok, I will answer to my question :) It doesn't make sense, because it already is always positive for each $i$ using the formula:

$$ D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2} $$

Here we get the following result: $$ D_{KL}(P,Q) = \sum_{i}\left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) $$

Where $$ \left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) \geq 0, \ \ \forall i$$

so each addend is non-negative (we assume $ Q(i) \ \text{and} \ P(i) > 0,\ \ \forall i $)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.