# Kullback-Leibler Divergence as the class separation measure

I am trying to use KL divergence as the separation measure between the classes. I have the positive, negative samples for 2 distributions and want to adjust the algorithm parameters to get the best possible separation defined by the maximum (or minimum) of the measure.

Previously I was trying with Bhattacharyya distance, but the normal distribution does not reflect the nature of my data and the results were not the best. This is why I switched to KL divergence, created histograms from the data samples and the method worked pretty nicely. The KL formula:

$$D_{KL}(P||Q) = \sum_{i}P(i) \log\left(\frac{Q(i)}{P(i)}\right)$$

To make it the symmetric I am using it as:

$$D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2}$$

But if I want to measure the separation I could (theoretically) use

$$$$\label{eq1} D_{KL}(P,Q) = \sum_{i}^NP(i) \left|\log\left(\frac{Q(i)}{P(i)}\right)\right|, \text{where} \;\ \forall i \in [0,N], Q(i) > 0 \, \text{and} \, P(i) > 0$$$$

to get always positive addend for each $$i$$ to get the measure of disparity between distributions.

I would not need the property $$d(x,y)=d(y,x)$$, just would want to concentrate on avoiding negative addends. How would it look like from your perspective? Does it make sense?

• Have you tried to prove the triangle inequality for your proposed solution to it becoming a metric?
– Jan
Commented Oct 11, 2018 at 10:13
• I don't really need to make it a metric. As I will only compare the results on the same data, no absolute difference will be needed, only relative
– user223260
Commented Oct 11, 2018 at 10:15

Ok, I will answer to my question :) It doesn't make sense, because it already is always positive for each $$i$$ using the formula:
$$D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2}$$
Here we get the following result: $$D_{KL}(P,Q) = \sum_{i}\left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right)$$
Where $$\left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) \geq 0, \ \ \forall i$$
so each addend is non-negative (we assume $$Q(i) \ \text{and} \ P(i) > 0,\ \ \forall i$$)