Kullback-Leibler Divergence as the class separation measure

Question

I am trying to use KL divergence as the separation measure between the classes. I have the positive, negative samples for 2 distributions and want to adjust the algorithm parameters to get the best possible separation defined by the maximum (or minimum) of the measure.

Previously I was trying with Bhattacharyya distance, but the normal distribution does not reflect the nature of my data and the results were not the best. This is why I switched to KL divergence, created histograms from the data samples and the method worked pretty nicely. The KL formula:

$$D_{KL}(P||Q) = \sum_{i}P(i) \log\left(\frac{Q(i)}{P(i)}\right) $$

To make it the symmetric I am using it as:

$$ D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2} $$

But if I want to measure the separation I could (theoretically) use

\begin{equation} \label{eq1} D_{KL}(P,Q) = \sum_{i}^NP(i) \left|\log\left(\frac{Q(i)}{P(i)}\right)\right|, \text{where} \;\ \forall i \in [0,N], Q(i) > 0 \, \text{and} \, P(i) > 0 \end{equation}

to get always positive addend for each $i$ to get the measure of disparity between distributions.

I would not need the property $d(x,y)=d(y,x)$, just would want to concentrate on avoiding negative addends. How would it look like from your perspective? Does it make sense?

Answer 1

Ok, I will answer to my question :) It doesn't make sense, because it already is always positive for each $i$ using the formula:

$$ D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2} $$

Here we get the following result: $$ D_{KL}(P,Q) = \sum_{i}\left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) $$

Where $$ \left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) \geq 0, \ \ \forall i$$

so each addend is non-negative (we assume $ Q(i) \ \text{and} \ P(i) > 0,\ \ \forall i $)