I am trying to use KL divergence as the separation measure between the classes. I have the positive, negative samples for 2 distributions and want to adjust the algorithm parameters to get the best possible separation defined by the maximum (or minimum) of the measure.

Previously I was trying with Bhattacharyya distance, but the normal distribution does not reflect the nature of my data and the results were not the best. This is why I switched to KL divergence, created histograms from the data samples and the method worked pretty nicely. The KL formula:

$$D_{KL}(P||Q) = \sum_{i}P(i) \log\left(\frac{Q(i)}{P(i)}\right) $$

To make it the symmetric I am using it as:

$$ D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2} $$

But if I want to measure the separation I could (theoretically) use

\begin{equation} \label{eq1} D_{KL}(P,Q) = \sum_{i}^NP(i) \left|\log\left(\frac{Q(i)}{P(i)}\right)\right|, \text{where} \;\ \forall i \in [0,N], Q(i) > 0 \, \text{and} \, P(i) > 0 \end{equation}

to get always positive addend for each $i$ to get the measure of disparity between distributions.

I would not need the property $d(x,y)=d(y,x)$, just would want to concentrate on avoiding negative addends. How would it look like from your perspective? Does it make sense?

  • Have you tried to prove the triangle inequality for your proposed solution to it becoming a metric? – Jan Oct 11 at 10:13
  • I don't really need to make it a metric. As I will only compare the results on the same data, no absolute difference will be needed, only relative – Jendker Oct 11 at 10:15
up vote 0 down vote accepted

Ok, I will answer to my question :) It doesn't make sense, because it already is always positive for each $i$ using the formula:

$$ D_{KL}(P,Q) = \frac{D_{KL}(P||Q) + D_{KL}(Q||P)}{2} $$

Here we get the following result: $$ D_{KL}(P,Q) = \sum_{i}\left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) $$

Where $$ \left(Q(i)-P(i)\right)\log\left(\frac{Q(i)}{P(i)}\right) \geq 0, \ \ \forall i$$

so each addend is non-negative (we assume $ Q(i) \ \text{and} \ P(i) > 0,\ \ \forall i $)

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.