2
$\begingroup$

Can anyone suggest minimum and maximum values that the relative entropy/KL divergence can range when computed from two time-series sequences?

I have calculated the relative entropy for two time-series sequences and it gives me greater than zero values. From the literature, I came to know that relative entropy will never have less than zero value. Moreover, if the value is zero then it suggests that these two time-series sequences are identical.

$\endgroup$
2
  • $\begingroup$ What exactly is your question? Please add further context, and please incorporate your question into the body of your post. $\endgroup$ May 10, 2021 at 11:05
  • $\begingroup$ My question is how do I interpret the relative entropy values? Like I want to understand that how much one probability distribution (e.g. first time series) differs from another probability distribution (e.g. second time-series). In this case, I have applied relative entropy on these two sequences. Hope this will clear the question. $\endgroup$ May 10, 2021 at 11:14

1 Answer 1

3
$\begingroup$

The relative entropy is always nonnegative. However, it is unbounded: there is no maximum value.

As an extremely simple example, suppose we are looking at two discrete distributions with only two outcomes, where $$ P(0)=P(1)=\frac{1}{2}, \quad Q(0)=q, \quad Q(1)=1-q \quad\text{for }0<q<1. $$ Then the entropy is $$ \sum_{x\in\mathcal{X}} P(x)\log\frac{P(x)}{Q(x)} = \frac{1}{2}\bigg(\log\frac{1}{2q}+\log\frac{1}{2(1-q)}\bigg) = \frac{1}{2}\big(-2\log 2-\log q-\log(1-q)\big).$$

And $-\log q-\log(1-q)$ is unbounded for $q$ near $0$ or $1$:

entropy

R code:

qq <- seq(.001,.999,by=.001)
plot(qq,0.5*(-2*log(2)-log(qq)-log(1-qq)),type="l")
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.