Question: Is the concept of entropy rate defined for real-valued random processes? If so, does it have the same interpretation as in the discrete case; and how can it be estimated?

Background: As far as I know, the entropy rate of a random process is defined as:

$$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) \,.$$

In conventional, finite-alphabet information theory the entropy rate of a process quantifies how many bits this process generates per realization. That is different from the entropy of the realizations themselves $H(X_i)$.

However, if the $X_i$ are continuous variables the notion of entropy $H(X_i)$ doesn't have the same interpretation as in the discrete case, since it isn't non-negative and is not invariant under change of variables.


You can compute the differential entropy rate of a continuous process, but like differential entropy lacks the interpretability of entropy, it will not be as meaningful as entropy rate.

To estimate the (differential) entropy rate, you can you exploit the fact that for a stationary process

$$\lim_{n \rightarrow \infty} \frac{1}{n} H[X_1, ..., X_n] = \lim_{n \rightarrow \infty} H[X_n \mid X_{n - 1}..., X_1].$$

This allows you to estimate the entropy rate by estimating the entropy of a conditional distribution. In practice, you will likely also make a Markov assumption and only take into account the last $m$ steps. Using an estimate of the conditional probability, $\hat P(x_{n} \mid x_{n - 1}, ..., x_{n - m})$, approximate the conditional entropy with the cross-entropy $$E_P[-\log \hat P(X_{n} \mid X_{n - 1}, ..., X_{n - m})].$$ This approximation is guaranteed to be an upper bound on the true entropy rate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.