# Is it true that entropy estimation is meaningless if samples are not i.i.d.?

Let $$X$$ be a discrete-support stochastic variable. Information entropy is a number defined as $$H(X)=-\sum_{n} p\left(x_{n}\right) \log p\left(x_{n}\right) \geq 0$$ Let $$\hat{H}$$ be an estimation of $$H$$. Is it true that $$\hat{H}$$ is meaningless if $$x_{n}$$ are not i.i.d.? Could a random walk be used as a source?

• What do you mean by "$x_n$ are not i.i.d"? In this context, $x_n$ are not samples from a distribution, they are the range of $X$. May 6, 2021 at 12:18
• @mhdadk: I mean that, if $\hat{H}$ can be defined in terms of frequencies (i.e., counts) $f\left(x_{n}\right)$ which go to replace $p\left(x_{n}\right)$ in $H(X)$, then $\hat{H}$ is meaningless if $x_{n}$ are not i.i.d..
– Mark
May 6, 2021 at 12:22
• Could you be more specific in your question about what $\hat{H}$ is? It could be $$\hat{H} = -\frac{1}{N} \sum_{i=1}^N \log{p(x_i)},$$ but there is no way to tell. May 6, 2021 at 12:24
• Actually, I have a time series I would like to estimate Shannon's entropy about. I wonder if this series should be considered a realization of an i.i.d. process to obtain a proper entropy's estimation
– Mark
May 6, 2021 at 12:56

## 2 Answers

Based on your comments, it seems that you are dealing with a random process/signal instead of a single random variable. Let this random signal be $$\mathbf{X} = \{X_1,X_2,...,X_N\}$$ such that the set of possible values of $$\mathbf{X}$$ is $$\mathbb{R}^N$$. Since each of $$X_1,X_2,...,X_N$$ are continuous, then we are interested in the differential entropy of $$\mathbf{X}$$, otherwise known as the joint differential entropy of $$X_1,X_2,...,X_N$$. This is defined as $$h(\mathbf{X}) = h(X_1,X_2,...,X_N) = -\int_{\mathbf{x} \in \mathbb{R}^N} p(\mathbf{x}) \cdot \log{p(\mathbf{x})} \ \text{d}\mathbf{x}$$ So, how well you estimate $$h(\mathbf{X})$$ depends heavily on how well you estimate the joint probability density function $$p(\mathbf{x})$$. The i.i.d assumption that you mention might be made to make it easier to estimate $$p(\mathbf{x})$$, but it may not necessarily be true.

Wikipedia offers a more detailed discussion of entropy estimation methods.

you probably refer to the asymptotic equipartition property? AEP Wikipedia $$X_1,...,X_n$$ have to be iid then, since $$-\frac{1}{n}\log p(X_1,...,X_n)=-\frac{1}{n}\sum_i\log p(X_i)\rightarrow -E[\log p(X)]$$ in probability, which is $$H(X)$$