Let $X$ be a discrete-support stochastic variable. Information entropy is a number defined as $$H(X)=-\sum_{n} p\left(x_{n}\right) \log p\left(x_{n}\right) \geq 0$$ Let $\hat{H}$ be an estimation of $H$. Is it true that $\hat{H}$ is meaningless if $x_{n}$ are not i.i.d.? Could a random walk be used as a source?
4
-
$\begingroup$ What do you mean by "$x_n$ are not i.i.d"? In this context, $x_n$ are not samples from a distribution, they are the range of $X$. $\endgroup$– mhdadkCommented May 6, 2021 at 12:18
-
$\begingroup$ @mhdadk: I mean that, if $\hat{H}$ can be defined in terms of frequencies (i.e., counts) $f\left(x_{n}\right)$ which go to replace $p\left(x_{n}\right)$ in $H(X)$, then $\hat{H}$ is meaningless if $x_{n}$ are not i.i.d.. $\endgroup$– MarkCommented May 6, 2021 at 12:22
-
$\begingroup$ Could you be more specific in your question about what $\hat{H}$ is? It could be $$\hat{H} = -\frac{1}{N} \sum_{i=1}^N \log{p(x_i)},$$ but there is no way to tell. $\endgroup$– mhdadkCommented May 6, 2021 at 12:24
-
$\begingroup$ Actually, I have a time series I would like to estimate Shannon's entropy about. I wonder if this series should be considered a realization of an i.i.d. process to obtain a proper entropy's estimation $\endgroup$– MarkCommented May 6, 2021 at 12:56
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Based on your comments, it seems that you are dealing with a random process/signal instead of a single random variable. Let this random signal be $$ \mathbf{X} = \{X_1,X_2,...,X_N\} $$ such that the set of possible values of $\mathbf{X}$ is $\mathbb{R}^N$. Since each of $X_1,X_2,...,X_N$ are continuous, then we are interested in the differential entropy of $\mathbf{X}$, otherwise known as the joint differential entropy of $X_1,X_2,...,X_N$. This is defined as $$ h(\mathbf{X}) = h(X_1,X_2,...,X_N) = -\int_{\mathbf{x} \in \mathbb{R}^N} p(\mathbf{x}) \cdot \log{p(\mathbf{x})} \ \text{d}\mathbf{x} $$ So, how well you estimate $h(\mathbf{X})$ depends heavily on how well you estimate the joint probability density function $p(\mathbf{x})$. The i.i.d assumption that you mention might be made to make it easier to estimate $p(\mathbf{x})$, but it may not necessarily be true.
Wikipedia offers a more detailed discussion of entropy estimation methods.
$\begingroup$
$\endgroup$
you probably refer to the asymptotic equipartition property? AEP Wikipedia $X_1,...,X_n$ have to be iid then, since $-\frac{1}{n}\log p(X_1,...,X_n)=-\frac{1}{n}\sum_i\log p(X_i)\rightarrow -E[\log p(X)] $ in probability, which is $H(X)$
