0
$\begingroup$

Let $X$ be a discrete-support stochastic variable. Information entropy is a number defined as $$H(X)=-\sum_{n} p\left(x_{n}\right) \log p\left(x_{n}\right) \geq 0$$ Let $\hat{H}$ be an estimation of $H$. Is it true that $\hat{H}$ is meaningless if $x_{n}$ are not i.i.d.? Could a random walk be used as a source?

$\endgroup$
4
  • $\begingroup$ What do you mean by "$x_n$ are not i.i.d"? In this context, $x_n$ are not samples from a distribution, they are the range of $X$. $\endgroup$
    – Mahmoud
    May 6, 2021 at 12:18
  • $\begingroup$ @mhdadk: I mean that, if $\hat{H}$ can be defined in terms of frequencies (i.e., counts) $f\left(x_{n}\right)$ which go to replace $p\left(x_{n}\right)$ in $H(X)$, then $\hat{H}$ is meaningless if $x_{n}$ are not i.i.d.. $\endgroup$
    – Mark
    May 6, 2021 at 12:22
  • $\begingroup$ Could you be more specific in your question about what $\hat{H}$ is? It could be $$\hat{H} = -\frac{1}{N} \sum_{i=1}^N \log{p(x_i)},$$ but there is no way to tell. $\endgroup$
    – Mahmoud
    May 6, 2021 at 12:24
  • $\begingroup$ Actually, I have a time series I would like to estimate Shannon's entropy about. I wonder if this series should be considered a realization of an i.i.d. process to obtain a proper entropy's estimation $\endgroup$
    – Mark
    May 6, 2021 at 12:56

2 Answers 2

1
$\begingroup$

Based on your comments, it seems that you are dealing with a random process/signal instead of a single random variable. Let this random signal be $$ \mathbf{X} = \{X_1,X_2,...,X_N\} $$ such that the set of possible values of $\mathbf{X}$ is $\mathbb{R}^N$. Since each of $X_1,X_2,...,X_N$ are continuous, then we are interested in the differential entropy of $\mathbf{X}$, otherwise known as the joint differential entropy of $X_1,X_2,...,X_N$. This is defined as $$ h(\mathbf{X}) = h(X_1,X_2,...,X_N) = -\int_{\mathbf{x} \in \mathbb{R}^N} p(\mathbf{x}) \cdot \log{p(\mathbf{x})} \ \text{d}\mathbf{x} $$ So, how well you estimate $h(\mathbf{X})$ depends heavily on how well you estimate the joint probability density function $p(\mathbf{x})$. The i.i.d assumption that you mention might be made to make it easier to estimate $p(\mathbf{x})$, but it may not necessarily be true.

Wikipedia offers a more detailed discussion of entropy estimation methods.

$\endgroup$
1
$\begingroup$

you probably refer to the asymptotic equipartition property? AEP Wikipedia $X_1,...,X_n$ have to be iid then, since $-\frac{1}{n}\log p(X_1,...,X_n)=-\frac{1}{n}\sum_i\log p(X_i)\rightarrow -E[\log p(X)] $ in probability, which is $H(X)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.