Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$.

Can we say this downsampling operation, always decrease the entropy, $H$, of the data: $H(\hat{X}) \leq H(X)$?

and, consequently, if $Y$ is a latent variable inferred from the data, the mutual information will also decrease: $I(Y,\hat{X})) \leq I(Y, X)$


Yes, the entropy decreases because, $\hat{X}$ is a subset of $X$, and we have the general entropy rule: $H(X)=H(\hat{X})+H(X|\hat{X})$. This is general version of $H(X,Y)=H(X)+H(Y|X)$, for two RVs.

And, $H(Y|\hat{X})\geq H(Y|X)$ because we're conditioning on more RVs. Conditioning decreases entropy. Therefore, for the mutual information, we also have:

$$I(Y;\hat{X})=H(Y)-H(Y|\hat{X}) \leq H(Y)-H(Y|X)\leq I(Y;X)$$

  • $\begingroup$ So we achieve the first equation because $H(\hat{X} | X) = 0$ ? right? $\endgroup$ – moh Sep 24 '18 at 13:02
  • $\begingroup$ Yep, it's like $H(X|X,Y)$ which is $0$ since $X$ is given. $\endgroup$ – gunes Sep 24 '18 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.