# Does downsampling decrease the entropy of the data?

Suppose we have an $$n-dim$$ time-series $$X={x_1, x_2, \cdots, x_n}$$ and we resample it to $$m-dim$$, $$\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$$, where $$m < n$$.

Can we say this downsampling operation, always decrease the entropy, $$H$$, of the data: $$H(\hat{X}) \leq H(X)$$?

and, consequently, if $$Y$$ is a latent variable inferred from the data, the mutual information will also decrease: $$I(Y,\hat{X})) \leq I(Y, X)$$

Yes, the entropy decreases because, $$\hat{X}$$ is a subset of $$X$$, and we have the general entropy rule: $$H(X)=H(\hat{X})+H(X|\hat{X})$$. This is general version of $$H(X,Y)=H(X)+H(Y|X)$$, for two RVs.
And, $$H(Y|\hat{X})\geq H(Y|X)$$ because we're conditioning on more RVs. Conditioning decreases entropy. Therefore, for the mutual information, we also have:
$$I(Y;\hat{X})=H(Y)-H(Y|\hat{X}) \leq H(Y)-H(Y|X)\leq I(Y;X)$$
• So we achieve the first equation because $H(\hat{X} | X) = 0$ ? right? – moh Sep 24 '18 at 13:02
• Yep, it's like $H(X|X,Y)$ which is $0$ since $X$ is given. – gunes Sep 24 '18 at 13:16