# Clean data and noisy data: which one has higher entropy?

I have a question regarding to the information theory. Between clean data and noisy data, which one has higher entropy? I think the noisy data has, am I right? But, noisy data does not have more information than clean data, it just contains more noise.

• Be careful about how "information" is defined. I think you're shifting meaning from the first time you use it (implicitly, when you equate it to entropy) to the second time, when you mean something else by information. The issue here isn't clean vs noisy data, it's simple equivocation. – Glen_b -Reinstate Monica Apr 7 '15 at 0:49

Entropy is not a measure of relevance, it is a measure of randomness. So your guess is correct.

Slide 5 of this lecture summarizes the notion of entropy in detail: http://www.cs.nyu.edu/~mohri/mls/lecture_14.pdf

The key idea is that entropy is a measure of the uncertainty of $X$. Thus, more noise increases the uncertainty and, therefore, the entropy increases.

Notice that entropy is maximal for a uniform distribution (i.e., complete noise):

$\newcommand{\E}{{\rm I\kern-.3em E}}$

$$H(x) = \E\Bigg[\log\frac{1}{p(X)}\Bigg] \leq \log \E \Bigg[\frac{1}{p(X)}\Bigg] = \log N$$

The inequality follows from Jensen's inequality. It assumes finite support.