I have a question regarding to the information theory. Between clean data and noisy data, which one has higher entropy? I think the noisy data has, am I right? But, noisy data does not have more information than clean data, it just contains more noise.

  • $\begingroup$ Be careful about how "information" is defined. I think you're shifting meaning from the first time you use it (implicitly, when you equate it to entropy) to the second time, when you mean something else by information. The issue here isn't clean vs noisy data, it's simple equivocation. $\endgroup$ – Glen_b -Reinstate Monica Apr 7 '15 at 0:49

Entropy is not a measure of relevance, it is a measure of randomness. So your guess is correct.


Slide 5 of this lecture summarizes the notion of entropy in detail: http://www.cs.nyu.edu/~mohri/mls/lecture_14.pdf

The key idea is that entropy is a measure of the uncertainty of $X$. Thus, more noise increases the uncertainty and, therefore, the entropy increases.

Notice that entropy is maximal for a uniform distribution (i.e., complete noise):

$\newcommand{\E}{{\rm I\kern-.3em E}}$

$$ H(x) = \E\Bigg[\log\frac{1}{p(X)}\Bigg] \leq \log \E \Bigg[\frac{1}{p(X)}\Bigg] = \log N $$

The inequality follows from Jensen's inequality. It assumes finite support.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.