Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

Lets say i want to calculate the information content of a particular message.What apart from the message itself has to be taken into account in doing so, and what data would i need to collect to perform my action?

share|improve this question

2 Answers

up vote 2 down vote

Using notions like entropy (like in Ashok's answer) only work if you believe the message is coming from a specific distribution. If all you have a single message, then the only measure of complexity that's meaningful is the Kolmogorov complexity of the message, which is sadly uncomputable.

share|improve this answer
Suresh is right -- in classical information theory, the information content of a single message isn't defined. You can only measure information content for some ensemble (distribution) of signals. – jpillow Jul 7 '11 at 5:29
up vote 1 down vote

May be, one needs to estimate the frequency $p(m)$ of occurrence of the message $m$. Then $\log \frac{1}{p(m)}$ is the information content of the message $m$.

share|improve this answer
This is a nice answer. The quantity $h(m) = \log 1/p(m)$ can be conceived as the "surprise" of seeing message $m$ (it's a decreasing function of the probability of encountering message $m$ -- if we always observe this message then $p(m)=1$ and surprise $h=0$.). This quantity averaged over the distribution of all messages $p(m)$ is the Shannon entropy. So the surprise of seeing $m$ is the contribution of that message to the total entropy, but it's not really an "information" in the classical sense. – jpillow Jul 7 '11 at 5:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.