Lets say i want to calculate the information content of a particular message.What apart from the message itself has to be taken into account in doing so, and what data would i need to collect to perform my action?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Using notions like entropy (like in Ashok's answer) only work if you believe the message is coming from a specific distribution. If all you have a single message, then the only measure of complexity that's meaningful is the Kolmogorov complexity of the message, which is sadly uncomputable.
answered Jun 7, 2011 at 4:18
-
$\begingroup$ Suresh is right -- in classical information theory, the information content of a single message isn't defined. You can only measure information content for some ensemble (distribution) of signals. $\endgroup$– jpillowCommented Jul 7, 2011 at 5:29
$\begingroup$
$\endgroup$
1
May be, one needs to estimate the frequency $p(m)$ of occurrence of the message $m$. Then $\log \frac{1}{p(m)}$ is the information content of the message $m$.
-
$\begingroup$ This is a nice answer. The quantity $h(m) = \log 1/p(m)$ can be conceived as the "surprise" of seeing message $m$ (it's a decreasing function of the probability of encountering message $m$ -- if we always observe this message then $p(m)=1$ and surprise $h=0$.). This quantity averaged over the distribution of all messages $p(m)$ is the Shannon entropy. So the surprise of seeing $m$ is the contribution of that message to the total entropy, but it's not really an "information" in the classical sense. $\endgroup$– jpillowCommented Jul 7, 2011 at 5:33