I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully).
Questions:
- Q1: how is my definition of information different than that of Claude Shannon?
- Q2: following my thread of thought (below), where did I go wrong?
My thread of thought
I think there are many different ways to look at entropy, but among the tiny subset of views I can think of, I prefer this view
- The entropy of a thing is a measure of information that the thing contains.
- Information of a thing is measured by total number of perfect questions that, if got answered, all ambiguity about the thing is resolved (i.e. the thing becomes fully known).
Let's see: You have a single 6-faced die that you have tossed. You have measured the outcome of this trusty fair die toss. I -on the other hand- don't know the outcome of your die toss. All what you have told me is this:
- This single die toss is independent of any other die toss you might have done in the past.
- Each die outcome $i$ has probability $\Pr(I=i)$.
Now, suppose that you tell me the outcome of the trusty die toss. Say, the outcome is 5. How much information did you give me by telling me "outcome is 5"?
The answer is (in my view): the quantity of information there is the quantity of perfect $n$-nary questions that I had to ask in order to know that know 5.
One could ask a single question "what is the number?", which is one hell of an $n$-nary question where $n$ is very very big (space of answers).
Alternatively one can ask many questions of the form "is the number greater than $x$?" which is a 2-nary (binary) question. Of course we will need to ask more of those questions.
A set of $n$-nary questions are said to be "perfect" if their numbers are minimum. Turns out that such questions are those that divide the space of answers in half at each time.
Also comparing information based on counts of questions of differing values of $n$ is not very meaningful (answers to questions of higher $n$ tell more about a thing than those of lower $n$). So we gotta stick $n$. Let's say that $n=2$.
Let's say the outcome was $1$; then these are the minimum/perfect questions that I need to ask to know the outcome:
- Is outcome greater than 3.5? Answer will be "no".
- Is outcome greater than 1? Answer will be "no"
Boom, it must be 1. I found it in 2 questions.
Now let's say that the outcome was 3:
- Is outcome greater than 3.5? Answer will be "no".
- Is outcome greater than 1? Answer will be "yes".
- Is outcome greater than 2? Answer will be "yes".
Boom, it must be 3. I found it in 3 questions.
It seems that outcomes 1, 4 need two questions, and others (2,3,5,6) need 3 questions.
Since all outcomes are equally likely to happen, expected number of questions is $\frac{2+2+3+3+3+3}{6}= 2.666666\ldots 6667$
Now what if we asked a different kind of perfect binary questions. Say that outcome is actually 1 and we ask:
- Is outcome greater than 3.5? Answer "no".
- Is outcome greater than 2? Answer is "no".
- Is outcome equal to 2? Answer is "no".
Boom, must be 1 then.
Let's say that actual outcome is 3:
- Is outcome greater than 3.5? Answer "no".
- Is outcome greater than 2? Answer is "yes".
Boom, it must be 3 then. In two questions.
I think with this kind of questions, outcomes 3 and 6 need 2 questions, while others (1,2,4,5) need 3. So average number of questions here is $\frac{2+2+3+3+3+3}{6}= 2.666666\ldots 6667$ as well.
But what about the grand average regardless of what kind of perfect questions do we ask? $\frac{(2+2+3+3+3+3)+(2+2+3+3+3+3)}{6+6}= 2.666666\ldots 6667$ obviously the same.
So I think that information contained in a single toss of a fair 6-faced die is 2.666667.
That is if we define information by total number of perfect $n$-nary questions that we need to ask.
But apparently Claude Shannon disagrees. He says: \begin{equation}\begin{split} \text{entropy of that fair die} &= \sum_{c=1}^6 \frac{1}{6}\log_2(\frac{6}{1})\\ &= \frac{1}{6} \times 6 \times \log_2(\frac{6}{1})\\ &= \log_2(\frac{6}{1})\\ &= 2.58492\\ \end{split}\end{equation}
