Summary:
Because it represents average total number of perfect questions that you need them to get answered in order to fully resolve all ambiguities in a data that you hadn't seen yet. A perfect question with $n$ possible answers is one that, when answered, the space of possibilities will be reduced by $n$ times.
Example:
Suppose that I rolled a $6$-faced fair dice and you were to predict its outcome. The space of possibilities is $6$. You could ask me questions like this binary one "is the outcome $1$?" (answer is either yes or no, i.e. $n=2$) and my answer could be "nopies!". Then the space of possibilities by just $1$. So this question is not a good one to ask.
Alternatively, you could ask better questions, such as this superior binary question "is it greater than $3.5$?", and my answer would be "yeppies!" -- then boom, the space of possibilities is reduced down by half! I.e. there are just $6/2=3$ candidates left (out of the originally 6). Hell yeah dude.
Now suppose that you keep recursively asking more of these good questions until you reach the case when the space of possibilities has only $1$ possibility, by which -by definition- there is no ambiguity left (you know the answer).
Let's do this:
- $6$ possibilities. Q: Is outcome $> 3.5$? A: Yes.
- $6/2=3$ possibilities left. Q: is outcome $\ge 5$? A: Yes.
- $6/2/2=1.5$ possibilities left. Q: is outcome $= 6$? A: Yes.
You conclude that the outcome must be number $6$, and you only needed to ask $3$ binary questions. I.e. $ceil(\log_2(6)) = ceil(2.58) = 3$
Now, obviously, number of binary questions are always a natural number. So why doesn't Shannon's entropy use $ceil$ function? Because it actually spits out the average number of good questions that need to be asked.
If you repeat this experiment (by writing a Python code), you will notice that on average you will need to ask $2.58$ perfect binary questions.
Of course, if you ask binary questions, you set the base of the log to that. So here $\log_2(...)$ because our questions were binary. If you ask questions that expect $n$ many possible answers, you will set the base to $n$ instead of $2$, i.e. $\log_n(...)$.
Simulation:
import random
total_questions = 0
TOTAL_ROUNDS = 10000
for i in range(0,TOTAL_ROUNDS):
outcome = random.randrange(1,7)
total_questions += 1
if outcome > 3.5:
total_questions += 1
if outcome >= 5:
total_questions += 1
if outcome == 5:
pass
else:
# must be 6! no need to ask
pass
else:
# must be 4! no need to ask
pass
else:
total_questions += 1
if outcome >= 2:
total_questions += 1
if outcome == 2:
pass
else:
# must be 3! no need to ask
pass
else:
# must be 1! no need to ask
pass
print 'total questions: ' + str(total_questions)
print 'average questions per outcome: ' + str(total_questions/float(TOTAL_ROUNDS))
Results:
total questions: 26634
average questions per outcome: 2.6634
Holy molly dude $2.6634 \ne \log_2(6) \ne 2.58$.
What's wrong? It's almost close, but not really close as I hoped. Is it Python's PRNG trying to say a slow joke? Or is it Shannon being wrong? Or is it -God forbid- my understanding is wrong? Either way HELP. S.O.S. already dude.