2
$\begingroup$

I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully).

Questions:

  • Q1: how is my definition of information different than that of Claude Shannon?
  • Q2: following my thread of thought (below), where did I go wrong?

My thread of thought

I think there are many different ways to look at entropy, but among the tiny subset of views I can think of, I prefer this view

  • The entropy of a thing is a measure of information that the thing contains.
  • Information of a thing is measured by total number of perfect questions that, if got answered, all ambiguity about the thing is resolved (i.e. the thing becomes fully known).

Let's see: You have a single 6-faced die that you have tossed. You have measured the outcome of this trusty fair die toss. I -on the other hand- don't know the outcome of your die toss. All what you have told me is this:

  • This single die toss is independent of any other die toss you might have done in the past.
  • Each die outcome $i$ has probability $\Pr(I=i)$.

Now, suppose that you tell me the outcome of the trusty die toss. Say, the outcome is 5. How much information did you give me by telling me "outcome is 5"?

The answer is (in my view): the quantity of information there is the quantity of perfect $n$-nary questions that I had to ask in order to know that know 5.

One could ask a single question "what is the number?", which is one hell of an $n$-nary question where $n$ is very very big (space of answers).

Alternatively one can ask many questions of the form "is the number greater than $x$?" which is a 2-nary (binary) question. Of course we will need to ask more of those questions.

A set of $n$-nary questions are said to be "perfect" if their numbers are minimum. Turns out that such questions are those that divide the space of answers in half at each time.

Also comparing information based on counts of questions of differing values of $n$ is not very meaningful (answers to questions of higher $n$ tell more about a thing than those of lower $n$). So we gotta stick $n$. Let's say that $n=2$.

Let's say the outcome was $1$; then these are the minimum/perfect questions that I need to ask to know the outcome:

  • Is outcome greater than 3.5? Answer will be "no".
  • Is outcome greater than 1? Answer will be "no"

Boom, it must be 1. I found it in 2 questions.

Now let's say that the outcome was 3:

  • Is outcome greater than 3.5? Answer will be "no".
  • Is outcome greater than 1? Answer will be "yes".
  • Is outcome greater than 2? Answer will be "yes".

Boom, it must be 3. I found it in 3 questions.

It seems that outcomes 1, 4 need two questions, and others (2,3,5,6) need 3 questions.

Since all outcomes are equally likely to happen, expected number of questions is $\frac{2+2+3+3+3+3}{6}= 2.666666\ldots 6667$

Now what if we asked a different kind of perfect binary questions. Say that outcome is actually 1 and we ask:

  • Is outcome greater than 3.5? Answer "no".
  • Is outcome greater than 2? Answer is "no".
  • Is outcome equal to 2? Answer is "no".

Boom, must be 1 then.

Let's say that actual outcome is 3:

  • Is outcome greater than 3.5? Answer "no".
  • Is outcome greater than 2? Answer is "yes".

Boom, it must be 3 then. In two questions.

I think with this kind of questions, outcomes 3 and 6 need 2 questions, while others (1,2,4,5) need 3. So average number of questions here is $\frac{2+2+3+3+3+3}{6}= 2.666666\ldots 6667$ as well.

But what about the grand average regardless of what kind of perfect questions do we ask? $\frac{(2+2+3+3+3+3)+(2+2+3+3+3+3)}{6+6}= 2.666666\ldots 6667$ obviously the same.

So I think that information contained in a single toss of a fair 6-faced die is 2.666667.

That is if we define information by total number of perfect $n$-nary questions that we need to ask.

But apparently Claude Shannon disagrees. He says: \begin{equation}\begin{split} \text{entropy of that fair die} &= \sum_{c=1}^6 \frac{1}{6}\log_2(\frac{6}{1})\\ &= \frac{1}{6} \times 6 \times \log_2(\frac{6}{1})\\ &= \log_2(\frac{6}{1})\\ &= 2.58492\\ \end{split}\end{equation}

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ As you discovered in your previous answer, your definition is different. So what are really trying to ask here? $\endgroup$ – whuber♦ Apr 8 '16 at 17:39
  • 2
    $\begingroup$ Here's a thought. Changing the base of a logarithm is just a rescaling of its value. For instance, using logarithms base $6$ is tantamount to multiplying all binary logarithms by $\log_6(2)\approx 0.387$. For any definition to have intrinsic meaning its values should not depend on the base used and should scale in exactly the same way. But if you could ask questions that have six, rather than two, options, then you would need exactly one question to determine the face of a die. Thus the log-6 "entropy" by your definition is $1$. Converted to binary that's $\log_2(6)$. $\endgroup$ – whuber♦ Apr 8 '16 at 19:27
  • 1
    $\begingroup$ The point of that previous thought is that your analysis of "entropy" depends fundamentally on which logarithmic base you choose to use. That arbitrariness shows that although you might be capturing something of interest about the combination of the nature of the questions (that is, binary instead of 6-ary) and the distribution, you are not expressing some intrinsic property of the distribution itself. $\endgroup$ – whuber♦ Apr 8 '16 at 19:30
  • 1
    $\begingroup$ No, I'm saying that Shannon's definition does have intrinsic meaning and yours obviously does not. His definition does not depend on the base, because the answer transforms appropriately. Consider this analogy: if you invent a thermometer that shows both degrees F and degrees C, but it gives a different conversion between F and C depending on what it's measuring, then it's a pretty weird thermometer, isn't it? Shannon's entropy converts consistently between different bases, no matter what the distribution may be, but yours does not. $\endgroup$ – whuber♦ Apr 8 '16 at 19:44
  • 1
    $\begingroup$ "Intrinsic" means it's a property of the distribution. We have no problems using multiple temperature measurement systems because there's a universal conversion between them; thus, what a thermometer measures is a property of the physical system it is measuring. We have no problems with Shannon's entropy for exactly the same reasons. Your calculation is problematic because it does not behave like this. Your calculation is describing the relationship between some system of asking questions and a distribution, but it is not giving clear information about the distribution itself. $\endgroup$ – whuber♦ Apr 8 '16 at 20:00
2
$\begingroup$

Your definition is different in that you are asking questions about a single dice throw as opposed to the limit. You can ask more informative questions about multiple throws. I believe your definition is what Huffman coding is finding a code for - the minimal expected length of the optimal compression for each symbol separately - so you might want to look at examples for inefficiency in that approach.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.