# Do information entropy probabilities have to sum to one?

My understanding of information entropy is that it requires the input probabilities to sum to 1.

So, for a sequence a,a,b,b you then have $$- \left(\frac12 \log_2 \frac12 + \frac12 \log_2 \frac12\right) = 1$$

Are there versions of information entropy that don't require probabilities to sum to 1? Or, is there a way to measure entropy that is also sensitive to the quantity of items, not only their probability? Or, is there an accepted way to derive a form of 'non-normalised' information entropy that somehow takes into account that the longer the information stream is, the more likely you'll come across various arrangements of information?

E.g., not that this is accurate, but to convey the question: Let's say you can compute a non-normalised entropy for the same sequence a,a,b,b as such:

$$-\left(\frac12 \log_2 \frac12+\frac12 \log_2 \frac12+\frac12 \log_2 \frac12+\frac12 \log_2 \frac12\right) = 2$$

Alternately, can you sum the information content over a string of information?

• For a,a,b,b you have four items at 1 bit of surprise each, therefore 4 total bits.
• For a,a,a,a,a,a,a,a,a,b you have 10 items at 0.469 bits average surprise, therefore 4.69 total bits?
• You can invent any formula you like. Whether it means anything or is useful for any purpose is a different matter. What is this "non-normalized information entropy" intended to reveal about the data?
– whuber
Jul 25 '14 at 13:25
• I'm trying to somehow quantify (preferably with some or another accepted method) that more information is contained in larger amounts of data, even if the probability of that information being (lets say) 1 or 0 doesn't really change with the size. So if you have 10 coin tosses, then you have 10 instances of 1 bit of information, whereas 20 coin tosses is 20 instances of 1 bit. Is it perhaps so simple as just saying 10 coin tosses = 10 bits and 20 coin tosses = 20 bits? Jul 25 '14 at 13:53
• In your question, then, you need to provide a quantitative, operational definition of your meaning of "information," because evidently it differs from the standard meaning as captured by the usual entropy calculation.
– whuber
Jul 25 '14 at 13:58