# Interpretation of the entropy with a coding length?

There is this interpretation of the entropy $-\sum_i p_i \log_2 p_i$ as the average length (in bits) per character when using an optimal encoding of a message.

Now, if we use the simple 3-letter case of A (80%), B (10%) and C(10%), the entropy is smaller than 1 bit, which seems strange (from a naive standpoint). How can this be reconciled with the interpretation above (if it can at all)?

• Arithmetic coding can use less than one bit per symbol (of course you still have to round up to the nearest bit for the whole message) – user253751 May 31 '16 at 23:10

AAA as 0