# Entropy of a block of characters

If a source provides us with a sequence chosen from 4 symbols (say A, C, G, T), then the maximum average information per symbol is 2 bits. If the source provides blocks of 3 of these symbols, then the maximum average information is 6 bits per block.

In the first case, each symbol has probability 1/4 therefore the entropy is 2 bits on average -(4*1/4*log(1/4)). The second case where we are sending blocks of 3 symbols is unclear. What is the meaning of sending block of 3 symbols? Are the symbols always different or could be the same? How do we calculate the entropy of group of symbols from a set?

We know, from the definition of joint entropy, that: $$H(X,Y,Z) = - \sum_x \sum_y \sum_z P(x,y,z) \log_2(P(x,y,z))$$ so you could calculate it directly, assuming you knew the joint distributions for $X$, $Y$, and $Z$.
Entropy is "subadditive", that is \begin{align} H(X) + H(Y) \lt H(X,Y) \hspace{0.1in} \textrm{if }& X \textrm{ and Y are dependent} \\ H(X) + H(Y) = H(X,Y) \hspace{0.1in} \textrm{if }& X \textrm{ and } Y \textrm{ are independent} \end{align} If you're willing to assume independence, the the entropy of a block is easy to calculate: it's the block length * the entropy of one element, so 3 symbols/block * 2 bits/symbol = 6 bits/block. If they are not independent or you do not know the joint distribution, then all you can say is that the entropy must be six bits or less.
The symbols could be both different and the same (any of the 64 triplets: AAA, AAC, ..., TTT). If the MAI (maximal average information) of one symbol is 2 bits, the MAI of 3 symbols is 3*2=6 bits. Another way to calculate the information is $-(64*1/64*log_2(1/64))=6$