(Sorry, this is the repost from StackOverflow)

I have a bunch of (DNA) sequences of each with length 10. DNA sequence can be A, C, G,T-- so if the sequence at each position is completely random the Shanon entropy of each position will be 2. Indeed many positions have entropy 2, but certain positions (position 4-8) have entropy approximately 1. Shannon entropy looks like 2,2,2,1.1,1.2,1.1,1.2,0.9,0.8,2

What is the interpretation of entropy reduced to half? Does this mean that there are approximately two clusters, such that within each cluster entropy of all the positions is 2?


As you pointed out, with 4 possibilities all equally likely, the entropy is 2, which is the maximum for 4 possible outcomes. This is because entropy is defined as: $$ H(X)=-\sum_ip(x_i)\log_bp(x_i) $$ In your example, you're using base 2 for the logarithm (i.e. $b=2$), which is a common choice and leads to an answer that has units of bits. For 4 equally likely outcomes, we have $p(x_i)=1/4$ for all $i$, so substituting that into the equation we get $H(X) = -\sum_i\frac{1}{4}log_2\frac{1}{4}=-4\times\frac{1}{4}\times-2=2$.

If the possible outcomes (base pairs) aren't (a priori) equally likely for a given position in the sequence, you can get an entropy anywhere between 0-2 bits. For instance, an entropy of 1 can result from knowing that only 2 base pairs are possible at that position (e.g. only A or C), and equally likely. Those get probability $1/2$, while the others have probability $0$, which results in: $H(X)=-2\times\frac{1}{2}\times\log_2\frac{1}{2}-2\times0\times\log_20=-1\times-1=1$.

Fractional numbers are also entirely possible. For instance, if you know that only 3 base pairs are possible, one with probability $1/2$ and the other two both with probability $1/4$, you get an entropy for that position of about 0.801.

In general, numbers closer to 2 in your example indicate that the probabilities of your outcomes are close to uniform, while numbers closer to 0 mean that the probability is more concentrated (with an entropy of 0 meaning only one base pair was possible at that location to begin with, so you have 0 surprise at observing that outcome).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.