# Entropy and information content

1. I am curious to know about the relation between Entropy and information content of a signal or trajectory of time series. When a system is at equilibrium, then it has maximum entropy.

Does entropy mean a measure of information loss ? Higher entropy == loss in information? In my understanding, Shannon's information theory entropy measures how uncertain we are about the next incoming symbol. Hence, Entropy = measure of uncertainty. Entropy is maximum when we are absolutely unsure of the outcome of the events. So, would this imply that Max Entropy = minimum information?

1. if we consider a trajectory in phase space and each element of the trajectory to be a symbol, then would the evolution of the trajectory mean that information is being created? How do we infer entropy for a trajectory?
• Commented Sep 2, 2017 at 20:18

Based on your phrasing, it seems you are equating thermodynamic entropy with information entropy. The concepts are related, but you have to be careful because they are used differently in the two fields.

Shannon entropy measures unpredictability. You are correct that entropy is maximum when the outcome is the most uncertain. An unbiased coin has maximum entropy (among coins), while a coin that comes up Heads with probability 0.9 has less entropy. Contrary to your next statement, however, max entropy = maximum information content.

Suppose we flip a coin 20 times. If the coin is unbiased, the sequence might look like this:

TTHTHHTTHHTHHTHTHTTH

If the coin comes up Heads with probability 0.9, it might look more like this:

HHHHHHHHHHTHHHHHHTHH

The second signal contains less information. Suppose we encode it using run length encoding, like this:

10T6T2

which we interpret as "10 heads, then 1 tail, then 6 heads, then a tail, then 2 heads". Compare this to the same encoding method applied to the first signal:

TT1T2TT2T2T1T1TT1

We can't compress the signal from the maximum entropy coin as much, because it contains more information.