In the world of physics, entropy seems to mean something different from stats + information theory world.

So, I assumed that there are two definitions to the word entropy. Indeed, I looked up how Shannon came up with the word and this was the story.

Shannon approached the great man with his idea of information-as-resolved uncertainty — which would come to stand at the heart of his work — and with an unassuming question. What should he call this thing? [John] Von Neumann answered at once: “say that information reduces ‘entropy.’ For one it is a good, solid physics word. And more importantly,” he went on, “no one knows what entropy really is, so in a debate you will always have the advantage.”

Is there any connection between the information theory "entropy" and the physics one? Are they two totally different things?


1 Answer 1


Jaynes demonstrated in 1957* that the interpretation of thermodynamic entropy in statistical terms, developed by Boltzmann and Gibbs in the late 19th century, is a specific case of the Shannon entropy defined almost a century later.

The initial definition of entropy had nothing (consciously) to do with statistics. As the first law of thermodynamics (conservation of energy) was being developed in the 19th century there was a problem of how to understand it in terms of the loss of useful energy via heat in processes like friction.

Clausius introduced the concept of entropy to solve that problem, a new type of macroscopic state variable akin to temperature or pressure. If a system goes through a cycle of changes and ends up at its initial state then its entropy, as a function of system state, has not changed. If any of the steps in the cycle had been irreversible, however, then the entropy of the rest of the world with which the system interacted has increased. That formed the basis of the second law of thermodynamics. Gibbs showed how to put the first and second laws of thermodynamics together by incorporating entropy along with other state variables as a measure of a system's macroscopic internal energy.

Boltzmann, applying the then-controversial (among physicists) concept of the atomic theory of matter to the macroscopic behavior of an ideal gas in an isolated container, showed that this macroscopic entropy would be proportional in that case to the logarithm of the number of microstates the individual atoms could occupy in terms of position and momentum. Gibbs greatly developed this interpretation of macroscopic phenomena in terms of probability distributions among microstates, forming the foundation of statistical mechanics.

A more general relationship between microstate probabilities $p_i$ and macroscopic entropy (usually denoted $S$), called the Gibbs entropy formula, is written in terms similar to Shannon entropy:

$$ S = - k_B \sum_i p_i \ln p_i,$$

in which $k_B$ is a physical constant, the Boltzmann constant, that relates the kinetic energy of a gas to its temperaure. Boltzmann's initial microscopic interpretation of macroscopic entropy then represents the situation in which all microstates are equally probable--the maximum entropy state, in the Shannon sense.

This late-19th-century work, however, needed to make some important assumptions as noted by Jaynes. These had to do with the classical understanding of mechanics underlying that work and the consequent necessity of working with continuous distributions, which pose problems in terms of defining Shannon entropy. The later development of quantum mechanics demonstrated that nature is fundamentally discrete, simplifying matters substantially.

Consistent with the current question, Jaynes acknowledged (p. 621):

The mere fact that the same mathematical expression $—\sum p_i \log p_i$ occurs both in statistical mechanics and in information theory does not in itself establish any connection between these fields. This can be done only by finding new viewpoints from which thermodynamic entropy and information-theory entropy appear as the same concept.

Jaynes went on to note (page 623):

in making inferences on the basis of partial information we must use that probability distribution which has maximum entropy subject to whatever is known. This is the only unbiased assignment we can make...

and then showed that the microscopic statistical-mechanical interpretation of the original macroscopic concept of entropy (and of other standard thermodynamic properties) could be seen as coming from the maximum-entropy solution (in the Shannon sense) to a macroscopic function of microstates about which there is no further information.

*E. T. Jaynes, Information Theory and Statistical Mechanics, Phys.Rev. 106, 620 (1957)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.