As I study information theory, I find myself increasingly perplexed by both the subject's relative obscurity and the fact that it's not studied together with probability theory.

It seems to me that the fundamental "idea" behind information theory can be summarized thus:

Given some (discrete) random variable X, we may ask ourselves what the "average probability" of the outcomes of X are. To do this properly, we need to take a geometric mean. Unfortunately, geometric means are obnoxious - conveniently for us, though, the logarithm of the geometric mean is the arithmetic mean of the logarithm.

Viewed this way, information theory really is "just" probability theory, except we've taken logarithms to turn multiplication into addition.

So, why aren't these studied together? Why is information theory generally only the purview of electrical engineering departments, rather than statistics departments? Why isn't entropy ever discussed as the logarithm of the "average probability," rather than the average of the logarithms?


First, let's be clear about terms. Information theory is a theory about the information content in messages, when generated or stored or retrieved or transmitted or received. Probability theory is a branch of pure mathematics, on which statistical methods are based. Statistics is shorthand for "statistical estimation theory" (or statistical methods, but that's not what you mean here), a theory about how unobservable parameters influence observations, and the basis for developing statistical methods. Both information theorists and statisticians use probability, with important exceptions, but their methods may differ and their applications typically do.

The main reason the two fields are separate is historical. Statistics was developed by scientists (most notably biologists) from mathematical probability over the course of a century or so, being pretty well established as a separate discipline by the 1920's. They wanted to be able to describe things that couldn't be directly observed (parameters).

Information theory was developed by computer scientists during World War II, by applying principles of thermodynamics to information and communications (messages). (This is why both use the term "entropy.") Physicists studying thermodynamics use statistical methods to describe systems containing billions of particles in probabilistic terms because they can't track individual particles; therefore, information theorists adopted the same methods for describing the long-run behavior of messages with certain attributes because they can't come up with a formula for each of the billions of possible messages.

So, the two theories started out in very different places and times, with divergent motivations, and have mostly stayed that way. It doesn't help that both use similar, non-technical terms in different ways. Most notably, the term "information" has contradictory (even opposite) definitions between the two paradigms.

Regarding Machine Learning, this is a new discipline that borrows from both theories and may be housed within statistics or engineering or mathematics or computer sciences, depending on the university. Many statisticians view ML as distinctly not statistical, because statisticians begin with theories and use data to build understandable models, whereas ML is seen (fairly or not) as using data to build algorithms without theory or understanding. (Again, note the difference between using statistical methods and applying statistical estimation theory.)

If you're also asking whether they should be a single discipline, I think so, but combining them requires more than just fixing vocabulary and syllabi. It requires the formal integration of the two underlying theories. As it is, estimation theory only recognizes one kind of information (Fisher information), while information theory allows for multiple concepts of information$-$but doesn't consider Fisher information to be actual "information." In fact, one branch of information theory, complexity theory, was designed specifically so that its concept of "algorithmic information" doesn't even require probability!

  • 1
    $\begingroup$ that was an excellent answer. thanks. $\endgroup$
    – mlofton
    Dec 7 '21 at 12:03
  • $\begingroup$ Absolutely fantastic answer; this has a lot of historical context I was missing, and validates a lot of frustrations I have had with different formalizations of the same underlying concepts. Thanks for taking the time to write this up. $\endgroup$ Dec 7 '21 at 16:48

Information theory has emerged as a means of solving engineering problems related to data compression and communication. Moreover, it is used for solving such problems to this day. It uses probability theory, but it is focused on a different kind of problems, just as probability theory uses calculus, but is not a subfield of calculus.

That said, I wouldn't agree with your premise. Information theory is commonly used in statistics in machine learning: for example, KL divergence is a common "loss" function used when training probabilistic models. There are also several statistics textbooks that cover information theory, for example Information Theory, Inference and Learning Algorithms by David J. C. MacKay.

  • $\begingroup$ How many major universities offer information theory courses from their statistics or mathematics departments, though? None of the three universities I spent time at did, and a quick browse through course curricula of a few others suggests it's really uncommon. Why? They're not so conceptually distant... $\endgroup$ May 17 '21 at 13:00
  • $\begingroup$ @user3716267 ask the universities. Information theory is used in statistics and machine learning, the question of why it is not important in the curriculums of some universities is to be asked to someone who created those curriculums. $\endgroup$
    – Tim
    May 17 '21 at 13:01
  • $\begingroup$ @user3716267. I think you make a good point and I don't know the answer. The same is sort of true with digital signal processing. The concepts of impulse response and convolution are building blocks of dsp. But, if you focus on time-series in many statistics departments ( or even econometrics departments ), you might not come across those concepts. I think what happens is that there are so many subfields of a field ( ( I'm talking about say, statistics being a field and time-series being a sub-field ) and it is hard to decide on what pieces of those sub-fields should be focused upon. $\endgroup$
    – mlofton
    May 17 '21 at 13:18
  • $\begingroup$ Also, what Tim is probably true. I bet that there are statistics departments that offer courses in information theory. $\endgroup$
    – mlofton
    May 17 '21 at 13:19
  • 1
    $\begingroup$ Digital signal processing definitely suffers from a similar fate. You get a lot of courses in EE departments that teach it (fuzzily and unrigorously), and it seems you can only find rigorous mathematical treatments of the subject if you really go hunting for them. I'm sure there do exist universities that "buck the trend," but I'm baffled as to why the trend is so strong. Is it just historical accident? $\endgroup$ May 17 '21 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.