Why is information theory studied separately from probability theory? As I study information theory, I find myself increasingly perplexed by both the subject's relative obscurity and the fact that it's not studied together with probability theory.
It seems to me that the fundamental "idea" behind information theory can be summarized thus:
Given some (discrete) random variable X, we may ask ourselves what the "average probability" of the outcomes of X are.  To do this properly, we need to take a geometric mean.  Unfortunately, geometric means are obnoxious - conveniently for us, though, the logarithm of the geometric mean is the arithmetic mean of the logarithm.
Viewed this way, information theory really is "just" probability theory, except we've taken logarithms to turn multiplication into addition.
So, why aren't these studied together?  Why is information theory generally only the purview of electrical engineering departments, rather than statistics departments?  Why isn't entropy ever discussed as the logarithm of the "average probability," rather than the average of the logarithms?
 A: Information theory has emerged as a means of solving engineering problems related to data compression and communication. Moreover, it is used for solving such problems to this day. It uses probability theory, but it is focused on a different kind of problems, just as probability theory uses calculus, but is not a subfield of calculus.
That said, I wouldn't agree with your premise. Information theory is commonly used in statistics in machine learning: for example, KL divergence is a common "loss" function used when training probabilistic models. There are also several statistics textbooks that cover information theory, for example Information Theory, Inference and Learning Algorithms by David J. C. MacKay.
A: First, let's be clear about terms. Information theory is a theory about the information content in messages, when generated or stored or retrieved or transmitted or received. Probability theory is a branch of pure mathematics, on which statistical methods are based. Statistics is shorthand for "statistical estimation theory" (or statistical methods, but that's not what you mean here), a theory about how unobservable parameters influence observations, and the basis for developing statistical methods. Both information theorists and statisticians use probability, with important exceptions, but their methods may differ and their applications typically do.
The main reason the two fields are separate is historical. Statistics was developed by scientists (most notably biologists) from mathematical probability over the course of a century or so, being pretty well established as a separate discipline by the 1920's. They wanted to be able to describe things that couldn't be directly observed (parameters).
Information theory was developed by computer scientists during World War II, by applying principles of thermodynamics to information and communications (messages). (This is why both use the term "entropy.") Physicists studying thermodynamics use statistical methods to describe systems containing billions of particles in probabilistic terms because they can't track individual particles; therefore, information theorists adopted the same methods for describing the long-run behavior of messages with certain attributes because they can't come up with a formula for each of the billions of possible messages.
So, the two theories started out in very different places and times, with divergent motivations, and have mostly stayed that way. It doesn't help that both use similar, non-technical terms in different ways. Most notably, the term "information" has contradictory (even opposite) definitions between the two paradigms.
Regarding Machine Learning, this is a new discipline that borrows from both theories and may be housed within statistics or engineering or mathematics or computer sciences, depending on the university. Many statisticians view ML as distinctly not statistical, because statisticians begin with theories and use data to build understandable models, whereas ML is seen (fairly or not) as using data to build algorithms without theory or understanding. (Again, note the difference between using statistical methods and applying statistical estimation theory.)
If you're also asking whether they should be a single discipline, I think so, but combining them requires more than just fixing vocabulary and syllabi. It requires the formal integration of the two underlying theories. As it is, estimation theory only recognizes one kind of information (Fisher information), while information theory allows for multiple concepts of information$-$but doesn't consider Fisher information to be actual "information." In fact, one branch of information theory, complexity theory, was designed specifically so that its concept of "algorithmic information" doesn't even require probability!
