Why is "Jensen's Inequality" Important in Probability?

Question

Why is Jensen's Inequality Important in Probability and Statistics?

I was reading the Wikipedia page on "Convex Functions" (https://en.wikipedia.org/wiki/Convex_function), and came across the following note:

Convex functions play an important role in many areas of mathematics. They are especially important in the study of optimization problems where they are distinguished by a number of convenient properties. In probability theory, a convex function applied to the expected value of a random variable is always bounded above by the expected value of the convex function of the random variable. This result, known as Jensen's inequality, can be used to deduce inequalities such as the arithmetic–geometric mean inequality and Hölder's inequality.

I have been trying to spend some more time understanding the above statement and trying to understand its relevance and importance.

From a probability theory perspective, "a convex function applied to the expected value of a random variable is always bounded above by the expected value of the convex function of the random variable."

I imagine that they are referring to a convex function such as "mean squared error" , i.e. Expected_Value((Y - Y*)^2) , where Y is a random variable being modelled by a machine learning model, and Y* is the prediction made by the machine learning model?

My Question: Can anyone explain why the fact that a convex function being bounded by the expected value of a random variable is of importance? To play devil's advocate: Suppose if Jensen's Inequality was not true - how would this complicate things in Probability and Statistics?

Thanks!

Answer 1

Much of probability and statistics is about deriving inequalities: proving that a random variable falls within an interval with specified probability, showing that one estimator is less biased than another, etc.

The two particular examples of using the Jensen inequality that immediately come to lind are:

• Non-negativity of Kullback-Leibler divergence, which is the basis if many information-theoretical considerations. E.g., Akaike information criterion for model selection is obtained by minumizing the KL divergence.
• Expectation-Maximization algorithm, which reduces likelihood maximization to a simpler problem, showing that true likelihood is guranteed to be greater than the one maximized (although one often sees the proofs that are based not directly on Jensen inequality, but on other equivalent inequalities, such as the above mentioned KL divergence.)