Why is Jensen's Inequality Important in Probability and Statistics?
I was reading the Wikipedia page on "Convex Functions" (https://en.wikipedia.org/wiki/Convex_function), and came across the following note:
Convex functions play an important role in many areas of mathematics. They are especially important in the study of optimization problems where they are distinguished by a number of convenient properties. In probability theory, a convex function applied to the expected value of a random variable is always bounded above by the expected value of the convex function of the random variable. This result, known as Jensen's inequality, can be used to deduce inequalities such as the arithmetic–geometric mean inequality and Hölder's inequality.
I have been trying to spend some more time understanding the above statement and trying to understand its relevance and importance.
From a probability theory perspective, "a convex function applied to the expected value of a random variable is always bounded above by the expected value of the convex function of the random variable."
I imagine that they are referring to a convex function such as "mean squared error" , i.e. Expected_Value((Y - Y*)^2) , where Y is a random variable being modelled by a machine learning model, and Y* is the prediction made by the machine learning model?
My Question: Can anyone explain why the fact that a convex function being bounded by the expected value of a random variable is of importance? To play devil's advocate: Suppose if Jensen's Inequality was not true - how would this complicate things in Probability and Statistics?