12
$\begingroup$

What is a Moment Generating Function (MGF)?

Can you explain it in layman's terms and along with a simple & easy example?

Please, limit using formal math notations as far as possible.

$\endgroup$
  • 1
    $\begingroup$ You want a simple, easy example ... but without mathematical notation? I'm not sure such a thing would be very easy to do - at least not without risking giving a misleading impression of what you're dealing with. I suppose one could give the mgf of a degenerate random variable that's always $0$ without needing much in the way of mathematical notation, but it's going to be unenlightening if you really want to understand mgfs. $\endgroup$ – Glen_b Oct 6 '16 at 14:58
  • 1
    $\begingroup$ I'm not sure if there's an intuitive way of understanding it, you might just think of it as a way of "encoding" a distribution (at least when it exists, this idea works a little bit better with characteristic functions). $\endgroup$ – dsaxton Oct 6 '16 at 15:01
  • 1
    $\begingroup$ A moment generating function - when it exists - is a way of encoding all the non-negative-integer moments of a random variable into a function, and from which they can be extracted again; mgfs can be used to do particular calculations that are sometimes not so easy to do in other ways. I don't expect that's much help. $\endgroup$ – Glen_b Oct 6 '16 at 15:06
  • 1
    $\begingroup$ I'm sure you have seen Joe Blitztein answer to the identical question on Quora $\endgroup$ – Antoni Parellada Oct 6 '16 at 17:51
17
$\begingroup$

Let's assume that an equation-free intuition is not possible, and still insist on boiling down the math to the very essentials to get an idea of what's going on: we are trying to obtain the statistical moments, which, after the obligatory reference to physics, we define as the expected value of a power of a random variable. For a continuous random variable, the raw $k$-th moment is by LOTUS:

\begin{align}\large \color{red}{\mathbb{E}\left[{X^k}\right]} &= \displaystyle\int_{-\infty}^{\infty}\color{blue}{X^k}\,\,\color{green}{\text{pdf}}\,\,\,dx\tag{1}\end{align}

The moment generating function, $M_X(t)$, is a way to walk around this integral by, instead, carrying out:

\begin{align} \large \color{blue}{\mathbb{E}\left[e^{\,tX}\right]}&=\displaystyle \int_{-\infty}^{\infty}\color{blue}{e^{tX}}\,\color{green}{\text{pdf}}\, dx\tag{2}\end{align}

Why? Because it's easier and there is a fantastic property of the MGF that can be seen by expanding the Maclaurin series of $\color{blue}{e^{\,tX}}$

$$e^{tX}=1+\frac{ X }{1!}\, t +\frac{ X^{2} }{2!}t^{2} +\frac{ X^{3} }{3!} t^{3} +\cdots$$

Taking the expectation of both sides of this power series:

$$\begin{align} M_X(t) &= \color{blue}{\mathbb{E}\left[e^{\,tX}\right]} \\[1.5ex] &=1 + \frac{\color{red}{\mathbb{E} \left[X\right]}}{1!} \, t \, + \frac{\color{red}{\mathbb{E} \left[X^2\right]}}{2!} \, t^2 \, + \frac{\color{red}{\mathbb{E} \left[X^3\right]}}{3!} \, t^3 \, + \cdots\tag{3} \end{align}$$

the moments appear "perched" on this polynomial "clothesline", ready to be culled by simply differentiating $k$ times and evaluating at zero once we go through the easier integration (in eq. (2)) just once for all moments! The fact that it is an easier integration is most apparent when the pdf is an exponential.

To recover the $k$-th moment:

$$M_X^{(k)}(0)=\frac{d^k}{dt^k}M_X(t)\Bigr|_{t=0}$$

The fact that eventually there is a need to differentiate makes it a not a free lunch - in the end it is a Laplacian transform.

This, in effect, gives us a physics avenue to the intuition. The Laplace transform is acting on the $\color{green}{\text{pdf}}$ and decomposing it into moments. The similarity to a Fourier transform is inescapable: a FT maps a function to a new function on the real line, and Laplace maps a function to a new function on the complex plane. The Fourier transform expresses a function or signal as a series of frequencies, while the Laplace transform resolves a function into its moments. In fact, a different way of obtaining moments is through a Fourier transform (characteristic function). The exponential term in the Laplace transform is in general of the form $e^{-st}$ with $s=\sigma + i\,\omega$, corresponding to the real exponentials and imaginary sinusoidals, and yielding plots such as this:


[From The Scientist and Engineer's Guide to Signal Processing by Steven W. Smith]


So we may be allowed to imagine that the $M_X(t)$ function decomposes the $\text{pdf}$ somehow into its "constituent frequencies" as spelled out in Eq $(3)$?


In response to the question under comments about the switching from $X^k$ to $e^{tx}$, this is a completely strategic move: one expression does not follow from the other. Here is an analogy: We have a car of our own and we are free to drive into the city every time we need to take care of some business (read, integrating Eq $(1)$ no matter how tough for every separate, single moment). Instead, we can do something completely different: we can drive to the nearest subway station (read, solve Eq $(2)$ just once), and from there use public transportation to reach every single place we need to visit (read, get any $k$ derivative of the integral in Eq $(2)$ to extract whichever $k$-th moment we need, knowing (thanks to Eq $(3)$) that all the moments are "hiding" in there and isolated by evaluating at $0$).

$\endgroup$
  • 2
    $\begingroup$ How does $E[e^{tX}]$ replace $E[X^k]$? (Out of the blue?) $\endgroup$ – user366312 Oct 7 '16 at 9:48
  • 2
    $\begingroup$ I wish the laymen who understand this answer were my students :) $\endgroup$ – Aksakal Jan 25 at 20:30
3
$\begingroup$

In the most layman terms it's a way to encode all characteristics of the probability distribution into one short phrase. For instance, if I know that MGF of the distribution is $$M(t)=e^{t\mu+1/2\sigma^2t^2}$$ I can find out the mean of this distribution by taking first term of Taylor expansion: $$\frac d {dt}M(t)|_{t=0}=\mu+\sigma^2t|_{t=0}=\mu$$ If you know what you're doing it's much faster than taking the expectation of the probability function.

Moreover, since this MGF encodes everything about the distribution, if you know how to manipulate the function, you can apply operations on all characteristics of the distribution at once! Why don't we always use MGF? First, it's not in every situation the MGF is the easiest tool. Second, MGF doesn't always exist.

Above layman

Suppose you have a standard normal distribution. You can express everything you know about it by stating its PDF: $$f(x)=\frac 1 {\sqrt{2\pi}}e^{-x^2/2}$$

You can calculate its moment such as mean and standard deviation, and use it on transformed variables and functions on random normals etc.

You can think of the MGF of normal distribution as an alternative to PDF. It contains the same amount of information. I already showed how to obtain the mean.

Why do we need an alternative way? As I wrote, sometimes it's just more convenient. For instance, try calculating the variance of the standard normal from PDF: $$\sigma^2=\int_{-\infty}^\infty x^2\frac 1 {\sqrt{2\pi}}e^{-x^2/2} dx=?$$ It's not that difficult, but it's much easier to do it with MGF $M(t)=e^{t^2/2}$: $$\sigma^2=\frac {d^2} {dt^2}M(t)|_{t=0}=\frac d {dt} t |_{t=0}=1$$

$\endgroup$
  • 1
    $\begingroup$ Can you please expand on the "everything" that it encodes about the distribution? $\endgroup$ – ColorStatistics Jan 25 at 18:53
  • 2
    $\begingroup$ To appreciate the point made by @ColorStatistics, please see stats.stackexchange.com/questions/25010. $\endgroup$ – whuber Jan 25 at 20:27
  • $\begingroup$ @whuber: Thank you, whuber. I'll study that reference. This is a topic I am looking to understand better. $\endgroup$ – ColorStatistics Jan 25 at 21:15
  • $\begingroup$ How can we prove that MGF & PDF contain the same amount of information? $\endgroup$ – Aaron Sep 16 at 4:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.