22
$\begingroup$

What is a Moment Generating Function (MGF)?

Can you explain it in layman's terms and along with a simple & easy example?

Please, limit using formal math notations as far as possible.

$\endgroup$
4
  • 1
    $\begingroup$ You want a simple, easy example ... but without mathematical notation? I'm not sure such a thing would be very easy to do - at least not without risking giving a misleading impression of what you're dealing with. I suppose one could give the mgf of a degenerate random variable that's always $0$ without needing much in the way of mathematical notation, but it's going to be unenlightening if you really want to understand mgfs. $\endgroup$
    – Glen_b
    Oct 6 '16 at 14:58
  • 1
    $\begingroup$ I'm not sure if there's an intuitive way of understanding it, you might just think of it as a way of "encoding" a distribution (at least when it exists, this idea works a little bit better with characteristic functions). $\endgroup$
    – dsaxton
    Oct 6 '16 at 15:01
  • 2
    $\begingroup$ A moment generating function - when it exists - is a way of encoding all the non-negative-integer moments of a random variable into a function, and from which they can be extracted again; mgfs can be used to do particular calculations that are sometimes not so easy to do in other ways. I don't expect that's much help. $\endgroup$
    – Glen_b
    Oct 6 '16 at 15:06
  • 2
    $\begingroup$ I'm sure you have seen Joe Blitztein answer to the identical question on Quora $\endgroup$ Oct 6 '16 at 17:51
31
$\begingroup$

Let's assume that an equation-free intuition is not possible, and still insist on boiling down the math to the very essentials to get an idea of what's going on: we are trying to obtain the statistical moments, which, after the obligatory reference to physics, we define as the expected value of a power of a random variable. For a continuous random variable, the raw $k$-th moment is by LOTUS:

\begin{align}\large \color{red}{\mathbb{E}\left[{X^k}\right]} &= \displaystyle\int_{-\infty}^{\infty}\color{blue}{X^k}\,\,\color{green}{\text{pdf}}\,\,\,dx\tag{1}\end{align}

The moment generating function, $$M_X(t):=\mathbb E\big[e^{tX}\big],$$ is a way to walk around this integral (Eq.1) by, instead, carrying out:

\begin{align} \large \color{blue}{\mathbb{E}\left[e^{\,tX}\right]}&=\displaystyle \int_{-\infty}^{\infty}\color{blue}{e^{tX}}\,\color{green}{\text{pdf}}\, dx\tag{2}\end{align}

Why? Because it's easier and there is a fantastic property of the MGF that can be seen by expanding the Maclaurin series of $\color{blue}{e^{\,tX}}$

$$e^{tX}=1+\frac{ X }{1!}\, t +\frac{ X^{2} }{2!}t^{2} +\frac{ X^{3} }{3!} t^{3} +\cdots$$

Taking the expectation of both sides of this power series:

$$\begin{align} M_X(t) &= \color{blue}{\mathbb{E}\left[e^{\,tX}\right]} \\[1.5ex] &=1 + \frac{\color{red}{\mathbb{E} \left[X\right]}}{1!} \, t \, + \frac{\color{red}{\mathbb{E} \left[X^2\right]}}{2!} \, t^2 \, + \frac{\color{red}{\mathbb{E} \left[X^3\right]}}{3!} \, t^3 \, + \cdots\tag{3} \end{align}$$

the moments appear "perched" on this polynomial "clothesline", ready to be culled by simply differentiating $k$ times and evaluating at zero once we go through the easier integration (in eq. (2)) just once for all moments! The fact that it is an easier integration is most apparent when the pdf is an exponential.

To recover the $k$-th moment:

$$M_X^{(k)}(0)=\frac{d^k}{dt^k}M_X(t)\Bigr|_{t=0}$$

The fact that eventually there is a need to differentiate makes it a not a free lunch - in the end it is a two-sided Laplace transform of the pdf with a changed sign in the exponent:

$$\mathcal L \{\text{pdf}(x)\}(s) =\int_{-\infty}^{\infty}e^{-sx}\text{pdf}(x) dx$$

such that $$M_X(t)=\mathcal L\{\text{pdf}(x)\}(-s)\tag 4.$$

This, in effect, gives us a physics avenue to the intuition. The Laplace transform is acting on the $\color{green}{\text{pdf}}$ and decomposing it into moments. The similarity to a Fourier transform is inescapable: a FT maps a function to a new function on the real line, and Laplace maps a function to a new function on the complex plane. The Fourier transform expresses a function or signal as a series of frequencies, while the Laplace transform resolves a function into its moments. In fact, a different way of obtaining moments is through a Fourier transform (characteristic function). The exponential term in the Laplace transform is in general of the form $e^{-st}$ with $s=\sigma + i\,\omega$, corresponding to the real exponentials and imaginary sinusoidals, and yielding plots such as this:


[From The Scientist and Engineer's Guide to Signal Processing by Steven W. Smith]


Therefore the $M_X(t)$ function decomposes the $\text{pdf}$ somehow into its "constituent frequencies" when $\sigma=0.$ From eq. (4):

\begin{align}\require{cancel} M_X(t)&=\mathbb E\big[e^{-sX}\big]\\[2ex] &=\displaystyle \int_{-\infty}^{\infty}{e^{-sx}}\,\text{pdf}(x)\, dx\\[2ex] &=\displaystyle \int_{-\infty}^{\infty}{e^{-(\sigma+i\omega)x}}\,\text{pdf}(x)\, dx\\[2ex] &=\displaystyle \int_{-\infty}^{\infty}\cancel{e^{-\sigma x}}\,\color{red}{e^{-i\omega x}\,\text{pdf}(x)\, dx} \end{align}

which leaves us with the improper integral of the part of the expression in red, corresponding to the Fourier transform of the pdf.

In general, the intuition of the Laplace transform poles of a function would be that they provide information of the exponential (decay) and frequency components of the function (in this case, the pdf).


In response to the question under comments about the switching from $X^k$ to $e^{tx}$, this is a completely strategic move: one expression does not follow from the other. Here is an analogy: We have a car of our own and we are free to drive into the city every time we need to take care of some business (read, integrating Eq $(1)$ no matter how tough for every separate, single moment). Instead, we can do something completely different: we can drive to the nearest subway station (read, solve Eq $(2)$ just once), and from there use public transportation to reach every single place we need to visit (read, get any $k$ derivative of the integral in Eq $(2)$ to extract whichever $k$-th moment we need, knowing (thanks to Eq $(3)$) that all the moments are "hiding" in there and isolated by differentiating and evaluating at $0$.

$\endgroup$
3
  • 2
    $\begingroup$ How does $E[e^{tX}]$ replace $E[X^k]$? (Out of the blue?) $\endgroup$
    – user366312
    Oct 7 '16 at 9:48
  • 3
    $\begingroup$ I wish the laymen who understand this answer were my students :) $\endgroup$
    – Aksakal
    Jan 25 '19 at 20:30
  • $\begingroup$ Hi, I really enjoy reading your answer! But one thing, "exponential (decay) and frequency components of the function (in this case, the pdf)." could you please explain what does this mean? $\endgroup$
    – JoZ
    Jul 1 '21 at 2:01
4
$\begingroup$

In the most layman terms it's a way to encode all characteristics of the probability distribution into one short phrase. For instance, if I know that MGF of the distribution is $$M(t)=e^{t\mu+1/2\sigma^2t^2}$$ I can find out the mean of this distribution by taking first term of Taylor expansion: $$\frac d {dt}M(t)|_{t=0}=\mu+\sigma^2t|_{t=0}=\mu$$ If you know what you're doing it's much faster than taking the expectation of the probability function.

Moreover, since this MGF encodes everything about the distribution, if you know how to manipulate the function, you can apply operations on all characteristics of the distribution at once! Why don't we always use MGF? First, it's not in every situation the MGF is the easiest tool. Second, MGF doesn't always exist.

Above layman

Suppose you have a standard normal distribution. You can express everything you know about it by stating its PDF: $$f(x)=\frac 1 {\sqrt{2\pi}}e^{-x^2/2}$$

You can calculate its moment such as mean and standard deviation, and use it on transformed variables and functions on random normals etc.

You can think of the MGF of normal distribution as an alternative to PDF. It contains the same amount of information. I already showed how to obtain the mean.

Why do we need an alternative way? As I wrote, sometimes it's just more convenient. For instance, try calculating the variance of the standard normal from PDF: $$\sigma^2=\int_{-\infty}^\infty x^2\frac 1 {\sqrt{2\pi}}e^{-x^2/2} dx=?$$ It's not that difficult, but it's much easier to do it with MGF $M(t)=e^{t^2/2}$: $$\sigma^2=\frac {d^2} {dt^2}M(t)|_{t=0}=\frac d {dt} t |_{t=0}=1$$

$\endgroup$
6
  • 3
    $\begingroup$ Can you please expand on the "everything" that it encodes about the distribution? $\endgroup$ Jan 25 '19 at 18:53
  • 3
    $\begingroup$ To appreciate the point made by @ColorStatistics, please see stats.stackexchange.com/questions/25010. $\endgroup$
    – whuber
    Jan 25 '19 at 20:27
  • $\begingroup$ @whuber: Thank you, whuber. I'll study that reference. This is a topic I am looking to understand better. $\endgroup$ Jan 25 '19 at 21:15
  • $\begingroup$ How can we prove that MGF & PDF contain the same amount of information? $\endgroup$
    – aerin
    Sep 16 '19 at 4:21
  • $\begingroup$ @aerin: If you can reconstruct the pdf from the pdf, then everything you can calculate from the pdf can be calculated from the mgf. So for a proof you need to find such a reconstruction formula ... $\endgroup$ Apr 30 '20 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.