We had couple of good discussions about Moment Generating Function(MGF), here and here.

But I still have questions on the applications of it and how can it be useful.

Specifically, I can understand that in real world, from data, we can get an estimation of the probability distribution. But how can we get Moment Generating Function(MGF) from data? If we cannot get it, where does it come from?

If it is from the Laplace transform of pdf, i.e., calculated from PDF, then it should be "less useful" than PDF, right?


Can we define an MGF from data?

The MGF of a random variables $X$ is defined to be

$$M(t) = \mathbf E\left[e^{tX}\right],$$

so given observed data $x_1,\ldots, x_n$ we can certainly define the empirical MGF to be

$$M(t; \underline x) = \frac1n \left( e^{tx_1} + \cdots + e^{tx_n}\right).$$

Is it useful?

The use of this empirical MGF is likely limited - in part due to it not admitting a simple closed formula, but also because many of the features that make the MGF useful for studying probability distributions, will not be relevant for the empirical MGF when we have small/moderate sample sizes.

I've set out a summary of some of the key reasons for studying MGFs of probability distributions at the end.

Is it less useful than the PDF?

Theoretically - no (in most cases).

Both the PDF and the MGF uniquelly determine a probability distribution - so neither contains any information that the other does not. Which is more useful depends on what you want to do with the distribution.

For sampling, the PDF will be more useful. To easily calculate the mean, variance and higher moments - the MGF may make this significantly easier.

It is worth noting, however, that not all distributions admit an MGF, for example the Cauchy distribution.

Probability vs Statistics

Finally - its worth noting that the value of the MGF is arguably higher to a probabilist than a statistician - where I'm informally using the convention that probabilists study abstract/theoretical distributions, whilst statisticians study data (and sometimes fit these to theoretical distributions).

Many of the properties I summarise below are more useful in this theoretical framework - for instance 4) the convergence property is key to proving the Central Limit Theorem.

Key Properties of the MGF

1) The key feature of an MGF is that its power series expansion is in terms of the distribution's moments:

$$ M(t) = 1 + t \mathbf E[X] + \frac{t^2}2 \mathbf E[X^2] + \frac{t^3}{3!} \mathbf E[X^3] + \cdots $$

For some distributuions evaluating this power series will be significantly easier than trying to compute these expectations directly through integration.

For instance if $X \sim N(0,1)$ then $M(t) = \exp(\frac12t^2)$, from which the standard Taylor expansion gives

$$M(t) = 1 + \left(\frac{t^2}{2}\right) + \frac12 \left(\frac{t^2}{2}\right)^2 + \frac{1}{3!} \left(\frac{t^2}{2}\right)^3 \cdots $$

we easily see that all odd moments of the distribution are 0, and also get a formula for all even moments:

$$ \mathbf E[X^{2n}] = \frac{(2n)!}{2^n n!},$$ (the right hand side is often denoted $n!!$, and is known as the double factorial).

2) The MGF of the sum of two independent variables, is the product of their respective MGFs:

$$M_{X+Y}(t) = M_X(t)M_Y(t),$$ again this make calculations easier.

3) The radius of convergence of the MGF can be used to deduce asymptotic properties of the moments of the distribution, via the Cauchy-Hadamard theorem.

4) The MGF (when it exists) uniquely determines a probability distribution. Moreover given a sequence of distributions, if their MGFs converge pointwise then this is equivalent to convergence in distribution.

| cite | improve this answer | |
  • 1
    $\begingroup$ thanks for the great answer!, so in your example (2) is more like convolution theorem that can simplify the computation? $\endgroup$ – Haitao Du May 3 at 12:34
  • 2
    $\begingroup$ Exactly, since you're familiar with Laplace/Fourier transforms (which is all that the MGF is) - this is exactly the property that convolution in the original domain is multiplication in the Laplace domain. $\endgroup$ – owen88 May 3 at 12:39

In parametric problems (i.e., where you have a specified distribution family indexed by a finite number of parameters), both the true density and MGF are functions of the parameters (assuming the latter exists). Both objects summarise the distribution and contain the same information, so neither is less useful in a strict sense (though the MGF is more difficult to interpret intuitively than the density). Estimation of either the density or the MGF can be done by estimating the unknown parameters, and substituting these into the required parametric function. For example, if we have IID normal data $x_1,...,x_n$ with sample mean $\bar{x}$ and sample variance $s^2$, we could estimate the MGF as:

$$\hat{m}_X(t) = \exp \Big( \bar{x} t + \frac{s^2}{2} \cdot t^2 \Big).$$

Alternatively, we can use non-parametric methods to estimate the MGF in cases where we do not want to assume a particular distributional family. The simplest estimator is the empirical MGF, which is:

$$\hat{m}_\mathbf{x}(t) = \frac{1}{n} \sum_{i=1}^n \exp (t x_i).$$

For IID data, if the MGF exists in a neighbourhood of $t \in \mathbb{R}$, then the law of large numbers ensures that $\hat{m}_\mathbf{x}(t) \rightarrow m_X(t)$. (Both the weak and strong law hold, so the convergence is "almost surely".)

Generally speaking, the empirical MGF is a more robust estimator, but it is less powerful than the parametric estimators in cases where you have correctly specified the distributional family. This is just one aspect of the more general statistical phenomenon that an assumption of a distributional family allows you to make your estimators more powerful, but at the expense of lack of robustness to outside distributions. You can read more about these various estimators, and their performance, in Gber and Collins (1989).

| cite | improve this answer | |

Just some additions to the excellent answer by @owen88. Some examples of empirical mgf's (emgf) (and comments on better ways to estimate them) can be found in answers here: How does saddlepoint approximation work?. One use is to approximate the bootstrap distribution, thereby making possible bootstrap without simulation! Related to the mgf is the probability generating function, see What is the difference between moment generating function and probability generating function?. And there is some literature about the use of the empirical moment generating function, for example this paper or this one. It is also possible to use emprical generating functions directly in inference, for example this paper.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.