24
$\begingroup$

We are typically introduced to method of moments estimators by "equating population moments to their sample counterpart" until we have estimated all of the population's parameters; so that, in the case of a normal distribution, we would only need the first and second moments because they fully describe this distribution.

$E(X) = \mu \implies \sum_{i=1}^n X_i/n = \bar{X}$

$E(X^2) = \mu^2 + \sigma^2 \implies \sum_{i=1}^n X_i^2/n$

And we could theoretically compute up to $n$ additional moments as:

$E(X^r) \implies \sum_{i=1}^nX_i^r /n$

How can I build intuition for what moments really are? I know they exist as a concept in physics and in mathematics, but I find neither directly applicable, especially because I don't know how to make the abstraction from the mass concept to a data point. The term seems to be used in a specific way in statistics, which differs from usage in other disciplines.

What characteristic of my data determines how many ($r$) moments there are overall?

Improve this question
$\endgroup$
7
  • 7
    $\begingroup$ The term means the same thing it does in physics, when applied to probability distribution. See here, which has the equation $\mu_n=\int r^n\,\rho(r)\,dr$, "where $\rho$ is the distribution of the density of charge, mass, or whatever quantity is being considered". When the "thing being considered" is the probability density, you have the corresponding moment in probability. Those are raw moments (moments about the origin). By comparison ... (ctd) $\endgroup$
    – Glen_b
    Commented Jan 10, 2015 at 14:31
  • 2
    $\begingroup$ Moments are parametrized features of the distribution of random variables, like quantiles. Moments are parameterized by the natural numbers, and completely characterize a distribution (see moment generating function). This does not rule out that for some distributions there might be perfect functional dependence among moments, so not all moments are always required to characterize the distribution. (1/2) $\endgroup$
    – tchakravarty
    Commented Jan 10, 2015 at 14:33
  • $\begingroup$ Moments $\geq 3$ are functionally dependent on the first two for the normal distribution, so the first two suffice to characterize the distribution, including the mean and variance. (2/2) $\endgroup$
    – tchakravarty
    Commented Jan 10, 2015 at 14:33
  • 5
    $\begingroup$ (ctd) ... moments in mathematics are the same ($\mu_n=\int_{-\infty}^\infty (x - c)^n\,f(x)\,dx$), except about $c$ rather than 0 (i.e. just a generalized form of the physics one - but since they're the same with a mere change of origin, a physicist would rightly say "how is that different?"). These are the same as in probability, when $f$ is a density. To me, all three are talking about the same thing when they say 'moments', not different things. $\endgroup$
    – Glen_b
    Commented Jan 10, 2015 at 14:35
  • 3
    $\begingroup$ I'm sure you can find answers in the many threads that have been posted about moments and intuition. Statistics uses moments in exactly the same way they are used in physics and mathematics--it is the same concept with the same definition in all three fields. $\endgroup$
    – whuber ♦
    Commented Jan 10, 2015 at 18:41

5 Answers 5

Reset to default
19
$\begingroup$

It's been a long time since I took a physics class, so let me know if any of this is incorrect.

General description of moments with physical analogs

Take a random variable, $X$. The $n$-th moment of $X$ around $c$ is: $$m_n(c)=E[(X-c)^n]$$ This corresponds exactly to the physical sense of a moment. Imagine $X$ as a collection of points along the real line with density given by the pdf. Place a fulcrum under this line at $c$ and start calculating moments relative to that fulcrum, and the calculations will correspond exactly to statistical moments.

Most of the time, the $n$-th moment of $X$ refers to the moment around 0 (moments where the fulcrum is placed at 0): $$m_n=E[X^n]$$ The $n$-th central moment of $X$ is: $$\hat m_n=m_n(m_1) =E[(X-m_1)^n]$$ This corresponds to moments where the fulcrum is placed at the center of mass, so the distribution is balanced. It allows moments to be more easily interpreted, as we'll see below. The first central moment will always be zero, because the distribution is balanced.

The $n$-th standardized moment of $X$ is: $$\tilde m_n = \dfrac{\hat m_n}{\left(\sqrt{\hat m_2}\right)^n}=\dfrac{E[(X-m_1)^n]} {\left(\sqrt{E[(X-m_1)^2]}\right)^n}$$ Again, this scales moments by the spread of the distribution, allowing for easier interpretation specifically of Kurtosis. The first standardized moment will always be zero, the second will always be one. This corresponds to the moment of the standard score (z-score) of a variable. I don't have a great physical analog for this concept.

Commonly used moments

For any distribution there are potentially an infinite number of moments. Enough moments will almost always fully characterize and distribution (deriving the necessary conditions for this to be certain is a part of the moment problem). Four moments are commonly talked about a lot in statistics:

  1. Mean - the 1st moment (centered around zero). It is the center of mass of the distribution, or alternatively it's proportional to the moment of torque of the distribution relative to a fulcrum at 0.
  2. Variance - the 2nd central moment. Interpreted as representing the degree to which the distribution of $X$ is spread out. It corresponds to the moment of inertia of a distribution balanced on its fulcrum.
  3. Skewness - the 3rd central moment (sometimes standardized). A measure of the skew of a distribution in one direction or another. Relative to a normal distribution (which has no skew), positively skewed distribution have a low probability of extremely high outcomes, negatively skewed distributions have a small probability of extremely low outcomes. Physical analogs are difficult, but loosely it measures the asymmetry of a distribution. As an example, the figure below is taken from Wikipedia. Skewness, taken from Wikipedia
  4. Kurtosis - the 4th standardized moment, usually excess Kurtosis, the 4th standardized moment minus three. Kurtosis measures the extent to which $X$ places more probability on the center of the distribution relative to the tails. Higher Kurtosis means less frequent larger deviations from the mean and more frequent smaller deviations. It is often interpreted relative to the normal distribution, which has a 4th standardized moment of 3, hence an excess Kurtosis of 0. Here a physical analog is even more difficult, but in the figure below, taken from Wikipedia, the distributions with higher peaks have greater Kurtosis. Kurtosis, also from WIkipedia

We rarely talk about moments beyond Kurtosis, precisely because there is very little intuition to them. This is similar to physicists stopping after the second moment.

Improve this answer
$\endgroup$
7
$\begingroup$

This is a bit of an old thread, but i wish to correct a misstatement in the comment by Fg Nu who wrote "Moments are parameterized by the natural numbers, and completely characterize a distribution".

Moments do NOT completely characterize a distribution. Specifically, knowledge of all infinite number of moments, even if they exist, does not necessarily uniquely determine the distribution.

Per my favorite probability book, Feller "An Introduction to Probability Theory and Its Applications Vol II" (see my answer at Real-life examples of common distributions ), section VII.3 example on pp. 227-228, the Lognormal is not determined by its moments, meaning that there are other distributions having all infinite number of moments the same as the Lognormal, but different distribution functions. As is widely known, the Moment Generating Function does not exist for the Lognormal, nor can it for these other distributions possessing the same moments.

As stated on p. 228, an essentially nonzero random variable $X$ is determined by its moments if they all exist and

$$\sum_{n=1}^{\infty} (\mathbb{E}[X^{2n}])^{-1/(2n)}$$

diverges. Note that this is not an if and only if. This condition does not hold for the Lognormal, and indeed it is not determined by its moments.

On the other hand, distributions (random variables) which share all infinite number of moments, can only differ by so much, due to inequalities which can be derived from their moments.

Improve this answer
$\endgroup$
9
  • $\begingroup$ This is considerably simplified when the distribution is bounded, in which case the moments always determine the distribution completely (uniquely). $\endgroup$
    – Alex R.
    Commented Apr 12, 2016 at 22:58
  • $\begingroup$ @Alex That's an immediate consequence of the result cited in Feller. $\endgroup$
    – whuber ♦
    Commented Apr 13, 2016 at 16:05
  • $\begingroup$ It is not completely correct to say that the moment generating function do not exist for the lognormal. The most useful theorems about mgf's assume it exist in an open interval containing zero, and in the strict sense it do not exist. But it do exist in a ray emanating from zero!, and that also give useful information. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 13, 2016 at 20:55
  • $\begingroup$ @ kjetil b halvorsen , can you describe (some of) the useful information you would get from the existence of the MGF of a lognormal on a ray emanating from zero? What ray would that be? $\endgroup$
    – Mark L. Stone
    Commented Apr 13, 2016 at 21:01
  • $\begingroup$ Bump of above comment as question to @kjetil b halvorsen.. $\endgroup$
    – Mark L. Stone
    Commented Jul 25, 2016 at 15:00
2
$\begingroup$

A corollary to Glen_b's remarks is that the first moment, the mean, corresponds to the center of gravity for a physical object, and the second moment around the mean, the variance, corresponds to its moment of inertia. After that, you're on your own.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I like the relation of the first moment and the mean...but the second moment is not the variance...the variance is the centered second moment...$E[x^2] = \int x^2f(x)dx$ $var[x]=E[(x-E[x])^2] = \int (x-E[x])^2f(x)dx$. $\endgroup$
    – Zachary Blumenfeld
    Commented Jan 10, 2015 at 21:53
0
$\begingroup$

A binomial tree has two branches each with a probably of 0.5. Actually, p=0.5, and q=1-0.5=0.5. This generates a normal distribution with an evenly distributed probability mass.

Actually, we have to assume that each tier in the tree is complete. When we break data up into bins, we get a real number from the division, but we round up. Well, that's a tier that is incomplete, so we don't end up with a histogram approximating the normal.

Change the branching probabilities to p=0.9999 and q=0.0001 and that gets us a skewed normal. The probability mass shifted. That accounts for skewness.

Having incomplete tiers or bins less than 2^n generate binomial trees with areas that have no probability mass. This gives us kurtosis.

Response to comment:

When I was talking about determining the number of bins, round up to the next integer.

Quincunx machines drop balls that come to eventually approximate the normal distribution via the binomial. Several assumptions are made by such a machine: 1) the number of bins is finite, 2) the underlying tree is binary, and 3) the probabilities are fixed. The Quincunx machine at the Museum of Mathematics in New York, lets the user dynamically change the probabilities. The probabilities can change at any time, even before the current layer is finished. Hence this idea about the bins not being filled.

Unlike what I said in my original answer when you have a void in the tree, the distribution demonstrates kurtosis.

I'm looking at this from the perspective of generative systems. I use a triangle to summarize decision trees. When a novel decision is made, more bins are added at the base of the triangle, and in terms of the distribution, in the tails. Trimming subtrees from the tree would leave voids in the distribution's probability mass.

I only replied to give you an intuitive sense. Labels? I've used Excel and played with the probabilities in the binomial and generated the expected skews. I have not done so with kurtosis, it doesn't help that we are forced to think about probability mass as being static while using language suggesting movement. The underlying data or balls cause the kurtosis. Then, we analyze it variously and attribute it to shape descriptive terms like center, shoulder, and tail. The only things we have to work with are the bins. Bins live dynamic lives even if the data can't.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ This is intriguing, but awfully sketchy. What are the labels on your binomial tree, for instance? It had better be an infinite tree if you want to get a normal distribution--but then the obvious labels (using a random walk or using binary representations of real numbers) don't lead to normal distributions at all. Without these details too much is left to readers' imaginations. Could you elaborate on them? $\endgroup$
    – whuber ♦
    Commented Jul 25, 2016 at 14:40
0
$\begingroup$

How can I build intuition for what moments really are?

In the context of computer vision, recognizing two-dimensional shapes such as letters or geometric objects from pixel data, a classic article exploiting moments is:

Ming-Kuei Hu, "Visual pattern recognition by moment invariants," in IRE Transactions on Information Theory, vol. 8, no. 2, pp. 179-187, February 1962.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.