19
$\begingroup$

We are typically introduced to method of moments estimators by "equating population moments to their sample counterpart" until we have estimated all of the population's parameters; so that, in the case of a normal distribution, we would only need the first and second moments because they fully describe this distribution.

$E(X) = \mu \implies \sum_{i=1}^n X_i/n = \bar{X}$

$E(X^2) = \mu^2 + \sigma^2 \implies \sum_{i=1}^n X_i^2/n$

And we could theoretically compute up to $n$ additional moments as:

$E(X^r) \implies \sum_{i=1}^nX_i^r /n$

How can I build intuition for what moments really are? I know they exist as a concept in physics and in mathematics, but I find neither directly applicable, especially because I don't know how to make the abstraction from the mass concept to a data point. The term seems to be used in a specific way in statistics, which differs from usage in other disciplines.

What characteristic of my data determines how many ($r$) moments there are overall?

$\endgroup$
  • 7
    $\begingroup$ The term means the same thing it does in physics, when applied to probability distribution. See here, which has the equation $\mu_n=\int r^n\,\rho(r)\,dr$, "where $\rho$ is the distribution of the density of charge, mass, or whatever quantity is being considered". When the "thing being considered" is the probability density, you have the corresponding moment in probability. Those are raw moments (moments about the origin). By comparison ... (ctd) $\endgroup$ – Glen_b Jan 10 '15 at 14:31
  • 2
    $\begingroup$ Moments are parametrized features of the distribution of random variables, like quantiles. Moments are parameterized by the natural numbers, and completely characterize a distribution (see moment generating function). This does not rule out that for some distributions there might be perfect functional dependence among moments, so not all moments are always required to characterize the distribution. (1/2) $\endgroup$ – tchakravarty Jan 10 '15 at 14:33
  • $\begingroup$ Moments $\geq 3$ are functionally dependent on the first two for the normal distribution, so the first two suffice to characterize the distribution, including the mean and variance. (2/2) $\endgroup$ – tchakravarty Jan 10 '15 at 14:33
  • 5
    $\begingroup$ (ctd) ... moments in mathematics are the same ($\mu_n=\int_{-\infty}^\infty (x - c)^n\,f(x)\,dx$), except about $c$ rather than 0 (i.e. just a generalized form of the physics one - but since they're the same with a mere change of origin, a physicist would rightly say "how is that different?"). These are the same as in probability, when $f$ is a density. To me, all three are talking about the same thing when they say 'moments', not different things. $\endgroup$ – Glen_b Jan 10 '15 at 14:35
  • 3
    $\begingroup$ I'm sure you can find answers in the many threads that have been posted about moments and intuition. Statistics uses moments in exactly the same way they are used in physics and mathematics--it is the same concept with the same definition in all three fields. $\endgroup$ – whuber Jan 10 '15 at 18:41
17
$\begingroup$

It's been a long time since I took a physics class, so let me know if any of this is incorrect.

General description of moments with physical analogs

Take a random variable, $X$. The $n$-th moment of $X$ around $c$ is: $$m_n(c)=E[(X-c)^n]$$ This corresponds exactly to the physical sense of a moment. Imagine $X$ as a collection of points along the real line with density given by the pdf. Place a fulcrum under this line at $c$ and start calculating moments relative to that fulcrum, and the calculations will correspond exactly to statistical moments.

Most of the time, the $n$-th moment of $X$ refers to the moment around 0 (moments where the fulcrum is placed at 0): $$m_n=E[X^n]$$ The $n$-th central moment of $X$ is: $$\hat m_n=m_n(m_1) =E[(X-m_1)^n]$$ This corresponds to moments where the fulcrum is placed at the center of mass, so the distribution is balanced. It allows moments to be more easily interpreted, as we'll see below. The first central moment will always be zero, because the distribution is balanced.

The $n$-th standardized moment of $X$ is: $$\tilde m_n = \dfrac{\hat m_n}{\left(\sqrt{\hat m_2}\right)^n}=\dfrac{E[(X-m_1)^n]} {\left(\sqrt{E[(X-m_1)^2]}\right)^n}$$ Again, this scales moments by the spread of the distribution, allowing for easier interpretation specifically of Kurtosis. The first standardized moment will always be zero, the second will always be one. This corresponds to the moment of the standard score (z-score) of a variable. I don't have a great physical analog for this concept.

Commonly used moments

For any distribution there are potentially an infinite number of moments. Enough moments will almost always fully characterize and distribution (deriving the necessary conditions for this to be certain is a part of the moment problem). Four moments are commonly talked about a lot in statistics:

  1. Mean - the 1st moment (centered around zero). It is the center of mass of the distribution, or alternatively it's proportional to the moment of torque of the distribution relative to a fulcrum at 0.
  2. Variance - the 2nd central moment. Interpreted as representing the degree to which the distribution of $X$ is spread out. It corresponds to the moment of inertia of a distribution balanced on its fulcrum.
  3. Skewness - the 3rd central moment (sometimes standardized). A measure of the skew of a distribution in one direction or another. Relative to a normal distribution (which has no skew), positively skewed distribution have a low probability of extremely high outcomes, negatively skewed distributions have a small probability of extremely low outcomes. Physical analogs are difficult, but loosely it measures the asymmetry of a distribution. As an example, the figure below is taken from Wikipedia. Skewness, taken from Wikipedia
  4. Kurtosis - the 4th standardized moment, usually excess Kurtosis, the 4th standardized moment minus three. Kurtosis measures the extent to which $X$ places more probability on the center of the distribution relative to the tails. Higher Kurtosis means less frequent larger deviations from the mean and more frequent smaller deviations. It is often interpreted relative to the normal distribution, which has a 4th standardized moment of 3, hence an excess Kurtosis of 0. Here a physical analog is even more difficult, but in the figure below, taken from Wikipedia, the distributions with higher peaks have greater Kurtosis. Kurtosis, also from WIkipedia

We rarely talk about moments beyond Kurtosis, precisely because there is very little intuition to them. This is similar to physicists stopping after the second moment.

$\endgroup$
6
$\begingroup$

This is a bit of an old thread, but i wish to correct a misstatement in the comment by Fg Nu who wrote "Moments are parameterized by the natural numbers, and completely characterize a distribution".

Moments do NOT completely characterize a distribution. Specifically, knowledge of all infinite number of moments, even if they exist, does not necessarily uniquely determine the distribution.

Per my favorite probability book, Feller "An Introduction to Probability Theory and Its Applications Vol II" (see my answer at Real-life examples of common distributions ), section VII.3 example on pp. 227-228, the Lognormal is not determined by its moments, meaning that there are other distributions having all infinite number of moments the same as the Lognormal, but different distribution functions. As is widely known, the Moment Generating Function does not exist for the Lognormal, nor can it for these other distributions possessing the same moments.

As stated on p. 228, an essentially nonzero random variable $X$ is determined by its moments if they all exist and

$$\sum_{n=1}^{\infty} (\mathbb{E}[X^{2n}])^{-1/(2n)}$$

diverges. Note that this is not an if and only if. This condition does not hold for the Lognormal, and indeed it is not determined by its moments.

On the other hand, distributions (random variables) which share all infinite number of moments, can only differ by so much, due to inequalities which can be derived from their moments.

$\endgroup$
  • $\begingroup$ This is considerably simplified when the distribution is bounded, in which case the moments always determine the distribution completely (uniquely). $\endgroup$ – Alex R. Apr 12 '16 at 22:58
  • $\begingroup$ @Alex That's an immediate consequence of the result cited in Feller. $\endgroup$ – whuber Apr 13 '16 at 16:05
  • $\begingroup$ It is not completely correct to say that the moment generating function do not exist for the lognormal. The most useful theorems about mgf's assume it exist in an open interval containing zero, and in the strict sense it do not exist. But it do exist in a ray emanating from zero!, and that also give useful information. $\endgroup$ – kjetil b halvorsen Apr 13 '16 at 20:55
  • $\begingroup$ @ kjetil b halvorsen , can you describe (some of) the useful information you would get from the existence of the MGF of a lognormal on a ray emanating from zero? What ray would that be? $\endgroup$ – Mark L. Stone Apr 13 '16 at 21:01
  • $\begingroup$ Bump of above comment as question to @kjetil b halvorsen.. $\endgroup$ – Mark L. Stone Jul 25 '16 at 15:00
2
$\begingroup$

A corollary to Glen_b's remarks is that the first moment, the mean, corresponds to the center of gravity for a physical object, and the second moment around the mean, the variance, corresponds to its moment of inertia. After that, you're on your own.

$\endgroup$
  • 3
    $\begingroup$ I like the relation of the first moment and the mean...but the second moment is not the variance...the variance is the centered second moment...$E[x^2] = \int x^2f(x)dx$ $var[x]=E[(x-E[x])^2] = \int (x-E[x])^2f(x)dx$. $\endgroup$ – Zachary Blumenfeld Jan 10 '15 at 21:53
0
$\begingroup$

A binomial tree has two branches each with a probably of 0.5. Actually, p=0.5, and q=1-0.5=0.5. This generates a normal distribution with an evenly distributed probability mass.

Actually, we have to assume that each tier in the tree is complete. When we break data up into bins, we get a real number from the division, but we round up. Well, that's a tier that is incomplete, so we don't end up with a histogram approximating the normal.

Change the branching probabilities to p=0.9999 and q=0.0001 and that gets us a skewed normal. The probability mass shifted. That accounts for skewness.

Having incomplete tiers or bins less than 2^n generate binomial trees with areas that have no probability mass. This gives us kurtosis.


Response to comment:

When I was talking about determining the number of bins, round up to the next integer.

Quincunx machines drop balls that come to eventually approximate the normal distribution via the binomial. Several assumptions are made by such a machine: 1) the number of bins is finite, 2) the underlying tree is binary, and 3) the probabilities are fixed. The Quincunx machine at the Museum of Mathematics in New York, lets the user dynamically change the probabilities. The probabilities can change at any time, even before the current layer is finished. Hence this idea about the bins not being filled.

Unlike what I said in my original answer when you have a void in the tree, the distribution demonstrates kurtosis.

I'm looking at this from the perspective of generative systems. I use a triangle to summarize decision trees. When a novel decision is made, more bins are added at the base of the triangle, and in terms of the distribution, in the tails. Trimming subtrees from the tree would leave voids in the distribution's probability mass.

I only replied to give you an intuitive sense. Labels? I've used Excel and played with the probabilities in the binomial and generated the expected skews. I have not done so with kurtosis, it doesn't help that we are forced to think about probability mass as being static while using language suggesting movement. The underlying data or balls cause the kurtosis. Then, we analyze it variously and attribute it to shape descriptive terms like center, shoulder, and tail. The only things we have to work with are the bins. Bins live dynamic lives even if the data can't.

$\endgroup$
  • 2
    $\begingroup$ This is intriguing, but awfully sketchy. What are the labels on your binomial tree, for instance? It had better be an infinite tree if you want to get a normal distribution--but then the obvious labels (using a random walk or using binary representations of real numbers) don't lead to normal distributions at all. Without these details too much is left to readers' imaginations. Could you elaborate on them? $\endgroup$ – whuber Jul 25 '16 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.