11
$\begingroup$

can someone provide an intuition on why the higher moments of a probability distribution p(x) like the third and fourth moments correspond to skewness and kurtosis, respectively?

specifically, why does the deviation about the mean raised to the 3rd or 4th power end up translating into a measure of skewness and kurtosis? Is there a way to relate this to the third or fourth derivatives of the function?

consider this definition of kurtosis:

$Kurtosis(X) = E[(x - \mu_{X})^4] / \sigma^4$

again, not clear why raising $(x-\mu)^4$ gives "peakedness" or why $(x-\mu)^3$ should give skew. seems magical and mysterious.

Edit: quick followup. what is the advantage of defining moments about the mean and not the median for metrics like kurtosis? what are the properties of estimators like:

$MedianKurtosis(X) = E[(x - \tilde{x})^4] / \sigma^4$

where $\tilde{x}$ is median. this would presumably be less sensitive to outliers in the distribution that throw off the mean and perhaps be a fairer measure of peaked-ness?

$\endgroup$
  • 4
    $\begingroup$ My intuition on skew is to note that the third power preserves negatives. So if you have more large negative deviations from the mean than you do positive (put very simply), then you end up with a negative skewed distribution. My intuition for the kurtosis is that the fourth power amplifies large deviations from the mean much more than the second power. This is why we think of kurtosis as a measure of how fat the tails of a distribution. Note that very large possibilities of x from the mean mu are raised to the forth power, which makes them amplified but ignores sign. $\endgroup$ – wolfsatthedoor Nov 9 '14 at 2:09
  • 2
    $\begingroup$ See stats.stackexchange.com/questions/84158/… $\endgroup$ – whuber Nov 9 '14 at 2:12
  • 1
    $\begingroup$ Since 4th powers are much more affected by outliers than 1st powers, I expect you'll gain little from looking at the fourth moment about the median -- at least if robustness was the aim. $\endgroup$ – Glen_b Nov 9 '14 at 8:01
  • 1
    $\begingroup$ First, note that these higher moments are not necessarily good/reliable measures of asymmetry/peakedness. That said, I think beams give a good physical intuition for the first three moments, e.g. mean = beam balance/scale, variance = cantilever flexure, skewness = seesaw. $\endgroup$ – GeoMatt22 Dec 10 '16 at 4:12
  • 1
    $\begingroup$ You are right, the interpretation of kurtosis as measuring "peakedness" is magical and mysterious. That's because it's not at all true. Kurtosis tells you absolutely nothing about the peak. It measures the tails (outliers) only. It is easy to prove mathematically that the observations near the peak contribute a miniscule amount to the kurtosis measure, regardless of whether the peak is flat, spiked, bimodal, sinusoidal, or bell-shaped. $\endgroup$ – Peter Westfall Nov 21 '17 at 1:22
4
$\begingroup$

There is a good reason for these definitions, which becomes clearer when you look at the general form for moments of standardised random variables. To answer this question, first consider the general form of the $n$th standardised central moment:$^\dagger$

$$\phi_n = \mathbb{E} \Bigg[ \Bigg( \frac{X - \mathbb{E}[X]}{\mathbb{S}[X]} \Bigg)^n \text{ } \Bigg].$$

The first two standardised central moments are the values $\phi_1=0$ and $\phi_2=1$, which hold for all distributions for which the above quantity is well-defined. Hence, we can consider the non-trivial standardised central moments that occur for values $n \geqslant 3$. To facilitate our analysis we define:

$$\begin{equation} \begin{aligned} \phi_n^+ &= \mathbb{E} \Bigg[ \Bigg| \frac{X - \mathbb{E}[X]}{\mathbb{S}[X]} \Bigg|^n \text{ } \Bigg| X > \mathbb{E}[X] \Bigg] \cdot \mathbb{P}(X > \mathbb{E}[X]), \\[8pt] \phi_n^- &= \mathbb{E} \Bigg[ \Bigg| \frac{X - \mathbb{E}[X]}{\mathbb{S}[X]} \Bigg|^n \text{ } \Bigg| X < \mathbb{E}[X] \Bigg] \cdot \mathbb{P}(X < \mathbb{E}[X]). \end{aligned} \end{equation}$$

These are non-negative quantities that give the $n$th absolute power of the standardised random variable conditional on it being above or below its expected value. We will now decompose the standardised central moment into these parts.


Odd values of $n$ measure the skew in the tails: For any odd value of $n \geqslant 3$ we have an odd power in the moment equation and so we can write the standardised central moment as $\phi_n = \phi_n^+ - \phi_n^-$. From this form we see that the standardised central moment gives us the difference between the $n$th absolute power of the standardised random variable, conditional on it being above or below its mean respectively.

Thus, for any odd power $n \geqslant 3$ we will get a measure that gives positive values if the expected absolute power of the standardised random variable is higher for values above the mean than for values below the mean, and gives negative values if the expected absolute power is lower for values above the mean than for values below the mean. Any of these quantities could reasonably be regarded as a measure of a type of "skewness", with higher powers giving greater relative weight to values that are far from the mean.

Since this phenomenon occurs for every odd power $n \geqslant 3$, the natural choice for an archetypal measure of "skewness" is to define $\phi_3$ as the skewness. This is a lower standardised central moment than the higher odd powers, and it is natural to explore lower-order moments before consideration of higher-order moments. In statistics we have adopted the convention of referring to this standardised central moment as the skewness, since it is the lowest standardised central moment that measures this aspect of the distribution. (The higher odd powers also measure types of skewness, but with greater and greater emphasis on values far from the mean.)


Even values of $n$ measure fatness of tails: For any even value of $n \geqslant 3$ we have an even power in the moment equation and so we can write the standardised central moment as $\phi_n = \phi_n^+ + \phi_n^-$. From this form we see that the standardised central moment gives us the sum of the $n$th absolute power of the standardised random variable, conditional on it being above or below its mean respectively.

Thus, for any even power $n \geqslant 3$ we will get a measure that gives non-negative values, with higher values occurring if the tails of the distribution of the standardised random variable are fatter. Note that this is a result with respect to the standardised random variable, and so a change in scale (changing the variance) has no effect on this measure. Rather, it is effectively a measure of the fatness of the tails, after standardising for the variance of the distribution. Any of these quantities could reasonably be regarded as a measure of a type of "kurtosis", with higher powers giving greater relative weight to values that are far from the mean.

Since this phenomenon occurs for every even power $n \geqslant 3$, the natural choice for an archetypal measure of kurtosis is to define $\phi_4$ as the kurtosis. This is a lower standardised central moment than the higher even powers, and it is natural to explore lower-order moments before consideration of higher-order moments. In statistics we have adopted the convention of referring to this standardised central moment as the "kurtosis", since it is the lowest standardised central moment that measures this aspect of the distribution. (The higher even powers also measure types of kurtosis, but with greater and greater emphasis on values far from the mean.)


$^\dagger$ This equation is well defined for any distribution whose first two moments exist, and which has non-zero variance. We will assume that the distribution of interest falls in this class for the rest of the analysis.

$\endgroup$
1
$\begingroup$

Similar question What's so 'moment' about 'moments' of a probability distribution? I gave a physical answer to that which addressed moments.

"Angular acceleration is the derivative of angular velocity, which is the derivative of angle with respect to time, i.e., $ \dfrac{d\omega}{dt}=\alpha,\,\dfrac{d\theta}{dt}=\omega$. Consider that the second moment is analogous to torque applied to a circular motion, or if you will an acceleration/deceleration (also second derivative) of that circular (i.e., angular, $\theta$) motion. Similarly, the third moment would be a rate of change of torque, and so on and so forth for yet higher moments to make rates of change of rates of change of rates of change, i.e., sequential derivatives of circular motion...."

See the link as this is perhaps easier to visualize this with physical examples.

Skewness is easier to understand than kurtosis. A negative skewness is a heavier left tail (or further negative direction outlier) than on the right and positive skewness the opposite.

Wikipedia cites Westfall (2014) and implies that high kurtosis arises either for random variables that have far outliers or for density functions with one or two heavy tails while claiming that any central tendency of data or density has relatively little effect on the kurtosis value. Low values of kurtosis would imply the opposite, i.e., a lack of $x$-axis outliers and the relative lightness of both tails.

$\endgroup$
  • $\begingroup$ Skewness is the balance point of the pdf of $Z^3$, and kurtosis is the balance point of the pdf of $Z^4$. Both transformations "stretch" the tails, kurtosis more. If the pdf of $Z^3$ falls to the right when a fulcrum is placed at 0, then there is positive skew in the original distribution. If the pdf of $Z^4$ falls to the right when a fulcrum is placed at 3.0, then the original distribution is heavier-tailed than the normal distribution. Here, "heaviness of tails" refers to more precisely to leverage than to mass. Moors' interpretation is not quite right wrt both mentions of "concentration." $\endgroup$ – Peter Westfall Feb 2 at 17:47
  • $\begingroup$ @PeterWestfall I agree that Moors' interpretation is imperfect. Precise language is not easily achievable without also being confusing. Take "leverage" for example. Leverage means first moment and one would have to invent something like "leveraged leverage" for the second moment, which might confuse more than illuminate. Your approach appears to invent a novel concept, i.e., "stretched leverage," which hints at geometric transforms for which one might also claim some advocates who favor it as self-consistent at the risk of also being controversial, and non-physical for others. $\endgroup$ – Carl Feb 2 at 23:20
  • $\begingroup$ "Leverage" refers to the first moment of the variable $U$, where $U = Z^4$. It's not rocket science. $\endgroup$ – Peter Westfall Feb 3 at 13:40
  • $\begingroup$ @PeterWestfall Not to be too punny, but you are leveraging leverage. Sure, you can still use the word, and if $Z^4$ were not a fourth dimensional object, as compared to a one dimensional distance, $Z$, it might be even be useful. The context here is that of moments, and creating a physical model for moments. There are several ways that can be done, for example, see my answer about that here. In other words, to put moments into any physical context, we have to do more than hand-waving and invocation of the fourth dimension. $\endgroup$ – Carl Feb 4 at 4:27
  • $\begingroup$ @PeterWestfall In the context of circular motion, we would call the second moment torque, and not the leverage of $Z^2$, which latter, although not incorrect, does not bring anything physical to mind. $\endgroup$ – Carl Feb 4 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.