7
$\begingroup$

I keep hearing these terms, and it seems like both refer to the same thing: a greater probability of an event occurring at the extreme values of a distribution, far away from the mean (more than 3 standard deviations away)

$\endgroup$
11
$\begingroup$

The "heaviness" of the tail refers to how quickly the probability decays as you move away from the center of the distribution, while skewness deals with symmetry or lack thereof. For instance, the exponential distribution is skewed but considered to have a fairly light tail, while the Cauchy distribution is perfectly symmetric but heavy-tailed.

$\endgroup$
6
$\begingroup$

As dsaxtron points out (+1), skewness refers to symmetry or asymmetry. Any symmetric distribution will have a skewness of zero - no matter how fat its tails. This is because of the third power in its definition, which allows deviations in both tails to cancel out.

Thus, there is no relationship between skewness and tail fatness.

However, and relatedly, I strongly recommend Westfall (2014), Kurtosis as Peakedness, 1905–2014. R.I.P. in The American Statistician, which extremely nicely debunks the common misconception (also found in the Wikipedia article) that kurtosis has anything to do with "peakedness". Instead, kurtosis measures the propensity to outliers, i.e., the fatness of tails of a distribution. This is because the kurtosis uses the fourth power of deviations from the mean, so positive and negative tails do not cancel out.

$\endgroup$
  • $\begingroup$ Hum. I just added the kurtosis tag to the question, since I believe it is helpful to understand the matter... and it turns out the tag wiki contains the same error. Gotta edit that. $\endgroup$ – Stephan Kolassa Jun 3 '15 at 15:34
  • 1
    $\begingroup$ Kurtosis is not peakedness, but it also isn't fat-tailedness unless you define it that way. For example, (-1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1) has a large kurtosis despite not having any "tail" at all past -1. This is very unlike the tail of a power-law distribution, for instance, where once your probability density goes over that predicted from a Gaussian, it stays there. $\endgroup$ – Rex Kerr Jun 3 '15 at 18:36
  • 1
    $\begingroup$ Wellllllll, only for certain common distributions. Even those which are strictly decreasing can decrease irregularly in such a way that the ultimate shape of the tail gives the opposite answer from the kurtosis. I would say rather that kurtosis tends to indicate the fatness of the tails, just as skewness tends to indicate asymmetry. (Because, of course, you can have zero third moment in a radically asymmetric distribution.) That is, unless you define fatness as what kurtosis tells you (e.g. exponential is "fat" compared to 0.9 Gaussian + 0.1 power law). $\endgroup$ – Rex Kerr Jun 3 '15 at 19:01
  • 1
    $\begingroup$ As demonstrated in Kendall and Stuart, moment kurtosis is neither peakedness nor heavy-tailedness, though it's associated with both. As Moors (Moors JJA (1986) The meaning of kurtosis: Darlington reexamined. The American Statistician 40 (4) 283-284) points out, kurtosis is a monotonic function of Var$(Z^2)$, (where if X is the variate under consideration, $Z = (X-\mu)/\sigma$ -- a standardized variate); so kurtosis then corresponds to the variation of $X$ about $\mu\pm\sigma$. As a result, low kurtosis is sometimes referred to as having "shoulders" and high kurtosis as "lack of shoulders". $\endgroup$ – Glen_b -Reinstate Monica Jun 4 '15 at 2:39
  • 1
    $\begingroup$ Yes; and originally developed out of Yule&Kendall I believe. But anyway, in Volume I of Advanced Theory of Statistics. At least it was in 2nd and 3rd edition to my recollection; they have a table that shows that neither heavier tail nor higher peak necessarily leads to more kurtosis. It's some decades since I looked at that though, so my recollection of the specifics of their counterexamples is hazy. I'd say that the ability of the tail to impact kurtosis is generally greater than the peak, but it's more than just the tail. $\endgroup$ – Glen_b -Reinstate Monica Jun 4 '15 at 10:26
1
$\begingroup$

Sorry I am late to this thread. There have been several points of view expressed in the comments that express confusion about outliers and tails.

Rex Kerr's comment that kurtosis is not fat-tailedness is misguided. His counterexample with no outliers (and therefore, as he claims, no fat tail) is $(-1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1)$. I will convert those data to the empirical distribution $x = (-1,0,1)$, with $p(x) = (1/11, 9/11, 1/11)$ and calculate excess kurtosis $k = 2.5$.

His comment is that this example shows "large kurtosis despite not having any tail."

To shed some more light on this, let's simplify the example. Consider instead the Bernoulli distribution $x = (0,1)$, $p(x) = (1-p,p)$. There is even less in the tail of this distribution, yet kurtosis tends to infinity as $p$ tends to $0$. For example, imagine that this is a model for a belief that the moon landing was faked, with $p(x) = (.94, .06)$. I think we would all agree that the person who believes that the moon landing was faked is an "outlier." The excess kurtosis of this distribution bears that out, with $k = 11.7$.

The degree of "outlier-ness" can be characterized by $z$-score: Here the person who thinks the moon landing was faked has $z$-score $z= (1 - .06)/\sqrt{.06*.94} = 3.95$, quite a ways into the tail of the $z$-distribution.

If belief in the hoax were more rare (which apparently it is not, according to polls), such as $0.1\%$, then that person would have a $z$-score of $z= (1 - .001)/\sqrt{.001*.999} = 31.61$, which, all would agree, is an outlier: If these data were from a normal distribution, the likelihood of seeing an observation $31.61$ standard deviations from the mean or more would be so small as to be called impossible. Also, the excess kurtosis is now $k = 995$.

So, despite the fact that the normal distribution has tails that extend to infinity, the Bernoulli distribution is arguably "heavier-tailed" for small $p$ in the sense that it can produce extreme observations greatly exceeding what the normal distribution is capable of.

Kurtosis is, by definition, the expected value of the $Z$ scores, each raised to the fourth power. When you have extreme $z$-scores (outliers), you have high kurtosis.

There are infinitely measures of tail extremity. Kurtosis is a measure of tail extremity that focuses on the $z$-scores, thus, by this measure, a distribution with finite support can be heavier-tailed than one with infinite support.

This definition is perfectly logical, and quite applied. The reason we care about tails is because we care about outliers. The normal distribution simply does not produce outliers 31 standard deviations from the mean, by any practical way of thinking about it. The Bernoulli distribution, on the other hand, produces such values quite easily.

The focus on outliers is quite applied because statistical procedures of all kinds are affected by outliers. Take the variance estimate, for example: Its accuracy depends strongly on kurtosis, because the value of the estimate is strongly dependent upon whether or not outliers are in that particular sample. Power of means tests are also affected by outliers. So the interpretation of kurtosis as a measure of outliers, and not peak or center, is not only correct, it also lines up correctly with statistical applications.

Back to Rex's "counterexample," he could make more extreme by letting $x = (-1,0,1)$, $p(x) = (.001, .998, .001)$. The excess kurtosis is now $k=497$. The reason is that the $+1$ and $-1$ responses are now extreme outliers, $22.4$ standard deviations from the mean. This distribution is heavier-tailed than the normal distribution in the sense that it produces occasional values $22.4$ standard deviations from the mean.

Also, while Rex's counterexample, and my enhanced version of it suggest that higher kurtosis corresponds to a "peaked" distribution, there are easy examples where the distribution is not peaked with the same kurtosis. Take, for example, $x = (-1000, -2,-1,0,+1,+2, +1000)$, $p(x) = (.001, .25,.20, .098, .20, .25, .001)$. This distribution is "U" shaped, not peaked, and there are occasional outliers. Its excess kurtosis is $k=496$, similar to my enhanced counterexample, and the most extreme values are similarly $22.3$ standard deviations from the mean.

In summary, kurtosis does measure the tail (potential outliers) of the distribution, because it is the expected value of $Z^4$. If you have some large $Z$-values, then you have large kurtosis.

I give three mathematical theorems (for which there obviously can be no counterexamples) in my TAS article to support the connection between kurtosis and tails. To my knowledge, there are no theorems connecting kurtosis to the shape of the peak, or even to the probability content in the $\mu \pm \sigma$ range. If anyone has such a theorem, I'd love to see it.

$\endgroup$
  • 1
    $\begingroup$ If we were to understand @RexKerr's comment as motivated by the concept of a heavy tail or a long tail, then it is fully justifiable. The kurtosis may be arbitrarily large even for the lightest-tailed distributions. That doesn't look "misguided" to me. $\endgroup$ – whuber Nov 21 '17 at 16:19
  • 1
    $\begingroup$ It is misguided in the sense of why we care about tails for practical purposes, which is that we care about outliers. $\endgroup$ – Peter Westfall Nov 21 '17 at 16:25
  • 2
    $\begingroup$ Or maybe we care about power law phenomena. Or maybe about lognormal distributions. Or possibly about Student t distributions with small degrees of freedom when we're doing ANOVA with only two or three replicates. Perhaps when studying extreme events like floods and finance. Or times for random walks to recur. I'm sure you can think of other applications, too. $\endgroup$ – whuber Nov 21 '17 at 16:48
  • 1
    $\begingroup$ Thanks whuber. Outliers are indeed important. The focus that many have placed on the central portion of the distribution as regards kurtosis, rather than on the outliers, is exactly the problem that I have been trying to point out. That interpretation of kurtosis is akin to the tail wagging the dog: Yes, you can understand kurtosis as a concentration of mass well inside the outliers, but it is more sensible and practically relevant to view kurtosis as outliers relative to the central mass. After all, to define something in terms of outliers is pretty unreliable. But central mass is stable. $\endgroup$ – Peter Westfall Nov 22 '17 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.