# Why is high positive kurtosis problematic for hypothesis tests?

I've heard (sorry cannot provide a link to a text, something I have been told) that a high positive kurtosis of residuals can be problematic for accurate hypothesis tests and confidence intervals (and therefore problems with statistical inference). Is this true and, if so, why? Would a high positive kurtosis of residuals not indicate that the majority of the residuals are near the residual mean of 0 and therefore less large residuals are present? (If you have an answer, please try to give an answer with not much indepth mathematics as I'm not highly mathematically inclined).

• I am guessing that you are focusing on models with ideal conditions of normal (Gaussian) error terms. (In many other contexts, high kurtosis of residuals could well be expected.) High kurtosis is most likely to imply a distribution fatter tailed than the normal, so some very high (+ or -) residuals. Even if there are many near zero, that is only the good news, and it is the possible bad news that needs attention. But in turn that could mean anything any number of things. A residual versus fitted plot is usually more informative. Jan 29, 2016 at 18:41
• Indeed, I was focusing on models with normality assumptions.
– DDK
Feb 1, 2016 at 8:59

heard [...] that a high positive kurtosis of residuals can be problematic for accurate hypothesis tests and confidence intervals (and therefore problems with statistical inference). Is this true and, if so, why?

For some kinds of hypothesis test, it's true.

Would a high positive kurtosis of residuals not indicate that the majority of the residuals are near the residual mean of 0 and therefore less large residuals are present?

No.

It looks like you're conflating the concept of variance with that of kurtosis. If the variance were smaller, then a tendency to more small residuals and fewer large residuals would come together. Imagine we hold the standard deviation constant while we change the kurtosis (so we're definitely talking about changes to kurtosis rather than to variance).

Compare different variances (but the same kurtosis):

with different kurtosis but the same variance:

(images from this post)

A high kurtosis is in many cases associated with more small deviations from the mean$$^\ddagger$$ -- more small residuals than you'd find with a normal distribution .. but to keep the standard deviation at the same value, we must also have more big residuals (because having more small residuals would make the typical distance from the mean smaller). To get more of both the big residuals and small residuals, you will have fewer "typical sized" residuals -- those about one standard deviation away from the mean.

$$\ddagger$$ it depends on how you define "smallness"; you can't simply add lots of large residuals and hold variance constant, you need something to compensate for it -- but for some given measure of "small" you can find ways to increase the kurtosis without increasing that particular measure. (For example, higher kurtosis doesn't automatically imply a higher peak as such)

A higher kurtosis tends to go with more large residuals, even when you hold the variance constant.

[Further, in some cases, the concentration of small residuals may actually lead to more of a problem than the additional fraction of the largest residuals -- depending on what things you're looking at.]

Anyway, let's look at an example. Consider a one-sample t-test and a sample size of 10.

If we reject the null hypothesis when the absolute value of the t-statistic is bigger than 2.262, then when the observations are independent, identically distributed from a normal distribution, and the hypothesized mean is the true population mean, we'll reject the null hypothesis 5% of the time.

Consider a particular distribution with substantially higher kurtosis than the normal: 75% of our population have their values drawn from a normal distribution and the remaining 25% have their values drawn from a normal distribution with standard deviation 50 times as large.

If I calculated correctly, this corresponds to a kurtosis of 12 (an excess kurtosis of 9). The resulting distribution is much more peaked than the normal and has heavy tails. The density is compared with the normal density below -- you can see the higher peak, but you can't really see the heavier tail in the left image, so I also plotted the logarithm of the densities, which stretches out the lower part of the image and compresses the top, making it easier to see both the peak and the tails.

The actual significance level for this distribution if you carry out a "5%" one-sample t-test with $$n=10$$ is below 0.9%. This is pretty dramatic, and pulls down the power curve quite substantially.

(You'll also see a substantive effect on the coverage of confidence intervals.)

Note that a different distribution with the same kurtosis as that will have a different impact on the significance level.

So why does the rejection rate go down? It's because the heavier tail leads to a few large outliers, which has slightly larger impact on the standard deviation than it does on the mean; this impacts the t-statistic because it leads to more t-values between -1 and 1, in the process reducing the proportion of values in the critical region.

If you take a sample that looks pretty consistent with having come from a normal distribution whose mean is just far enough above the hypothesized mean that it's significant, and then you take the observation furthest above the mean and pull it even further away (that is, make the mean even larger than under $$H_0$$), you actually make the t-statistic smaller.

Let me show you. Here's a sample of size 10:

 1.13 1.68 2.02 2.30 2.56 2.80 3.06 3.34 3.68 4.23


Imagine we want to test it against $$H_0: \mu=2$$ (a one-sample t-test). It turns out that the sample mean here is 2.68 and the sample standard deviation is 0.9424. You get a t-statistic of 2.282 -- just in the rejection region for a 5% test (p-value of 0.0484).

Now make that largest value 50:

      1.13 1.68 2.02 2.30 2.56 2.80 3.06 3.34 3.68 50


Clearly we pull the mean up, so it should indicate a difference even more than it did before, right? Well, no, it doesn't. The t-statistic goes down. It is now 1.106, and the p-value is quite large (close to 30%). What happened? Well, we did pull the mean up (to 7.257), but the standard deviation shot up over 15.

Standard deviations are a bit more sensitive to outliers than means are -- when you put in an outlier, you tend to push the one-sample t-statistic toward 1 or -1.

If there's a chance of several outliers, much the same happens only they can sometimes be on opposite sides (in which case the standard deviation is even more inflated while the impact on the mean is reduced compared to one outlier), so the t-statistic tends to move closer to 0.

Similar stuff goes on with a number of other common tests that assume normality -- higher kurtosis tends to be associated with heavier tails, which means more outliers, which means that standard deviations get inflated relative to means and so differences you want to pick up tend to get "swamped" by the impact of the outliers on the test. That is, low power.

• Wow, thanks so much for the very clear and elaborate answer. Your time is much appreciated!
– DDK
Feb 1, 2016 at 7:54
• It is also worth noting that, while the large-sample distribution of the sample mean does not depend on kurtosis (hence, the actual significance level of normality-assuming tests for means converges to the nominal level, typically .05, as n-> infinity, for all finite kurtosis), the same is not true for tests for variances. The large-sample distribution of the estimated variance depends on the kurtosis, so the actual significance level of classic, normality-assuming tests for variance does not converge to the nominal level as n -> infinity when the kurtosis is different from zero. Nov 10, 2017 at 0:26
• Also, higher kurtosis does not imply, mathematically, that there are "more small deviations from the mean." The only thing it tells you for sure is that there is more in the tail. Nov 10, 2017 at 0:31
• You cannot get more large deviations and hold the variance constant unless you also make more small deviations; if you don't hold the variance constant, more of your deviations become small relative to the new scale. So yes, when it comes to looking at kurtosis, mathematics does tell you that more large carries with it more small. Nov 10, 2017 at 0:49
• @Peter Let's take $Z$ as a standardized $X$. Kurtosis is $\kappa=E(Z^4)$, and $\sqrt{\kappa-1}=E(Z^2)$ is monotonic in $\kappa$. If I move probability further into the tail of $Z$, some probability must move toward the mean (or I can't hold $\text{Var}(Z)=1$). Similarly if I move probability further into the tail of $X$ & let the variance increase, $\mu\pm k\sigma$ is wider, and so for at least some values of $k$ more of the rest of the distribution will tend to fall inside those bounds; once you standardize the new $X$ ($X'$ to $Z'$ say), you have more smaller values in that direct sense. Nov 10, 2017 at 1:14

Kurtosis measures outliers. Outliers are problematic for the standard inferences (e.g., t-tests, t-intervals) that are based on the normal distribution. That's the end of story! And it's really a pretty simple story.

The reason this story is not well appreciated is because the ancient myth that kurtosis measures "peakedness" persists.

Here is a simple explanation showing why kurtosis measures outliers and not "peakedness".

Consider the following data set.

0, 3, 4, 1, 2, 3, 0, 2, 1, 3, 2, 0, 2, 2, 3, 2, 5, 2, 3, 1

Kurtosis is the expected value of the (z-values)^4. Here are the (z-values)^4:

6.51, 0.30, 5.33, 0.45, 0.00, 0.30, 6.51, 0.00, 0.45, 0.30, 0.00, 6.51, 0.00, 0.00, 0.30, 0.00, 27.90, 0.00, 0.30, 0.45

The average is 2.78, and that is an estimate of the kurtosis. (Subtract 3 if you want excess kurtosis.)

Now, replace the last data value with 999 so it becomes an outlier:

0, 3, 4, 1, 2, 3, 0, 2, 1, 3, 2, 0, 2, 2, 3, 2, 5, 2, 3, 999

Now, here are the (z-values)^4:

0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00,0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 360.98

The average is 18.05, and that is an estimate of the kurtosis. (Subtract 3 if you want excess kurtosis.)

Clearly, only the outlier(s) matter. Nothing about the "peak" or the data near the middle matters.

If you perform standard statistical analyses with the second data set, you should expect trouble. The large kurtosis alerts you to the problem.

Here is a paper that elaborates:

Westfall, P.H. (2014). Kurtosis as Peakedness, 1905 – 2014. R.I.P. The American Statistician, 68, 191–195.

• Why not just use nonparametric tests? For these types of problems they are likely to be superior.
– Carl
Jan 1, 2018 at 0:57
• Agreed, that is a possible avenue, IF you like testing, which is rapidly becoming less interesting in its classic form. But that is not really my concern. I am more interested in probabilistic modeling in general. One application: Maybe you really are interested in the mean, eg, in cases where the dependent variable is dollars earned, the process mean is more interesting than the process median. So, what do the data tell you about the process mean when the data are outlier-prone? It's a difficult problem, but an important one, and moment kurtosis is relevant to the answer. Not nonpar tests. Jan 7, 2018 at 23:07
• For the Cauchy distribution, the trimmed mean can be a better measure of the location than the median, and the ordinary mean would be not be a measure of location. What to use as a measure of location is dependent upon what the distribution is. An example for which kurtosis would not be helpful as an indicator is the uniform distribution for which the average extreme value is a better measure of location than both the median and the mean.
– Carl
Jan 8, 2018 at 2:57
• Not the point. If you are interested in totals, e.g., dollars, then the ordinary mean is the measure of location you want. Mar 4, 2018 at 0:20
• If you have a Cauchy distributed variable, you can make a case for total dollars earned, but the mean will not be an especially useful measure of location meaning that the "expected value" has no reasonable expectation associated with it.
– Carl
Mar 4, 2018 at 0:27