# Statistical power of t-test in mildly skewed dataset

I'm trying to learn when I can use t-test if data set is not normal "enough".

Here are the things that I know, please verify:

1. T-test is still robust when data is mildly skewed and light tailed.
2. Mann-Whitney test for medians is a better choice when data has high skewness, or heavy tailed
3. T-test can still be used even with high skewness, and heavy tails, as long as you have large sample sizes. But it has less statistical power than Mann-Whitney test.

From here, they say:

It is helpful to note that as the sample size n increases, the T ratio: $$T = \frac{\bar{X} - \mu}{s/\sqrt{N}}$$ approaches an approximate normal distribution regardless of the distribution of the original data.

Question 1: How does the equation for $$T$$ approach normal distribution?

Question 2: What is the "statistical power" that people talk about? Does it give information about how many sample sizes that I need to have a meaningful result from t-test?

Question 3: Is there any way to determine sample size needed for t-test? Like, if you have skewness of $$x$$ and kurtosis of $$y$$, you need $$n$$ sample size for the result from t-test to be valid, even when your data is non-normal

• If assumptions for a t test are met, then you can use the noncentral t distribution to find the power of a t test against various alternatives. When assumptions are not met, power formulas may be difficult or impossible to find, but usually it is possible to do a simulation to approximate the power against a specific alternative. – BruceET Jul 5 '19 at 1:27
• About (3): Wilcoxon signed rank test is appropriate for one-sample tests. Because this test uses ranks, mild skewness is not a problem. In many cases, loss of power (compared with t test) is small. See Addendum to my Answer. – BruceET Jul 5 '19 at 16:31

I will address the computation of the power of a one-sample t test. Suppose we wish to use $$n = 20$$ observations from a normal distribution to test $$H_0: \mu = 110$$ against $$H_a: \mu < 110$$ at the 5% level. Then we will reject $$H_0$$ when the t statistic $$T = \frac{\bar X - 110}{S/\sqrt{20}} < -1.729,$$ where $$S$$ is the sample standard deviation and $$-1.729$$ cuts probability $$0.05$$ from the lower tail of the distribution $$\mathsf{T}(\nu = n-1 = 19).$$ [Computation in R.]

c = qt(.05, 19);  c
 -1.729133


In order to do a power computation, we need to make a guess at the unknown sample standard deviation $$\sigma$$ and to choose a particular value $$\mu_a < \mu_0 = 100$$ for the computation. If we use $$\sigma = 15$$ and $$\mu_a = 100,$$ then we can run a simulation based on many samples of size $$n = 10$$ from $$\mathsf{Norm}(\mu_a = 100, \sigma = 15)$$ and see in what proportion of the samples we reject with $$T < c = -1.729.$$

set.seed(1776);  m=10^6; n=20; mu.0=110; mu.a=100; sg=15
t = replicate(m,
t.test(rnorm(n, mu.a, sg), mu = mu.0, alt="less")$stat) mean(t <= c)  0.890277  So the power is about 89%. Obviously, in this simulation the t statistics computed by t.test and captured using $-notation do not have Student's t distribution with 19 degrees of freedom.

lbl = "Simulated Alternative Dist'n of T"
hist(t, prob=T, br=30, col="skyblue2", main=lbl)
abline(v = -1.728, col="red", lwd=2, lty="dotted") The actual distribution of the t statistic $$T$$ is a noncentral t distribution with 19 degrees of freedom and noncentrality parameter $$\lambda = \sqrt{n}(\mu_0 - \mu_a)/\sigma = -2.9814,$$ the density function of which is plotted above.

lam = sqrt(20)*(-10)/15 = lam
 -2.981424


This means that we can find the exact power $$0.8902$$ in R, without simulation, using the code below:

pt(c, 19, lam)
 0.8902459


Thus by using the noncentral t distribution, you can make a power curve, showing the power against a sequence of alternative values $$\mu_a.$$ Also, by trying various sample sizes, you can find the $$n$$ required to achieve the desired power against a particular alternative.

However, if your data are not normal, then neither the regular nor the noncentral t distribution is applicable. It may be difficult to find a formula for the exact power. Nevertheless, you can use the simulation method with appropriate distributions to find approximate power. Similar simulation methods could be used to investigate the power of a nonparametric test.

Addendum. (1) Using a Wilcoxon signed rank test instead of a t test with 20 observations from a normal population.

Suppose you worry that normal data are not normal and use a one-sample test instead of a t test. What happens to the power? We use the same null and alternative hypotheses above, and seek power against the alternative that the distribution is centered at 100.

We don't seek a formula in terms of the distribution of the Wilcoxon test statistic, so we use simulation. Specifically, we use the implementation of the Wilcoxon test in R, capture its P-value at each iteration, and express rejection in terms of P-values below $$0.05.$$ The power is 87.4%, compared with power 89.0% for the t test.

set.seed(2019)
pv = replicate(10^6,
wilcox.test(rnorm(20,100,15), mu=110, alt="l")$p.val) mean(pv < .05)  0.874229  (2) Using a Wilcoxon signed rank test instead of a t test when 20 observations are from a moderately right-skewed distribution. A slightly, but noticeably, skewed distribution results from taking the third power of a normal sample with positive elements. Very roughly speaking, the cube of observations from $$\mathsf{Norm}(\mu=4.63, \sigma=0.232)$$ has $$\mu \approx 100, \sigma \approx 15.$$ Also the cube of observations from $$\mathsf{Norm}(4.783, .0.217)$$ has $$\mu \approx 110, \sigma \approx 15.$$ The following simulation illustrates that with $$H_0: \mu = 110$$ and $$H_a: \mu < 100$$ a t.test at the 5% level, using such slightly skewed data, has power about 88% against the alternative $$\mu_a = 100.$$ Similarly, a Wilcoxon signed rank test also has power about 88%. So for 20 observations from the moderately skewed population mentioned above, the t test loses its slight power advantage over the Wilcoxon test. set.seed(705) pv = replicate(10^6, t.test(rnorm(20,4.63,.232)^3, mu=110, alt="l")$p.val)
mean(pv < .05)
 0.877868

set.seed(705)
pv = replicate(10^6,
wilcox.test(rnorm(20,4.63,.232)^3, mu=110, alt="l")$p.val) mean(pv < .05)  0.877091  The first iteration in each simulation is shown below: set.seed(705) x = rnorm(20, 4.63, .232)^3 ; mean(x); sd(x)  104.0221  15.34042 t.test(x, mu=110, alt="less") One Sample t-test data: x t = -1.7427, df = 19, p-value = 0.04877 alternative hypothesis: true mean is less than 110 95 percent confidence interval: -Inf 109.9534 sample estimates: mean of x 104.0221 wilcox.test(x, mu=110, alt="less") Wilcoxon signed rank test data: x V = 61, p-value = 0.0527 alternative hypothesis: true location is less than 110  • (+1) The cube of observations of a$N(4.63, 0.732)$rv have mean 106.7 and an sd of 49.4. To get a mean of 100 and an sd of 15 for the cubed observations, you'd need$N(4.64, 0.232)$. Probably just a typo. – COOLSerdash Jul 5 '19 at 21:37 • Thanks for catching this! I had > x = rnorm(10^6, 4.63, .232)^3; mean(x); sd(x), which returns 100.005 and 14.9921, but mistyped from my bad handrwiting. Fortunately I got it right in the simulation code. Made correction in text. – BruceET Jul 5 '19 at 22:38 • You give$\mu = 110$and$\mu < 110$in the first paragraph. Should it be$100$for both? – AkselA Jul 6 '19 at 13:52 • No, but there was a typo in the$T$-statistic just below that. Now fixed. Thanks. – BruceET Jul 6 '19 at 17:28 • (1) Judgment that it's not a t distribution based of its obviously asymmetrical shape. But that's no surprise, because dist'n of t stat is noncentral (not ordinatry) t when alternative is true. (2) What is 'good' power depends on how much risk you can bear of not rejecting when$H_0\$ false. Also on budget: higher powers require larger sample sizes. (3) Hard to compare 95% confidence with 80% power. Roughly, more data leads to shorter CIs and higher powers. // Some people test hypotheses by looking at CIs: If null value not included in 95% CI, then Reject null hypothesis at level 5%. – BruceET Jul 7 '19 at 4:22

This has been discussed at length on this site. The t-test is not very robust to skewness. For example, with the log-normal distribution a sample size of 50,000 is not large enough for the t-based method to be sufficiently accurate. The Wilcoxon signed-rank one-sample test does not test a median. The Wilcoxon-Mann-Whitney test is a two-sample test. The Wilcoxon tests are 0.95 as efficient as the t-based methods if normality holds, and can be arbitrarily more powerful than this parametric counterpart when normality does not hold. I suggest reading an intro nonparametric statistics book.

• I'm aware that t-test is not robust to skewness, because they were discussed at length on this site, and I read a lot of them. But again, it all depends on how bad the skewness is. If skewness is mild, it can be overcome by large sample size (or so I read). The question is, how do I know what "mild" is? And how many sample size do I need to overcome that? Can the needed sample size be quantified in context of statistical power? – Eric Kim Jul 5 '19 at 16:21
• Everyone says it "depends on how bad the skewness is", but no body suggest an way to define what "mild skew vs bad skew" quantitatively. As you said, with log-normal distribution, sample size of 50,000 is not enough because its completely asymmetric, but what if skewness is small? Then 50,000 is certainly good enough (though not perfect) – Eric Kim Jul 5 '19 at 16:29
• I would use methods that don't care, staying away when possible from using any method that assumes the standard deviation is a good measure of dispersion. SD assumes symmetry. – Frank Harrell Jul 5 '19 at 20:05

How does the equation for 𝑇 approach normal distribution?

Well, that it isn't quite true. The sampling distribution for that statistic approaches a standard normal in the limit as n grows large. That is very different. T

What is the "statistical power" that people talk about? Does it give information about how many sample sizes that I need to have a meaningful result from t-test?

Power is the probability of correctly rejecting the null hypothesis when it is false. If there really is a difference between groups, then a high powered test will correctly identify that there is a difference upon repetition of the experiment.

Power is an essential part of a sample size calculation. The sample size depends on a few other things (namely the effect size and the noise in the data), but generally speaking more power requires more samples (all else constant).

Is there any way to determine sample size needed for t-test? Like, if you have skewness of 𝑥 and kurtosis of 𝑦, you need 𝑛 sample size for the result from t-test to be valid, even when your data is non-normal

Not to my knowledge. The equations for sample sizes lean heavily on the assumptions of the t-test, and so I don't think there are formulae which make use of higher moments.