0
$\begingroup$

So, for an experiment I analyzed behavioral data of two different treatment groups with $n = 30$ for each of two groups, so 60 subjects in total. I used a computational model to analyze the data and calculate a specific score that describes the motivation to obtain a specific reward.

This score is a continuous variable, but according to a Shapiro-Wilk test, the distribution of the data is non-normal. According to a two-sided Wilcoxon rank sum test, the difference between both groups is not significant ($p \approx .08$). However, after performing an alternative (model-free) analysis by using logistic regression, I actually found a significant effect of treatment group on behavior.

Additionally, after performing a two sided t-test with the computational data, I also found a significant difference between both groups ($p \approx .04$).

Now the information that I have are quite contradictory. Some say that a sample size of 30 is sufficient to use a t-test, even if the data is not normally distributed. Other sources say that I still should use a u-test, even for "larger" sample sizes.

So I'm a bit confused. Of course, the results of the t-test fit with my hypothesis and the model-free findings. On the other hand, I want to use the "appropriate" test, not the one that confirms my hypothesis.

What do you think? Is my sample size big enough to use a t-test? Or should I go with the u-test, even though it contradicts the results of my alternative method?

EDIT: This is how the boxplot of my data looks like (adapted to the code below) Boxplot

$\endgroup$
4
  • 4
    $\begingroup$ I am afraid that, by running so many tests in search of significance, you may have tainted your analysis. XKCD has a good cartoon along these lines. $\endgroup$
    – Dave
    Commented Dec 28, 2021 at 17:12
  • 1
    $\begingroup$ Your two p-values, .04 and .08 really aren't that much different. Both the t-test and the Wilcoxon test work fine at any sample size, though a practical minimum might be 4 or 5 observations per group for both tests. The Wilcoxon test doesn't really fix the problem of unequal variances; it works best when the the variances are equal but the distribution of the tails is higher than for the normal distribution. You might consider Welch's modification of a t-test, designed for unequal variances. You might describe the distribution of your values a little more. $\endgroup$ Commented Dec 28, 2021 at 18:34
  • $\begingroup$ The plot was made in R and R's t.test function defaults to Welch correction so chances are, that the t test result already was a Welch test result. $\endgroup$
    – Bernhard
    Commented Dec 29, 2021 at 10:55
  • $\begingroup$ Depending of the units of the x-axis it may be that a difference between means of approx. 0.6 is not worth a discussion about its statistical significance. $\endgroup$
    – cdalitz
    Commented Dec 29, 2021 at 16:28

1 Answer 1

2
$\begingroup$

You should not expect an accurate result from a two-sample t test on samples that are sufficiently far from normal to fail Shapiro-Wilk tests of normality. The P-value about 4% would be just barely significant even if accurate.

If the two samples have approximately the same shape, a Wilcoxon rank sum test might tell you whether population medians are significantly different. However, this test is not quite as powerful as a t test. In any case a P-value about 8% is not impressive evidence for a significant difference between population locations.

@Dave has a good point that you have done too many tests on the data. Cherry picking the smallest P-value of two 2-sample tests would be "P-hacking."

Consider the following fictitious data:

set.seed(1234)
x1 = rexp(30, 1/10);  x2 = rexp(30, 1/15)
mean(x1); mean(x2)
[1] 9.384906
[1] 17.75834

Means are quite different. The issue is whether the difference is statistically significant at, say, the 5% level. Boxplots show strongly right-skewed samples and apparently different dispersions.

x = c(x1,x2);  g = rep(1:2, each=30)
boxplot(x~g, horizontal=T, col="skyblue2")

enter image description here

Normal probability plots are clearly not linear, so the data should not be assumed normal. The Welch t test may or may not give useful results with sample sizes as large as $n_1=n_2 = 30.$

enter image description here

R code for figure:

par(mfrow=c(1,2))
 qqnorm(x1); qqline(x1, col="blue")
 qqnorm(x2); qqline(x2, col="blue")
par(mfrow=c(1,1))

My first (and only) test would be a permutation test using the Welch t statistic as metric. This test does not assume that data are normal, nor that the t statistic has a t distribution. It approximates the distribution of the t statistic for our data. [We look at P-values here because the Welch t test tends to have slightly different degrees of freedom at each iteration.]

pv.obs = t.test(x~g)$p.val; pv.obs
[1] 0.02797518
pv = replicate(10^5, t.test(x~sample(g))$p.val)
mean(pv <= pv.obs)
[1] 0.02633  # Sim. P-value of permutation text

So the permutation test finds a significant difference at the 3% level.

Because the boxplots show different shapes (dispersions), I would stop there.

If you want to know what the pooled t test, and Wilcoxon rank sum test would have given, here are the results. But we have done a valid test already, so these results are to satisfy curiosity, not as valid test results.

t.test(x1,x2, var.eq=T)$p.val # Pooled
[1] 0.02622975
wilcox.test(x1,x2)$p.val      # Wilcoxon SR
[1] 0.2358858

Note: My fictitious right-skewed data for this Answer were sampled from exponential populations. If you know that data are exponential, then there is an exact test. See this Q&A, where it is stated that means of two independent exponential samples, each of size $n$ have $\frac{\bar X_1}{\bar X_2} \sim \mathsf{F}(2n,2n).$ So, for our data with $\bar X_2 > \bar X_1,$ the P-value of an exact 2-sided test is $0.015.$

f = mean(x2)/mean(x1); f
[1] 1.892223
2*(1 - pf(f, 60, 60))
[1] 0.01470998
$\endgroup$
8
  • 1
    $\begingroup$ We don't actually care whether the samples are Normally distributed, only whether the sample mean (difference, in this case) is close to Normally distributed, because it is what is being tested. For example, a sample mean from 12 Uniform variates was at one time close enough to Normally distributed to be used as a random number generator... If I generate two sets of size 30 iid Exponential variates and take their difference, the t-statistic for testing that the mean $\neq 0$ (in my run of 100K experiments) rejected H0 5.023% of the time. $\endgroup$
    – jbowman
    Commented Dec 28, 2021 at 23:14
  • 2
    $\begingroup$ For normal data the sample mean $\bar X$ and the sample SD $S$ are independent. So, with non-normal data it is not quite enough to have $\bar X$ approx normal. The t statistic can fail to have a t dist'n because of correlation of it's numerator and denominator. A t test may have 5% rej. rate for exponentials with equal population means, yet have poor power because num. and denom. of t stat are highly correlated. Of course, the exact F-test behaves better. For my fictitious data Welch t rejected with P-val 0.027 and the exact test with P-val 0.015. Robustness of t is '"legendary," not ideal $\endgroup$
    – BruceET
    Commented Dec 29, 2021 at 7:52
  • 1
    $\begingroup$ @jbowman "We don't care whether the samples are Normally distributed, only whether the sample mean is close to Normally distributed" - not quite true. If you have a certain percentage of outliers but finite variance in the underlying distribution, the mean will be asymptotically normal about the distribution mean, but the distribution mean will be affected by outliers and therefore usually not be a sensible estimation target. $\endgroup$ Commented Dec 29, 2021 at 9:36
  • 1
    $\begingroup$ There are fundamental problems with the way the issues were set up. Testing for normality is bad statistical practice. The Wilcoxon test works fine if the data are normality distributed. Running multiple tests of a hypothesis is a no-no. Logistic regression is not model-free but is a viable way to test group differences. $\endgroup$ Commented Dec 29, 2021 at 12:46
  • 1
    $\begingroup$ @BruceET - oh, sure, if you can construct an exact test that's going to be better. My point was with specific reference to the statement about the Normality of the data, but as you observe the robustness of the t-test at the null hypothesis is no guarantee of near-optimal power. $\endgroup$
    – jbowman
    Commented Dec 29, 2021 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.