# How to choose between Pooled and Satterthwaite methods in t test for 2 small samples with marginally different variances?

I compare means (or medians) of the numeric variable X for two groups composed of 6 and 18 subjects (using SAS). How do I choose between Pooled and Satterthwaite methods if the F-test for Equality of Variances is 0,046, i.e., marginally significant while using the Pooled method the p-value is 0.056 and in the case of Satterthwaite it is 0.007, i.e. they are very different.

And is the t-test the right test to use with this data taking into consideration that Kolmogorov-Smirnov and other criteria for normality of the distribution of the of 6 subjects in the first group show p-values around 0.04?

I also performed the Mann-Whitney U-test and it gave p=0.07 (Wilcoxon) and p=0.053 (Kruskal). But the shape of histograms for two groups looks quite different, so my concern is that it means that one of the assumptions of Mann-Whitney U test is also violated. However, may I conclude from all these tests that the p-value is above 0.5? Or is it possible that the p-value is actually < 0.05?

The data is the following.

Group 1: 0.35894, 0.38882, 0.36004, 0.37289, 0.54223, 0.50994.
Group 2: 0.44332, 0.82616, 0.65995, 0.30866, 0.36630, 0.31708, 0.83976, 0.75948, 0.45998, 0.62960, 0.87457, 0.57932, 0.68573, 0.54461, 0.76902, 0.39784, 0.99413, 0.38794.

• My guess is that you have too little data for any test to give convincing results. But I can't be sure because I can't exactly follow what you say about test results. And I can't reconcile your histograms with what you say about the data. // If you have two small datasets, you might get a more coherent answer by posting the data, and letting us try to decide what to do based on data instead of your results from unknown software. Nov 25 '20 at 16:50
• Bruce, many thanks! I just added the data.
– Mark
Nov 25 '20 at 17:51
• It is my belief that the assumption of equal variance is pretty much always wrong, so use the Satterthwaite method unless you have reason to believe the two groups have equal variance (such as performing a simulation where you know the variance). The fact that the default in R is to use the Satterthwaite approximation makes me confident that my approach is a good one.
– Dave
Nov 25 '20 at 19:08
• @Dave: Totally agree. Nov 25 '20 at 19:27
• @Dave: Another where equal variance is reasonable is when you have a randomized experiment, see stats.stackexchange.com/questions/434928/… Nov 25 '20 at 21:37

Thanks for providing the data. I will jump ahead and say that I'd be uncomfortable saying, at the 5% level, that the two samples come from populations with different locations. At the 6% level, Yes. If you intended a one-sided test that population 2 is centered above population 1, Yes at the 5% level.

So this is a case where there is not enough data to claim significant difference of a two-sided test at the 5% level. The main difficulties are that x1 is so small and has four "low" values along with two "high" ones: the location of the center of its population is unclear.

In such situations, the principled course of action is no admit that there is not quite enough data for a convincing result. If the issue at hand is of great importance, perhaps there is enough evidence of a possible difference to warrant a second study, perhaps with more observations.

A less-principled course of action is to 'shop around' among borderline plausible tests until you find one that gives a P-value barely below 5%. This is a path that has led to many published "false discoveries" that no one has been able to replicate with an appropriately powered subsequent study.

x1 =  c(0.35894, 0.38882, 0.36004, 0.37289, 0.54223, 0.50994)
x2 =  c(0.44332, 0.82616, 0.65995, 0.30866, 0.36630, 0.31708, 0.83976, 0.75948, 0.45998,
0.62960, 0.87457, 0.57932, 0.68573, 0.54461, 0.76902, 0.39784, 0.99413, 0.38794)


Examining the data, I notice that even though x1 has only six observations, it fails a Shapiro-Wilk test of normality with P-value around 3.6%.

shapiro.test(x1)$p.val [1] 0.03581921  One should be wary of using t tests (pooled or Welch) with such a small non-normal dataset. That might suggest using a nonparametric 2-sample Wilcoxon (rank sum) test. But an assumption of a straightforward use of this test is that the two samples have similar 'shapes', which includes having roughly equal variances. However, your x1 obviously has a much smaller variance than does you x2. stripchart(list(x1,x2), ylim=c(.5,.2.5), pch="|")  So we can use the Wilcoxon signed rank test to check for 'stochastic dominance' but not specifically for unequal medians. At the 6% level one can say that one population dominates the other. At the 3% level one can say that x2 has significantly larger values than x1. wilcox.test(x1,x2)$p.val
[1] 0.05585604


This dominance is illustrated by empirical CDFs (ECDFs) of the two samples: The plot for x2 lies (mostly) below and to the right of the plot for x1.

plot(ecdf(x2), col="brown", main="ECDF Plots of X1 (blue) and X2 (brown)")
lines(ecdf(x1), col="blue")


Other than for 'P-hacking', I find no reason to use a Welch two-sample t test, on account of the (aggressive) non-normality of x, mentioned above. However, for full disclosure, here are its (significant) results:

t.test(x1,x2)

Welch Two Sample t-test

data:  x1 and x2
t = -3.0067, df = 21.159, p-value = 0.00668
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-0.30489818 -0.05564293
sample estimates:
mean of x mean of y
0.4221433 0.6024139


If one tries to argue that the population that produce x1 might be normal after all, then the issue is whether it's normal with mean around 3.7 or around 5.2 (and with what variance?), and what results those scenarios might yield. We simply don't know much about population 1.

It would be difficult to argue that any one analysis of such a dataset is the correct one, and there are enough issues with your data that others may recommend alternative analyses. Possibly, different analyses may appear on this page. However, I would not want to claim a significant difference between the two populations at the 5% level based on your data.

• Many thanks Bruce for such a detailed answer! Very useful! You said there are not enough evidence to stay that p=<0.05. But that it is possible to state that p=<0.06. While I agree with you intuitively I’m wondering that they may say there are not enough evidence even to state that p=<0.06 because from formal point of view none of these methods which were used for calculation of p-value is applicable due to violations of some assumptions about data. Regarding the last line of code. You probably meant mean(pv <= 0.05). BTW for pv<=0.0067 we get 50% because of normality data you suggested.
– Mark
Nov 26 '20 at 7:57

It is my belief that the assumption of equal variance is pretty much always wrong, so use the Satterthwaite method unless you have reason to believe the two groups have equal variance (such as performing a simulation where you know the variance). The fact that the default in R is to use the Satterthwaite approximation makes me confident that my approach is a good one. Watch out for this in the software you use, because R might be unusual in this way (though I think they made the right decision).

• Dave, thank you! If I choose Satterthwaite method the p value for the difference of means will be 0.007, i.e. very small, specially in comparison with alternaive Pooled method which gives p =0.056. I need strong arguments for such a strong decision. I tried to find support from other approaches -- I used Mann-Whitney U-test, regression of the measurement on the group, logistic regression with crossvalidation but all these methods which are probably also not perfect for these data produced p-values in the interval 0.5-0.7. I appreciate your comment very much!
– Mark
Nov 25 '20 at 21:01