Test two groups with only sample statistics or without distributional assumptions

Question

I have two sets of samples, A and B. I want to find whether the underlying mean (i.e. if the sample size was infinite) of A is greater than that of B, to a certain confidence (95%).

There are two scenarios for this:

1. All I know is the mean, standard deviation and sample size for both. For A, these are (0.73, 1.34, 30), and for B, these are (0.67, 1.21, 30). I then make the assumption that samples are normally-distributed.

2. I have the values for all the individual samples, and make no assumption about the underlying distribution.

How can I do the hypothesis test for each of these two scenarios?

Answer 1

1) You'd use a t-test. In this case sample sizes are identical, so there's not even a need to worry about Welch adjustment; the ordinary t-test would do.

2) That depends; if you're prepared to assume identical distributions up to a location shift, then if population means are finite you could certainly use a Wilcoxon-Mann-Whitney for that. However, it's considerably more broadly applicable for a test of equality of means than that*. [One advantage of restriction to a location-shift alternative is interpretability, but it's not necessary for the test to be suitable. Scale shift alternatives - and a host of other transformations that imply ordered means - would all be included]

If you assume only identical distributions under the null (no need to restrict it to location shift alternative or even to a stochastic-dominance alternative), you could do a permutation test on the difference in means. (There are still other options though)

What software are you using?

---

* Here's an illustration that it can be broader and still be about means.

(i) Assume $X_i$, $Y_j$ are both continuous from $F_X$ and $F_Y$ respectively (each with finite mean), with the usual independence assumptions and assume for this argument that both variables are non-negative. Let $S_X=1-F_X$. Let's take our hypotheses to be

$H_0: F_X=F_Y$ vs $H_1: F_X<F_Y$

Then that's equivalent to

$H_0: S_X=S_Y$ vs $H_1: S_X>S_Y$

Integrate the expressions on both sides on the positive half line and we get:

$H_0: E(X)=E(Y)$ vs $H_1: E(X)>E(Y)$

as long as $\int_0^\infty S_X(t)-S_Y(t) dt >0$ and we can interchange the integration and the difference.

[This sort of argument can be easily extended to a situation with a lower bound on the variables other than zero.]

(ii) the restriction to positive random variables was a convenience in order to be able to bring in the argument about the relation between survival functions and expectation; it's not necessary. For example, second order stochastic dominance implies the expectations are ordered.

I'm not currently clear on what the weakest restrictions (on top of the assumptions already made for the WMW) above first order stochastic dominance would need to be to imply ordering of expectations, but it's whatever minimum assumption is needed for $F_X-F_Y<0$ in some region to imply $E(X)-E(Y)>0$. (I think if the inequality $F_X-F_Y\leq 0$ is somewhere strict in such a way that their integrals differ in that region, it should be possible to show that $E(X)-E(Y)>0$ follows.)

I hope to come back to this and say some more once I've thought about it (and maybe learned some more). The implication is that the Wilcoxon-Mann-Whitney is useful for testing means under much less restrictive assumptions than are often suggested.

As gung points out, we still need some assumptions. The Wilcoxon-Mann-Whitney is sensitive to an even broader set of alternatives than I'm discussing here, so without additional assumptions, rejection doesn't necessarily suggest a difference in means.

Answer 2

To clarify on @Glen_b's first point, we typically think of calculating the $t$-test by feeding the data into some software (the wide availability of advanced, easy to use software has spoiled people, the kids today...). But in the old days, we ran $t$-tests by hand (after walking twelve miles uphill through the driving snow), and to do that you would first calculate the means and SDs, and then enter those into the formula.
$$ t = \frac{\bar x_2 - \bar x_1 - (\rm null\ difference)}{\sqrt{\frac{s_p^2}{n_1}+\frac{s_p^2}{n_2}}} $$ where, $$ s_p = \sqrt{\frac{(n_1-1)s_1 + (n_2-1)s_2}{n_1+n_2-2}} $$ If you want to use software, there are often options that let you input sample statistics instead of data. For example, in R you could use ?tsum.test in the BSDA package.

library(BSDA)
tsum.test(mean.x=.73, s.x=1.34, n.x=30,
          mean.y=.67, s.y=1.21, n.y=30, alternative="greater", var.equal=TRUE)
#         Standard Two-Sample t-Test
# 
# data:  Summarized x and y
# t = 0.182, df = 58, p-value = 0.4281
# alternative hypothesis: true difference in means is greater than 0
# 95 percent confidence interval:
#  -0.4909958         NA
# sample estimates:
# mean of x mean of y 
#      0.73      0.67 

Note that you could not combine your two scenarios. That is, if you only have sample statistics, you must assume that the data are normally distributed to perform a $t$-test.