I'm wondering if t test maybe used for a really small sample size. I have a set of data with only 3 entries for each group and I need to compare whether the two groups are significantly different.
1 Answer
The permutation test will have insufficient power. (There just aren't enough different ways to split six samples into two groups of three.) But If the assumptions of the t-test hold, then its results are valid.
Many thoughtful readers will question whether such a situation could actually arise. Let me share a real story. It concerns cleaning up lead contamination in a field: for years, a farmer accepted "recycled" batteries and dumped them behind his house. Eventually the environmental regulators caught up with him. They caused the "responsible party" to go through three phases of cleanup work: (1) sample the soils to estimate the amount and extent of lead contamination; (2) remove the soils in thin layers, taking samples in the process, until it was clear that clean soils had been reached; (3) independently sample all remaining soil and formally test whether the mean lead concentration is below the environmental standard.
The procedure for (3) was designed and approved before the cleanup began. It called for random sampling of all soils exposed during the excavation, analysis of the samples by a certified laboratory, and applying a Student t test. Equivalently, to demonstrate success, a suitable upper confidence limit (UCL) of the mean had to be less than the standard. It did not specify how many samples to take: that would be up to the responsible party to decide.
Almost a thousand samples were obtained and analyzed during the first two phases. Although these allowed the (univariate and spatial) distributions of the lead concentrations to be characterized reliably, they of course did not represent the remaining concentrations. However, they did suggest the shape of the (univariate) distribution of the remaining concentrations. Physical and chemical theory, soils science, and experience with remediating lead in soils elsewhere all provided support for this statistical characterization.
The cleanup was so thorough and successful that the likely mean concentration was negligible--more than an order of magnitude less than the standard. Power analyses, based on pessimistic (high) estimates of the standard deviation, all suggested that a random sample of only two or three would be needed.
There were many potential complications: for instance, any areas that might have been overlooked during the excavation could introduce large outlying values. To detect these, a large number of samples were obtained at random locations, and then composited in groups to produce just five physical samples for laboratory testing. All the values were low. As expected, a t-test of any two of those samples would still have demonstrated attainment.
Sprinkled within this brief case study are examples of various ways in which we might be assured that a t-test is appropriate even with tiny samples: experience; theory; preliminary related sampling; making pessimistic assumptions; and sample compositing all played a role--and any one of them might have sufficed to justify the t-test.
Incidentally, there are versions of the t-test that work with just a single observation. They are based on obtaining independent estimates of the variance or, lacking that, mathematical theory. Could this ever make sense in reality? The classic situation of compositing the blood from hundreds of soldiers to test for venereal disease provides one possible application.
t.test(x=c(4.5,4.6),y=5.7, var.equal=TRUE)
(sample sizes 2 and 1). It works. However, power may be quite low. $\endgroup$