5
$\begingroup$

I'm wondering if t test maybe used for a really small sample size. I have a set of data with only 3 entries for each group and I need to compare whether the two groups are significantly different.

$\endgroup$
  • $\begingroup$ With three cases per group I wouldn't be conducting any test, but rather conduct a qualitative analysis (i.e. describe the groups and the differences in your report, case by case). Check: stats.stackexchange.com/questions/121852/… $\endgroup$ – Tim Sep 21 '17 at 10:09
  • $\begingroup$ Thanks. I read somewhere though that I can use a permutation test, but I am sort of looking for something that is a bit simpler and may be used by middle school students. Gathering more data though is not feasible. $\endgroup$ – cren Sep 21 '17 at 10:46
  • $\begingroup$ You can still do it, but it's not very accurate. $\endgroup$ – SmallChess Sep 21 '17 at 13:12
  • 3
    $\begingroup$ Conceptually, the permutation test is much simpler than the t-test. It might be the better pedagogical choice. You needn't feel obliged to test at a 5% level, either. If you could accept 10% as the standard of significance, then a permutation test can be a nice illustration. $\endgroup$ – whuber Sep 21 '17 at 13:36
  • 1
    $\begingroup$ see the links at stats.stackexchange.com/questions/294682/… . In R try t.test(x=c(4.5,4.6),y=5.7, var.equal=TRUE) (sample sizes 2 and 1). It works. However, power may be quite low. $\endgroup$ – Glen_b Sep 21 '17 at 22:32
6
$\begingroup$

The permutation test will have insufficient power. (There just aren't enough different ways to split six samples into two groups of three.) But If the assumptions of the t-test hold, then its results are valid.

Many thoughtful readers will question whether such a situation could actually arise. Let me share a real story. It concerns cleaning up lead contamination in a field: for years, a farmer accepted "recycled" batteries and dumped them behind his house. Eventually the environmental regulators caught up with him. They caused the "responsible party" to go through three phases of cleanup work: (1) sample the soils to estimate the amount and extent of lead contamination; (2) remove the soils in thin layers, taking samples in the process, until it was clear that clean soils had been reached; (3) independently sample all remaining soil and formally test whether the mean lead concentration is below the environmental standard.

The procedure for (3) was designed and approved before the cleanup began. It called for random sampling of all soils exposed during the excavation, analysis of the samples by a certified laboratory, and applying a Student t test. Equivalently, to demonstrate success, a suitable upper confidence limit (UCL) of the mean had to be less than the standard. It did not specify how many samples to take: that would be up to the responsible party to decide.

Almost a thousand samples were obtained and analyzed during the first two phases. Although these allowed the (univariate and spatial) distributions of the lead concentrations to be characterized reliably, they of course did not represent the remaining concentrations. However, they did suggest the shape of the (univariate) distribution of the remaining concentrations. Physical and chemical theory, soils science, and experience with remediating lead in soils elsewhere all provided support for this statistical characterization.

The cleanup was so thorough and successful that the likely mean concentration was negligible--more than an order of magnitude less than the standard. Power analyses, based on pessimistic (high) estimates of the standard deviation, all suggested that a random sample of only two or three would be needed.

There were many potential complications: for instance, any areas that might have been overlooked during the excavation could introduce large outlying values. To detect these, a large number of samples were obtained at random locations, and then composited in groups to produce just five physical samples for laboratory testing. All the values were low. As expected, a t-test of any two of those samples would still have demonstrated attainment.

Sprinkled within this brief case study are examples of various ways in which we might be assured that a t-test is appropriate even with tiny samples: experience; theory; preliminary related sampling; making pessimistic assumptions; and sample compositing all played a role--and any one of them might have sufficed to justify the t-test.


Incidentally, there are versions of the t-test that work with just a single observation. They are based on obtaining independent estimates of the variance or, lacking that, mathematical theory. Could this ever make sense in reality? The classic situation of compositing the blood from hundreds of soldiers to test for venereal disease provides one possible application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.