I've learnt that small sample size may lead to insufficient power and type 2 error. However, I have the feeling that small samples just may be generally unreliable and may lead to any kind of result by chance. Is that true?

  • $\begingroup$ I have an aversion to unnecessary mathematical notation, so I've edited the title, could you please check that I did not change the meaning by changing it? $\endgroup$ – mpiktas Apr 18 '11 at 7:21
  • 1
    $\begingroup$ Be sure also to talk about hypothesis testing (Neyman-Pearson tests) and not significance testing (Fisher tests). These approaches are commonly mixed even if there is no notion of error in the second one, and proper usages should be different because they lead to different kinds of conclusion. $\endgroup$ – Seb May 8 '11 at 11:25
  • $\begingroup$ If you're using an asymptotic test then, yes, it is possible. Otherwise, no - the test is defined to control the type 1 error rate (i.e. $\alpha$). $\endgroup$ – Macro Feb 8 '12 at 3:50
  • $\begingroup$ But isn't it true, if you are flipping coins twice, you are more likely to result in skewed result (2 same sides (100%)), than when you are flipping 100 times, which will most likely result in approx 1/2, 1/2. Doesn't this indicate that the smaller the size, the more likely you may incur type I error? $\endgroup$ – user31513 Oct 15 '13 at 11:43

As a general principle, small sample size will not increase the Type I error rate for the simple reason that the test is arranged to control the Type I rate. (There are minor technical exceptions associated with discrete outcomes, which can cause the nominal Type I rate not to be achieved exactly especially with small sample sizes.)

There is an important principle here: if your test has acceptable size (= nominal Type I rate) and acceptable power for the effect you're looking for, then even if the sample size is small it's ok.

The danger is that if we otherwise know little about the situation--maybe these are all the data we have--then we might be concerned about "Type III" errors: that is, model mis-specification. They can be difficult to check with small sample sets.

As a practical example of the interplay of ideas, I will share a story. Long ago I was asked to recommend a sample size to confirm an environmental cleanup. This was during the pre-cleanup phase before we had any data. My plan called for analyzing the 1000 or so samples that would be obtained during cleanup (to establish that enough soil had been removed at each location) to assess the post-cleanup mean and variance of the contaminant concentration. Then (to simplify greatly), I said we would use a textbook formula--based on specified power and test size--to determine the number of independent confirmation samples that would be used to prove the cleanup was successful.

What made this memorable was that after the cleanup was done, the formula said to use only 3 samples. Suddenly my recommendation did not look very credible!

The reason for needing only 3 samples is that the cleanup was aggressive and worked well. It reduced average contaminant concentrations to about 100 give or take 100 ppm, consistently below the target of 500 ppm.

In the end this approach worked because we had obtained the 1000 previous samples (albeit of lower analytical quality: they had greater measurement error) to establish that the statistical assumptions being made were in fact good ones for this site. That is how the potential for Type III error was handled.

One more twist for your consideration: knowing the regulatory agency would never approve using just 3 samples, I recommended obtaining 5 measurements. These were to be made of 25 random samples of the entire site, composited in groups of 5. Statistically there would be only 5 numbers in the final hypothesis test, but we achieved greater power to detect an isolated "hot spot" by taking 25 physical samples. This highlights the important relationship between how many numbers are used in the test and how they were obtained. There's more to statistical decision making than just algorithms with numbers!

To my everlasting relief, the five composite values confirmed the cleanup target was met.

| cite | improve this answer | |
  • 1
    $\begingroup$ (+1) great story about aggressive cleanup and type III error, would be nice if this would be also relevant for economic time series. For deterministic models or models with low noise ratio small sample size IMHO won't be the greatest problem (compared to huge set of very noisy likely independent large sample data, even principal components are hard with these ones). $\endgroup$ – Dmitrij Celov Apr 18 '11 at 6:57
  • 1
    $\begingroup$ +1, for those who are interested in further understanding the "technical exceptions associated with discrete outcomes" mentioned in the first paragraph, I discuss those here: Comparing and contrasting p-values, significance levels, and type I error. $\endgroup$ – gung - Reinstate Monica Jan 5 '13 at 19:19
  • 1
    $\begingroup$ +1, great example of why you can't take a wild stab at a useful sample size without key info. $\endgroup$ – Freya Harrison Jan 17 '13 at 12:35

Another consequence of a small sample is the increase of type 2 error.

Nunnally demonstrated in the paper "The place of statistics in psychology", 1960, that small samples generally fail to reject a point null hypothesis. These hypothesis are hypothesis having some parameters equals zero, and are known to be false in the considered experience.

On the opposite, too large samples increase the type 1 error because the p-value depends on the size of the sample, but the alpha level of significance is fixed. A test on such a sample will always reject the null hypothesis. Read "The insignificance of statistical significance testing" by Johnson and Douglas (1999) to have an overview of the issue.

This is not a direct answer to the question but these considerations are complementary.

| cite | improve this answer | |
  • $\begingroup$ +1 for the calling out the issue of large samples and Type I error $\endgroup$ – Josh Hemann May 5 '11 at 19:43
  • 8
    $\begingroup$ -1, the comment that "too large samples increase the type 1 error" is incorrect. You may be confusing statistical significance & practical significance, in that a situation can exist where the true effect is not exactly 0, but so small that it is inconsequential, & we would consider the null 'true' for practical purposes. In this case, the null would be rejected more than (eg) 5% of the time, & more often w/ increasing N. However, strictly speaking, the null hypothesis that the true effect is exactly 0 is, by stipulation, false. Thus, these rejections aren't actually type I errors. $\endgroup$ – gung - Reinstate Monica Jan 5 '13 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.