I've learnt that small sample size may lead to insufficient power and type 2 error. However, I have the feeling that small samples just may be generally unreliable and may lead to any kind of result by chance. Is that true?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
As a general principle, small sample size will not increase the Type I error rate for the simple reason that the test is arranged to control the Type I rate. (There are minor technical exceptions associated with discrete outcomes, which can cause the nominal Type I rate not to be achieved exactly especially with small sample sizes.)
There is an important principle here: if your test has acceptable size (= nominal Type I rate) and acceptable power for the effect you're looking for, then even if the sample size is small it's ok.
The danger is that if we otherwise know little about the situation--maybe these are all the data we have--then we might be concerned about "Type III" errors: that is, model mis-specification. They can be difficult to check with small sample sets.
As a practical example of the interplay of ideas, I will share a story. Long ago I was asked to recommend a sample size to confirm an environmental cleanup. This was during the pre-cleanup phase before we had any data. My plan called for analyzing the 1000 or so samples that would be obtained during cleanup (to establish that enough soil had been removed at each location) to assess the post-cleanup mean and variance of the contaminant concentration. Then (to simplify greatly), I said we would use a textbook formula--based on specified power and test size--to determine the number of independent confirmation samples that would be used to prove the cleanup was successful.
What made this memorable was that after the cleanup was done, the formula said to use only 3 samples. Suddenly my recommendation did not look very credible!
The reason for needing only 3 samples is that the cleanup was aggressive and worked well. It reduced average contaminant concentrations to about 100 give or take 100 ppm, consistently below the target of 500 ppm.
In the end this approach worked because we had obtained the 1000 previous samples (albeit of lower analytical quality: they had greater measurement error) to establish that the statistical assumptions being made were in fact good ones for this site. That is how the potential for Type III error was handled.
One more twist for your consideration: knowing the regulatory agency would never approve using just 3 samples, I recommended obtaining 5 measurements. These were to be made of 25 random samples of the entire site, composited in groups of 5. Statistically there would be only 5 numbers in the final hypothesis test, but we achieved greater power to detect an isolated "hot spot" by taking 25 physical samples. This highlights the important relationship between how many numbers are used in the test and how they were obtained. There's more to statistical decision making than just algorithms with numbers!
To my everlasting relief, the five composite values confirmed the cleanup target was met.
Another consequence of a small sample is the increase of type 2 error.
Nunnally demonstrated in the paper "The place of statistics in psychology", 1960, that small samples generally fail to reject a point null hypothesis. These hypothesis are hypothesis having some parameters equals zero, and are known to be false in the considered experience.
On the opposite, too large samples increase the type 1 error because the p-value depends on the size of the sample, but the alpha level of significance is fixed. A test on such a sample will always reject the null hypothesis. Read "The insignificance of statistical significance testing" by Johnson and Douglas (1999) to have an overview of the issue.
This is not a direct answer to the question but these considerations are complementary.