Should small sample sizes of 5 and less be included in reports? This is an area of medical training that is currently being reported only to the course level trainers and statistically not included in reports to upper management. All of the data is mined from course-level surveys.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ You should give some details about the test you want to do. This would give more adapted and focused answers. $\endgroup$– robin girardCommented Dec 13, 2010 at 18:48
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
If you are doing things properly the worst thing you can do is not to reject $H_0$ (does not mean $H_0$ is true, only mean you do not have enough evidence to reject)
Anyway, there can be something significant with 5 observations: If you need to test, for example, wether your random variable has a negative mean under the assumption of variance one, then having 5 observation over 10 is highly significant ....
Just be carefull not to use a test that uses something like asymptotic normality or other assumption that cannot be satisfyed or checked with only five observations
answered Dec 13, 2010 at 15:48
$\begingroup$
$\endgroup$
It depends what you're testing.
If you are testing for variance and your $H_0$ is that $V = 1$, but your values are $0, +1000, -1000, +2000, -2000$, then it's clear that you should reject $H_0$. If you are testing the hypothesis that the mean is not 2, then you do not have enough data to reject it.
For most things, though, 5 is too little.
(I didn't run through the math above to build intuition. I'd generally recommend a Bayes ratio as the best quantitative test).
$\begingroup$
$\endgroup$
Five is pretty small, but not so small that it doesn't contain some information. If you get 5 course evaluations that agree an instructor is ineffective, that's pretty strong evidence. That said, the only time the evidence will be conclusive if it is rather extreme, e.g. all zeroes or all tens. Still, it seems foolish to toss it out.
You may also have a later opportunity to pool evidence, say if the same instructor or course is offered multiple times. Then those fives start adding up.
