# Can you do statistics with 4 data points?

I was just involved in a Q/A were a poor guy was requested to run a statistical test to prove that algorithm A is better than two other algorithms. However, he has only 4 data points. Does it really make sense to make a statistical test on 4 points? Where is the limit? On three?

To clarify, I understand that there are 12 numbers reported but for me it looks more like either 4 three-dimensional data points, or 3 four-dimensional data points.

In their answers, authors introduce some assumptions about the underline distributions in order to artificially increase the number of data points and in the process compute mean values of four numbers; or perform t-tests on pairs of algorithms (comparing 8 numbers in total for each pair) and again making unfounded assumptions about underline distribution.

How reliable is this process when you don't know the underline distribution and because you don't have enough data you can't hope to be able to infer/validate it? Isn't it more fair to just say that there isn't much you can do with so few data?

• FYI, even though your question might stand on its own, I looked at the link you provided and it doesn't look that the person only has 3 data points... – Patrick Coulombe Jan 24 '14 at 0:04
• Thanks, indeed it is 4 datasets, not 3. I don't think it changes much about the question though... – iliasfl Jan 24 '14 at 0:08
• He doesn't have 4 data points, he has 12. And the question has answers at the link. So, what is new about your question? If nothing, then I think this should be closed. – Peter Flom - Reinstate Monica Jan 24 '14 at 0:12
• You can certainly do a t-test with 2 groups of 4. – Peter Flom - Reinstate Monica Jan 24 '14 at 1:01
• You can do an honest-to-goodness test with one value (and modest, realistic assumptions): stats.stackexchange.com/a/1836. BTW, downvotes are not intended to reflect offense or annoyance, but rather to indicate (as the hovertext indicates) that a question is ill-posed or badly researched. I think there's a decent question here but as it stands readers have to wade through the comments to understand what you are asking: could you please edit it? Note, too, that the last paragraph may be perceived as a misplaced rant (which it is) and ought to be deleted. – whuber Jan 24 '14 at 16:58

I have a friend who used to work for the US defense department (long time ago, cold war era) and was once asked to answer a question using a single data point. When he insisted that he needed more data he was told that the person who had provided the single data point had been caught and executed for espionage shortly after providing the single data point, so there would be no more data coming. That is when my friend started to learn about Bayesian statistics.

I also remember seeing an article several years ago, possibly in the American Statistician, possibly in Chance, that derived a way to compute a confidence interval for a mean based on a single data point (the 95% interval from a value of x was something like -x to 3*x) if you were willing to make certain assumptions (and the usual diagnostics were not of any help with only 1 point).

So, yes, you can do valid statistics with very small sample sizes, but you will tend to have low power/precision and large sample properties will not help you, so violations of any assumptions will have a potentially much larger impact.

• I believe you may be talking about this paper - D. Edelman (1990) "A Confidence Interval for the Center of an Unknown Unimodal Distribution Based on a Sample of Size 1", The American Statistician, Vol. 44, No. 4, pp. 285-287 – Glen_b -Reinstate Monica Jan 24 '14 at 6:21
• That is indeed interesting. Thanks for sharing the story and the paper. – Underminer Jan 24 '14 at 21:50
• @Glen_b I provided a more recent (2001) reference to TAS in my answer at stats.stackexchange.com/a/1836/919. It references the Edelman paper (see the bottom right of the first page). – whuber Jan 24 '14 at 22:14
• Thanks @whuber - that would be a better fit to 'several years ago'. A very readable paper, too. – Glen_b -Reinstate Monica Jan 25 '14 at 0:05
• I think the one referenced by @whuber is the one I was remembering. – Greg Snow Jan 25 '14 at 1:45