Can you do statistics with 4 data points?

Question

I was just involved in a Q/A were a poor guy was requested to run a statistical test to prove that algorithm A is better than two other algorithms. However, he has only 4 data points. Does it really make sense to make a statistical test on 4 points? Where is the limit? On three?

To clarify, I understand that there are 12 numbers reported but for me it looks more like either 4 three-dimensional data points, or 3 four-dimensional data points.

In their answers, authors introduce some assumptions about the underline distributions in order to artificially increase the number of data points and in the process compute mean values of four numbers; or perform t-tests on pairs of algorithms (comparing 8 numbers in total for each pair) and again making unfounded assumptions about underline distribution.

How reliable is this process when you don't know the underline distribution and because you don't have enough data you can't hope to be able to infer/validate it? Isn't it more fair to just say that there isn't much you can do with so few data?

Answer 1

I have a friend who used to work for the US defense department (long time ago, cold war era) and was once asked to answer a question using a single data point. When he insisted that he needed more data he was told that the person who had provided the single data point had been caught and executed for espionage shortly after providing the single data point, so there would be no more data coming. That is when my friend started to learn about Bayesian statistics.

I also remember seeing an article several years ago, possibly in the American Statistician, possibly in Chance, that derived a way to compute a confidence interval for a mean based on a single data point (the 95% interval from a value of x was something like -x to 3*x) if you were willing to make certain assumptions (and the usual diagnostics were not of any help with only 1 point).

So, yes, you can do valid statistics with very small sample sizes, but you will tend to have low power/precision and large sample properties will not help you, so violations of any assumptions will have a potentially much larger impact.

Answer 2

Short answer: yes, but your results will usually be useless.

Long answer: Statistics often involves forming some kind of inference about underlying parameters based on data, with constraints on the probability of a False-Positive and/or of a False-Negative. In a typical test, i.e. testing if a sample came from a given distribution, we put an upper bound (called alpha) on the probability of a Type-I Error (False Positive), mostly for two reasons:

• In practice that is the only kind of error you can put a bound on, b/c of the nature of your null hypothesis
• False Positives are usually considered more terrible than False Negatives (a corollary of Occam's Razor)

Holding alpha constant, beta (upper bound of probability of False-Negative) is generally larger for smaller data sets. And when beta is large, your overall probability of producing a Positive is very small, and so your test will almost always return Negative, which is not very different from just accepting your Null Hypothesis from the get-go. In this situtation we say the statistical test is not very powerful.

Answer 3

I helped with a geological project where the researchers had a single data point, accompanied by a very reliable uncertainty bound. They were interested in testing a geological model (a set of differential equations describing the evolution of tectonic plates) which made a very specific prediction for the value of that single datum. Given its uncertainty distribution, we could straightforwardly calculate a p.value, given the model is true, and reject the null hypothesis convincingly. So, in that case, I would argue that we successfully 'did statistics' with a single data point (and its uncertainty).