5
$\begingroup$

I was just involved in a Q/A were a poor guy was requested to run a statistical test to prove that algorithm A is better than two other algorithms. However, he has only 4 data points. Does it really make sense to make a statistical test on 4 points? Where is the limit? On three?

To clarify, I understand that there are 12 numbers reported but for me it looks more like either 4 three-dimensional data points, or 3 four-dimensional data points.

In their answers, authors introduce some assumptions about the underline distributions in order to artificially increase the number of data points and in the process compute mean values of four numbers; or perform t-tests on pairs of algorithms (comparing 8 numbers in total for each pair) and again making unfounded assumptions about underline distribution.

How reliable is this process when you don't know the underline distribution and because you don't have enough data you can't hope to be able to infer/validate it? Isn't it more fair to just say that there isn't much you can do with so few data?

$\endgroup$
  • $\begingroup$ FYI, even though your question might stand on its own, I looked at the link you provided and it doesn't look that the person only has 3 data points... $\endgroup$ – Patrick Coulombe Jan 24 '14 at 0:04
  • $\begingroup$ Thanks, indeed it is 4 datasets, not 3. I don't think it changes much about the question though... $\endgroup$ – iliasfl Jan 24 '14 at 0:08
  • 3
    $\begingroup$ He doesn't have 4 data points, he has 12. And the question has answers at the link. So, what is new about your question? If nothing, then I think this should be closed. $\endgroup$ – Peter Flom - Reinstate Monica Jan 24 '14 at 0:12
  • 2
    $\begingroup$ You can certainly do a t-test with 2 groups of 4. $\endgroup$ – Peter Flom - Reinstate Monica Jan 24 '14 at 1:01
  • 1
    $\begingroup$ You can do an honest-to-goodness test with one value (and modest, realistic assumptions): stats.stackexchange.com/a/1836. BTW, downvotes are not intended to reflect offense or annoyance, but rather to indicate (as the hovertext indicates) that a question is ill-posed or badly researched. I think there's a decent question here but as it stands readers have to wade through the comments to understand what you are asking: could you please edit it? Note, too, that the last paragraph may be perceived as a misplaced rant (which it is) and ought to be deleted. $\endgroup$ – whuber Jan 24 '14 at 16:58
13
$\begingroup$

I have a friend who used to work for the US defense department (long time ago, cold war era) and was once asked to answer a question using a single data point. When he insisted that he needed more data he was told that the person who had provided the single data point had been caught and executed for espionage shortly after providing the single data point, so there would be no more data coming. That is when my friend started to learn about Bayesian statistics.

I also remember seeing an article several years ago, possibly in the American Statistician, possibly in Chance, that derived a way to compute a confidence interval for a mean based on a single data point (the 95% interval from a value of x was something like -x to 3*x) if you were willing to make certain assumptions (and the usual diagnostics were not of any help with only 1 point).

So, yes, you can do valid statistics with very small sample sizes, but you will tend to have low power/precision and large sample properties will not help you, so violations of any assumptions will have a potentially much larger impact.

$\endgroup$
  • 6
    $\begingroup$ I believe you may be talking about this paper - D. Edelman (1990) "A Confidence Interval for the Center of an Unknown Unimodal Distribution Based on a Sample of Size 1", The American Statistician, Vol. 44, No. 4, pp. 285-287 $\endgroup$ – Glen_b -Reinstate Monica Jan 24 '14 at 6:21
  • $\begingroup$ That is indeed interesting. Thanks for sharing the story and the paper. $\endgroup$ – Underminer Jan 24 '14 at 21:50
  • 2
    $\begingroup$ @Glen_b I provided a more recent (2001) reference to TAS in my answer at stats.stackexchange.com/a/1836/919. It references the Edelman paper (see the bottom right of the first page). $\endgroup$ – whuber Jan 24 '14 at 22:14
  • $\begingroup$ Thanks @whuber - that would be a better fit to 'several years ago'. A very readable paper, too. $\endgroup$ – Glen_b -Reinstate Monica Jan 25 '14 at 0:05
  • $\begingroup$ I think the one referenced by @whuber is the one I was remembering. $\endgroup$ – Greg Snow Jan 25 '14 at 1:45
1
$\begingroup$

Short answer: yes, but your results will usually be useless.

Long answer: Statistics often involves forming some kind of inference about underlying parameters based on data, with constraints on the probability of a False-Positive and/or of a False-Negative. In a typical test, i.e. testing if a sample came from a given distribution, we put an upper bound (called alpha) on the probability of a Type-I Error (False Positive), mostly for two reasons:

  • In practice that is the only kind of error you can put a bound on, b/c of the nature of your null hypothesis
  • False Positives are usually considered more terrible than False Negatives (a corollary of Occam's Razor)

Holding alpha constant, beta (upper bound of probability of False-Negative) is generally larger for smaller data sets. And when beta is large, your overall probability of producing a Positive is very small, and so your test will almost always return Negative, which is not very different from just accepting your Null Hypothesis from the get-go. In this situtation we say the statistical test is not very powerful.

$\endgroup$
1
$\begingroup$

I helped with a geological project where the researchers had a single data point, accompanied by a very reliable uncertainty bound. They were interested in testing a geological model (a set of differential equations describing the evolution of tectonic plates) which made a very specific prediction for the value of that single datum. Given its uncertainty distribution, we could straightforwardly calculate a p.value, given the model is true, and reject the null hypothesis convincingly. So, in that case, I would argue that we successfully 'did statistics' with a single data point (and its uncertainty).

$\endgroup$
  • 1
    $\begingroup$ Was this result published somewhere? It would be useful to be able to give doubters an example that such cases can legitimately occur in research. $\endgroup$ – Glen_b -Reinstate Monica Jun 13 '18 at 4:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.