4
$\begingroup$

I feel like this question has a very silly (simple) answer, so I apologize. I have a data set that only has 5 points

41.9
32.2
113.3
110.2
102.6

Clearly something significant happens between the 2nd and 3rd data points (but not between the 1st and 2nd, 3rd and 4th, etc.). What statistical test could I employ to show this in a more rigorous sense?

EDIT: Note, I was not clear with my question (sorry). The data is an ordered set. I would like to determine that the difference between 2-3 is statistically larger than 1-2, 3-4, and 4-5.

Thanks!

$\endgroup$
  • 1
    $\begingroup$ Some statistical writers have compared your hypothesis to randomly shooting five bullets at a large wall, circling two that happened to be close to each other (and far from the other three), and saying "Wow! What were the chances of that happening!" $\endgroup$ – whuber Jun 16 '15 at 20:44
  • $\begingroup$ Well said, clearly (if I understand you correctly) not much in a statistical sense can be inferred here. $\endgroup$ – Shinobii Jun 16 '15 at 20:46
  • $\begingroup$ That's right. However, if before collecting these data you had stated exactly the same hypothesis--is the difference at 2-3 the largest of all (four) differences?--then you could get more traction, because your hypothesis would be independent of the data. Consequently it would be valid to use the data to test that hypothesis. As your intuition might tell you, there's a one in four chance that the 2-3 difference is the largest. That's a tiny bit unusual, but not enough to get anybody's attention. $\endgroup$ – whuber Jun 16 '15 at 20:51
7
$\begingroup$

Of the $5! = 120$ distinct sequences that can be formed of those five numbers,

  • $4\times 2!\times 3! = 48$ of them will have the small values $41.9$ and $32.2$ next to each other. (There are four places for this pair to occur, $2!$ ways of ordering them, and $3!$ ways to order the other three numbers.)

  • Yet another $2! \times 3! = 12$ sequences will alternate between a high value in $\{102.6, 110.2, 113.3\}$ and a low value in $\{32.2, 41.9\}$.

  • Another $2! \times 3! = 12$ will bracket the three high values with a low value on either end.

I have enumerated $72$ so far, which is $60\%$ of all the possible sequences. Thus, depending on what kinds of patterns might catch your notice (which is a matter for your psychologist to explore), the total number of such "clearly something significant" sequences could easily be more frequent than sequences that do not clearly have something significant! From this we may draw two conclusions:

  1. Not a single one of these patterns is rare enough to be considered "statistically significant" at a conventional ($5\%$, or $6/120$) level.

  2. Any conclusion about "significance" derived after recognizing a "clear, significant" pattern when exploring dataset must be considered subjective.

(This is not to say such conclusions are without value. It only maintains that statistics, correctly applied, will not sanctify the conclusions of an open-ended exploratory analysis with any level of "significance," because it cannot.)


Such quantitative reasoning leads generally to the following statistico-psychological metatheorem:

In any collection of random patterns, the majority will be unusual.

Those of you familiar with Garrison Keillor may recognize an echo of the Lake Wobegon population: "... and all the children are above average." However, I privately refer to this as the Shirley MacLaine principle, in honor of her well-known work as a "spiritual missionary," a seer of things and causes that do not exist.

$\endgroup$
  • $\begingroup$ Sorry, I should have been more clear. They must remain ordered, i.e. there are only 4 possibilities (1-2, 2-3, 3-4, 4-5). I would like to show (if possible) that the difference between 2-3 is significantly larger than 1-2, 3-4, and 4-5. $\endgroup$ – Shinobii Jun 16 '15 at 20:38
  • 1
    $\begingroup$ They do remain ordered. That's what a permutation is. When you produce a sequence of five values and ask about its "significance," you are wondering about how those five values compare to what they could have been had they appeared in some other order--because if those five values were actually independent, all orders would be equally likely. I have, in effect, conducted a post hoc statistical test that demonstrates that if we pretended a formal test were applicable, your p-value would have to be considered to be $60\%$ or greater. $\endgroup$ – whuber Jun 16 '15 at 20:41
  • $\begingroup$ Ah, I see. Clearly need to brush up on my statistics. It is the one subject that is required for ALL fields of study, yet not emphasized enough in University. I appreciate your very detailed answer. $\endgroup$ – Shinobii Jun 16 '15 at 20:44
  • $\begingroup$ This is the fun part of statistics: the math is easy, the stakes low, and the fundamental ideas--which can be truly counter-intuitive--come to the fore. Quite a few of the quotations at stats.stackexchange.com/questions/726/… allude to this situation where we develop a hypothesis from our data and then are tempted to use the same data to confirm that hypothesis. $\endgroup$ – whuber Jun 16 '15 at 20:48
  • $\begingroup$ If we assume that we would only try to find significance we saw 2 low values followed by 3 high values or 3 high values followed by two low values, then it's only 24 sequences. Then if we apply a two-sample t-test to the observed sequence and adjust the p-value via Bonferroni, we would still end up with a pretty small p-value. $\endgroup$ – James Jun 17 '15 at 13:51
2
$\begingroup$

It looks like the simplest way is to use Chow test. If your sequence was assumed to be iid with no covariates, then it's probably equivalent to a two-sample test (e.g. a t-test) for the equality of means with 2 and 3 observations per sample. However, based on the Chow article, I don't see how it adjusts for data snooping, i.e. for the fact that the split into the two groups is suggested by the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.