A null Chi-square test – what could be done next?

Question

I have a rating (from 1 to 5) under two conditions (let’s call them "blue" and "red"). I tested whether the distribution of the ratings is significantly different using a chi squared test for goodness of fit. The test results were null - meaning I could not say that the rating distribution of the blue and red groups is significantly different.

Would looking at only a partial sub-group of the data be considered a legitimate calculation?

Example: asking whether the distribution of blue and red is different only for the “1” and “5” ratings (without the rating 2-3-4).

• If yes: What can I test further?
• If no: What is the problem with such an examination?

Answer 1

There are indeed situations in which sub-group analysis is valid, but there must be a clearly defined scientific justification for it beyond that your main test did not work. your question suggests that your desire does not arise from a well thought out argument in which case the answer is no, do not test the sub group.

Because you have already carried out a statistical test and you are looking to apply another you would need to apply a multiple hypothesis correction such as FDR or Bonferroni's. This is because each test carries a risk of a false positive and the more tests carried out the more chances of a spurious rejection of a null hypothesis.

Answer 2

You can do it, but the chi-square test on a subset necessarily doesn't know about the rest of the data.

The main problem is that you lay yourself open to accusations or suspicions that you are being selective and emphasising patterns shown only in part of the data.

Consider the following extreme analog[ue]. Two variables on a scatter plot show only a weak relationship. So you select top right and bottom left data points, and then compute and announce a perfect correlation of $+1$. Good idea or not?

You're not proposing anything so extreme, but you are considering a step in the same direction.

I've sympathy with those who would argue that the significance test has no useful meaning and the $P$-value no validity if you just focus on a subset without an independent reason for doing that. A minimum protocol is to explain exactly what you did and exactly why it makes both scientific and statistical sense. On the face of it, it doesn't.

Answer 3

It would have been fine had you prespecified the subgroup - that is had a reason to investigate it, and said so before you looked at the data (and accounted for multiple testing etc)

Now you’ve looked at the data, further analysis is data dredging - while further analysis might lead to hypothesis generation, you shouldn’t draw any conclusions.

If you want to draw conclusions, you’ll need to collect a fresh set of data.

(Incidentally, what’s the problem with showing there’s no significant effect? That’s often useful to know)