Can chi-squared tests be used for making statements like this or is this an inaccurate analysis of my data?

The majority of the groups (n=9) affected by disease ranged in prevalence from 5.56% to 12.73% with the exception of three groups (groups A, B and C) which produced markedly higher prevalence rates. Chi-squared tests were performed on these however, and it was determined that neither the 39.13% of group A (χ2= 1.08, df=1, P=0.297) nor the 21.05% from group B (χ2= 6.36, df= 1 P=0.12) were statistically significant. The 23.19% from group C (χ2= 59.52, df=1, P=<0.05) was significant thereby marking it out as unique in this assemblage.

Is this acceptable? Or would a more accurate way be to do a chi-square on a 2x9 contingency table? And if it is, how then do you know which group is significantly different from the others?

  • 4
    $\begingroup$ It's not clear what you compared C with ... but more importantly, it reads like you worked out which hypotheses to test by looking at the data. If that's the case, then you can't say much about significance at all, because your p-values will be wrong. $\endgroup$ – Glen_b Jan 18 '14 at 15:00
  • $\begingroup$ Thank you for your answer. I didn't actually compare Group C (or in fact groups A or B)with anything else. The results I quote were from doing a chi-square on a 2x1 contingency table, comprised of the results from the group (i.e. number individuals affected by disease, number of individuals clear of disease). I must admit, I learnt it from a 'how2stats' YouTube video without truly knowing its appropriate application! I think I have done what you said: chose which group to test primarily by looking at the data. I wanted to know if the high percentages I was seeing in one group was significant. $\endgroup$ – Denise Jan 18 '14 at 15:24
  • $\begingroup$ Why would it be interesting to test if the proportion of individuals with the disease is 50%? That seems an odd thing to be testing. $\endgroup$ – Glen_b Jan 19 '14 at 0:23

Statistical significance is a description of the magnitude of evidence for a particular alternative hypothesis in a classical statistical test. Hence, it is best practice to refer to statistically significant evidence of such-and-such a hypothesis (at such-and-such a significance level). It is generally bad practice to refer to groups of data or effects as being "statistically significant". When you use this term, you should always be referring to "statistically significant evidence" of such-and-such.

In view of this, I would suggest rephrasing your explanation to refer clearly to the hypotheses you were testing and use language that ascribes statistical significance (or lack thereof) to the evidence in those tests. You should also bear in mind that it is generally improper to choose groups with high outcomes and then test those groups using individual tests, since this gives rise to a problem of hypothesis-selection bias, or multiple comparisons bias.

  • $\begingroup$ "evidence for a particular alternative hypothesis" I was under the impression that it was a measure of evidence against the null hypothesis rather than for a hypothesis. $\endgroup$ – COOLSerdash Mar 24 '18 at 7:58
  • 1
    $\begingroup$ If you have statistical significance, that means you had a low p-value, which constitutes evidence in favour of the alternative, against the null. So you would say something like, "We found statistically significant evidence (at the one-percent level) of a difference in population means between the two groups". $\endgroup$ – Ben Mar 24 '18 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.