0
$\begingroup$

Mock data:

df = pd.DataFrame({
    'Treatment': [1, 2, 3],
    'Success': [25, 29, 25],
    'Fail': [48, 118, 92]
})

Where 'Treatment' represents three independent groups of subjects who received one of three treatments, 'Success' the number of people who were cured by the treatment, and 'Fail' the number of subjects who were not cured by the treatment.

We store the results in the following 3x2 array:

results = np.array([[25, 48], [29, 118], [25, 92]])

And we finally run a Chi-square test for independence to test whether there is an association between treatment type (1, 2, or 3) and outcome of treatment (success or fail):

chi_square, p_value, df, matrix = stats.chi2_contingency(observed=results)
print(chi_square)
print(p_value)
print(df)
print(matrix)

Out:

6.158896567124848
0.0459846201224755
2
[[ 17.11275964  55.88724036]
 [ 34.45994065 112.54005935]
 [ 27.4272997   89.5727003 ]]

I have two questions:

  1. Since we are comparing three (not two) groups/treatments, do we need to apply a correction for multiple comparisons? E.g. Bonferroni's, Šidák's, etc., and if not, why? Note that p-value is just under .05
  2. What Python function can we use to see which of the comparisons were significant, i.e. not just the overall result? Or, can we print the Chi-square statistic and p-value for each of the three comparisons? And this, without running three separate comparisons.

Many thanks!

$\endgroup$
2
$\begingroup$

You did one chi-squared test, albeit with $3-1=2$ degrees of freedom, so the answer to your Q1 is no, no further adjustment is needed for multiple testing as the degrees of freedom have already been taken into account. There was one chi-squared statistic and one $p$-value from this test.

If you reject the null hypothesis that the three underlying probabilities of success were all equal, that simply suggests the alternative that they were not all equal, and does not tell which were special in some sense.

If you had wanted to know that, then you should have set up multiple tests at the beginning with a suitable adjustment, before your chi-squared test; having looked at the data, it would be bad practice to choose a new test now.

Even if you had planned to do that in advance and carried it out, you might have got a confusing result that Treatment 1 was significantly more successful in the experiment than Treatment 2 but Treatment 1 was not significantly more successful in the experiment than Treatment 3, while the experimental data for Treatments 2 and 3 were not significantly different. Good luck interpreting that.

$\endgroup$
2
  • $\begingroup$ Many thanks for this, Henry, however this does not reply my second (and more important) question. Please note this is mock data (i.e. created to ask this question), not actual results from a study. You say I might have set up multiple tests from start. Well, this is exactly what I am asking: How do you set up these tests? You also mention difficulty in interpreting results. This is also part of my question: precisely, how would you interpret such results? Also note I set up my data precisely to illustrate that (a result close to .05 level and with contrasting results depending on treatment). $\endgroup$
    – johnjohn
    Mar 1 at 22:33
  • $\begingroup$ "...and does not tell which were special in some sense". Hence my question. How do you test this at the beginning (and possible correcting for multiple comparisons), when you are conducting analysis on real data (and not on mock data as in my example). By the way, I edited question to reflect this is mock data. Thanks $\endgroup$
    – johnjohn
    Mar 1 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.