# Interpretting a chi-sq correctly

I apologize for what may seem a very simple question. I can't seem to find an answer to how I can interpret chi-sq results other than rejecting the null hypothesis.

Pearson's Chi-squared test with Yates' continuity correction

X-squared = 5.2538, df = 1, p-value = 0.0219

I understand that I am able to reject the null hypothesis given the p-value. Can I also interpret the results saying that Procedure B is more likely to lead to a malignant result given the observed results are greater than expected?

For interpreting the test itself you should stick to interpreting the p-value, however nobody stops you from saying that Procedure B led to more malignant results than expected under the independence null hypotheses. You may even get away with the interpretation that you suggested, however this is not directly what the test says.

The test is defined by null and alternative hypothesis, test statistic and result (p-value). None of these says anything in particular about what results are "likely" with Procedure B. My point is that on top of that you can explain and interpret how your data led to this result. Nothing wrong with that, but don't mix the term "significant" in there; "likely" is kind of borderline and one can discuss for long whether this is justified. I wouldn't use it.

In addition to what is highlighted be @Lewian, the end of the statistical procedure (here a chi-squared test) and what you do next may seem unrelated, but there is an interaction that needs to be considered.

But first a mention of the statistical test results. This is predicated on the idealized nature of the test, and requires the assumptions to be met. A non-zero number of low p-values probably occur because of treatments being chosen non-randomly, that is the choice is in some way influenced by characteristics of a given individual.

But lets suppose this is not an issue with the data above, and the assumptions are met. Then if you do a subsequent hypothesis, we know that (because of the low p-value) there is a difference between the expected frequencies and the observed frequencies in the two categories. Although it is natural to want to interpret this difference, in some cases (which we do not know which) in the absence of outside information (such as known biological mechanisms) we can only say so much. A simple example is if the low p-value was just through chance (that is if the treatments each have the same effect, for example if neither treatment has an effect) then further information obtained from the data is still related to the chi-squared test result that it is unknown of whether the result was due to chance. A low p-value tells us something about the null hypothesis is wrong, it does not tell us of it is real, due to chance, or because of a violation of the test assumptions. Uncertainty remains even though low p-values are often interpreted as strong evidence, from which strong causes can be interpreted.

The Wikipedia page on HARKing classifies this situation as THARKing, transparently hypothesizing after the results are known, rather than the secretive, undisclosed, HARKing that was first proposed by Kerr (1998). Of the 12 potential costs of HARKing given by Kerr, one is translating Type I errors into hard-to-eradicate theory.

• We of course can't know whether a significant p-value represents a real signal or just a result due to random chance, but you can control how often you're willing to accept random chance results by appropriate selection of the significance threshold. The p-value indicates that the observed data would be very unlikely to occur if there really was no difference in treatment effects. By setting the threshold of alpha to your desired significance, you can "know" that a significant result is not due to chance to whatever confidence you like. Commented Aug 2, 2021 at 20:50

To begin, I have checked your original chi-squared analysis of these data. The P-value of your chi-squared test is $$0.02 < 0.025 = 2.5\%;$$ you have a result significant at the 2.5% level. Thus, there is a significant difference between groups A and B with fewer malignancies in A. If your original purpose was to see whether preferred Procedure A has fewer malignancies, it would have been appropriate to do a one-sided test from the start.

If you had found a P-value somewhat above 5% for the inherently two-sided chi-squared procedure, then it would have been inappropriate to notice that you could have gotten a P-value just below 5% by deciding ad hoc to do a one sided test against the alternative that A has fewer malignancies,

Especially in medical research it is routine for a prospectus to precede experimentation, in which the hypotheses (one or two sided) are clearly declared, and to have a power-and-sample size analysis showing how many subjects must be used in order to have reasonable power for rejecting against an alternative of interest.

I have no way to know your original purpose and expectations for this research. However, with a P-value below 2.5%, if you find a difference in a two-sided test, it seems reasonable to notice and report the direction of the effect, mentioning the P-value of the two-sided test. In that context, it seems appropriate to do a one-sided test at the 2.5% level.

You could look at a one-sided test for a difference in proportions (implemented in R as prop.test).

prop.test(c(9,20),c(61,58), alt="less")

2-sample test for equality of proportions
with continuity correction

data:  c(9, 20) out of c(61, 58)
X-squared = 5.2538, df = 1, p-value = 0.01095
alternative hypothesis: less
95 percent confidence interval:
-1.00000000 -0.05351627
sample estimates:
prop 1    prop 2
0.1475410 0.3448276


There are several versions of this test of $$H_0: p_A=p_B$$ against $$H_a: p_A < p_B,$$ depending on (a) whether the standard error for the the estimated difference is computed assuming $$H_0$$ is true (pooling the two groups) or whether the standard errors for $$\hat p_A$$ and $$\hat p_b$$ are computed separately and (b) whether a continuity correction is used.

For a two-sided test, the version implemented in R happens to be identical to the chi-squared test; both give exactly the same P-value:

prop.test(c(9,20),c(61,58))$p.val # 2-sided prop.test [1] 0.02189937 TBL = rbind(c(9,52),c(20,38)) chisq.test(TBL)$p.val              # 2-sided chisq.test
[1] 0.02189937


The table of expected values used in chisq.test just above is essentially the same as you show in your Question:

chisq.test(TBL)exp [,1] [,2] [1,] 14.86555 46.13445 [2,] 14.13445 43.86555  Thus, in that sense, I suppose you might say the the one-sided test of proportions (with P-value $$0.01095 < 0.25 = 2.5\%)$$ is a "one-sided" version of the chi-squared test on the $$2 \times 2$$ table. However---with or without considering the relationship to the chi-squared test---tests of two binomial proportion are widely used. NIST shows the pooled version. A look at power and sample size: Suppose you have 60 subjects each in Groups A and B, and you are hoping to find a difference in malignancy proportions as large as $$p_A = .15$$ and $$p_B = 0.35.$$ The power of a one-sided test at the 2.5% level to detect this difference in proportions of malignancies is about 65%, so it would be difficult to dismiss the current result as being due to luck. Even so, if you are doing future work along these lines, you might consider using a few more subjects. [An additional simulation shows that 80 in each group would give power 80% in the same circumstances.] set.seed(2021) n = 60 pv = replicate(10^5, prop.test(c(rbinom(1,n,.15),rbinom(1,n,.35)), c(n,n), alt="less")p.val)
mean(pv <= 0.025)
[1] 0.65219

• Useful to know that the two-sided prop.test gives the same result as the two-sided chi-squared in R. Is this obvious? Or only applies sometimes, such as for $2x2$ contingency tables? I had either forgotten or was unaware of this. Commented Aug 3, 2021 at 17:05