I'll first give an example and afterwards a more formal definition of my problem.


Let's assume I'm looking at balls with two properties: their color can be black or white, their weight heavy or light.

If I have a collection of n balls and I know the frequency of each property (say 50% black balls, 50% heavy balls), I can easily calculate the expected frequency of both events at the same time assuming the events are independent (here: 25% black and heavy balls).

Then I can simply count to evaluate the observed frequency of balls with both properties (black and heavy) and compare it to the expected frequency.

Now assume I have m such collections. They all have different, but known base frequencies of the individual properties. Thus, I can calculate for each collection the expected frequency of double events and compare to the observed frequency of actual.

If color and weight of a ball were truly independent, I would expect observed frequencies scattering around expected frequencies in both directions. However, what I see in my data (m = 45) is that observed frequencies are always lower than expected frequencies.

Formal Definition:

I have a sample $S$ with two binary features $A$ and $B$. I know the base frequencies for both features $p_S(A)$ and $p_S(B)$. Assuming independent features $A$ and $B$, the frequency of both features occurring at the same time would be $p_S(AB) = p_S(A)*p_S(B)$. Then I can compare this expected frequency to the actually observed frequency $\hat{p_S}(AB)$ in sample $S$.

In a set of $m$ samples $S_1, ..., S_m$ with varying $p_{S_i}(A)$ and $p_{S_i}(B)$, I find that $p_{S_i}(AB) > \hat{p_{S_i}}(AB)$ for all $i$.

How can I test if this is a significant deviation from the expected independence of $A$ and $B$?

  • $\begingroup$ Given $m$= 45, do you want to test $p_S(AB) = p_S(A)*p_S(B)$ 45 times (one by one), or test 45 null hypothesis simultaneously (one test)? $\endgroup$
    – user158565
    Commented Oct 29, 2018 at 18:15
  • $\begingroup$ I'm looking for a single test. In any individual case, $p_S(AB)$ is only slightly lower than $p_S(A)*p_S(B)$. I would not expect significant differences for individual cases. The remarkable thing is that all 45 samples are deviating in the same direction: $p_{S_i}(AB) > \hat{p_{S_i}}(AB)$. To me, this does not at all look like random deviations. $\endgroup$
    – Andreas
    Commented Oct 30, 2018 at 7:57
  • $\begingroup$ Suppose you have different P(A) and P(B) from different collection. Calculate the expected number of AB under the assumption of A and B are independent, E = NP(A)P(B) for each collection. Calculate X=(E-O)*(E-O)/E for each collection, where O is the number of AB observed in the collection. Add all X together, compare with Chi Square distribution with m degree of freedom to get your p value. $\endgroup$
    – user158565
    Commented Oct 30, 2018 at 15:01
  • $\begingroup$ Thanks for your help. I went a bit deeper into the theory behind the Chi Square test and now this makes a lot of sense. If you want to make this an answer instead of a comment, I will flag it as the accepted answer. $\endgroup$
    – Andreas
    Commented Nov 21, 2018 at 12:22

1 Answer 1


Suppose you have different $\Pr(A)$ and $\Pr(B)$ from different collection. Calculate the expected number of $A\cap B$ under the assumption of $A$ and $B$ are independent, $E = N\Pr(A)\Pr(B)$ for each collection, where $N$ is total number of events. Calculate $X=(E-O)^2/E$ for each collection, where $O$ is the number of $A\cap B$ observed in the collection. $\sum_X$ follows Chi-Square distribution with $m$ degree of freedom under the null hypothesis that $A$ and $B$ are independent, where $m$ is the number of $X$s. Comparing $\sum_X$ with Chi Square distribution with $m$ degree of freedom to get your $p$ value. If $p$ value is small, then the null hypothesis of independent will be rejected and $A$ and $B$ are related will be accepted.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.