# Expected vs observed frequency of two events at the same time

I'll first give an example and afterwards a more formal definition of my problem.

## Example:

Let's assume I'm looking at balls with two properties: their color can be black or white, their weight heavy or light.

If I have a collection of n balls and I know the frequency of each property (say 50% black balls, 50% heavy balls), I can easily calculate the expected frequency of both events at the same time assuming the events are independent (here: 25% black and heavy balls).

Then I can simply count to evaluate the observed frequency of balls with both properties (black and heavy) and compare it to the expected frequency.

Now assume I have m such collections. They all have different, but known base frequencies of the individual properties. Thus, I can calculate for each collection the expected frequency of double events and compare to the observed frequency of actual.

If color and weight of a ball were truly independent, I would expect observed frequencies scattering around expected frequencies in both directions. However, what I see in my data (m = 45) is that observed frequencies are always lower than expected frequencies.

## Formal Definition:

I have a sample $$S$$ with two binary features $$A$$ and $$B$$. I know the base frequencies for both features $$p_S(A)$$ and $$p_S(B)$$. Assuming independent features $$A$$ and $$B$$, the frequency of both features occurring at the same time would be $$p_S(AB) = p_S(A)*p_S(B)$$. Then I can compare this expected frequency to the actually observed frequency $$\hat{p_S}(AB)$$ in sample $$S$$.

In a set of $$m$$ samples $$S_1, ..., S_m$$ with varying $$p_{S_i}(A)$$ and $$p_{S_i}(B)$$, I find that $$p_{S_i}(AB) > \hat{p_{S_i}}(AB)$$ for all $$i$$.

How can I test if this is a significant deviation from the expected independence of $$A$$ and $$B$$?

• Given $m$= 45, do you want to test $p_S(AB) = p_S(A)*p_S(B)$ 45 times (one by one), or test 45 null hypothesis simultaneously (one test)? – user158565 Oct 29 '18 at 18:15
• I'm looking for a single test. In any individual case, $p_S(AB)$ is only slightly lower than $p_S(A)*p_S(B)$. I would not expect significant differences for individual cases. The remarkable thing is that all 45 samples are deviating in the same direction: $p_{S_i}(AB) > \hat{p_{S_i}}(AB)$. To me, this does not at all look like random deviations. – Andreas Oct 30 '18 at 7:57
• Suppose you have different P(A) and P(B) from different collection. Calculate the expected number of AB under the assumption of A and B are independent, E = NP(A)P(B) for each collection. Calculate X=(E-O)*(E-O)/E for each collection, where O is the number of AB observed in the collection. Add all X together, compare with Chi Square distribution with m degree of freedom to get your p value. – user158565 Oct 30 '18 at 15:01
• Thanks for your help. I went a bit deeper into the theory behind the Chi Square test and now this makes a lot of sense. If you want to make this an answer instead of a comment, I will flag it as the accepted answer. – Andreas Nov 21 '18 at 12:22

Suppose you have different $$\Pr(A)$$ and $$\Pr(B)$$ from different collection. Calculate the expected number of $$A\cap B$$ under the assumption of $$A$$ and $$B$$ are independent, $$E = N\Pr(A)\Pr(B)$$ for each collection, where $$N$$ is total number of events. Calculate $$X=(E-O)^2/E$$ for each collection, where $$O$$ is the number of $$A\cap B$$ observed in the collection. $$\sum_X$$ follows Chi-Square distribution with $$m$$ degree of freedom under the null hypothesis that $$A$$ and $$B$$ are independent, where $$m$$ is the number of $$X$$s. Comparing $$\sum_X$$ with Chi Square distribution with $$m$$ degree of freedom to get your $$p$$ value. If $$p$$ value is small, then the null hypothesis of independent will be rejected and $$A$$ and $$B$$ are related will be accepted.