What is the effect on a Chi Squared test when one variable is nearly ubiquitous?

Question

I am working with survey data that includes a number of yes/no questions about some behaviours that may or may not be linked to each other.

In particular, there is one behaviour (we'll call it behaviour A) that has been shown to be linked to another behaviour (call it B) in other studies. However, in our study, a Chi Squared test is not showing any association between A and B. However, I think that it may be difficult to detect an association for a couple of reasons. In this particular study, 98% of the 558 participants engaged in behaviour A. And and those who didn't engaged in other related behaviours.

I am wondering whether having a sample that, by nature, has such a high prevalence of A would make it difficult to find correlations between A and other behaviours that might exist if we had a broader sample where A wasn't so prevalent (as per the other studies).

Answer 1

Unless the totals in the other direction ($B$, not-$B$) are also very highly unbalanced, the chi-squared value will be largely driven by the two numbers in the smaller category (i.e. in the not-$A$ row).

It looks like it will have about 11 values total. If the $B$ categories are close to evenly split you'd need the smaller of the two not-$A$ values to be 0 or 1 to attain significance at the 5% level -- a very strong imbalance in the not-$A$'s.

If $B$ and not-$B$ totals are not close to evenly split then it can be even worse (depending on which direction the effect is in), and even a small imbalance also reduces one of the not-$A$ expected values below 5, which may make the chi-squared approximation to the distribution of the test statistic also something of an issue.

In short, yes, the imbalance in the $A$/not-$A$ totals is a problem because you need a fairly strong effect to pick anything up.