# Correlate two variables, with many (0,0) values?

Suppose you have a large but finite collection of tweets. You want to know whether talking about football tends to correlate with talking about basketball. You can generate a table for a few hundred users with x's of "NFL" mentions, and y's of "NBA" mentions for each user. Now consider the case where over half of them are (0,0). I actually have such tables for many word pairs: some graphs look like a messy y=mx, some look as if bounded by y=1/mx, some are one quadrant of a shotgun blast.

Q: is there any mathematically sound way of describing the statistics, the correlations, when so many values are (0,0)?

Intuitively speaking, I've run into two problems:

1) Using a simple linear correlation function in a spreadsheet, I seem to get similar correlation (r^2) values whether "I can tell" it's a shotgun or it's a y=1/x bounded system (i.e., exclusivity). I.d like a measure that distinguishes between exclusivity and no relation at all.

2) Sometimes I've generated graphs which look like y=1/x, and proves a case of exclusivity (such as sheep vs. goats) which I already believe to be true. Other times for very similar concepts, however, I see the same graph shape which implies exclusivity, a discrepancy that seems illogical (such as "football" vs. "NFL"), unless I've somehow discovered distinct populations that use different words to describe a similar interest. I'm wondering if what my intuitive response to these exclusivity graphs is ignoring hundreds of points squished at the origin : (1,1)'s.

I hope for a statistical operation that would take my gut feel out of this analysis. Thanks

• How can something be bounded by $1/mx$ if you have (0,0) values? – Sid May 21 '15 at 2:08
• Take for example, number of times someone voted Democratic vs. Republican. It is likely that the spread of points will be BOUNDED by y=1/x, as in most points hugging the two axes. Forgive if I used wrong term, but (0,0) is among the set of points fenced in by y=1/x. – pterandon May 21 '15 at 10:12
• Here are examples of the three cases. imgur.com/fYEGgcY Perhaps I should start this over? Consider how one of these plots has a point of (700,5), "Mathematically", I would guess that this is similar to (700,0), so I wouldn't want to but this point in same bucket as the (1,1). So I think bucketing isn't best approach. – pterandon May 22 '15 at 11:54
• Your figure helps understand things. The figures suggest that for a given x, (a) the mean of y is increasing linearly and (b) the variance is also increasing linearly. This suggests you could use a poisson generalized linear model. The regression coefficients might match up with your intutions – Sid May 22 '15 at 17:49
• Okay, great, can you point me to a primer on that? A way to find the mu, or ideally an Excel formula? – pterandon May 23 '15 at 11:15

$$P(User_{NFL} | User_{NBA}) = \frac{P(User_{NFL} \cap User_{NBA})}{P(User_{NBA})}$$
\begin{align} allConfidence(A,B) &= min \big\{P(A|B), P(B|A)\big\}\\ maxConfidence(A,B) &= max \big\{P(A|B), P(B|A)\big\}\\ Kulczynski(A,B) &= \frac{1}{2}\big(P(A|B) + P(B|A) \big)\\ \end{align}