Likeness of brands using tweets - is Chi-square appropriate? I'm trying to determine if brand x is more similar to brand y or brand z using tweets. I have a data set for the words that occurred in each brand's tweets, and I'm considering using the chi-square test with the main brand's (x) word distribution being used as my expected values, and secondary brands (y, z) being used as observed values.
Is that the right test to use? 
Also, for a set of 5,000 tweets I found 2200 unique words with average occurrences of 5 and a median of 1. Does the >5 expected values rule of thumb hold with large sets of values?
 A: Several points:


*

*the "main brand"'s word distribution is also based on a sample. You should treat it as one, rather than as a population. Which is to say, rather than a goodness-of-fit statistic, it's a homogeneity-of-proportion statistic that you'd be better served by.

*Having a moderate proportion of values with expectation below 5 might not be such a problem if there's a lot of them with nearly equal expectation. However, if there are many with expectation below 1 (especially if there's a good deal of variation in expected value), you'll likely find the chi-square approximation to the distribution of the test statistic becomes quite poor. That said, these issues are (mostly) moot points because:

*The question "which is closer?" is not answered by a statistical test of whether two distributions over categories are different. 
Such a question might be better answered by measuring, rather than testing how alike they are.
You could use the chi-square statistic to measure that (but not the test, which answers a different question).
If there are many categories with small expectation (getting below 1, say, especially if there's a lot of variation in expected values), the chi-square statistic may put relatively too much weight on those (i.e. a few small-expectation categories may dominate your measure of deviation, making the statistic a very noisy measure). If the problem isn't too severe, some regularization may help (perhaps of the Agresti-Coull type, where in a $2xk$ comparison, "2" would be added to all the entries). You may alternatively find a slight improvement in using the likelihood ratio statistic as a measure of deviation, but that will still have problems if there are substantial numbers of small expected values, so you might consider whether that's the measure of deviation you really want (there Gelman discusses a paper by Perkins et al[1] which at least in the case of goodness of fit applications suggests summing the squared deviations between proportion and expected proportion, which they refer to as 'Euclidean distance'). This is not the only possible alternative idea, since there are many ways to measure deviation; you might be best served by considering what might be a good measure.
As an alternative to trying to deal with the noise associated with large numbers of small expected values, you might consider whether it's feasible to divide low-frequency words into (pre-specified) groups of similar words, which would have the effect of reducing the impact of small expected values -- and should tend to improve the ability to distinguish differences in distribution as whuber points out in comments, while simultaneously making the information substantially more interpretable. The value of being able to interpret the source of differences should not be underestimated.
[1]: Perkins, Tygert and Ward (2011),
"$\chi^2$ and classical exact tests often wildly misreport
significance; the remedy lies in computers,"
arXiv preprint, http://arxiv.org/abs/1108.4126
See also a shorter 2012 version here
