I want to find the most statistically distinct phrases in texts across different classes, where the texts have already been classified.

Suppose I have ~100,000 short text documents, where each document has a label. Let's say I have 100,000 text messages labeled with the gender of the author (male or female). I want to find what words are most associated with each gender. I am trying to do something similar to this.

I have tried combining all classes into one document and running TF-IDF, but this doesn't work very well, as the total number of documents is just 2!

My next plan is to try logistic regression with word counts as my features and look at the coefficients for each word.

Are there standard techniques for this type of problem or are there other things to try?

  • $\begingroup$ Why not divide the documents into their classes and run tf-idf on each collection? $\endgroup$
    – yeedle
    Mar 3, 2017 at 18:16

1 Answer 1


If I correctly understood your question, you want to find out which phrases - i.e., n-grams from n=1 to some length - in your binary labeled messages are more likely to be associated with either label (male or female), a.k.a., how gender-specific any n-gram found in the 100,000 documents might be.

That kind of problem is easily solved by univariate statistics, e.g., a Chi-square test [1]. You typically will count (separately per group, naturally!) how often you see each n-gram in a document. You might do well to first normalize all counts with any TF-IDF strategy you see fit (because your documents are very short: just log-dampening your counts will be fine), and then apply a Chi-square test to each n-gram and its counts. This test figures out which n-grams are more significant for one of the two groups (which in your case would be equivalent to the gender-specific words) [2].

Chi-square does so by evaluating the sampling distribution of each n-gram among your two sets of labeled documents against the expected distribution (that each n-gram is seen approximately at the same frequency in both labeled document collection, and with some statistical rigor), and you can produce a p-value to describe how likely it is to have observed the distribution you sampled, assuming the null hypothesis is true (i.e., that the n-gram is independent of the labels, aka. in your case, that the n-gram is not gender specific) [3]. That is, you will need four counts for any n-gram: the number of documents with label 1 (say, female) containing the n-gram $N_{11}$, the number of documents with label 1 that don't have that n-gram $N_{10}$, the number of documents with label 0 (say, male) containing the n-gram $N_{01}$, and finally the number of documents with label 0 but without that n-gram $N_{00}$. So $N$ is your total count of documents. Then you calculate

$$\chi^2 = \frac{N \times (N_{11} N_{00} - N_{10} N_{01})^2} {(N_{11}+N_{01})(N_{11}+N_{10})(N_{10}+N_{00})(N_{01}+N_{00})}$$

You can then look for the lower bound entry of this critical value ($\chi^2$) for one degree of freedom (you only have two classes, after all!) on any table for the chi-square distribution to find the p-value [4] and decide for yourself where to place the cutoff for classifying the n-gram as a gender-specific phrase. In my opinion, any $\chi^2 > 3.84$ would be a clear indicator of such a gender-specific phrase (and I can be at least 95% certain that decision holds true even if you were to find another 100,000 texts).

"Armed" with this example, it should also be easy for any reader to figure out that and how this can be "scaled" to more than just two classes [5].


That being said, given the latest published research on this particular issue, I think it's necessary to mention that for the past years, $\chi^2$ testing (and log-likelihood ratio tests, e.g., for collocations) has (have) been contested for not treating each text (document) independently, but as a whole, and for only looking at a 2x2 contingency table of word pairs ("bag-of-words"). That is, $\chi^2$ is effectively only looking at independent word-pairs, while other techniques like the Boostrap, t-test, or Wilcoxon's (aka., rank-sum) would evaluate all the word frequencies with a single (frequency-ordered) list [6].

More recently published research by the same authors, "Significance testing of word frequencies in corpora" [7], therefore shows that those other tests might provide a more adequate way of of finding statistically significant, distinct words among classes. Indeed, that work contains a concrete example regarding gender example in the question, comparing $\chi^2$ and the Boostrap (see Section 6, Tables 8 & 9). The authors also show that even our good old favorite, the "eat-it-all" t-tools work reassuringly well on word frequency data. However, do note the author's caveat: If your corpora are small, applying the Boostrap is probably the the only option among their other suggestions (t-test, rank-sum).

  1. http://scikit-learn.org/stable/modules/feature_selection.html
  2. http://www.learn4master.com/machine-learning/chi-square-test-for-feature-selection
  3. http://nlp.stanford.edu/IR-book/html/htmledition/feature-selectionchi2-feature-selection-1.html
  4. http://passel.unl.edu/Image/Namuth-CovertDeana956176274/chi-sqaure%20distribution%20table.PNG
  5. https://en.wikipedia.org/wiki/Chi-squared_test
  6. https://users.ics.aalto.fi/lijffijt/presentations/ICAME2012.pdf
  7. https://scholar.google.com/scholar?cluster=12251684576638059241

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.