I'm working on a multimillion-token corpus of conversational speech trying to determine how words differ in terms of being able to form a complete utterance by themselves. To that end I've computed two frequency lists:

  1. F_standalone: the frequency of how often a word did stand alone in the corpus.
  2. F_overall: the frequency of the respective words overall in the corpus.

Basically, one should assume that a word's potential of forming a complete utterance by itself can be read off the ratio of F_standalone divided by F_overall. While this may make sense for many words it makes much less sense for words that are ultra rare in the corpus overall: you get a (maximum) ratio of 1 if a word that occurs just once in the whole corpus (of many million tokens) happens to occur as a stand-alone word.

A reproducible sample of the data I have is this:

mysample <- data.frame(
  Word = c("vesuvius", "cruel","pentonville","mortuary","yuck","bollocks","yeah","mm","pardon"),
  F_standalone = c(1,1,1,2,7,26,22875,11576,584),
  F_overall = c(1,35,2,3,58,140,60158,21954,877),
  Ratio = c(1.000000000,0.028571429,0.500000000,0.666666667,0.1206896552,0.18571429,0.3802487,0.5272843,0.6659065)

         Word F_standalone F_overall      Ratio
1    vesuvius            1         1 1.00000000
2       cruel            1        35 0.02857143
3 pentonville            1         2 0.50000000
4    mortuary            2         3 0.66666667
5        yuck            7        58 0.12068966
6    bollocks           26       140 0.18571429
7        yeah        22875     60158 0.38024870
8          mm        11576     21954 0.52728430
9      pardon          584       877 0.66590650

As can be seen form the sample, vesuviusoccurs just once in either condition (as stand-alone and in the corpus as a whole) and thus has a ratio of 1; pentonville occurs once as a stand-alone utterance but twice overall, yielding a ratio of 0.50000000. On the other hand, words such as yeah, mm, or pardonhave both high frequencies as stand alone items and overall and get ratios between 0.3. and 0.7.

Given the much higher observed frquencies, yeah, mm, and pardon seem to have a much higher capability of forming an utterance by themselves than vesuvius and pentonville. So the ratio surely is an unreliable metric. How can an item's capability of forming a complete utterance be determined more reliably and with more statistical rigor?


You are facing a sparsity problem. You would benefit from smoothing. For example, you could add $C=10$ or $C=50$ to each F_overall. For example, if you use $C=50$, then "vesuvius" has probability $1/51 \approx 0.020$ and "bollocks" has probability $26/190 \approx 0.14$ to appear standalone in an utterance.

This scheme does not alter much the probability of overall frequent words, but it lowers the probability of overall rare words quite a lot, which is what you want.

See additive smoothing on wikipedia for more information, interpretations, etc.

  • $\begingroup$ Thanks for your answer. Would a Fisher's exact test be appropriate? If yes, with or without the additive smoothing? $\endgroup$ Jun 26 '20 at 8:09
  • $\begingroup$ Sorry, I have no idea if and how significance tests work when you "modify" your data via smoothing... I was thinking only from a traditional NLP perspective, where you have i.e. some positive examples of "standalone" words. Based on that, you can smooth and then compute a threshold that selects all these words, which should generalize nicely. $\endgroup$
    – bomzh
    Jun 26 '20 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.