I'm working on a multimillion-token corpus of conversational speech trying to determine how words differ in terms of being able to form a complete utterance by themselves. To that end I've computed two frequency lists:
F_standalone: the frequency of how often a word did stand alone in the corpus.
F_overall: the frequency of the respective words overall in the corpus.
Basically, one should assume that a word's potential of forming a complete utterance by itself can be read off the ratio of
F_standalone divided by
F_overall. While this may make sense for many words it makes much less sense for words that are ultra rare in the corpus overall: you get a (maximum) ratio of 1 if a word that occurs just once in the whole corpus (of many million tokens) happens to occur as a stand-alone word.
A reproducible sample of the data I have is this:
mysample <- data.frame( Word = c("vesuvius", "cruel","pentonville","mortuary","yuck","bollocks","yeah","mm","pardon"), F_standalone = c(1,1,1,2,7,26,22875,11576,584), F_overall = c(1,35,2,3,58,140,60158,21954,877), Ratio = c(1.000000000,0.028571429,0.500000000,0.666666667,0.1206896552,0.18571429,0.3802487,0.5272843,0.6659065) ) mysample Word F_standalone F_overall Ratio 1 vesuvius 1 1 1.00000000 2 cruel 1 35 0.02857143 3 pentonville 1 2 0.50000000 4 mortuary 2 3 0.66666667 5 yuck 7 58 0.12068966 6 bollocks 26 140 0.18571429 7 yeah 22875 60158 0.38024870 8 mm 11576 21954 0.52728430 9 pardon 584 877 0.66590650
As can be seen form the sample,
vesuviusoccurs just once in either condition (as stand-alone and in the corpus as a whole) and thus has a ratio of
pentonville occurs once as a stand-alone utterance but twice overall, yielding a ratio of
0.50000000. On the other hand, words such as
pardonhave both high frequencies as stand alone items and overall and get ratios between 0.3. and 0.7.
Given the much higher observed frquencies,
pardon seem to have a much higher capability of forming an utterance by themselves than
pentonville. So the ratio surely is an unreliable metric. How can an item's capability of forming a complete utterance be determined more reliably and with more statistical rigor?