# How to determine the stand-alone potential of words

I'm working on a multimillion-token corpus of conversational speech trying to determine how words differ in terms of being able to form a complete utterance by themselves. To that end I've computed two frequency lists:

1. F_standalone: the frequency of how often a word did stand alone in the corpus.
2. F_overall: the frequency of the respective words overall in the corpus.

Basically, one should assume that a word's potential of forming a complete utterance by itself can be read off the ratio of F_standalone divided by F_overall. While this may make sense for many words it makes much less sense for words that are ultra rare in the corpus overall: you get a (maximum) ratio of 1 if a word that occurs just once in the whole corpus (of many million tokens) happens to occur as a stand-alone word.

A reproducible sample of the data I have is this:

mysample <- data.frame(
Word = c("vesuvius", "cruel","pentonville","mortuary","yuck","bollocks","yeah","mm","pardon"),
F_standalone = c(1,1,1,2,7,26,22875,11576,584),
F_overall = c(1,35,2,3,58,140,60158,21954,877),
Ratio = c(1.000000000,0.028571429,0.500000000,0.666666667,0.1206896552,0.18571429,0.3802487,0.5272843,0.6659065)
)

mysample
Word F_standalone F_overall      Ratio
1    vesuvius            1         1 1.00000000
2       cruel            1        35 0.02857143
3 pentonville            1         2 0.50000000
4    mortuary            2         3 0.66666667
5        yuck            7        58 0.12068966
6    bollocks           26       140 0.18571429
7        yeah        22875     60158 0.38024870
8          mm        11576     21954 0.52728430
9      pardon          584       877 0.66590650


As can be seen form the sample, vesuviusoccurs just once in either condition (as stand-alone and in the corpus as a whole) and thus has a ratio of 1; pentonville occurs once as a stand-alone utterance but twice overall, yielding a ratio of 0.50000000. On the other hand, words such as yeah, mm, or pardonhave both high frequencies as stand alone items and overall and get ratios between 0.3. and 0.7.

Given the much higher observed frquencies, yeah, mm, and pardon seem to have a much higher capability of forming an utterance by themselves than vesuvius and pentonville. So the ratio surely is an unreliable metric. How can an item's capability of forming a complete utterance be determined more reliably and with more statistical rigor?

You are facing a sparsity problem. You would benefit from smoothing. For example, you could add $$C=10$$ or $$C=50$$ to each F_overall. For example, if you use $$C=50$$, then "vesuvius" has probability $$1/51 \approx 0.020$$ and "bollocks" has probability $$26/190 \approx 0.14$$ to appear standalone in an utterance.