I am wondering how to best approximate the information value of a word $x$ in general human speech. By information value I literally mean its entropy:

$H(X) = \mathbb{E}_{X} [I(x)] = -\sum_{x \in \mathbb{X}} p(x) \log p(x).$

with $p(x)$ the probability of the word $x$ occurring in general human speech (assuming nothing of the participants or their context).

But, what is the state-of-the-art in the study of $p(x)$ and its properties? Are there any known "good estimates" of $p(x)$? (at least in standard English)

I would imagine that the number of websites returned by Google or statistics of word frequency in Wikipedia can help approximate $p(x)$. But has the statistics or computational linguistics community ever tried to estimate $p(x)$? If so, how?

From what I understand, the security community resorts to approximations of $p(x)$ for estimating password strength (e.g. see this Cryptography thread XKCD #936: Short complex password, or long dictionary passphrase?). So I imagine that good sampling distributions of $p(x)$ exist and are used at least by some communities.

Other notes:

  • According to Wikipedia, the Oxford English Dictionary had 301,100 main entries as of 30 November, 2005. So $p(x)$ can be understood as a vector with ~300K values between 0 and 1.
  • 3
    $\begingroup$ This other xkcd page may be of interest. $\endgroup$ – Sycorax says Reinstate Monica Apr 1 '13 at 18:22
  • 1
    $\begingroup$ I'm concerned that the question asking for repositories holding estimates of $p(x)$ seems to be straying pretty solidly toward this item in the faq "Questions about obtaining particular datasets are off-topic". Further, it's not the 'statistics community' that would seek to obtain such data, but researchers in a specific area (such as speech researchers); they might tend to be users of (rather than developers of) statistical methodology. $\endgroup$ – Glen_b -Reinstate Monica Apr 1 '13 at 22:24
  • $\begingroup$ @Glen_b I see the point on the repository. I will rephrase the question. $\endgroup$ – Amelio Vazquez-Reina Apr 1 '13 at 22:26

This is a surprisingly frustrating thing to pin down.

Shannon looked at this in one of the earliest infomation theory papers (Shannon, 1951) and estimated the entropy of printed text at around 1 bit/character, using a neat 'guessing game' paradigm. In the same paper, he estimates the entropy of a word at around 12 bits. Shannon, however, used a relatively small data set[*] and it turns out that the entropy depends on many factors. @Lmorin mentioned time above, but other relevant factors include the topic (children's books have a limited vocabulary, for example), modality, context, author's style, and so on!

The general term for $P(\textrm{word})$ is a language model and computational linguists/natural language processing researchers spend a lot of time building them because they're very useful [**]. The models contain the per-character or per-word probability. A language model also often contains information about transitions between words. A trigram (or 3rd-order model) looks like $P(\textrm{Word}_n | \textrm{Word}_{n-1}, \textrm{Word}_{n-2}$). However, the probabilites usually aren't taken directly from the data---it's exceedingly sparse---so there are various smoothing/interpolation/back-off methods designed to produce reasonable probability distributions.

Any decent NLP textbook should have a chapter on language modelling. You might start with Chapter 6 of Manning and Schutze's "dice book" or Chapter 4 of Jurafsky and Martin. However, language models are so useful that they'll also show up in contexts as diverse as speech recognition, information retrieval, and even bioinformatics. This slide deck might be a good place to start if you want to read more.

There's also a fair bit of literature about human language models. Noam Chompsky famously ranted about how "the notion of 'probability of a sentence' is an entirely useless one, under any known interpretation of this term.” but a lot of people have subsequently disagreed. If you're interested in this, you may want to look for papers on 'statistical learning' (not machine learning; psychologists use the term a bit differently).

[*] It was the 50s and he was presumably doing most of this manually, so…fair enough!

[**] In particular, it can help resolve ambguities. Suppose you can't tell if a blob is actually a 'T' or an 'I' by itself. If one alternative produces a common word and one doesn't (Iherefore vs Therefore), it's pretty clear which one you should pick.


The better answer I can give you:



As you can see $p(x)$ is in fact $p(x,t)$, I think there's a lot of interesting (or funny) to do with this information. (what happened to parenthesis in the 17° century ? http://books.google.com/ngrams/graph?content=%5B%28%5D%2C%5B%29%5D&year_start=1600&year_end=2000&corpus=15&smoothing=3&share=)


I don't know if you can get all that data easily.

You only have a percentage of books in which the ngrams appears, it's not really what you wanted. I think that the 2 probability are linked but the link will be hard to find without making questionnable assumptions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.