could someone please give a concrete example to illustrate the Dirichlet distribution prior for bag-of-words?

Question

I am aware of the notion of the Dirichlet distribution, a multivariate generalization of the beta distribution.

To get parameters of the Dirichlet distribution prior for bag-of-words, this CMU Machine Learning Course gives an approach by computing word frequencies.

that course also gives this example, to give the parameters of the Dirichlet distribution prior for the word "aardvarks", it is reasonable to look at a hundred million webpages, and counts how many times the word "aardvarks" shows up.

so far so good to me, and then

that course says at 46:09

the number of total hallucinated words will be 50,000.

which is NOT convincing, the value of 50,000 is the number of vocabulary of most adults who speak English fluently, rather than the total number of words in those a hundred million webpages.

am I missing something?

could someone please give a concrete example to illustrate this approach with a particular dataset, such as 20 newsgroups dataset?

Answer 1

There's certainly a degree of subjectivity when choosing a priors. Dirichlet distribution is a generalization of beta distribution, so you should probably start with reading about them, and how their parameters influence the distribution.

Using the notation introduced in the course

$$ \hat P(X = \text{'aardvark'}) = \frac{ \text{# observed 'aardvark'} + \text{# hallucinated 'aardvark'} }{ \text{# observed words} + \text{# hallucinated words} } $$

The key concept in here are the pseudocounts, i.e. the counts of "hallucinated" words, which are parameters of your prior. As noticed in the course, the popular strategy is just use Laplace smoothing and set $\text{# hallucinated 'aardvark'} = 1$ (uniform prior). In such case, before seeing the data you assume to observed each word single time. If you then observe it once, the single observation has same "weight" as your prior, if you observe it more then once, the likelihood starts to overpower the prior. Imagine what would happen if you set $\text{# hallucinated 'aardvark'} = 100$, in such case observing some word once, or even ten times, would not change the estimate of probability almost at all. The other extreme, is to set it to zero, in such case the prior would not influence the estimate at all, but it would make you estimate the probabilities of unseen words to zeros, and this would not work in Naive Bayes algorithm, since it would zero-out everything.

The suggestion from the audience was to use informative prior, based on the frequencies from previous observations of word frequencies, normalized by the factor $\text{# hallucinated words} = 50\,000$. This value itself is not that important, as what matters is rather the individual pseudocounts. Recall that if Dirichlet's distribution parameter $\alpha_i < 1$ then the distribution of probabilities is pushed to extremes, while $\alpha_i \gg 1$, pushes the distribution of probabilities to be accumulated around the mode. So you should choose the parameter wisely, based on the values you would assume to see as observed counts and the degree of smoothing that you want to apply. For example, if you expect to see large counts, then the pseudocounts can be higher. If you expect to see each word at most once, then using the pseudocounts of one may (but does not have to) be too high.

Moreover, the "total number of words" is unlimited. For computational reasons, when working with language data, you would most likely focus on $n$ most frequent words. In such case, you can assign the $\text{# hallucinated words} = n$.