0
$\begingroup$

I am reading about smoothing methods for language model ( I am working on unigram model).
If you are not familiar with unigram model, it is closely related to multinomial distribution (with the multinomial coefficient ignored in most cases ). We want to model unigram with a parameter vector $\theta$.
It is well-known that unigram model suffers from the zero probability phenomenon where a lot of MLE estimated $\theta _i$ equal to zero (due to the sparsity of the data ), which resorts to smoothing methods so as to assign non-zero probabilities to $\theta_ i$s.
Bayesian smoothing with Dirichlet prior is a preferred smoothing technique in the field of information retrieval. It "encodes" pseudo-counts through the prior in the form of a Dirichlet distribution ( $\theta \sim Dir(\alpha)$ - the parameter $\alpha$ is useful because it can favors probability masses for different $\theta$s).
The $\theta$ is then determined by:
$$p(\theta|D)\propto p(D|\theta )\times p(\theta |\alpha )$$
($D=(d_ 1,d_2,...,d_n)$ comprises the counts of different words in the data document D- and is also a value vector of the multinomial distribution ).
Leveraging the fact the Dirichlet distribution is a conjugate prior of the multinomial distribution, one can determine the posterior distribution by the form of another Dirichlet distribution:
$$p(\theta |D)\sim Dir(D+\alpha)$$
The estimated $\theta$ is eventually chosen to be the mean of the $Dir(D+\alpha)$ (the mean of a Dirichlet distribution has a closed-form formula).

My question is: Why do people choose the mean instead of the mode of the posterior (why don't they perform MAP while the mode of the posterior Dirichlet distribution also has a closed-form ?)

Thank you for your attention to my question.

$\endgroup$
  • 2
    $\begingroup$ Choosing between mean and mode requires defining a notion of optimality to separate point estimators. Otherwise all estimators are acceptable. $\endgroup$ – Xi'an Oct 15 '17 at 11:10
1
$\begingroup$

Choosing appropriate statistic for getting point estimate from posterior distribution is like choosing appropriate statistic for describing your sample. We use mean when we want our estimator to be sensitive to outliers, what leads to solution that is in the middle of the mass of the distribution, i.e. we are interested in the expected value. Median is the value that splits your data in such way that 50% of it lays on each side of the median. Mode is the most "likely" value (the most commonly observed in discrete distributions). More formally, mean minimizes the squared loss (L2), while median estimated the absolute loss (L1) and mode minimizes the 0-1 loss. People often choose mean, i.e. minimizing the squared errors, because everybody does so and by convenience, but also because it has some nice properties. On another hand, median provides more robustness since it is less sensitive to outliers. As about MAP, it is also popular because MAP with uniform prior is equivalent to maximizing the likelihood, so it gives you results that are the same as frequentist equivalents. So basically it depends on what kind of properties would you like your estimator to have, there is no one-size-fits-all solution.

$\endgroup$
  • 1
    $\begingroup$ A fundamental problem with the MAP is that it depends on the choice of the dominating measure. $\endgroup$ – Xi'an May 20 '18 at 19:47
  • $\begingroup$ @Xi'an , as a frequent reader of your blog, can I ask you a somewhat unrelated question(an extension on your comment): if you happen to have time, could you please kindly recommend a Measure theory-centered publication that explains and discusses at length the concept of dominating measure (i.e. something of your choice)? $\endgroup$ – Jen Nov 12 '18 at 11:32
  • 1
    $\begingroup$ @Jen: actually, not really. I learned measure theory from courses at the Uni, including Malliavin, and did not study any particular book. I never taught it myself and hence never had to recommend a book... $\endgroup$ – Xi'an Nov 12 '18 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.