I know that the multinomial distribution gives the likelihood of some vector D of occurrences to happen given a probability vector (parameters) P' i.e. P(D|P'). Now with a Dirichlet prior we are introducing a prior for those parameters. Up to my understanding, this Dirichlet prior is used for the posterior Dirichlet-Multinomial P''=P(P'|D) which gives the probability of the parameters.

If we took the MAP estimate of P'' we will get again P', but we already have P'. Shall we assume that the parameters P' are unknown in the multinomial distribution or...?

  • $\begingroup$ Your question & notation is not very clear. Is your question: why do we use priors? $\endgroup$ – Tim Feb 21 '19 at 10:09
  • $\begingroup$ @Tim yeah you can say that, but please stick to my question i.e. multinomial scenario and Dirichlet prior as I am not familiar with all distributions $\endgroup$ – Mosab Shaheen Feb 21 '19 at 10:11

Say that you have an urn with red, green, and blue balls, you draw $n$ balls from the urn with replacement. The distribution of the counts of the red, green, and blue balls, $(x_1, x_2, x_3)$, would follow multinomial distribution parametrized by probabilities $(\pi_1, \pi_2, \pi_3)$ such that $\sum_{j=1}^3 \pi_j = 1$ for drawing red, green, and blue balls respectively,

$$ (x_1, x_2, x_3) \sim \mathcal{M}(n, \,\pi_1, \pi_2, \pi_3) $$

The values of $\pi_i$ are unknown and you want to estimate them from your data (counts of the drawn balls). There are different ways of estimating the probabilities, for example you could take the maximum likelihood estimate $\hat\pi_i = \tfrac{x_i}{n}$. Another possibility is to use Bayesian approach, where instead of looking only at the data, you also assume a prior for the probabilities and then use Bayes theorem to update the prior to obtain the posterior estimate of the parameters. In case of multinomial distribution, the most popular choice for prior is Dirichlet distribution, so as a prior for $\pi_i$'s we assume

$$ (\pi_1, \pi_2, \pi_3) \sim \mathcal{D}(\alpha_1, \alpha_2, \alpha_3) $$

where $\alpha_1, \alpha_2, \alpha_3$ such that $\forall\,\alpha_i > 0$ are the parameters of the Dirichlet distribution. Because this is a conjugate prior, updating the prior to posterior is straightforward, because the posterior distribution of the estimated parameters is

$$ (\pi_1, \pi_2, \pi_3) \mid (x_1, x_2, x_3) \sim \mathcal{D}(\alpha_1 + x_1, \alpha_2 + x_2, \alpha_3 + x_3) $$

If you want a point estimate for the probabilities, you can take mean of the posterior distribution

$$ \hat\pi_i = \frac{\alpha_i + x_i}{\sum_{j=1}^3 \alpha_j + x_j} $$

If you want a practical example where it is useful, for example in natural language processing you can use Laplace smoothing, i.e. estimate the probabilities of occurrences of words using Dirichlet-multinomial model with uniform prior. It helps for the fact that when training and then predicting using a machine learning model, if in the test set you find a word that was not seen in training set, then with maximum likelihood approach you would conclude that the probability of observing such word is zero (it was not seen in training set), while in case of Bayesian estimate it is nonzero

$$ \hat\pi_i = \frac{\alpha_i + 0}{\sum_{j=1}^3 \alpha_j + x_j} $$

This makes a difference in many cases, for example with Naive Bayes algorithm you multiply all the probabilities, so multiplying by zero would zero-out everything.

| cite | improve this answer | |
  • $\begingroup$ Nice explanation, thanks Tim. I understand from this is that we do not use the posterior if we already know the probabilities, rather we use it to estimate the probabilities π. However I am still confused, If you have one instance X(x1,x2,...) how can you estimate the probabilities π using only one instance X (i.e. this instance may be an anomaly and does not reflect the actual experiment. Shouldn't be an iterative approach. If so could you please mention it in your answer. Also when you write (π1,π2,π3)∼D(α1,α2,α3) what does this symbol "∼" mean (I am not a mathematician so pls bear with me) $\endgroup$ – Mosab Shaheen Feb 21 '19 at 11:12
  • 1
    $\begingroup$ @MosabShaheen $\sim$ means "is distributed as". As about example, it was given in the answer above, just replace $(x_1, x_2, x_3)$ and $(\alpha_1, \alpha_2, \alpha_3)$ with some numbers. $\endgroup$ – Tim Feb 21 '19 at 11:23
  • $\begingroup$ thanks, I mean X(x1,x2,x3) should be Xs (i.e. many instances) because we want to train using many instances not only one. I do not know if you mean by (x1,x2,x3) a one instance but that's what I understood it. So shouldn't be many instances? $\endgroup$ – Mosab Shaheen Feb 21 '19 at 11:37
  • 1
    $\begingroup$ @MosabShaheen (π1,π2,π3)∼D(α1,α2,α3) is the definition of prior. Posterior is defined in next equation. As about multiple instances, just use totals of all the instances, so $n$ is the total number of balls drawn and $x_1$ is the total number of blue balls that you drawn. You assume the data is i.i.d. so it doesn't matter that you draw them in multiple experiments. Sum of multinomials is mumtinomial. $\endgroup$ – Tim Feb 21 '19 at 11:58
  • 1
    $\begingroup$ @MosabShaheen it means that they are random variables that follow such distribution. $\endgroup$ – Tim Feb 21 '19 at 12:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.