Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a question about the Dirichlet posterior distribution. Given a multinomial likelihood function it's known that the posterior is $Dir({\alpha_i + N_i})$, where $N_i$ is the number of times we've seen $i^{th}$ observation.

What happens if we start to decrease $\alpha$s for a given fixed data $D$? It seems from the form of the posterior that after some point $\alpha$s will stop affecting posterior at all. But wouldn't it be right to say that when we make $\alpha$s very small the probability mass moves to the corners of the simplex and the posterior must be affected to a greater extent? What statement is the correct one?

share|improve this question
add comment

1 Answer 1

For me the most helpful way to envision the effect of the parameters for Dirichlet is the Polya urn. Imagine you have an urn containing n different colors, with $\alpha_i$ of each color in the urn (note that you can have fractions of a ball). You reach in and draw a ball, then replace it along with another of the same color. You then repeat this an infinite amount of times and the final proportion constitutes a sample from the a Dirichlet distribution. If you have very small values for $\alpha$, it should be clear that the added ball will heavily weight you towards the color of that first draw, which explains why the mass moves to the corners of the simplex. If you have large $\alpha's$, then that first draw doesn't impact the final proportion as much.

What your posterior is essentially saying is that you started with $\alpha_i$ balls of color $i$, did a bunch of draws, and happened to draw out that color $N_i$ times. You can then imagine samples from the posterior being generated with the same process and imagine the effects the initial $\alpha$ along with the counts $N$ will have on those samples. Clearly a small value for $\alpha$ will have less effect on the posterior.

Another way to think about it is that the parameters to your Dirichlet control how much you trust your data. If you have small values of $\alpha$, then you trust your data almost entirely. Conversely, if you have large values for $\alpha$, then you are less trusting of your data and will smooth the posterior a bit more.

In summary, you are correct to say that as you decrease the $\alpha's$, they will have less effect on the posterior, but at the same time the prior will have most of its mass on the corners of the simplex.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.