What does "a distribution over distributions" mean? I am reading a paper about Dirichlet Processes, and it said "A Dirichlet Process is also a distribution over distributions." What does that mean?
 A: Suppose we are going to play a game in which I will flip a coin. If the coin is heads (H) then you win, if the coin is tails (T) then I win. To figure out whether to play the game, you would like to know the probability of H, P(H), and the probability of tails, P(T).
We could write down these two probabilities in a list format, just for record keeping: [P(H), P(T)]. So now, we have a discrete distribution over the possible outcomes, P(H) for H and P(T) for T. Let's call this list "L", so L = [P(H), P(T)] and if we know what L is then we know the distribution over the possible outcomes of the game.
But let's go one step further. Let's say that because I've spent a long time studying math in my life, there is a 3/4 chance that I wake up cranky on any given day. So there is a 1/4 chance that I wake up feeling happy.
Let's say that if I wake up cranky in the morning, I will pick a coin that has P(H) = 1/10 and P(T) = 9/10. But if I wake up happy, I will pick a coin with P(H) = 1/2 and P(T) = 1/2.
In that case, there would be $L_{cranky}$ = [1/10, 9/10], and $L_{happy}$ = [1/2, 1/2].
So what will the actual list of probabilities, plain old L, be for this game?
With a 3/4 chance, the list L will be $L_{cranky}$ and with a 1/4 chance L will be $L_{happy}$.
So here we have a discrete probability distribution but the values that it describes are themselves lists (L) containing probabilities. So this distribution is like a "meta" distribution to the eventual coin-based game we will play.
It is a probability distribution over a space of outcomes where the outcomes are each a probability distribution.
A: Suppose there are boxes with chocolates, with some portion of dark and sweet chocolates. And you are interested in eating them (chocolates, not - boxes).
You pick at random one of the boxes. (Some kinds of boxes can be more common than others.) Then, you can pick at random one of the chocolates.
So you have a distribution (a collection of boxes) of distributions (chocolates in a box).
