# What does "a distribution over distributions" mean?

I am reading a paper about Dirichlet Processes, and it said "A Dirichlet Process is also a distribution over distributions." What does that mean?

• The finite Dirichlet distribution is also a distribution over distributions, namely over multinomial distributions. That means if you draw a sample from it, you get a random distribution. A biased (weighted) die can be described by a multinomial distribution. A machine that makes biased die with some random error can be described by a Dirichlet distribution--a distribution over distributions. Feb 19 '14 at 22:45

## 2 Answers

Suppose there are boxes with chocolates, with some portion of dark and sweet chocolates. And you are interested in eating them (chocolates, not - boxes).

You pick at random one of the boxes. (Some kinds of boxes can be more common than others.) Then, you can pick at random one of the chocolates.

So you have a distribution (a collection of boxes) of distributions (chocolates in a box).

Suppose we are going to play a game in which I will flip a coin. If the coin is heads (H) then you win, if the coin is tails (T) then I win. To figure out whether to play the game, you would like to know the probability of H, P(H), and the probability of tails, P(T).

We could write down these two probabilities in a list format, just for record keeping: [P(H), P(T)]. So now, we have a discrete distribution over the possible outcomes, P(H) for H and P(T) for T. Let's call this list "L", so L = [P(H), P(T)] and if we know what L is then we know the distribution over the possible outcomes of the game.

But let's go one step further. Let's say that because I've spent a long time studying math in my life, there is a 3/4 chance that I wake up cranky on any given day. So there is a 1/4 chance that I wake up feeling happy.

Let's say that if I wake up cranky in the morning, I will pick a coin that has P(H) = 1/10 and P(T) = 9/10. But if I wake up happy, I will pick a coin with P(H) = 1/2 and P(T) = 1/2.

In that case, there would be $L_{cranky}$ = [1/10, 9/10], and $L_{happy}$ = [1/2, 1/2].

So what will the actual list of probabilities, plain old L, be for this game?

With a 3/4 chance, the list L will be $L_{cranky}$ and with a 1/4 chance L will be $L_{happy}$.

So here we have a discrete probability distribution but the values that it describes are themselves lists (L) containing probabilities. So this distribution is like a "meta" distribution to the eventual coin-based game we will play.

It is a probability distribution over a space of outcomes where the outcomes are each a probability distribution.

• So in your coin example, playing with you is identical to playing with a coin re-biased to include your crankiness in it. Is there ever an example where meta-distributions come into play and the observer cannot interpret them as "alternate probability distributions", but actually is forced to recognize their meta quality? Feb 26 at 4:16
• @frogeyedpeas the claim you can reduce it to a single bias is not correct. Though the bias of the coin (which is a random variable) does have an expected value, the actual distribution of outcomes for the game is also a random variable. For example, what if I needed to make a wager based on the standard deviation in this game? That requires taking an expected value across distributions. More generally, any aspect that factors in "risk" in some sense will require knowledge of the actual distribution of distributions.
– ely
Feb 26 at 13:46
• One other item you may find interesting is the Dirichlet distribution (and the simpler Beta distribution), since these models exist explicitly for the use case of modeling a distribution of distributions (i.e. "collapsing" to just the expected distribution isn't useful or good enough). Beta distributions in particular are heavily used in industry all over the place.
– ely
Feb 26 at 13:48