What exactly is the alpha in the Dirichlet distribution?

Question

I'm fairly new to Bayesian statistics and I came across a corrected correlation measure, SparCC, that uses the Dirichlet process in the backend of it's algorithm. I have been trying to go through the algorithm step-by-step to really understand what is happening but I am not sure exactly what the alpha vector parameter does in a Dirichlet distribution and how it normalizes the alpha vector parameter?

The implementation is in Python using NumPy: https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.dirichlet.html

The docs say:

alpha : array Parameter of the distribution (k dimension for sample of dimension k).

My questions:

1. How do the alphas affect the distribution?;

2. How are the alphas being normalized?; and

3. What happens when the alphas are not integers?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Reproducibility
np.random.seed(0)

# Integer values for alphas
alphas = np.arange(10)
# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

# Dirichlet Distribution
dd = np.random.dirichlet(alphas) 
# array([ 0.        ,  0.0175113 ,  0.00224837,  0.1041491 ,  0.1264133 ,
#         0.06936311,  0.13086698,  0.15698674,  0.13608845,  0.25637266])

# Plot
ax = pd.Series(dd).plot()
ax.set_xlabel("alpha")
ax.set_ylabel("Dirichlet Draw")

Answer 1

The Dirichlet distribution is a multivariate probability distribution that describes $k\ge2$ variables $X_1,\dots,X_k$, such that each $x_i \in (0,1)$ and $\sum_{i=1}^N x_i = 1$, that is parametrized by a vector of positive-valued parameters $\boldsymbol{\alpha} = (\alpha_1,\dots,\alpha_k)$. The parameters do not have to be integers, they only need to be positive real numbers. They are not "normalized" in any way, they are parameters of this distribution.

The Dirichlet distribution is a generalization of the beta distribution into multiple dimensions, so you can start by learning about the beta distribution. Beta is a univariate distribution of a random variable $X \in (0,1)$ parameterized by parameters $\alpha$ and $\beta$. The nice intuition about it comes if you recall that it is a conjugate prior for the binomial distribution and if we assume a beta prior parameterized by $\alpha$ and $\beta$ for the binomial distribution's probability parameter $p$, then the posterior distribution of $p$ is also a beta distribution parameterized by $\alpha' = \alpha + \text{number of successes}$ and $\beta' = \beta + \text{number of failures}$. So you can think of $\alpha$ and $\beta$ as of pseudocounts (they do not need to be integers) of successes and failures (check also this thread).

In the case of the Dirichlet distribution, it is a conjugate prior for the multinomial distribution. If in the case of the binomial distribution we can think of it in terms of drawing white and black balls with replacement from the urn, then in case of the multinomial distribution we are drawing with replacement $N$ balls appearing in $k$ colors, where each of colors of the balls can be drawn with probabilities $p_1,\dots,p_k$. The Dirichlet distribution is a conjugate prior for $p_1,\dots,p_k$ probabilities and $\alpha_1,\dots,\alpha_k$ parameters can be thought of as pseudocounts of balls of each color assumed a priori (but you should read also about the pitfalls of such reasoning). In Dirichlet-multinomial model $\alpha_1,\dots,\alpha_k$ get updated by summing them with observed counts in each category: $\alpha_1+n_1,\dots,\alpha_k+n_k$ in similar fashion as in case of beta-binomial model.

The higher value of $\alpha_i$, the greater "weight" of $X_i$ and the greater amount of the total "mass" is assigned to it (recall that in total it must be $x_1+\dots+x_k=1$). If all $\alpha_i$ are equal, the distribution is symmetric. If $\alpha_i < 1$, it can be thought of as anti-weight that pushes away $x_i$ toward extremes, while when it is high, it attracts $x_i$ toward some central value (central in the sense that all points are concentrated around it, not in the sense that it is symmetrically central). If $\alpha_1 = \dots = \alpha_k = 1$, then the points are uniformly distributed.

This can be seen on the plots below, where you can see trivariate Dirichlet distributions (unfortunately we can produce reasonable plots only up to three dimensions) parameterized by (a) $\alpha_1 = \alpha_2 = \alpha_3 = 1$, (b) $\alpha_1 = \alpha_2 = \alpha_3 = 10$, (c) $\alpha_1 = 1, \alpha_2 = 10, \alpha_3 = 5$, (d) $\alpha_1 = \alpha_2 = \alpha_3 = 0.2$.

The Dirichlet distribution is sometimes called a "distribution over distributions" since it can be thought of as a distribution of probabilities themselves. Notice that since each $x_i \in (0,1)$ and $\sum_{i=1}^k x_i = 1$, then $x_i$'s are consistent with the first and second axioms of probability. So you can use the Dirichlet distribution as a distribution of probabilities for discrete events described by distributions such as categorical or multinomial. It is not true that it is a distribution over any distributions, for example it is not related to probabilities of continuous random variables, or even some discrete ones (e.g. a Poisson distributed random variable describes probabilities of observing values that are any natural numbers, so to use a Dirichlet distribution over their probabilities, you'd need an infinite number of random variables $k$).

Answer 2

Disclaimer: I have never worked with this distribution before. This answer is based on this wikipedia article and my interpretation of it.

---

The Dirichlet distribution is a multivariate probability distribution with similar properties to the Beta distribution.

The PDF is defined as follows:

$$\{x_1, \dots, x_K\} \sim\frac{1}{B(\boldsymbol{\alpha})}\prod_{i=1}^Kx_i^{\alpha_i - 1}$$

with $K \geq 2$, $x_i \in (0,1)$ and $\sum_{i=1}^Kx_i = 1$.

If we look at the closely related Beta distribution:

$$\{x_1, x_2 (=1-x_1)\} \sim \frac{1}{B(\alpha,\beta)}x_1^{\alpha-1}x_2^{\beta-1}$$

we can see that these two distributions are the same if $K=2$. So let's base our interpretation on that first and then generalise to $K>2$.

---

In Bayesian statistics, the Beta distribution is used as a conjugate prior for binomial parameters (See Beta distribution). The prior can be defined as some prior knowledge on $\alpha$ and $\beta$ (or in line with the Dirichlet distribution $\alpha_1$ and $\alpha_2$). If some binomial trial then has $A$ successes and $B$ failures, the posterior distribution is then as follows: $\alpha_{1,pos} = \alpha_1 + A$ and $\alpha_{2,pos}=\alpha_2 + B$. (I won't work this out, as this is probably one of the first things you learn with Bayesian statistics).

So the Beta distribution then represents some posterior distribution on $x_1$ and $x_2 (=1-x_1)$, which can be interpreted as the probability of successes and failures respectively in a Binomial distribution. And the more data ($A$ and $B$) you have, the narrower this posterior distribution will be.

---

Now we know how the distribution works for $K=2$, we can generalise it to work for a multinomial distribution instead of a binomial. Which means that instead of two possible outcomes (success or failure), we will allow for $K$ outcomes (see why it generalises to Beta/Binom if $K=2$?). Each of these $K$ outcomes will have a probability $x_i$, which sums to 1 as probabilities do.

$\alpha_i$ then takes a similar role to the $\alpha_1$ and $\alpha_2$ in the Beta distribution as a prior for $x_i$ and gets updated in a similar fashion.

So now to get to your questions:

How do the alphas affect the distribution?

The distribution is bounded by the restrictions $x_i \in (0,1)$ and $\sum_{i=1}^Kx_i = 1$. The $\alpha_i$ determine which parts of the $K$-dimensional space get the most mass. You can see this in this image (not embedding it here because I don't own the picture). The more data there is in the posterior (using that interpretation) the higher the $\sum_{i=1}^K\alpha_i$, so the more certain you are of the value of $x_i$, or the probabilities for each of the outcomes. This means that the density will be more concentrated.

How are the alphas being normalized?

The normalisation of the distribution (making sure the integral equals 1) goes through the term $B(\boldsymbol{\alpha})$:

$$B(\boldsymbol{\alpha}) = \frac{\prod_{i=1}^K\Gamma(\alpha_i)}{\Gamma(\sum_{i=1}^K\alpha_i)}$$

Again if we look at the case $K=2$ we can see that the normalising factor is the same as in the Beta distribution, which used the following:

$$B(\alpha_1, \alpha_2) = \frac{\Gamma(\alpha_1)\Gamma(\alpha_2)}{\Gamma(\alpha_1+\alpha_2)}$$

This extends to

$$B(\boldsymbol{\alpha}) = \frac{\Gamma(\alpha_1)\Gamma(\alpha_2)\dots\Gamma(\alpha_K)}{\Gamma(\alpha_1+\alpha_2+\dots+\alpha_K)}$$

What happens when the alphas are not integers?

The interpretation doesn't change for $\alpha_i>1$, but as you can see in the image I linked before, if $\alpha_i < 1$ the mass of the distribution accumulates at the edges of the range for $x_i$. $K$ on the other hand has to be an integer and $K\geq2$.