# Limiting Distribution of Average of iid Gamma Variables

It can be shown that if

$X \sim Gamma(\alpha, \theta)$,

then

$\overline{X_{n}} \sim Gamma(\alpha n, \theta/n)$.

where $X_{1}, X_{2} \dots X_{n}$ are all iid and follow the distribution of $X$.

The proof of this argues that the MGF of $X$ is $M_{X}(t) = \frac{1}{(1-\theta t)^{\alpha}}$ and the MGF of an average is just $M_{X}(\frac{t}{n})^{n}$.

I was playing around and wondered that if we took the limit of $M_{\overline{X_{n}}}(t)$, then we would have

$$lim_{n \to \infty} \frac{1}{(1 - \frac{\theta t}{n})^{\alpha n}}$$

which can be show to $= e^{\theta \alpha t}$.

By a theorem in probability theory:

$M_{X_{n}}(t) \rightarrow M_{X}(t) \implies X_{n} \rightarrow X$ in distribution.

Thus, $\overline{X_{n}}$ converges to the constant random variable, $Q = \alpha \theta$.

What does this mean? That, in the limit, the average of $n$ trials of a gamma distribution will have at least $\alpha n \approx \infty$ occurences with waiting time $\theta/n \approx 0$?

TLDR: in this case $\bar{X}$ is always a gamma random variable of a certain form. But when $n$ is large, this particular type of gamma random variable "is close to" a (degenerate) normal random variable.

The expectation of a $\text{Gamma}(\alpha n, \theta /n)$ random variable, the way you have it parametrized, is $\frac{\alpha n \theta}{n} = \alpha \theta$. The variance will be $\alpha n \theta^2n^{-2}$. This is the distribution of $\bar{X}$ for any integer $n$, small or large. So, to answer your question, $E[\bar{X}] = \alpha \theta$ even for small $n$.

You don't need a large sample to get results about the mean and distribution. But if you do have a large sample, it's approximately normal. That's because the MGF looks different now after you take the limit. It's the MGF of a normal (see here) with mean $\alpha \theta$ and variance $0$. Or in other words, $P(\bar{X} \in A) \to \delta_{\alpha\theta}(A)$.

This agrees with intuition because, in the case of the normal, the parameters represent the mean and variance, and you get these numbers when you take the limit of the gamma rv's mean and variance in the above paragraph.

• I disagree with the second sentence above. This is a question about the laws of large numbers, not the Central Limit Theorem. -1 for unnecessary verbiage. – Dilip Sarwate Nov 8 '16 at 21:59
• Yes, but it's about CLT too. I'll change that sentence though by adding a parenthetical thing. – Taylor Nov 8 '16 at 22:00

Since you mention waiting times.
Assume that $\alpha$ is an integer, and that $X_1$ is the time of occurrence of (equivalently, the waiting time for) the $\alpha$-th arrival after $t = 0$ in a Poisson process of rate $\lambda = \theta^{-1}$. Recall that in a Poisson process of rate $\lambda$, there are (on average) $\lambda$ arrivals in any time interval of unit length, and the average spacing between arrivals (average waiting time) is $\lambda^{-1} = \theta$. Furthermore, $X_1$ is a Gamma random variable with parameters $(\text{order}, \text{scale}) = (\alpha,\theta)$ and so its mean $E[X_1] = \alpha\theta$ is the average time of occurrence of the $\alpha$-th arrival. Next, define $X_2$ the time elapsed between the occurrence of the $\alpha$-th arrival and the $(2\alpha)$-th arrival, and more generally, $X_k$ as the time elapsed between the occurrence of the $((k-1)\alpha)$-th arrival and the $(k\alpha)$-th arrival. Then the $X_i$ are independent $Gamma$ random variables with parameters $(\text{order}, \text{scale}) = (\alpha,\theta)$.

OK, with that as prologue, consider that $$Y_n = X_1 + X_2 + \cdots + X_n$$ is the waiting time for the $(n\alpha)$-th arrival after $t=0$. Over this long period of time, a total of $(n\alpha)$ arrivals have occurred, and so $$Z_n = \frac{Y_n}{n} = \frac{X_1 + X_2 + \cdots + X_n}{n}$$ is an estimate of the waiting time for $\alpha$ arrivals to occur, obtained by taking $n$ measurements over a long period of time and averaging the results; a method dear to the heart of every statistician. What you have determined is that as $n\to\infty$, the distribution of $Z_n$ converges to a degenerate random variable (called a constant by statistically illiterate folks) that takes on value $\alpha\theta$ with probability $1$. What this result is saying is that when $n$ is very large, $Y_n$, the waiting time for the occurrence of the $(n\alpha)$-th arrival will be approximately $n\alpha\theta$ while $Z_n$ will be very very close to $\alpha\theta$. Note the difference in wording here. $Y_n$ can have substantial deviations e.g. $O(\sqrt n)$) from the nominal value $n\alpha\theta$ which itself is increasing without bound as $n\to\infty$, but $Z_n$ will deviate from its nominal value only very slightly with the deviation being $O(\frac{1}{\sqrt{n}})$ which is getting smaller as $n \to \infty$.