The relationship between the gamma distribution and the normal distribution

Question

I recently found it necessary to derive a pdf for the square of a normal random variable with mean 0. For whatever reason, I chose not to normalise the variance beforehand. If I did this correctly then this pdf is as follows:

$$ N^2(x; \sigma^2) = \frac{1}{\sigma \sqrt{2 \pi} \sqrt{x}} e^{\frac{-x}{2\sigma^2}} $$

I noticed this was in fact just a parametrisation of a gamma distribution:

$$ N^2(x; \sigma^2) = \operatorname{Gamma}(x; \frac{1}{2}, 2 \sigma^2) $$

And then, from the fact the sum of two gammas (with the same scale parameter) equals another gamma, it follows that the gamma is equivalent to the sum of $k$ squared normal random variables.

$$ N^2_\Sigma(x; k, \sigma^2) = \operatorname{Gamma}(x; \frac{k}{2}, 2 \sigma^2) $$

This was a bit surprising to me. Even though I knew the $\chi^2$ distribution -- a distribution of the sum of squared standard normal RVs -- was a special case of the gamma, I didn't realise the gamma was essentially just a generalisation allowing for the sum of normal random variables of any variance. This also leads to other characterisations I had not come across before, such as the exponential distribution being equivalent to the sum of two squared normal distributions.

This is all somewhat mysterious to me. Is the normal distribution fundamental to the derivation of the gamma distribution, in the manner I outlined above? Most resources I checked make no mention that the two distributions are intrinsically related like this, or even for that matter describe how the gamma is derived. This makes me think some lower-level truth is at play that I have simply highlighted in a convoluted way?

Answer 1

As Prof. Sarwate's comment noted, the relations between squared normal and chi-square are a very widely disseminated fact - as it should be also the fact that a chi-square is just a special case of the Gamma distribution:

$$X \sim N(0,\sigma^2) \Rightarrow X^2/\sigma^2 \sim \mathcal \chi^2_1 \Rightarrow X^2 \sim \sigma^2\mathcal \chi^2_1= \text{Gamma}\left(\frac 12, 2\sigma^2\right)$$

the last equality following from the scaling property of the Gamma.

As regards the relation with the exponential, to be accurate it is the sum of two squared zero-mean normals each scaled by the variance of the other, that leads to the Exponential distribution:

$$X_1 \sim N(0,\sigma^2_1),\;\; X_2 \sim N(0,\sigma^2_2) \Rightarrow \frac{X_1^2}{\sigma^2_1}+\frac{X_2^2}{\sigma^2_2} \sim \mathcal \chi^2_2 \Rightarrow \frac{\sigma^2_2X_1^2+ \sigma^2_1X_2^2}{\sigma^2_1\sigma^2_2} \sim \mathcal \chi^2_2$$

$$ \Rightarrow \sigma^2_2X_1^2+ \sigma^2_1X_2^2 \sim \sigma^2_1\sigma^2_2\mathcal \chi^2_2 = \text{Gamma}\left(1, 2\sigma^2_1\sigma^2_2\right) = \text{Exp}( {1\over {2\sigma^2_1\sigma^2_2}})$$

But the suspicion that there is "something special" or "deeper" in the sum of two squared zero mean normals that "makes them a good model for waiting time" is unfounded: First of all, what is special about the Exponential distribution that makes it a good model for "waiting time"? Memorylessness of course, but is there something "deeper" here, or just the simple functional form of the Exponential distribution function, and the properties of $e$? Unique properties are scattered around all over Mathematics, and most of the time, they don't reflect some "deeper intuition" or "structure" - they just exist (thankfully).

Second, the square of a variable has very little relation with its level. Just consider $f(x) = x$ in, say, $[-2,\,2]$:

...or graph the standard normal density against the chi-square density: they reflect and represent totally different stochastic behaviors, even though they are so intimately related, since the second is the density of a variable that is the square of the first. The normal may be a very important pillar of the mathematical system we have developed to model stochastic behavior - but once you square it, it becomes something totally else.

Answer 2

Let us address the question posed, This is all somewhat mysterious to me. Is the normal distribution fundamental to the derivation of the gamma distribution...? No mystery really, it is simply that the normal distribution and the gamma distribution are members, among others of the exponential family of distributions, which family is defined by the ability to convert between equational forms by substitution of parameters and/or variables. As a consequence, there are many conversions by substitution between distributions, a few of which are summarized in the figure below.

LEEMIS, Lawrence M.; Jacquelyn T. MCQUESTON (February 2008). "Univariate Distribution Relationships" (PDF). American Statistician. 62 (1): 45–53. doi:10.1198/000313008x270448 cite

Here are two normal and gamma distribution relationships in greater detail (among an unknown number of others, like via chi-squared and beta).

First A more direct relationship between the gamma distribution (GD) and the normal distribution (ND) with mean zero follows. Simply put, the GD becomes normal in shape as its shape parameter is allowed to increase. Proving that that is the case is more difficult. For the GD, $$\text{GD}(z;a,b)=\begin{array}{cc} & \begin{cases} \dfrac{b^{-a} z^{a-1} e^{-\dfrac{z}{b}}}{\Gamma (a)} & z>0 \\ 0 & \text{other} \\ \end{cases} \,. \\ \end{array}$$

As the GD shape parameter $a\rightarrow \infty$, the GD shape becomes more symmetric and normal, however, as the mean increases with increasing $a$, we have to left shift the GD by $(a-1) \sqrt{\dfrac{1}{a}} k$ to hold it stationary, and finally, if we wish to maintain the same standard deviation for our shifted GD, we have to decrease the scale parameter ($b$) proportional to $\sqrt{\dfrac{1}{a}}$.

To wit, to transform a GD to a limiting case ND we set the standard deviation to be a constant ($k$) by letting $b=\sqrt{\dfrac{1}{a}} k$ and shift the GD to the left to have a mode of zero by substituting $z=(a-1) \sqrt{\dfrac{1}{a}} k+x\ .$ Then $$\text{GD}\left((a-1) \sqrt{\frac{1}{a}} k+x;\ a,\ \sqrt{\frac{1}{a}} k\right)=\begin{array}{cc} & \begin{cases} \dfrac{\left(\dfrac{k}{\sqrt{a}}\right)^{-a} e^{-\dfrac{\sqrt{a} x}{k}-a+1} \left(\dfrac{(a-1) k}{\sqrt{a}}+x\right)^{a-1}}{\Gamma (a)} & x>\dfrac{k(1-a)}{\sqrt{a}} \\ 0 & \text{other} \\ \end{cases} \\ \end{array}\,.$$

Note that in the limit as $a\rightarrow\infty$ the most negative value of $x$ for which this GD is nonzero $\rightarrow -\infty$. That is, the semi-infinite GD support becomes infinite. Taking the limit as $a\rightarrow \infty$ of the reparameterized GD, we find

$$\lim_{a\to \infty } \, \frac{\left(\frac{k}{\sqrt{a}}\right)^{-a} e^{-\frac{\sqrt{a} x}{k}-a+1} \left(\frac{(a-1) k}{\sqrt{a}}+x\right)^{a-1}}{\Gamma (a)}=\dfrac{e^{-\dfrac{x^2}{2 k^2}}}{\sqrt{2 \pi } k}=\text{ND}\left(x;0,k^2\right)$$

Graphically for $k=2$ and $a=1,2,4,8,16,32,64$ the GD is in blue and the limiting $\text{ND}\left(x;0,\ 2^2\right)$ is in orange, below

Second Let us make the point that due to the similarity of form between these distributions, one can pretty much develop relationships between the gamma and normal distributions by pulling them out of thin air. To wit, we next develop an "unfolded" gamma distribution generalization of a normal distribution.

Note first that it is the semi-infinite support of the gamma distribution that impedes a more direct relationship with the normal distribution. However, that impediment can be removed when considering the half-normal distribution, which also has a semi-infinite support. Thus, one can generalize the normal distribution (ND) by first folding it to be half-normal (HND), relating that to the generalized gamma distribution (GD), then for our tour de force, we "unfold" both (HND and GD) to make a generalized ND (a GND), thusly.

The generalized gamma distribution

$$\text{GD}\left(x;\alpha ,\beta ,\gamma ,\mu \right)=\begin{array}{cc} & \begin{cases} \dfrac{\gamma e^{-\left(\dfrac{x-\mu }{\beta }\right)^{\gamma }} \left(\dfrac{x-\mu }{\beta }\right)^{\alpha \gamma -1}}{\beta \,\Gamma (\alpha )} & x>\mu \\ 0 & \text{other} \\ \end{cases} \\ \end{array}\,,$$

Can be reparameterized to be the half-normal distribution,

$$\text{GD}\left(x;\frac{1}{2},\frac{\sqrt{\pi }}{\theta },2,0 \right)=\begin{array}{cc} & \begin{cases} \dfrac{2 \theta e^{-\dfrac{\theta ^2 x^2}{\pi }}}{\pi } & x>0 \\ 0 & \text{other} \\ \end{cases} \\ \end{array}\,\,\,=\text{HND}(x;\theta)$$

Note that $\theta=\frac{\sqrt{\pi}}{\sigma\sqrt{2}}.$ Thus, $$\text{ND}\left(x;0,\sigma^2\right)=\frac{1}{2}\text{HND}(x;\theta)+\frac{1}{2}\text{HND}(-x;\theta)=\frac{1}{2}\text{GD}\left(x;\frac{1}{2},\frac{\sqrt{\pi }}{\theta },2,0 \right)+\frac{1}{2}\text{GD}\left(-x;\frac{1}{2},\frac{\sqrt{\pi }}{\theta },2,0 \right)\,,$$

which implies that

$$ \begin{align} \text{GND}(x;\mu,\alpha,\beta) &= \frac{1}{2}\text{GD}\left(x;\frac{1}{\beta},\alpha,\beta,\mu \right)+\frac{1}{2}\text{GD}\left(-x;\frac{1}{\beta},\alpha,\beta,\mu \right)\\ &= \frac{\beta e^{-\left(\dfrac{\left|x-\mu\right|}{\alpha }\right)^{\mathrm{\Large{\beta}}}}}{2 \alpha \Gamma \left(\dfrac{1}{\beta }\right)}\\ \end{align} \,,$$

is a generalization of the normal distribution, where $\mu$ is the location, $\alpha>0$ is the scale, and $\beta>0$ is the shape and where $\beta=2$ yields a normal distribution. It includes the Laplace distribution when $\beta=1$. As $\beta\rightarrow\infty$, the density converges pointwise to a uniform density on $(\mu-\alpha,\mu+\alpha)$. Below is the generalized normal distribution plotted for $\alpha =\frac{\sqrt{\pi} }{2}\,,\beta=1/2,1,4$ in blue with the normal case $\alpha =\frac{\sqrt{\pi} }{2},\,\beta=2$ in orange.

The above can be seen as the generalized normal distribution Version 1 and in different parameterizations is known as the exponential power distribution, and the generalized error distribution, which are in turn one of several other generalized normal distributions.

Answer 3

The derivation of the chi-squared distribution from the normal distribution is much analogous to the derivation of the gamma distribution from the exponential distribution.

We should be able to generalize this:

• If the $X_i$ are independent variables from a generalized normal distribution with power coefficient $m$ then $Y = \sum_{i}^n {X_i}^m$ can be related to some scaled Chi-squared distribution (with "degrees of freedom" equal to $n/m$).

---

The analogy is as following:

Normal and Chi-squared distributions relate to the sum of squares

• The joint density distribution of multiple independent standard normal distributed variables depends on $\sum x_i^2$
  $f(x_1, x_2, ... ,x_n) = \frac{\exp \left( {-0.5\sum_{i=1}^{n}{x_i}^2}\right)}{(2\pi)^{n/2}}$

• If $X_i \sim N(0,1)$

  then $\sum_{i=1}^n {X_i}^2 \sim \chi^2(\nu)$

Exponential and gamma distributions relate to the regular sum

• The joint density distribution of multiple independent exponential distributed variables depends on $\sum x_i$

  $f(x_1, x_2, ... ,x_n) = \frac{\exp \left( -\lambda\sum_{i=1}^{n}{x_i} \right)}{\lambda^{-n}}$

• If $X_i \sim Exp(\lambda)$

  then $\sum_{i=1}^n X_i \sim \text{Gamma}(n,\lambda)$

---

The derivation can be done by a change of variables integrating not over all $x_1,x_2,...x_n$ but instead only over the summed term (this is what Pearson did in 1900). This unfolds very similar in both cases.

For the $\chi^2$ distribution:

$$\begin{array}{rcl} f_{\chi^2(n)}(s) ds &=& \frac{e^{-s/2}}{\left( 2\pi \right)^{n/2}} \frac{dV}{ds} ds\\ &=& \frac{e^{-s/2}}{\left( 2\pi \right)^{n/2}} \frac{\pi^{n/2}}{\Gamma(n/2)}s^{n/2-1} ds \\ &=& \frac{1}{2^{n/2}\Gamma(n/2)}s^{n/2-1}e^{-s/2} ds \\ \end{array}$$

Where $V(s) = \frac{\pi^{n/2}}{\Gamma (n/2+1)}s^{n/2}$ is the n-dimensional volume of an n-ball with squared radius $s$.

For the gamma distribution:

$$\begin{array}{rcl} f_{G(n,\lambda)}(s) ds &=& \frac{e^{-\lambda s}}{\lambda^{-n}} \frac{dV}{ds} ds\\ &=& \frac{e^{-\lambda s}}{\lambda^{-n}} n \frac{s^{n-1}}{n!}ds \\ &=& \frac{\lambda^{n}}{ \Gamma(n)} s^{n-1} e^{-\lambda s} ds \\ \end{array}$$

Where $V(s) = \frac{s^n}{n!}$ is the n-dimensional volume of a n-polytope with $\sum x_i < s$.

---

The gamma distribution can be seen as the waiting time $Y$ for the $n$-th event in a Poisson process which is the distributed as the sum of $n$ exponentially distributed variables.

As Alecos Papadopoulos already noted there is no deeper connection that makes sums of squared normal variables 'a good model for waiting time'. The gamma distribution is the distribution for a sum of generalized normal distributed variables. That is how the two come together.

But the type of sum and type of variables may be different. While the gamma distribution, when derived from the exponential distribution (p=1), gets the interpretation of the exponential distribution (waiting time), you can not go reverse and go back to a sum of squared Gaussian variables and use that same interpretation.

The density distribution for waiting time which falls of exponentially, and the density distribution for a Gaussian error falls of exponentially (with a square). That is another way to see the two connected.