From Wikipedia and probably all statistics textbooks, we know that in the density of a Gamma random variable $$f(x; k, \theta) = \frac{1}{\Gamma(k)\theta^k}x^{k - 1}e^{-\frac{x}{\theta}}, \quad x > 0; \theta > 0, k > 0, \tag{1}$$ $k$ is called shape parameter and $\theta$ is called scale parameter. The reason that $\theta$ is referred as scale parameter is quite obvious: if $X \sim \text{Gamma}(k, \theta)$, then for any $c > 0$, $cX \sim \text{Gamma}(k, c\theta)$. In other words, for fixed $k$, the Gamma$(k, \theta)$ family is invariant under scale transformation $X \mapsto cX$. This argument makes much sense and I followed this convention for many years.

However, when I was answering this question tonight and tried using Gamma distribution as an example, I found some conflict if we considered Gamma distribution as a member of exponential-family distributions. Conventionally, an exponential family has the following representation (for example, see Chapter $7$ of this book): $$f(x; \phi, \varphi) = \exp\left[\frac{x\phi - \gamma(\phi)}{\varphi} + \tau(x, \varphi)\right].\tag{2}$$ In this representation, $\varphi$ is called scale parameter. Now let's transform $(1)$ into the form $(2)$ and see what will happen.

By letting $\varphi = \frac{1}{k}$ and $\phi = -\frac{1}{k\theta}$, $(1)$ can be written as $$f(x; \phi, \varphi) = \exp\left[\frac{x\phi - (-\log(-\phi))}{\varphi} -\frac{\log \varphi}{\varphi} + \left(\frac{1}{\varphi} - 1\right)\log x - \log\Gamma(\varphi^{-1})\right] \tag{3}$$

Comparing $(2)$ and $(3)$, it is that $\varphi = \frac{1}{k}$ should be called as scale parameter! So based on the nomenclature inside exponential family, it seems also makes sense to refer $k$ as scale parameter.

I understand that this small collision may be due to reparameterization and the fact the word "scale" is too busy in statistics and in probability. Can anyone give other explanations and is it possible that this unfortunate collision can be fixed?

  • $\begingroup$ Not important, but the gamma distribution is missing from many, I would guess most, statistical texts. It would be a rare introductory text that mentioned it, although there are some, and it features in only some monographs. So "probably all" is exaggeration. Your question is unaffected. $\endgroup$ – Nick Cox Dec 3 '15 at 9:36
  • $\begingroup$ @NickCox : I think the statement that the Gamma distribution is missing being mentioned from most textbooks is a bit extreme! $\endgroup$ – Xi'an Dec 3 '15 at 10:07
  • $\begingroup$ Most means to me "more than half", not "almost all". But I notionally count here almost all introductory texts. I don't think typical texts on regression, categorical data, time series, statistical graphics, multivariate statistics and several other fields would have cause to mention the gamma. Absolutely no dispute that it's key for other large parts of statistics, including anything Bayesian. The metric here is #texts only, and not a proxy for importance or centrality of role, which is more nearly a matter of judgment. (I can't imagine omitting the gamma from my own applications.) $\endgroup$ – Nick Cox Dec 3 '15 at 10:50

A scale parameter is defined as a parameter $\sigma$ such that, if $X\sim F(x;\mu,\sigma)$, then $\sigma^{-1}X \sim F(x;\mu,1)$, that is, when $\sigma^{-1}X$ is independent from $\sigma$.

This means $\theta$ is a scale parameter for the Gamma $\text{Ga}(\kappa,\theta)$ distribution.

When looking at the exponential family representation of the Gamma $\text{Ga}(\kappa,\theta)$ distribution with density $$f(x; \kappa, \theta) = \dfrac{1}{\Gamma(\kappa)\theta^\kappa}x^{\kappa - 1}e^{-\frac{x}{\theta}}$$ we get $$f(x; \kappa, \theta) = \exp\left\{ -\log \Gamma(\kappa)+\kappa\log\theta+\{\kappa - 1\}\log x - \theta^{-1}x\right\}$$ or $$f(x; \kappa, \theta) = \exp\left\{ (\log x\ x)'(\kappa \ \theta^{-1}) -\log \Gamma(\kappa)-\kappa\log\theta^{-1} -\log x \right\}$$ it has no extra-scale parameter where you could write $$f(x; \kappa, \theta,\varphi) = \exp\left\{ \varphi^{-1}(\log x\ x)'(\kappa \ \theta^{-1}) -\Psi(\kappa,\theta,\varphi) -\log x \right\}$$ as $\varphi$ is superfluous and not identifiable, that is, only $(\kappa \ \theta^{-1})/\varphi$ is identifiable.

As a note, Ferguson published a famous paper in 1962 where he proves that the only exponential family with location-scale parameterisation is the Normal family.


Comparing (2) and (3), it is that φ=1/k should be called as scale parameter!

According the equation you wrote it seems that both of your phis (you really need to use different letters for parameters) are part of the scale parameter. Why don't you write it in standard form so that the density is

$$f(x) \propto \exp(ax + b\log x)$$

Then it will be clear that $\frac{1}{a}$ is your scale parameter. I bet it will be a function of both of your phis.

You have:

$$f(x; \phi, \varphi) \propto \exp\left[\frac{x\phi}{\varphi} + \left(\frac{1}{\varphi} - 1\right)\log x\right] \tag{from 3}$$

so your scale parameter is $\frac{\varphi}\phi = -\theta$ as desired.

  • 1
    $\begingroup$ "I bet" is not really a proof... Maybe you could expand it and provide arguments? $\endgroup$ – Tim Dec 3 '15 at 10:05
  • 1
    $\begingroup$ @tim: It's just a question of calculation. I'm just in the middle of something. I'll check back in a few minutes to see if Solitary has given it a shot. $\endgroup$ – Neil G Dec 3 '15 at 10:06
  • 1
    $\begingroup$ @tim: (Done, as requested) $\endgroup$ – Neil G Dec 3 '15 at 10:31

Both alternatives are (as mentioned prior) given here, one with $\frac{x}{\theta }$, where $\theta$ is indeed a scale parameter, and $\beta x$, where $\beta$ is a rate scale parameter, the reciprocal of $\theta$. $\theta$ is the scale factor. Similarly for exponential distributions, $\frac{x}{\theta }$, where $\theta$ is the scale factor.

However, $\beta$ is often used for practical reasons. As mentioned, it is not a scale factor, it is a rate scaling factor. As mentioned, the gamma distribution (GD) becomes an exponential distribution (ED) when the GD shape parameter ${\alpha }$ is 1, i.e., $\frac{\beta ^{\alpha }}{\Gamma (\alpha )}x^{\alpha -1} e^{-\beta x}\to \beta e^{-\beta x}$. For time series, both the gamma and exponential distributions often use $\beta$ because it is more normally distributed than ${\theta }$, and using $\frac{x}{\theta }$ would introduce a discontinuity at $\beta = 0$. Thus, for time series ${\theta }$ is problematic for regression analysis, and $\beta$ is far more practical a measure, even if it is not "statistically" appealing from an abstract theoretical point of view.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.