# What is distribution parameterization?

I encountered this term in the Stanford notes about softmax regression:

we will begin by expressing the multinomial as an exponential family distribution. To parameterize a multinomial over k possible outcomes, one could use k parameters...

I searched online for an explanation and I could see it is a recurrent theme but no explanation for dummies. What does it mean to parameterize a distribution and what is the end goal?

• From a modeling perspective it means to "state one's probability model as...". The $k$ parameters for $k$ outcomes seems like the obvious choice as it is the saturated model. But if $k$ is large, the saturated model does not perform well. So you could parameterize the same process as a hierarchical model and estimate it with a decision tree provided there is some scientific rationale for that approach. – AdamO Dec 18 '17 at 15:38

Reparameterization means the substitution of a function for a parameter, where the parameters are the coefficients of a distribution. References on this do not help much. Parameterization is the explicit form for a distribution. For example, the gamma distribution has two different parameterizations that are in common use:

1) The probability density function in the shape-rate parametrization is

$$f(x;\alpha,\beta) = \frac{ \beta^\alpha x^{\alpha-1} e^{-\beta x}}{\Gamma(\alpha)} \quad \text{ for } x > 0 \text{ and } \alpha, \beta > 0\;,$$ where ${\Gamma(\alpha)}$ is a complete gamma function.

2) The probability density function using the shape-scale parametrization is

$$f(x;k,\theta) = \frac{x^{k-1}e^{-\frac{x}{\theta}}}{\theta^k\Gamma(k)} \quad \text{ for } x > 0 \text{ and } k, \theta > 0.$$

From this we can see that $\beta=\dfrac{1}{\theta}$, and $k=\alpha$ from which we can state that the shape-scale parametrization ($k,\theta$, respectively) can be reparameterized to be the shape-rate parametrization ($\alpha,\beta$) by substituting the $\beta$ parameter for the reciprocal of the $\theta$ parameter. However, $k=\alpha$ is not a reparameterization, it is just a different label for the same thing; the shape parameter.

Why do we reparameterize? Suppose for parameterization number 1 above that we have $\beta$ values that are close to zero. Then regression fitting of that distribution using that parameterization would be frequently more robust than using the reciprocal parameter $\theta=\dfrac{1}{\beta}$, which alternative between iterations might make huge jumps, e.g., from 10000 to 100000. Why the increased robustness? Suppose that during fitting we make a slight transient incursion into negative $\beta$-values, for example, $-10^{-8}$ for one of the iterations. For the first parameterization, a slightly negative value is usually rectified during the next iteration. For the second parameterization above, that would yield $\theta=-100000000$, and thereafter we might be forever stuck in negative territory because of the $\pm\infty$ discontinuity at $\theta=\dfrac{1}{\beta}$ when $\beta\rightarrow0^+,0^-$, respectively.

Caution. There is a different context for parametric equations, that may cause confusion. This has nothing to do with the meaning of parameterization here.

It means to use a parameter or a set of parameters to describe a probability distribution.

The easiest example would be the Bernoulli distribution with one parameter $p$: Suppose we want to have a probability distribution on the discrete outcome of a coin flip. We use $p$ to represent the probability of getting a HEAD(H), in other words, the probability of getting TAIL(T) is $1-p$. Therefore the Probability Mass Function is

$$P(X)=\begin{cases} p & \text{for }X=H \\ 1-p & \text{for }X=T \end{cases}$$

and it is parameterized by $p$.

"multinomial over k possible outcomes" is very similar, but with more parameters.

BTW, I personally think the term "multinomial" is confusing. People uses "Multinoulli Distribution" or "Categorical Distribution" to describe a distribution with multiple outcomes.

See page 62 of this book.

• Who says "multinoulli" is a standard term rather than multinomial? IMHO, it is a horrible hybrid. – Nick Cox Dec 18 '17 at 19:00
• @NickCox I've never heard of "multinoulli" before but I kinda like it. – JimB Dec 18 '17 at 19:26
• @JimB I agree with you in not having heard of it before. – Nick Cox Dec 18 '17 at 19:27