What is distribution parameterization?

Question

I encountered this term in the Stanford notes about softmax regression:

we will begin by expressing the multinomial as an exponential family distribution. To parameterize a multinomial over k possible outcomes, one could use k parameters...

I searched online for an explanation and I could see it is a recurrent theme but no explanation for dummies. What does it mean to parameterize a distribution and what is the end goal?

Answer 1

Reparameterization means the substitution of a function for a parameter, where the parameters are the coefficients of a distribution. References on this do not help much. Parameterization is the explicit form for a distribution. For example, the gamma distribution has two different parameterizations that are in common use:

1) The probability density function in the shape-rate parametrization is

$$f(x;\alpha,\beta) = \frac{ \beta^\alpha x^{\alpha-1} e^{-\beta x}}{\Gamma(\alpha)} \quad \text{ for } x > 0 \text{ and } \alpha, \beta > 0\;,$$ where ${\Gamma(\alpha)}$ is a complete gamma function.

2) The probability density function using the shape-scale parametrization is

$$f(x;k,\theta) = \frac{x^{k-1}e^{-\frac{x}{\theta}}}{\theta^k\Gamma(k)} \quad \text{ for } x > 0 \text{ and } k, \theta > 0.$$

From this we can see that $\beta=\dfrac{1}{\theta}$, and $k=\alpha$ from which we can state that the shape-scale parametrization ($k,\theta$, respectively) can be reparameterized to be the shape-rate parametrization ($\alpha,\beta$) by substituting the $\beta$ parameter for the reciprocal of the $\theta$ parameter. However, $k=\alpha$ is not a reparameterization, it is just a different label for the same thing; the shape parameter.

Why do we reparameterize? One good reason to parameterize a particular way it to use the form that produces more normal and less skewed distributions of the parameter values that occur using that form. Thus, the reader will find that the exponential and gamma distributions are frequently parameterized in the rate form (e.g., number 1 above), as opposed to the scale form (e.g., number 2 above). Also, suppose for parameterization number 1 above that we have $\beta$ values that are close to zero. Then regression fitting of that distribution using that parameterization would be frequently more robust than using the reciprocal parameter $\theta=\dfrac{1}{\beta}$, which alternative between iterations might make huge jumps, e.g., from 10000 to 100000. Why the increased robustness? Suppose that during fitting we make a slight transient incursion into negative $\beta$-values, for example, $-10^{-8}$ for one of the iterations. For the first parameterization, a slightly negative value is usually rectified during the next iteration. For the second parameterization above, that would yield $\theta=-100000000$, and thereafter we might be forever stuck in negative territory because of the $\pm\infty$ discontinuity at $\theta=\dfrac{1}{\beta}$ when $\beta\rightarrow0^+,0^-$, respectively.

Caution. There is a different context for parametric equations, that may cause confusion. This has nothing to do with the meaning of parameterization here.

Answer 2

It means to use a parameter or a set of parameters to describe a probability distribution.

The easiest example would be the Bernoulli distribution with one parameter $p$: Suppose we want to have a probability distribution on the discrete outcome of a coin flip. We use $p$ to represent the probability of getting a HEAD(H), in other words, the probability of getting TAIL(T) is $1-p$. Therefore the Probability Mass Function is

$$ P(X)=\begin{cases} p & \text{for }X=H \\ 1-p & \text{for }X=T \end{cases} $$

and it is parameterized by $p$.

"multinomial over k possible outcomes" is very similar, but with more parameters.

---

BTW, I personally think the term "multinomial" is confusing. People uses "Multinoulli Distribution" or "Categorical Distribution" to describe a distribution with multiple outcomes.

See page 62 of this book.