6
$\begingroup$

I encountered this term in the Stanford notes about softmax regression:

we will begin by expressing the multinomial as an exponential family distribution. To parameterize a multinomial over k possible outcomes, one could use k parameters...

I searched online for an explanation and I could see it is a recurrent theme but no explanation for dummies. What does it mean to parameterize a distribution and what is the end goal?

$\endgroup$
  • 1
    $\begingroup$ From a modeling perspective it means to "state one's probability model as...". The $k$ parameters for $k$ outcomes seems like the obvious choice as it is the saturated model. But if $k$ is large, the saturated model does not perform well. So you could parameterize the same process as a hierarchical model and estimate it with a decision tree provided there is some scientific rationale for that approach. $\endgroup$ – AdamO Dec 18 '17 at 15:38
5
$\begingroup$

Reparameterization means the substitution of a function for a parameter, where the parameters are the coefficients of a distribution. References on this do not help much. Parameterization is the explicit form for a distribution. For example, the gamma distribution has two different parameterizations that are in common use:

1) The probability density function in the shape-rate parametrization is

$$f(x;\alpha,\beta) = \frac{ \beta^\alpha x^{\alpha-1} e^{-\beta x}}{\Gamma(\alpha)} \quad \text{ for } x > 0 \text{ and } \alpha, \beta > 0\;,$$ where ${\Gamma(\alpha)}$ is a complete gamma function.

2) The probability density function using the shape-scale parametrization is

$$f(x;k,\theta) = \frac{x^{k-1}e^{-\frac{x}{\theta}}}{\theta^k\Gamma(k)} \quad \text{ for } x > 0 \text{ and } k, \theta > 0.$$

From this we can see that $\beta=\dfrac{1}{\theta}$, and $k=\alpha$ from which we can state that the shape-scale parametrization ($k,\theta$, respectively) can be reparameterized to be the shape-rate parametrization ($\alpha,\beta$) by substituting the $\beta$ parameter for the reciprocal of the $\theta$ parameter. However, $k=\alpha$ is not a reparameterization, it is just a different label for the same thing; the shape parameter.

Why do we reparameterize? Suppose for parameterization number 1 above that we have $\beta$ values that are close to zero. Then regression fitting of that distribution using that parameterization would be frequently more robust than using the reciprocal parameter $\theta=\dfrac{1}{\beta}$, which alternative between iterations might make huge jumps, e.g., from 10000 to 100000. Why the increased robustness? Suppose that during fitting we make a slight transient incursion into negative $\beta$-values, for example, $-10^{-8}$ for one of the iterations. For the first parameterization, a slightly negative value is usually rectified during the next iteration. For the second parameterization above, that would yield $\theta=-100000000$, and thereafter we might be forever stuck in negative territory because of the $\pm\infty$ discontinuity at $\theta=\dfrac{1}{\beta}$ when $\beta\rightarrow0^+,0^-$, respectively.

Caution. There is a different context for parametric equations, that may cause confusion. This has nothing to do with the meaning of parameterization here.

$\endgroup$
3
$\begingroup$

It means to use a parameter or a set of parameters to describe a probability distribution.

The easiest example would be the Bernoulli distribution with one parameter $p$: Suppose we want to have a probability distribution on the discrete outcome of a coin flip. We use $p$ to represent the probability of getting a HEAD(H), in other words, the probability of getting TAIL(T) is $1-p$. Therefore the Probability Mass Function is

$$ P(X)=\begin{cases} p & \text{for }X=H \\ 1-p & \text{for }X=T \end{cases} $$

and it is parameterized by $p$.

"multinomial over k possible outcomes" is very similar, but with more parameters.


BTW, I personally think the term "multinomial" is confusing. People uses "Multinoulli Distribution" or "Categorical Distribution" to describe a distribution with multiple outcomes.

See page 62 of this book.

$\endgroup$
  • 7
    $\begingroup$ Who says "multinoulli" is a standard term rather than multinomial? IMHO, it is a horrible hybrid. $\endgroup$ – Nick Cox Dec 18 '17 at 19:00
  • 1
    $\begingroup$ @NickCox I've never heard of "multinoulli" before but I kinda like it. $\endgroup$ – JimB Dec 18 '17 at 19:26
  • $\begingroup$ @JimB I agree with you in not having heard of it before. $\endgroup$ – Nick Cox Dec 18 '17 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.