In Section 3.4 of Casella and Berger's Statistical Inference, an exponential family is defined to be a set of pdfs or pmfs such that for each member $f(x | \boldsymbol{\theta})$ of the set, $$ f(x | \boldsymbol{\theta}) = h(x)c(\boldsymbol{\theta})\exp\left(\sum_{i = 1}^k w_i(\boldsymbol{\theta})t_i(x)\right), $$ where $h$ and $c$ are nonnegative and $w_1, \ldots, w_k, t_1, \ldots, t_k$ are real-valued. The natural parameterization is given as $$ f(x | \boldsymbol{\eta}) = h(x)c^*(\boldsymbol{\eta})\exp\left(\sum_{i = 1}^k \eta_i t_i(x)\right), $$ where $h$ and the $t_i$'s are the same as before. The natural parameter space is defined as $$ \left\{\eta \in \mathbb{R}^k : \int_{-\infty}^{\infty} h(x)\exp\left(\sum_{i = 1}^k \eta_i t_i(x)\right)\,dx < \infty\right\}. $$ As far as I can tell, the book doesn't give any reason why the word "natural" is used to describe the natural parameterization and the natural parameter space. What are the reasons why that word is used to describe these?
1 Answer
"Natural" is the qualification chosen for this particular class of exponential families, so one can accept it as a definition without seeking further reasons. (The alternative qualification "canonical" is also sometimes employed.)
If one looks at the generic production of exponential families, one starts$ ^\dagger$ with a reference measure corresponding to $h(\cdot)$ and a collection of $k$ statistics (aka functions) $t_i(\cdot)$. The ensuing exponential family is then created as $$f(x|\mathbf{\eta}) \propto h(x)\,\exp\{\eta_1t_1(x)+\cdots+\eta_kt_k(x)\}\tag{1}$$ i.e., by considering all densities using a linear combination of the $t_i$'s (with the exponential ensuring the function is positive). Using a linear combination of the $t_i$'s may appear as the "natural" choice. The largest range of possible linear combinations is "naturally" made of all parameters $\mathbf{\eta}$ ensuring that (1) is integrable, i.e., defines a probability density.
In historical terms, the reverse happened, namely, standard distributions such as the Normal or the Poisson distributions, were observed (by R.A. Fisher, I presume) to share this exponential structure, albeit possibly requiring a transform of the standard parameter $\mathbf{\theta}$ (such as the mean~x~variance parameter for the Normal distribution): $$f(x|\mathbf{\theta}) \propto h(x)\,\exp\{w(\theta_1)t_1(x)+\cdots+\omega(\theta_k)t_k(x)\}\tag{2}$$ Since (2) can be expressed as (1) by a reparameterisation $$\eta_1=w(\theta_1),\ldots,~\eta_k=\omega(\theta_k)$$ the parameterisation in $\eta_1,\ldots,\eta_k$ can be seen as "natural" in that it is sufficient (enough) to characterise the distribution (in the sense that the reparameterisation is not necessarily bijective). It is also the case that for this parameterisation that the cumulants can be derived from the normalising constant.
$ ^\dagger$ To quote from Morris and Lock (2014), "Starting with a solitary member distribution of an NEF, all possible distributions within that NEF can be generated via five operations: using linear functions (translations and re-scalings), convolution and division (division being the inverse of convolution), and exponential generation..."
