Skip to main content
added 234 characters in body
Source Link
User1865345
  • 10.3k
  • 12
  • 23
  • 40

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness$^\dagger$: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

--

$\dagger$ link expired. See whuber's comment.

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness$^\dagger$: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

--

$\dagger$ link expired. See whuber's comment.

deleted 4 characters in body
Source Link
whuber
  • 333.5k
  • 63
  • 792
  • 1.3k

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which findfinds stationary points of the likelihood, among which will be found anall interior global maximummaxima if itone exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which find stationary points of the likelihood, among which will be found an interior global maximum if it exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

edited body
Source Link
whuber
  • 333.5k
  • 63
  • 792
  • 1.3k

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i;\theta))}$$$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which find stationary points of the likelihood, among which will be found an interior global maximum if it exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i;\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which find stationary points of the likelihood, among which will be found an interior global maximum if it exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which find stationary points of the likelihood, among which will be found an interior global maximum if it exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

added 3 characters in body
Source Link
whuber
  • 333.5k
  • 63
  • 792
  • 1.3k
Loading
Source Link
whuber
  • 333.5k
  • 63
  • 792
  • 1.3k
Loading