2
$\begingroup$

I'm working in a self study fashion preparing for a course I'm going to take this semester in generalised linear models. The question is, given that the Y random variable belongs to the exponential family, show that: $$ E(\frac{\partial L}{\partial \theta}) = 0 $$

$$ E(\frac{\partial^2 L}{\partial \theta^2}) = -E((\frac{\partial L}{\partial \theta})^2) $$

I'm a bit rusty in this type of exercise, but this is what I've managed so far.

For the first part, it is easy to differentiate $L(\theta)$, where $L$ is the log likelihood. The exact parametrization of the exponential family I'm using is (treating $\phi$ as known) the following:

$$ f(y; \theta, \phi) = exp[\phi(y\theta - b(\theta)) + c(y;\phi)] $$

And $Y$ is the random variable distributed by $f$.

I can arrive at $\frac{\partial L}{\partial \theta} = \phi y - \phi b'(\theta)$ (the functions $b$ and $c$ are differentiable). However, in order to conclude that $E(\frac{\partial L}{\partial \theta}) = 0$ I need to assume that $b'(\theta) = E(Y) = \mu$ so that I can use the properties of expectation an eliminate it altogether. And it feels like I'm cheating, since I don't have this assumption in the first place.

Calculating $E(Y) = \int_{\mathbb{R}}yf(y)dy$ just doesn't work out nicely.

The second part also culminates in me having to calculate $E(b''(\theta))$ in the same fashion.


In McCullagh and Nelder's book [1], they say the the relations $E(\frac{\partial L}{\partial \theta}) = 0$ and $E(\frac{\partial^2 L}{\partial \theta^2}) = -E((\frac{\partial L}{\partial \theta})^2)$ are well known (p. 28) and use it to establish $E(Y)$, so the result I'm trying to prove apparently precedes the $E(Y)$ calculation.

1: Generalized Linear Models, 2nd edition P. McCullagh and. J.A. Nelder (1989)

$\endgroup$
  • $\begingroup$ Your expression is for the exponential dispersion family and not for the exponential family. $\endgroup$ – Sextus Empiricus Aug 16 '20 at 21:51
1
$\begingroup$

However, in order to conclude that $E(\frac{\partial L}{\partial \theta}) = 0$ I need to assume that $b'(\theta) = E(Y) = \mu$ so that I can use the properties of expectation an eliminate it altogether. And it feels like I'm cheating, since I don't have this assumption in the first place.

$b(\theta)$ is the log partition function and it's derivatives relate to the moments of $y$.

For the relation with $\mu$ see https://en.m.wikipedia.org/wiki/Partition_function_(mathematics)#Expectation_values


More general

Let the distribution be described by:

$$f(x,\theta) \propto e^{g(x,\theta)}$$

or with a factor $z(\theta) = \int e^{g(x,\theta)} dx $ to normalize it

$$f(x,\theta) = \frac{e^{g(x,\theta)}}{\int e^{g(x,\theta)} dx} = \frac{e^{g(x,\theta)}}{z(\theta)}$$

Then we have (where the prime $'$ denotes differentiation to $\theta$)

$$\begin{array}{}\frac{\partial}{\partial \theta} \log \left[ f(x,\theta) \right] &=& \log \left[ f(x,\theta) \right]' & =& \frac{f'(x,\theta)}{f(x,\theta)}\\ &&&=& \frac{\left(-z'(\theta)/z(\theta)^2 + g'(x,\theta)/ z(\theta) \right) \, e^{g(x,\theta)}} { e^{g(x,\theta)}/z(\theta)}\\ &&&=& \frac{-z'(\theta)}{z(\theta)} + g'(x,\theta) \end{array}$$

And now the question is whether

$$\frac{z'(\theta)}{z(\theta)} = E\left[ g'(x,\theta) \right]$$

If we can express

$$z'(\theta) = \frac{\partial}{\partial \theta} \int e^{g(x,\theta)} dx = \int \frac{\partial}{\partial \theta} e^{g(x,\theta)} dx = \int g'(x,\theta) e^{g(x,\theta)} dx$$

then

$$\frac{z'(\theta)}{z(\theta)} = \frac{\int g'(x,\theta) e^{g(x,\theta)} dx}{\int e^{g(x,\theta)} dx} = E\left[ g'(x,\theta) \right]$$

A similar derivation, more direct without the exponent is here: https://en.wikipedia.org/wiki/Score_(statistics)#Mean

$\endgroup$
1
$\begingroup$

The identities you state are completely general and are indeed well known. They apply to any likelihood function provided the log-likelihood is twice continuously differentiable and the support of the distribution doesn't depend on $\theta$. There is no need to assume an exponential family or exponential dispersion model or anything about $\mu$.

If $f(y;\theta)$ is probability density function, then by definition it satisfies $$\int f(y;\theta)dy=1$$ Writing this in terms of the log-likelihood function $L(\theta;y)=\log f(y;\theta)$ gives $$\int \exp L(\theta;y)dy=1$$ Differentiating both sides with respect to $\theta$ gives $$\int \frac{\partial L}{\partial\theta}\exp L(\theta;y)dy=0$$ which is the first identity $$E\left(\frac{\partial L}{\partial\theta}\right)=0.$$

Differentiating both sides a second time gives the second identity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.