If Y has an exponential family distribution show that $E(\frac{\partial L}{\partial \theta}) = 0$

Question

I'm working in a self study fashion preparing for a course I'm going to take this semester in generalised linear models. The question is, given that the Y random variable belongs to the exponential family, show that: $$ E\left(\frac{\partial L}{\partial \theta}\right) = 0 $$

$$ E\left(\frac{\partial^2 L}{\partial \theta^2}\right) = -E\left(\left(\frac{\partial L}{\partial \theta}\right)^2\right) $$

I'm a bit rusty in this type of exercise, but this is what I've managed so far.

For the first part, it is easy to differentiate $L(\theta)$, where $L$ is the log likelihood. The exact parametrization of the exponential family I'm using is (treating $\phi$ as known) the following:

$$ f(y; \theta, \phi) = \exp[\phi(y\theta - b(\theta)) + c(y;\phi)] $$

And $Y$ is the random variable distributed by $f$.

I can arrive at $\frac{\partial L}{\partial \theta} = \phi y - \phi b'(\theta)$ (the functions $b$ and $c$ are differentiable). However, in order to conclude that $E(\frac{\partial L}{\partial \theta}) = 0$ I need to assume that $b'(\theta) = E(Y) = \mu$ so that I can use the properties of expectation an eliminate it altogether. And it feels like I'm cheating, since I don't have this assumption in the first place.

Calculating $E(Y) = \int_{\mathbb{R}}yf(y)dy$ just doesn't work out nicely.

The second part also culminates in me having to calculate $E(b''(\theta))$ in the same fashion.

---

In McCullagh and Nelder's book [1], they say the the relations $E(\frac{\partial L}{\partial \theta}) = 0$ and $E(\frac{\partial^2 L}{\partial \theta^2}) = -E((\frac{\partial L}{\partial \theta})^2)$ are well known (p. 28) and use it to establish $E(Y)$, so the result I'm trying to prove apparently precedes the $E(Y)$ calculation.

1: Generalized Linear Models, 2nd edition P. McCullagh and. J.A. Nelder (1989)

Answer 1

However, in order to conclude that $E(\frac{\partial L}{\partial \theta}) = 0$ I need to assume that $b'(\theta) = E(Y) = \mu$ so that I can use the properties of expectation an eliminate it altogether. And it feels like I'm cheating, since I don't have this assumption in the first place.

$b(\theta)$ is the log partition function and it's derivatives relate to the moments of $y$.

For the relation with $\mu$ see https://en.m.wikipedia.org/wiki/Partition_function_(mathematics)#Expectation_values

---

More general

Let the distribution be described by:

$$f(x,\theta) \propto e^{g(x,\theta)}$$

or with a factor $z(\theta) = \int e^{g(x,\theta)} dx $ to normalize it

$$f(x,\theta) = \frac{e^{g(x,\theta)}}{\int e^{g(x,\theta)} dx} = \frac{e^{g(x,\theta)}}{z(\theta)}$$

Then we have (where the prime $'$ denotes differentiation to $\theta$)

$$\begin{array}{}\frac{\partial}{\partial \theta} \log \left[ f(x,\theta) \right] &=& \log \left[ f(x,\theta) \right]' & =& \frac{f'(x,\theta)}{f(x,\theta)}\\ &&&=& \frac{\left(-z'(\theta)/z(\theta)^2 + g'(x,\theta)/ z(\theta) \right) \, e^{g(x,\theta)}} { e^{g(x,\theta)}/z(\theta)}\\ &&&=& \frac{-z'(\theta)}{z(\theta)} + g'(x,\theta) \end{array}$$

And now the question is whether

$$\frac{z'(\theta)}{z(\theta)} = E\left[ g'(x,\theta) \right]$$

If we can express

$$z'(\theta) = \frac{\partial}{\partial \theta} \int e^{g(x,\theta)} dx = \int \frac{\partial}{\partial \theta} e^{g(x,\theta)} dx = \int g'(x,\theta) e^{g(x,\theta)} dx$$

then

$$\frac{z'(\theta)}{z(\theta)} = \frac{\int g'(x,\theta) e^{g(x,\theta)} dx}{\int e^{g(x,\theta)} dx} = E\left[ g'(x,\theta) \right]$$

A similar derivation, more direct without the exponent is here: https://en.wikipedia.org/wiki/Score_(statistics)#Mean

Answer 2

The identities you state are completely general and are indeed well known. They apply to any likelihood function provided the log-likelihood is twice continuously differentiable and the support of the distribution doesn't depend on $\theta$. There is no need to assume an exponential family or exponential dispersion model or anything about $\mu$.

If $f(y;\theta)$ is probability density function, then by definition it satisfies $$\int f(y;\theta)\,dy=1$$ Writing this in terms of the log-likelihood function $L(\theta;y)=\log f(y;\theta)$ gives $$\int \exp L(\theta;y)\,dy=1$$ Differentiating both sides with respect to $\theta$ gives $$\int \frac{\partial L}{\partial\theta}\exp L(\theta;y)\,dy=0$$ which is the first identity $$E\left(\frac{\partial L}{\partial\theta}\right)=0.$$

Differentiating both sides a second time gives the second identity.