# If Y has an exponential family distribution show that $E(\frac{\partial L}{\partial \theta}) = 0$

I'm working in a self study fashion preparing for a course I'm going to take this semester in generalised linear models. The question is, given that the Y random variable belongs to the exponential family, show that: $$E\left(\frac{\partial L}{\partial \theta}\right) = 0$$

$$E\left(\frac{\partial^2 L}{\partial \theta^2}\right) = -E\left(\left(\frac{\partial L}{\partial \theta}\right)^2\right)$$

I'm a bit rusty in this type of exercise, but this is what I've managed so far.

For the first part, it is easy to differentiate $$L(\theta)$$, where $$L$$ is the log likelihood. The exact parametrization of the exponential family I'm using is (treating $$\phi$$ as known) the following:

$$f(y; \theta, \phi) = \exp[\phi(y\theta - b(\theta)) + c(y;\phi)]$$

And $$Y$$ is the random variable distributed by $$f$$.

I can arrive at $$\frac{\partial L}{\partial \theta} = \phi y - \phi b'(\theta)$$ (the functions $$b$$ and $$c$$ are differentiable). However, in order to conclude that $$E(\frac{\partial L}{\partial \theta}) = 0$$ I need to assume that $$b'(\theta) = E(Y) = \mu$$ so that I can use the properties of expectation an eliminate it altogether. And it feels like I'm cheating, since I don't have this assumption in the first place.

Calculating $$E(Y) = \int_{\mathbb{R}}yf(y)dy$$ just doesn't work out nicely.

The second part also culminates in me having to calculate $$E(b''(\theta))$$ in the same fashion.

In McCullagh and Nelder's book [1], they say the the relations $$E(\frac{\partial L}{\partial \theta}) = 0$$ and $$E(\frac{\partial^2 L}{\partial \theta^2}) = -E((\frac{\partial L}{\partial \theta})^2)$$ are well known (p. 28) and use it to establish $$E(Y)$$, so the result I'm trying to prove apparently precedes the $$E(Y)$$ calculation.

1: Generalized Linear Models, 2nd edition P. McCullagh and. J.A. Nelder (1989)

• Your expression is for the exponential dispersion family and not for the exponential family. Aug 16, 2020 at 21:51

However, in order to conclude that $$E(\frac{\partial L}{\partial \theta}) = 0$$ I need to assume that $$b'(\theta) = E(Y) = \mu$$ so that I can use the properties of expectation an eliminate it altogether. And it feels like I'm cheating, since I don't have this assumption in the first place.

$$b(\theta)$$ is the log partition function and it's derivatives relate to the moments of $$y$$.

For the relation with $$\mu$$ see https://en.m.wikipedia.org/wiki/Partition_function_(mathematics)#Expectation_values

### More general

Let the distribution be described by:

$$f(x,\theta) \propto e^{g(x,\theta)}$$

or with a factor $$z(\theta) = \int e^{g(x,\theta)} dx$$ to normalize it

$$f(x,\theta) = \frac{e^{g(x,\theta)}}{\int e^{g(x,\theta)} dx} = \frac{e^{g(x,\theta)}}{z(\theta)}$$

Then we have (where the prime $$'$$ denotes differentiation to $$\theta$$)

$$\begin{array}{}\frac{\partial}{\partial \theta} \log \left[ f(x,\theta) \right] &=& \log \left[ f(x,\theta) \right]' & =& \frac{f'(x,\theta)}{f(x,\theta)}\\ &&&=& \frac{\left(-z'(\theta)/z(\theta)^2 + g'(x,\theta)/ z(\theta) \right) \, e^{g(x,\theta)}} { e^{g(x,\theta)}/z(\theta)}\\ &&&=& \frac{-z'(\theta)}{z(\theta)} + g'(x,\theta) \end{array}$$

And now the question is whether

$$\frac{z'(\theta)}{z(\theta)} = E\left[ g'(x,\theta) \right]$$

If we can express

$$z'(\theta) = \frac{\partial}{\partial \theta} \int e^{g(x,\theta)} dx = \int \frac{\partial}{\partial \theta} e^{g(x,\theta)} dx = \int g'(x,\theta) e^{g(x,\theta)} dx$$

then

$$\frac{z'(\theta)}{z(\theta)} = \frac{\int g'(x,\theta) e^{g(x,\theta)} dx}{\int e^{g(x,\theta)} dx} = E\left[ g'(x,\theta) \right]$$

A similar derivation, more direct without the exponent is here: https://en.wikipedia.org/wiki/Score_(statistics)#Mean

The identities you state are completely general and are indeed well known. They apply to any likelihood function provided the log-likelihood is twice continuously differentiable and the support of the distribution doesn't depend on $$\theta$$. There is no need to assume an exponential family or exponential dispersion model or anything about $$\mu$$.

If $$f(y;\theta)$$ is probability density function, then by definition it satisfies $$\int f(y;\theta)\,dy=1$$ Writing this in terms of the log-likelihood function $$L(\theta;y)=\log f(y;\theta)$$ gives $$\int \exp L(\theta;y)\,dy=1$$ Differentiating both sides with respect to $$\theta$$ gives $$\int \frac{\partial L}{\partial\theta}\exp L(\theta;y)\,dy=0$$ which is the first identity $$E\left(\frac{\partial L}{\partial\theta}\right)=0.$$

Differentiating both sides a second time gives the second identity.