Why are Likelihood Functions from Exponential Family Convex?

Question

I am trying to understand why the Likelihood Function of any distribution from the Exponential Family is Convex. This convexity property makes optimization/parameter estimation easier when working with regression models (e.g. GLM) that are based on probability distributions from the Exponential Family.

I spent the day making notes about this, here is what I have so far.

Part 1 - Function Normalization: In general, suppose we have a function $f(x)$. Now, define a new function $g(x) = \frac{f(x)}{\int f(x) dx}$. This new function $g(x)$ will integrate to 1:

$$\int \frac{f(x)}{\int f(x) \, dx} \, dx = \frac{1}{\int f(x) \, dx} \cdot \int f(x) \, dx = 1$$

In the context of the Exponential Family, define (incomplete) $f^*(x|\theta) = h(x) \exp\left(\eta(\theta)T(x)\right)$. This still needs to be normalized and we want to create a normalized function $f(x|\theta)$. Using the same logic, we should be able to do this as:

$$f(x|\theta) = \frac{f^*(x|\theta)}{\int f^*(x|\theta) dx}$$

Substituting $f^*(x|\theta)$:

$$f(x|\theta) = \frac{h(x) \exp\left(\eta(\theta)T(x)\right)}{\int h(x) \exp\left(\eta(\theta)T(x)\right) dx}$$

We can simplify ( $a = e^{\log a}$ and $\frac{e^a}{e^b} = e^{a-b}$), and rewrite as:

$$f(x|\theta) = \frac{h(x) \exp\left(\eta(\theta)T(x)\right)}{\exp\left(\log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx\right)}$$

$$f(x|\theta) = h(x) \exp\left(\eta(\theta)T(x) - \log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx\right)$$

If we define $A(\theta) = \log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx$, we can see that the original function is now normalized:

$$f(x|\theta) = h(x) \exp\left(\eta(\theta)T(x) - A(\theta)\right)$$

Part 2 - Derivatives of the Likelihood from the Exponential Family (Score Function) :

One of the advantages of the Exponential Family is that any distribution belonging to this family has the same structure of likelihood function and derivatives of the likelihood:

$$p(x;\theta) = h(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) \right)$$

Earlier, we defined the log-partition function $A(\theta)$ as:

$$A(\theta) = \log \int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx$$

The log-likelihood function for an exponential family distribution is given by:

$$\ell(\theta; x) = \log p(x;\theta) = \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) + \log h(x)$$

Taking the derivative of the log-likelihood with respect to $\theta$, we get:

$$\frac{\partial \ell(\theta; x)}{\partial \theta} = T(x) - \frac{\partial A(\theta)}{\partial \theta}$$

The second derivative (univariate case) of the log-likelihood is:

$$\frac{\partial^2 \ell(\theta; x)}{\partial \theta^2} = - \frac{\partial^2 A(\theta)}{\partial \theta^2}$$

Part 3: Properties of the Derivatives - "Hand-wavy" part

This part is not too clear for me, but I think I can understand the general idea. Focusing on the log-partition function $A(\theta)$, we see:

$$ A(\theta) = \log \int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx $$

The first derivative of the log partition function is:

$$ \frac{\partial A(\theta)}{\partial \theta} = \frac{1}{\int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx} \int_X T(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx $$

We know that $x$ has a probability distribution of $p(x;\theta) = h(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) \right)$. Thus:

$$ p(x;\theta) = \frac{h(x) \exp(\theta T(x))}{\exp(A(\theta))} = \frac{h(x) \exp(\theta T(x))}{\exp \left[ \log \int \exp(\theta T(x)) h(x) \, dx \right]} = \frac{h(x)\exp(\theta T(x))}{\int \exp(\theta T(x)) h(x) \, dx} $$

And in general:

$$E[T(x)] = \int T(x) p(x;\theta) \ dx$$

So we can observe that:

$$ \frac{\partial A(\theta)}{\partial \theta} = E[T(X)] $$

The second derivative of the log partition function is (I haven't worked this out myself yet):

$$ \frac{\partial^2 A(\theta)}{\partial \theta^2} = Var[T(X)] $$

Since variance is always non-negative, the second derivative of the log partition function $A(\theta)$ must be non-negative. And from Calculus, if the second derivative of a function is non-negative, the function itself is either convex or concave - but certainly the function will not be non-convex (not sure if this is a correct application for a multi-variable case).

In our case, our function is the likelihood of the Exponential Family. After doing all this work, I believe that the likelihood function from any distribution belonging to the Exponential Family will always be convex or concave - thus greatly simplifying the optimization/parameter estimation process.

Is my analysis correct?

Answer 1

counterexample

Say we have a normal distribution with fixed $\sigma = 1$ and $\mu = \text{sign}(\theta) |\theta|^{0.5}$

$$f(x|\theta) = \frac{1}{\sqrt{2\pi}} \exp \left( - \frac{x-\text{sign}(\theta)|\theta|^{0.5}}{2} \right)^2$$

If we observe $x=0$ then the likelihood looks like:

---

If instead you are talking about the natural exponential family instead of the exponential family (a switch that you make between part 1 and part 2), then your analysis is correct.