# Why are Likelihood Functions from Exponential Family Convex?

I am trying to understand why the Likelihood Function of any distribution from the Exponential Family is Convex. This convexity property makes optimization/parameter estimation easier when working with regression models (e.g. GLM) that are based on probability distributions from the Exponential Family.

I spent the day making notes about this, here is what I have so far.

Part 1 - Function Normalization: In general, suppose we have a function $$f(x)$$. Now, define a new function $$g(x) = \frac{f(x)}{\int f(x) dx}$$. This new function $$g(x)$$ will integrate to 1:

$$\int \frac{f(x)}{\int f(x) \, dx} \, dx = \frac{1}{\int f(x) \, dx} \cdot \int f(x) \, dx = 1$$

In the context of the Exponential Family, define (incomplete) $$f^*(x|\theta) = h(x) \exp\left(\eta(\theta)T(x)\right)$$. This still needs to be normalized and we want to create a normalized function $$f(x|\theta)$$. Using the same logic, we should be able to do this as:

$$f(x|\theta) = \frac{f^*(x|\theta)}{\int f^*(x|\theta) dx}$$

Substituting $$f^*(x|\theta)$$:

$$f(x|\theta) = \frac{h(x) \exp\left(\eta(\theta)T(x)\right)}{\int h(x) \exp\left(\eta(\theta)T(x)\right) dx}$$

We can simplify ( $$a = e^{\log a}$$ and $$\frac{e^a}{e^b} = e^{a-b}$$), and rewrite as:

$$f(x|\theta) = \frac{h(x) \exp\left(\eta(\theta)T(x)\right)}{\exp\left(\log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx\right)}$$

$$f(x|\theta) = h(x) \exp\left(\eta(\theta)T(x) - \log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx\right)$$

If we define $$A(\theta) = \log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx$$, we can see that the original function is now normalized:

$$f(x|\theta) = h(x) \exp\left(\eta(\theta)T(x) - A(\theta)\right)$$

Part 2 - Derivatives of the Likelihood from the Exponential Family (Score Function) :

One of the advantages of the Exponential Family is that any distribution belonging to this family has the same structure of likelihood function and derivatives of the likelihood:

$$p(x;\theta) = h(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) \right)$$

Earlier, we defined the log-partition function $$A(\theta)$$ as:

$$A(\theta) = \log \int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx$$

The log-likelihood function for an exponential family distribution is given by:

$$\ell(\theta; x) = \log p(x;\theta) = \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) + \log h(x)$$

Taking the derivative of the log-likelihood with respect to $$\theta$$, we get:

$$\frac{\partial \ell(\theta; x)}{\partial \theta} = T(x) - \frac{\partial A(\theta)}{\partial \theta}$$

The second derivative (univariate case) of the log-likelihood is:

$$\frac{\partial^2 \ell(\theta; x)}{\partial \theta^2} = - \frac{\partial^2 A(\theta)}{\partial \theta^2}$$

Part 3: Properties of the Derivatives - "Hand-wavy" part

This part is not too clear for me, but I think I can understand the general idea. Focusing on the log-partition function $$A(\theta)$$, we see:

$$A(\theta) = \log \int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx$$

The first derivative of the log partition function is:

$$\frac{\partial A(\theta)}{\partial \theta} = \frac{1}{\int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx} \int_X T(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx$$

We know that $$x$$ has a probability distribution of $$p(x;\theta) = h(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) \right)$$. Thus:

$$p(x;\theta) = \frac{h(x) \exp(\theta T(x))}{\exp(A(\theta))} = \frac{h(x) \exp(\theta T(x))}{\exp \left[ \log \int \exp(\theta T(x)) h(x) \, dx \right]} = \frac{h(x)\exp(\theta T(x))}{\int \exp(\theta T(x)) h(x) \, dx}$$

And in general:

$$E[T(x)] = \int T(x) p(x;\theta) \ dx$$

So we can observe that:

$$\frac{\partial A(\theta)}{\partial \theta} = E[T(X)]$$

The second derivative of the log partition function is (I haven't worked this out myself yet):

$$\frac{\partial^2 A(\theta)}{\partial \theta^2} = Var[T(X)]$$

Since variance is always non-negative, the second derivative of the log partition function $$A(\theta)$$ must be non-negative. And from Calculus, if the second derivative of a function is non-negative, the function itself is either convex or concave - but certainly the function will not be non-convex (not sure if this is a correct application for a multi-variable case).

In our case, our function is the likelihood of the Exponential Family. After doing all this work, I believe that the likelihood function from any distribution belonging to the Exponential Family will always be convex or concave - thus greatly simplifying the optimization/parameter estimation process.

Is my analysis correct?

• Hint: Check the Hessian of $A(\theta).$ Is it convex? If so, what can you say about the log-likelihood? Check its Hessian. Jan 14 at 5:28
• In part 2 you seem to be talking about the natural exponential family. Jan 14 at 20:17

## 1 Answer

counterexample

Say we have a normal distribution with fixed $$\sigma = 1$$ and $$\mu = \text{sign}(\theta) |\theta|^{0.5}$$

$$f(x|\theta) = \frac{1}{\sqrt{2\pi}} \exp \left( - \frac{x-\text{sign}(\theta)|\theta|^{0.5}}{2} \right)^2$$

If we observe $$x=0$$ then the likelihood looks like:

If instead you are talking about the natural exponential family instead of the exponential family (a switch that you make between part 1 and part 2), then your analysis is correct.

• here is the likelihood function for the pdf you wrote : $$L(\theta|x) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi}} \exp \left( - \frac{(x_i-\text{sign}(\theta)|\theta|^{0.5})^2}{2} \right)$$ Jan 15 at 1:14
• Here is some R code to plot the likelihood for x=0 at different values of theta ... i will post it in parts so its easier to run : library(ggplot2) Jan 15 at 1:16
• likelihood <- function(theta) { x <- 0 mu <- sign(theta) * abs(theta)^0.5 return(dnorm(x, mean = mu, sd = 1)) } Jan 15 at 1:16
• theta <- seq(-10, 10, by = 0.1) Jan 15 at 1:16
• L <- sapply(theta, likelihood) Jan 15 at 1:16