3
$\begingroup$

I am trying to understand why the Likelihood Function of any distribution from the Exponential Family is Convex. This convexity property makes optimization/parameter estimation easier when working with regression models (e.g. GLM) that are based on probability distributions from the Exponential Family.

I spent the day making notes about this, here is what I have so far.

Part 1 - Function Normalization: In general, suppose we have a function $f(x)$. Now, define a new function $g(x) = \frac{f(x)}{\int f(x) dx}$. This new function $g(x)$ will integrate to 1:

$$\int \frac{f(x)}{\int f(x) \, dx} \, dx = \frac{1}{\int f(x) \, dx} \cdot \int f(x) \, dx = 1$$

In the context of the Exponential Family, define (incomplete) $f^*(x|\theta) = h(x) \exp\left(\eta(\theta)T(x)\right)$. This still needs to be normalized and we want to create a normalized function $f(x|\theta)$. Using the same logic, we should be able to do this as:

$$f(x|\theta) = \frac{f^*(x|\theta)}{\int f^*(x|\theta) dx}$$

Substituting $f^*(x|\theta)$:

$$f(x|\theta) = \frac{h(x) \exp\left(\eta(\theta)T(x)\right)}{\int h(x) \exp\left(\eta(\theta)T(x)\right) dx}$$

We can simplify ( $a = e^{\log a}$ and $\frac{e^a}{e^b} = e^{a-b}$), and rewrite as:

$$f(x|\theta) = \frac{h(x) \exp\left(\eta(\theta)T(x)\right)}{\exp\left(\log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx\right)}$$

$$f(x|\theta) = h(x) \exp\left(\eta(\theta)T(x) - \log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx\right)$$

If we define $A(\theta) = \log \int h(x) \exp\left(\eta(\theta)T(x)\right) dx$, we can see that the original function is now normalized:

$$f(x|\theta) = h(x) \exp\left(\eta(\theta)T(x) - A(\theta)\right)$$

Part 2 - Derivatives of the Likelihood from the Exponential Family (Score Function) :

One of the advantages of the Exponential Family is that any distribution belonging to this family has the same structure of likelihood function and derivatives of the likelihood:

$$p(x;\theta) = h(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) \right)$$

Earlier, we defined the log-partition function $A(\theta)$ as:

$$A(\theta) = \log \int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx$$

The log-likelihood function for an exponential family distribution is given by:

$$\ell(\theta; x) = \log p(x;\theta) = \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) + \log h(x)$$

Taking the derivative of the log-likelihood with respect to $\theta$, we get:

$$\frac{\partial \ell(\theta; x)}{\partial \theta} = T(x) - \frac{\partial A(\theta)}{\partial \theta}$$

The second derivative (univariate case) of the log-likelihood is:

$$\frac{\partial^2 \ell(\theta; x)}{\partial \theta^2} = - \frac{\partial^2 A(\theta)}{\partial \theta^2}$$

Part 3: Properties of the Derivatives - "Hand-wavy" part

This part is not too clear for me, but I think I can understand the general idea. Focusing on the log-partition function $A(\theta)$, we see:

$$ A(\theta) = \log \int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx $$

The first derivative of the log partition function is:

$$ \frac{\partial A(\theta)}{\partial \theta} = \frac{1}{\int_X \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx} \int_X T(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) \right) h(x) dx $$

We know that $x$ has a probability distribution of $p(x;\theta) = h(x) \exp \left( \sum_{i=1}^{s} \theta_i T_i(x) - A(\theta) \right)$. Thus:

$$ p(x;\theta) = \frac{h(x) \exp(\theta T(x))}{\exp(A(\theta))} = \frac{h(x) \exp(\theta T(x))}{\exp \left[ \log \int \exp(\theta T(x)) h(x) \, dx \right]} = \frac{h(x)\exp(\theta T(x))}{\int \exp(\theta T(x)) h(x) \, dx} $$

And in general:

$$E[T(x)] = \int T(x) p(x;\theta) \ dx$$

So we can observe that:

$$ \frac{\partial A(\theta)}{\partial \theta} = E[T(X)] $$

The second derivative of the log partition function is (I haven't worked this out myself yet):

$$ \frac{\partial^2 A(\theta)}{\partial \theta^2} = Var[T(X)] $$

Since variance is always non-negative, the second derivative of the log partition function $A(\theta)$ must be non-negative. And from Calculus, if the second derivative of a function is non-negative, the function itself is either convex or concave - but certainly the function will not be non-convex (not sure if this is a correct application for a multi-variable case).

In our case, our function is the likelihood of the Exponential Family. After doing all this work, I believe that the likelihood function from any distribution belonging to the Exponential Family will always be convex or concave - thus greatly simplifying the optimization/parameter estimation process.

Is my analysis correct?

$\endgroup$
2
  • 2
    $\begingroup$ Hint: Check the Hessian of $A(\theta).$ Is it convex? If so, what can you say about the log-likelihood? Check its Hessian. $\endgroup$ Jan 14 at 5:28
  • 1
    $\begingroup$ In part 2 you seem to be talking about the natural exponential family. $\endgroup$ Jan 14 at 20:17

1 Answer 1

4
$\begingroup$

counterexample

Say we have a normal distribution with fixed $\sigma = 1$ and $\mu = \text{sign}(\theta) |\theta|^{0.5}$

$$f(x|\theta) = \frac{1}{\sqrt{2\pi}} \exp \left( - \frac{x-\text{sign}(\theta)|\theta|^{0.5}}{2} \right)^2$$

If we observe $x=0$ then the likelihood looks like:

convex likelihood for distribution from a exponential family


If instead you are talking about the natural exponential family instead of the exponential family (a switch that you make between part 1 and part 2), then your analysis is correct.

$\endgroup$
12
  • $\begingroup$ here is the likelihood function for the pdf you wrote : $$L(\theta|x) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi}} \exp \left( - \frac{(x_i-\text{sign}(\theta)|\theta|^{0.5})^2}{2} \right)$$ $\endgroup$ Jan 15 at 1:14
  • $\begingroup$ Here is some R code to plot the likelihood for x=0 at different values of theta ... i will post it in parts so its easier to run : library(ggplot2) $\endgroup$ Jan 15 at 1:16
  • $\begingroup$ likelihood <- function(theta) { x <- 0 mu <- sign(theta) * abs(theta)^0.5 return(dnorm(x, mean = mu, sd = 1)) } $\endgroup$ Jan 15 at 1:16
  • $\begingroup$ theta <- seq(-10, 10, by = 0.1) $\endgroup$ Jan 15 at 1:16
  • $\begingroup$ L <- sapply(theta, likelihood) $\endgroup$ Jan 15 at 1:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.