# Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are:

1. Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why?
2. Furthermore, what constraints are there on the link function to insure convexity?

My understanding of GLMs is that they maximize a highly nonlinear likelihood function. Thus, I would imagine that there are several local maxima and the parameter set you converge to depends on the initial conditions for the optimization algorithm. However, after doing some research I have not found a single source which indicates that there are multiple local maxima. Furthermore, I am not so familiar with optimization techniques, but I know the Newton-Raphson method and IRLS algorithm are highly prone to local maxima.

Please explain if possible both on an intuitive and mathematical basis!

EDIT: dksahuji answered my original question, but I want to add the followup question [2] above. ("What constraints are there on the link function to insure convexity?")

• I think some restrictions must be required before that could be so. What is the source for the statement? Feb 24, 2014 at 4:26
• Several sites seemed to imply it however I couldnt find anything which mentioned it outright, so I also welcome its disproof! Feb 24, 2014 at 14:59
• as long as the likelihood is well defined everywhere on the domain (and ignoring some tangential numerical issues) I think yes. Under those conditions, the hessian is <0 everywhere on the domain so the likeihood is globally concave. Btw, the function are not 'highly non-linear' in the parameters and that's what matters. Feb 24, 2014 at 15:30
• Logistic, Poisson, and Gaussian regressions are often convex given a "good" link function. However, with arbitrary link function, they are not convex. Feb 25, 2014 at 22:07
• Just a note about my edit (which DankMasterDan brought up): the OP talked about minimizing the likelihood, which was obviously wrong; one would either maximize the likelihood or minimize the deviance. Feb 26, 2014 at 0:32

The definition of exponential family is:

$$p(x|\theta) = h(x)\exp(\theta^T\phi(x) - A(\theta)),$$

where $A(\theta)$ is the log partition function. Now one can prove that the following three things hold for 1D case (and they generalize to higher dimensions--you can look into properties of exponential families or log partition):

1. $\frac{dA}{d\theta} = \mathbb{E}[\phi(x)]$

2. $\frac{d^2A}{d\theta^2} = \mathbb{E}[\phi^2(x)] -\mathbb{E}[\phi(x)]^2 = {\rm var}(\phi(x))$

3. $\frac{ \partial ^2A}{\partial\theta_i\partial\theta_j} = \mathbb{E}[\phi_i(x)\phi_j(x)] -\mathbb{E}[\phi_i(x)] \mathbb{E}[\phi_j(x)] = {\rm cov}(\phi(x)) \Rightarrow \Delta^2A(\theta) = {\rm cov}(\phi(x))$

The above result prove that $A(\theta)$ is convex(as ${\rm cov}(\phi(x))$ is positive semidefinite). Now we take a look at likelihood function for MLE:

\begin{align} p(\mathcal{D}|\theta) &= \bigg[\prod_{i=1}^{N}{h(x_i)}\bigg]\ \exp\!\big(\theta^T[\sum_{i=1}^{N}\phi(x_i)] - NA(\theta)\big) \\ \log\!\big(p(\mathcal{D}|\theta)\big) &= \theta^T\bigg[\sum_{i=1}^{N}\phi(x_i)\bigg] - NA(\theta) \\ &= \theta^T[\phi(\mathcal{D})] - NA(\theta) \end{align}

Now $\theta^T[\phi(\mathcal{D})]$ is linear in theta and $-A(\theta)$ is concave. Therefore, there is a unique global maximum.

There is a generalized version called curved exponential family which would also be similar. But most of the proofs are in canonical form.

• so does this mean that GLM have a unique global minima nomatter which link function is chosen (including the noncanonical ones)? Feb 26, 2014 at 2:25
• I will try to answer as far as I percieve it. $p(x|\theta) = h(x)exp(\eta(\theta)^T\phi(x) - A(\eta(\theta)))$ is the case you are talking about. This still is concave in $\eta$ but may not be in $\theta$ so $\eta$ should be such that the whole log likelihood is concave in $\theta$. Feb 26, 2014 at 5:17
• Note that the question asks about convergence, rather than just existence, but with a few restrictions, that, too, may be doable. Feb 26, 2014 at 6:22
• As an example, there'd be a need to say we can't have things like complete separation. The convergence criteria will in practice leave us some distance from the optimum and I've seen cases where that distance can be quite large. That sort of thing might be regarded as distinct from the theoretical convergence. Feb 26, 2014 at 13:14
• @Glen_b That might be true in general but I am not able to see any reason for concave function to not converge to optima within small tolerable value. But I would say that I dont have any practical experience with these and I have just started. :) Feb 26, 2014 at 15:27

I was investigating this heavily during my thesis. The answer is that the GLM likelihood is not always convex, it is only convex under the right assumptions. A very good investigation of this was made by Nelder and Wedderburn in their paper "On the Existence and Uniqueness of the Maximum Likelihood Estimates for Certain Generalized Linear Models" which can be found at https://www.jstor.org/stable/2335080