# Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are:

1. Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why?
2. Furthermore, what constraints are there on the link function to insure convexity?

My understanding of GLMs is that they maximize a highly nonlinear likelihood function. Thus, I would imagine that there are several local maxima and the parameter set you converge to depends on the initial conditions for the optimization algorithm. However, after doing some research I have not found a single source which indicates that there are multiple local maxima. Furthermore, I am not so familiar with optimization techniques, but I know the Newton-Raphson method and IRLS algorithm are highly prone to local maxima.

Please explain if possible both on an intuitive and mathematical basis!

EDIT: dksahuji answered my original question, but I want to add the followup question  above. ("What constraints are there on the link function to insure convexity?")

• I think some restrictions must be required before that could be so. What is the source for the statement? – Glen_b -Reinstate Monica Feb 24 '14 at 4:26
• Several sites seemed to imply it however I couldnt find anything which mentioned it outright, so I also welcome its disproof! – DankMasterDan Feb 24 '14 at 14:59
• as long as the likelihood is well defined everywhere on the domain (and ignoring some tangential numerical issues) I think yes. Under those conditions, the hessian is <0 everywhere on the domain so the likeihood is globally concave. Btw, the function are not 'highly non-linear' in the parameters and that's what matters. – user603 Feb 24 '14 at 15:30
• @user603 what is your source/proof that the hessian is <0 everywhere? – DankMasterDan Feb 25 '14 at 21:10
• Logistic, Poisson, and Gaussian regressions are often convex given a "good" link function. However, with arbitrary link function, they are not convex. – Memming Feb 25 '14 at 22:07

The definition of exponential family is:

$$p(x|\theta) = h(x)\exp(\theta^T\phi(x) - A(\theta)),$$

where $A(\theta)$ is the log partition function. Now one can prove that the following three things hold for 1D case (and they generalize to higher dimensions--you can look into properties of exponential families or log partition):

1. $\frac{dA}{d\theta} = \mathbb{E}[\phi(x)]$

2. $\frac{d^2A}{d\theta^2} = \mathbb{E}[\phi^2(x)] -\mathbb{E}[\phi(x)]^2 = {\rm var}(\phi(x))$

3. $\frac{ \partial ^2A}{\partial\theta_i\partial\theta_j} = \mathbb{E}[\phi_i(x)\phi_j(x)] -\mathbb{E}[\phi_i(x)] \mathbb{E}[\phi_j(x)] = {\rm cov}(\phi(x)) \Rightarrow \Delta^2A(\theta) = {\rm cov}(\phi(x))$

The above result prove that $A(\theta)$ is convex(as ${\rm cov}(\phi(x))$ is positive semidefinite). Now we take a look at likelihood function for MLE:

\begin{align} p(\mathcal{D}|\theta) &= \bigg[\prod_{i=1}^{N}{h(x_i)}\bigg]\ \exp\!\big(\theta^T[\sum_{i=1}^{N}\phi(x_i)] - NA(\theta)\big) \\ \log\!\big(p(\mathcal{D}|\theta)\big) &= \theta^T\bigg[\sum_{i=1}^{N}\phi(x_i)\bigg] - NA(\theta) \\ &= \theta^T[\phi(\mathcal{D})] - NA(\theta) \end{align}

Now $\theta^T[\phi(\mathcal{D})]$ is linear in theta and $-A(\theta)$ is concave. Therefore, there is a unique global maximum.

There is a generalized version called curved exponential family which would also be similar. But most of the proofs are in canonical form.

• so does this mean that GLM have a unique global minima nomatter which link function is chosen (including the noncanonical ones)? – DankMasterDan Feb 26 '14 at 2:25
• I will try to answer as far as I percieve it. $p(x|\theta) = h(x)exp(\eta(\theta)^T\phi(x) - A(\eta(\theta)))$ is the case you are talking about. This still is concave in $\eta$ but may not be in $\theta$ so $\eta$ should be such that the whole log likelihood is concave in $\theta$. – dksahuji Feb 26 '14 at 5:17
• Note that the question asks about convergence, rather than just existence, but with a few restrictions, that, too, may be doable. – Glen_b -Reinstate Monica Feb 26 '14 at 6:22
• @Glen_b Can you elaborate? I dont know any such restrictions. Maybe something like restrictions on stepsize in a gradient based optimizer to gaurantee convergence in case of concave function. – dksahuji Feb 26 '14 at 11:11
• @Glen_b That might be true in general but I am not able to see any reason for concave function to not converge to optima within small tolerable value. But I would say that I dont have any practical experience with these and I have just started. :) – dksahuji Feb 26 '14 at 15:27