My questions are:

  1. Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why?
  2. Furthermore, what constraints are there on the link function to insure convexity?

My understanding of GLMs is that they maximize a highly nonlinear likelihood function. Thus, I would imagine that there are several local maxima and the parameter set you converge to depends on the initial conditions for the optimization algorithm. However, after doing some research I have not found a single source which indicates that there are multiple local maxima. Furthermore, I am not so familiar with optimization techniques, but I know the Newton-Raphson method and IRLS algorithm are highly prone to local maxima.

Please explain if possible both on an intuitive and mathematical basis!

EDIT: dksahuji answered my original question, but I want to add the followup question [2] above. ("What constraints are there on the link function to insure convexity?")

  • $\begingroup$ I think some restrictions must be required before that could be so. What is the source for the statement? $\endgroup$
    – Glen_b
    Commented Feb 24, 2014 at 4:26
  • $\begingroup$ Several sites seemed to imply it however I couldnt find anything which mentioned it outright, so I also welcome its disproof! $\endgroup$ Commented Feb 24, 2014 at 14:59
  • $\begingroup$ as long as the likelihood is well defined everywhere on the domain (and ignoring some tangential numerical issues) I think yes. Under those conditions, the hessian is <0 everywhere on the domain so the likeihood is globally concave. Btw, the function are not 'highly non-linear' in the parameters and that's what matters. $\endgroup$
    – user603
    Commented Feb 24, 2014 at 15:30
  • 1
    $\begingroup$ Logistic, Poisson, and Gaussian regressions are often convex given a "good" link function. However, with arbitrary link function, they are not convex. $\endgroup$
    – Memming
    Commented Feb 25, 2014 at 22:07
  • 1
    $\begingroup$ Just a note about my edit (which DankMasterDan brought up): the OP talked about minimizing the likelihood, which was obviously wrong; one would either maximize the likelihood or minimize the deviance. $\endgroup$
    – StasK
    Commented Feb 26, 2014 at 0:32

2 Answers 2


The definition of exponential family is:

$$ p(x|\theta) = h(x)\exp(\theta^T\phi(x) - A(\theta)), $$

where $A(\theta)$ is the log partition function. Now one can prove that the following three things hold for 1D case (and they generalize to higher dimensions--you can look into properties of exponential families or log partition):

  1. $ \frac{dA}{d\theta} = \mathbb{E}[\phi(x)]$

  2. $ \frac{d^2A}{d\theta^2} = \mathbb{E}[\phi^2(x)] -\mathbb{E}[\phi(x)]^2 = {\rm var}(\phi(x)) $

  3. $ \frac{ \partial ^2A}{\partial\theta_i\partial\theta_j} = \mathbb{E}[\phi_i(x)\phi_j(x)] -\mathbb{E}[\phi_i(x)] \mathbb{E}[\phi_j(x)] = {\rm cov}(\phi(x)) \Rightarrow \Delta^2A(\theta) = {\rm cov}(\phi(x))$

The above result prove that $A(\theta)$ is convex(as ${\rm cov}(\phi(x))$ is positive semidefinite). Now we take a look at likelihood function for MLE:

\begin{align} p(\mathcal{D}|\theta) &= \bigg[\prod_{i=1}^{N}{h(x_i)}\bigg]\ \exp\!\big(\theta^T[\sum_{i=1}^{N}\phi(x_i)] - NA(\theta)\big) \\ \log\!\big(p(\mathcal{D}|\theta)\big) &= \theta^T\bigg[\sum_{i=1}^{N}\phi(x_i)\bigg] - NA(\theta) \\ &= \theta^T[\phi(\mathcal{D})] - NA(\theta) \end{align}

Now $\theta^T[\phi(\mathcal{D})]$ is linear in theta and $-A(\theta)$ is concave. Therefore, there is a unique global maximum.

There is a generalized version called curved exponential family which would also be similar. But most of the proofs are in canonical form.

  • $\begingroup$ so does this mean that GLM have a unique global minima nomatter which link function is chosen (including the noncanonical ones)? $\endgroup$ Commented Feb 26, 2014 at 2:25
  • 3
    $\begingroup$ I will try to answer as far as I percieve it. $ p(x|\theta) = h(x)exp(\eta(\theta)^T\phi(x) - A(\eta(\theta)))$ is the case you are talking about. This still is concave in $\eta$ but may not be in $\theta$ so $\eta$ should be such that the whole log likelihood is concave in $\theta$. $\endgroup$
    – dksahuji
    Commented Feb 26, 2014 at 5:17
  • $\begingroup$ Note that the question asks about convergence, rather than just existence, but with a few restrictions, that, too, may be doable. $\endgroup$
    – Glen_b
    Commented Feb 26, 2014 at 6:22
  • 1
    $\begingroup$ As an example, there'd be a need to say we can't have things like complete separation. The convergence criteria will in practice leave us some distance from the optimum and I've seen cases where that distance can be quite large. That sort of thing might be regarded as distinct from the theoretical convergence. $\endgroup$
    – Glen_b
    Commented Feb 26, 2014 at 13:14
  • 1
    $\begingroup$ @Glen_b That might be true in general but I am not able to see any reason for concave function to not converge to optima within small tolerable value. But I would say that I dont have any practical experience with these and I have just started. :) $\endgroup$
    – dksahuji
    Commented Feb 26, 2014 at 15:27

I was investigating this heavily during my thesis. The answer is that the GLM likelihood is not always convex, it is only convex under the right assumptions. A very good investigation of this was made by Nelder and Wedderburn in their paper "On the Existence and Uniqueness of the Maximum Likelihood Estimates for Certain Generalized Linear Models" which can be found at https://www.jstor.org/stable/2335080


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.