When performing classification using logistic regression, can the score function (the gradient of the log-likelihood with respect to the weights) and/or the Fisher information matrix (inverse of the log-likelihood Hessian taken w.r.t. to weights) be used to compare the quality of fit of two sets of weights (without calculating the log-likelihood)? I know that if the score function is zero for the first set of weights, then it is optimal and the log-likelihood is greater than or equal to the log-likelihood of the second set of weights. Can a more general statement be made for suboptimal weights?


Yes, the score test is asymptotically equivalent to the likelihood ratio test. That means it is a test for nested models. It is evaluated using the score and information under the null hypothesis. That is an important advantage over the LRT and Wald tests.

| cite | improve this answer | |
  • $\begingroup$ So if I understand this correctly, if I choose, for instance, two random points in weight space, $\mathbf{x}_1$ and $\mathbf{x}_2$, I would calculate $S(\mathbf{x}) = \ell'(\mathbf{x})^\mathrm{T} \boldsymbol{\mathcal{I}}(\mathbf{x})^{-1} \ell'(\mathbf{x})$ at both $\mathbf{x}_1$ and $\mathbf{x}_2$ where $\ell(\mathbf{x})$ is the log-likelihood and $\boldsymbol{\mathcal{I}}(\mathbf{x}) = \langle \ell''(\mathbf{x}) \rangle_y$ (averaged over all $y_t \in [0,1]$). If $S(\mathbf{x}_1) < S(\mathbf{x}_2)$, then $\mathbf{x}_1$ will have a lower log-likelihood? $\endgroup$ – HSK Dec 13 '16 at 18:34
  • $\begingroup$ @HSK no your notation makes no sense. If $\mathbf{x}_1$ is the maximum likelihood estimate under the null hypothesis, e.g. constrained MLE, then all the information and score functions are evaluated at $\mathbf{x}_1$. The score statistic is based on the slope of the likelihood function at the null hypothesis, if it is very steep, the likelihood is definitely not at an extrema there, so you reject the null hypothesis. $\endgroup$ – AdamO Dec 13 '16 at 18:37
  • $\begingroup$ I see, this might be a matter of miscommunication. In the original question, the logistic regression is not comparing nested models. Instead, assume only one dataset is used (no change in the samples and features) and the logistic regression is performed twice using two different random initializations but the regression stops prematurely for both maximizations such the $\mathbf{x}_1$ and $\mathbf{x}_2$ are suboptimal points in weight space. Is it possible to determine based on the score and possibly the information matrix which of the suboptimal solutions is closest to the MLE? $\endgroup$ – HSK Dec 13 '16 at 18:49
  • $\begingroup$ @HSK If you don't have a likelihood how can you be sure one is closer to an MLE? There might not be an MLE. It sounds like you are dealing with a boundary problem. Logistic regression is well-behaved irrespective of initialization when the information is non-singular. No test suffices. $\endgroup$ – AdamO Dec 13 '16 at 21:38
  • $\begingroup$ The MLE can be calculated and likelihoods can in principal be calculated for the suboptimal weights as well. The motivation for avoiding the likelihood calculation for the suboptimal weights has to do with this (overly specific) question: logistic-rank. Of course, if there is no alternative to calculating the likelihoods, as I suspect, then the problem in the link can only be solved by trying all solutions. $\endgroup$ – HSK Dec 13 '16 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.