# Interpreting score function and information matrix in logistic regression

When performing classification using logistic regression, can the score function (the gradient of the log-likelihood with respect to the weights) and/or the Fisher information matrix (inverse of the log-likelihood Hessian taken w.r.t. to weights) be used to compare the quality of fit of two sets of weights (without calculating the log-likelihood)? I know that if the score function is zero for the first set of weights, then it is optimal and the log-likelihood is greater than or equal to the log-likelihood of the second set of weights. Can a more general statement be made for suboptimal weights?

• So if I understand this correctly, if I choose, for instance, two random points in weight space, $\mathbf{x}_1$ and $\mathbf{x}_2$, I would calculate $S(\mathbf{x}) = \ell'(\mathbf{x})^\mathrm{T} \boldsymbol{\mathcal{I}}(\mathbf{x})^{-1} \ell'(\mathbf{x})$ at both $\mathbf{x}_1$ and $\mathbf{x}_2$ where $\ell(\mathbf{x})$ is the log-likelihood and $\boldsymbol{\mathcal{I}}(\mathbf{x}) = \langle \ell''(\mathbf{x}) \rangle_y$ (averaged over all $y_t \in [0,1]$). If $S(\mathbf{x}_1) < S(\mathbf{x}_2)$, then $\mathbf{x}_1$ will have a lower log-likelihood? – HSK Dec 13 '16 at 18:34
• @HSK no your notation makes no sense. If $\mathbf{x}_1$ is the maximum likelihood estimate under the null hypothesis, e.g. constrained MLE, then all the information and score functions are evaluated at $\mathbf{x}_1$. The score statistic is based on the slope of the likelihood function at the null hypothesis, if it is very steep, the likelihood is definitely not at an extrema there, so you reject the null hypothesis. – AdamO Dec 13 '16 at 18:37
• I see, this might be a matter of miscommunication. In the original question, the logistic regression is not comparing nested models. Instead, assume only one dataset is used (no change in the samples and features) and the logistic regression is performed twice using two different random initializations but the regression stops prematurely for both maximizations such the $\mathbf{x}_1$ and $\mathbf{x}_2$ are suboptimal points in weight space. Is it possible to determine based on the score and possibly the information matrix which of the suboptimal solutions is closest to the MLE? – HSK Dec 13 '16 at 18:49