# How to calculate p values in logistic regression with gradient descent algorithm

In logistic regression, the gradient descent algorithm for calculating coefficients can be described this way:

Until convergence, do $$\beta := \beta + \alpha \frac{\partial L}{\partial \beta}$$

where L is the log likelihood function.

Coefficients can be got quite easily here, but how can I get the p values with this algorithm?

-

The answer provided by @user43310 is generally correct, but incomplete, as @WHuber pointed out.

Once you've declared the algorithm to have converged, compute the Hessian $\textbf{H}$, which effectively tells you how "peaked" the parameter surface is at some parameter values. The matrix $(-\textbf{H})^{-1}$ is the variance-covariance estimates of the parameters at their (approximate) maxima. Therefore, the vector $\sqrt{\text{diag(}(-\textbf{H})^{-1}{)}}$ is the estimate of the standard error of each parameter value at its maximum. Under the assumption that the sampling distribution of the parameters is approximately normal in the limit of infinite sample size, then we test the hypothesis that the parameters are deviates from a normal distribution with mean zero and s.d. given by this procedure, e.g. that the parameter $z$ satisfies $|z|\ge 1.96\text{se}$ at a typical level. Alternatively, one can compare the quantity $$\frac{(\hat{z}-z_0)^2}{\text{var}(\hat{z})}$$ to a $\chi^2$ distribution with degrees of freedom determined from the number of observations less the number of parameters estimated.

More information on this method of determining the variance of parameter estimates can be found in Gary King, Unifying Political Methodology, pp. 89-90.

As @gung writes, this approach is called a Wald test. Asymptotically, it is equivalent to the likelihood ratio test. However, the Wald test is sensitive to the parameterization, while the LRT is not: any monotonic transformation of $z$ will work for the LRT.

As an added bonus, the wiki link currently cites professor & CV contributor Frank Harrell in several places, which I think is neat.

-
$H$ is Hessian of the log likelihood function $L$, right? – qed Apr 14 '14 at 15:14
That's correct. And I just found the citation I was looking for; it's now recorded in the answer. – General Abrial Apr 15 '14 at 1:35

Compute the Hessian evaluated at the obtained coefficients. Then negate the Hessian, invert it, and thats the variance-covariance matrix of the coefficients.

-
...And how are the p-values obtained? (Perhaps more importantly, when can this approach--which is approximate--be expected to work and why?) – whuber Apr 7 '14 at 21:38