Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In logistic regression, the gradient descent algorithm for calculating coefficients can be described this way:

Until convergence, do $$ \beta := \beta + \alpha \frac{\partial L}{\partial \beta} $$

where L is the log likelihood function.

Coefficients can be got quite easily here, but how can I get the p values with this algorithm?

share|improve this question
add comment

2 Answers 2

up vote 6 down vote accepted

The answer provided by @user43310 is generally correct, but incomplete, as @WHuber pointed out.

Once you've declared the algorithm to have converged, compute the Hessian $\textbf{H}$, which effectively tells you how "peaked" the parameter surface is at some parameter values. The matrix $(-\textbf{H})^{-1}$ is the variance-covariance estimates of the parameters at their (approximate) maxima. Therefore, the vector $\sqrt{\text{diag(}(-\textbf{H})^{-1}{)}}$ is the estimate of the standard error of each parameter value at its maximum. Under the assumption that the sampling distribution of the parameters is approximately normal in the limit of infinite sample size, then we test the hypothesis that the parameters are deviates from a normal distribution with mean zero and s.d. given by this procedure, e.g. that the parameter $z$ satisfies $|z|\ge 1.96\text{se}$ at a typical level. Alternatively, one can compare the quantity $$\frac{(\hat{z}-z_0)^2}{\text{var}(\hat{z})}$$ to a $\chi^2$ distribution with degrees of freedom determined from the number of observations less the number of parameters estimated.

More information on this method of determining the variance of parameter estimates can be found in Gary King, Unifying Political Methodology, pp. 89-90.

As @gung writes, this approach is called a Wald test. Asymptotically, it is equivalent to the likelihood ratio test. However, the Wald test is sensitive to the parameterization, while the LRT is not: any monotonic transformation of $z$ will work for the LRT.

As an added bonus, the wiki link currently cites professor & CV contributor Frank Harrell in several places, which I think is neat.

share|improve this answer
    
$H$ is Hessian of the log likelihood function $L$, right? –  qed Apr 14 at 15:14
    
That's correct. And I just found the citation I was looking for; it's now recorded in the answer. –  user777 Apr 15 at 1:35
add comment

Compute the Hessian evaluated at the obtained coefficients. Then negate the Hessian, invert it, and thats the variance-covariance matrix of the coefficients.

share|improve this answer
4  
...And how are the p-values obtained? (Perhaps more importantly, when can this approach--which is approximate--be expected to work and why?) –  whuber Apr 7 at 21:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.