5
$\begingroup$

The log-likelihood function for logistic function is $$l(\theta) = \sum_{i=1}^m(y^{(i)}\log h(x^{(i)}) + (1-y^{(i)})\log(1 - h(x^{(i)})))$$, where $$h(x^{(i)}) = \frac{1}{1 + e^{-\theta^Tx^{(i)}}}\,.$$

In order to obtain maximum likelihood estimation, I implemented fitting the logistic regression model using Newton's method. I encountered 2 problems:

  1. I try to fit the model to my data, but during the iterations, a singular Hessian matrix is encountered, what do I do with this kind of problem?

  2. With different initial guess $\theta$, will the model converge to different results?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Whenever you are implementing a well-known statistical procedure, it is a great idea to compare your results to those produced by working software. When you run your data through a stats package, what does it report? Does it have the same problem or not? $\endgroup$
    – whuber ♦
    Commented Oct 17, 2013 at 12:41
  • $\begingroup$ @whuber, yes, that's a good idea. $\endgroup$
    – avocado
    Commented Oct 17, 2013 at 12:55
  • 1
    $\begingroup$ Have you considered the modified newton's method? It was a method built specifically (as far as I know) to deal with running into singular matrix. You simply make an $n$ by $n$ identidy matrix which you subtract from your Hessian to avoid a matrix that does not invert. As for your second question, Newton's method can run into problems that it finds local and not global maximum, so you should try different starting points and see what you get. It can also start moving in one direction (the wrong one) and simply never converge, so starting point selection is very important. $\endgroup$
    – Eric Peterson
    Commented Oct 17, 2013 at 14:55
  • $\begingroup$ I found this. I believe that the method that I was suggesting here is the one that they're calling the Levenberg-Marquadt method. Discovery - Unconstrained Optimization 24 So that • ˆH(x) is symmetric p.d. • ˆH(x) is not too close to singular, i.e., its smallest eigenvalue is bounded below by a constant bigger than zero. Popular methods: • Greenstadt’s method: Modify eigenvalues. • Levenberg-Marquardt method: Add a scaled identity matrix • Modified Cholesky Stratigies: Perform Choleskey factorization of the Hessian and modify the diagonal elements $\endgroup$
    – Eric Peterson
    Commented Oct 17, 2013 at 15:04

2 Answers 2

Reset to default
4
$\begingroup$

One trick that often helps for logistic regression type problems is to realize that:

$1 - h(x^{(i)}) = h(-x^{(i)})$

and that $h(-x^{(i)})$ is more numerically stable than $1 - h(x^{(i)})$.

You can find a discussion of that here. This is an article in the Stata Journal so the examples are in Stata/Mata, but the problem has to do with the way computers store numbers and is thus more general. For example, I have been able to reproduce the first anomalous example exactly in R, i.e. not just the general pattern but the exact values.

Improve this answer
$\endgroup$
0
$\begingroup$

Just wanna tell that what @Eric Peterson said about creating (constant*identity matrix) to make our hessian matrix invertible works perfectly. I just tried it in my code. But, instead of subtracting it from heassian, it should be adding it. We can think that it's similar with what we do in regression when we add regularization.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.