1
$\begingroup$

I'm trying to calculate logistic regression coefficients by defining the log-likelihood function and using maximum likelihood.

In some cases when the initial (start) values I gave to the maximum likelihood were not correct I got wrong results for the logistic regression (different from the ones I get when using glm for example).

Given the input data and y values, what should be the optimum initial values for logistic regression (or, in other words, what are the values that are being used in glm)?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Typically ${\bf \beta}={\bf 0}$ works well when there are no collinearity problems. Could you provide more information about the sample size and the number of covariates? $\endgroup$
    – user10525
    Aug 7, 2012 at 16:14
  • $\begingroup$ I have around 20,000 vectors with ~20 features per vector. The features has some dependencies between them. y values are 0 or 1 with around 1-5% or ones (hope that help, because I can't add all the data here... $\endgroup$
    – user5497
    Aug 7, 2012 at 16:21
  • $\begingroup$ @user5497 How different are the results? Are we talking difference at the 10th decimal place or at the 1st? Can you perhaps provide the output from R (obfuscated if need be)? $\endgroup$
    – Jason Morgan
    Aug 8, 2012 at 1:44
  • $\begingroup$ I'm not using glm, so as I understand safeBinaryRegression can't help me. What I've done is implemented my own log likelihood function and used maximum likelihood (R's mle2 - bbmle) to find the coefficients. I've compared the results to regular glm (which works without any problems). mle2 should get start values. Currently I'm using zeros for all the coefficients. When I'm using those start values, in some cases I get very weird results (the differences between the mle2 results and the glm results are big). So I wanted to know what are the start values that glm uses $\endgroup$
    – user5497
    Aug 8, 2012 at 8:15
  • $\begingroup$ for example, here are the first 7 coefficients in the glm run: -6.2307913208130525, 6.110187257533295e-06, -2.0577042478307273, 0.4786093240660332, 0.38126727104872804, -0.625615435816033, 0.04482479648912922 And the first 7 coefficients in the mle2 run (with zeros start values): -1.4625013759985311e-08, -0.0005046844214205488, 1.2295071793926704e-08, 1.632233091079531e-08, -1.8873962762517583e-08, -6.607669091467728e-09. You can see that it seems that it was "stuck" in a local minimum near 0,0,0,0,0,.... $\endgroup$
    – user5497
    Aug 8, 2012 at 8:22

1 Answer 1

Reset to default
5
$\begingroup$

I believe there exists no optimum initial value. As stated by user10525, the value β=0 works well and is the default choice for glm.

In order to check what's going on with glm I would follow the basic steps:

  1. Try to change the number of iterations in the Newton-Raphson algorithm in glm adding control=glm.control(maxit=Y), where Y is the number of desired iterations. I would even begin with Y=1 to check stability of the algorithm at β=0.

  2. Adding start=c(a,b,c,...) you can change the initial value in the regression. Note that the length of start must be equal to $p+1$, where $p$ denotes the number of covariates in your logistic regression.

  3. Analyse the stability of the regression for more choices of the above parameters; you could probably find at least a range of initial values corresponding to "convergent" logistic regressions for a given number of iterations.

  4. Andrew Gelman discusses a nice example of divergence of glm in presence of "bad" initial value choices in his blog: http://andrewgelman.com/2011/05/04/whassup_with_gl/

  5. Please note that glm "explodes" in presence of complete separation as the MLE does not exist: this is also something to check.

I hope this can be applied to your case.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.