2
$\begingroup$

It seems to be common to take a "high" condition number as a sign for multicolinearity in regression analysis. For linear models I'm totally convinced that this is a good idea, but is there any analysis on how colinearity influences the condition number of logistic regressio models?

I can't find any and simple simulations seem to indicate that there is no monotonically increasing relationship between the two.

Would be great if anyone could point out a reference or give a short analysis of this!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

This UCLA page covers this issue in the context of logistic regression. But the phrasing of this question suggests some misunderstanding of multicollinearity and condition numbers.

The condition number and multicollinearity are functions of the design matrix of independent variable values. They bear no relation to the dependent variable or to the type of regression. A high condition number or multicollinearity means that some of the predictor variables are close to being linear combinations of each other. Thus in any linear modeling there will be ambiguity in determining which is the "true" predictor variable among a set of collinear variables. It doesn't matter whether the regression is linear, logistic, or any other type of generalized linear model.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ okay, well then I misunderstood what the term condition number referred to. What I am asking for is a relationship between the multicollinearty in the data and the condition number of the objective function's second derivative, i.e. hessian matrix which is of course different depending on the type of regression you do. $\endgroup$
    – Jonasson
    Commented Jan 29, 2017 at 0:57
  • 2
    $\begingroup$ The Hessian in logistic regression involves factors based on the probabilities of the individual observations given the regression coefficient values (see this page for example), so I don't know that a general answer can be given. Regardless of any numerical problems arising from ill-conditioning of the Hessian, however, your model will still pose difficulties in interpretation if the design matrix is poorly conditioned/predictors are multicollinear. Coefficients will have large errors and differ from sample to sample. $\endgroup$
    – EdM
    Commented Jan 29, 2017 at 21:39
1
$\begingroup$

When we estimate GLM/Logit/... by IRLS, then is based on weighted least squares. The analogue to the condition number of the design matrix would be the condition number of the weighted design matrix which does depend on the distributional assumption for the response.

In other nonlinear models, e.g nonlinear least squares, we could look at the condition number for the jacobian or the matrix of contributions to the score.

Another direct measure would be to look at the hessian directly. As far as I know, Stata computes and reports the rank of the estimated covariance of the parameter estimate in some cases.

I do not remember any reference, but it is kind of obvious that identification of the parameters will fail if the estimating equations are collinear (when we don't have additional "overidentifying" restrictions).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.