0
$\begingroup$

I am trying to make a logistic regression model based on 5 predictors: 2 of these are categorical and 3 are numerical. The output is simply 1 or 0, and upon performing the Matlab function glmfit(x,y,'binomial'), a warning message appears:

Warning: X is ill conditioned, or the model is overparameterised, and some coefficients are not identifiable. You should use caution in making predictions.

Seeing this message, the instinct is to take delete different predictors (blindly and randomly for now) and I saw that I indeed get no warning for other combinations of predictors.

Now there is a barrage of questions going through in my mind

  1. How do I know which parameters are important? Do I simply look at the p-values of the model and decide from there? Or do I look at the coefficients of the original model where the error/warning appeared and check which coefficients are zero. The zero coefficients decide that those particular predictions can be expelled.

  2. Now every time I playfully delete/add predictors, I am creating a 'new' model, am I not? Is there a way to rank the models from 'least' useful to 'most' useful using AIC or BIC? What then is the use of those I asked in question 1? If we can just use AIC/BIC and other criteria to rank models (seeing the predictors and interactions that are useful), why bother with the p-values in the first question?

I am so confused, I appreciate your insights.

$\endgroup$
1
$\begingroup$

How do I know which parameters are important? Do I simply look at the p-values of the model and decide from there?

No, for the reasons I talk about here.

I highly suspect you are experiencing seperation of your data. There likely exists some hyperplane which perfectly separates the positive and negative cases. This leads to unidentifiability, which I discuss in this blog post as a motivating topic. In this case, regularization is one approach you can take, or you can decide which variable to remove a priori (that is, without referencing model fits).

Now every time I playfully delete/add predictors, I am creating a 'new' model, am I not? Is there a way to rank the models from 'least' useful to 'most' useful using AIC or BIC?

People like to do this. It isn't my preference, but it has been done and seems to be inoffensive. Again, I would encourage you not to do this if your goal is inference. Instead, I would say think about the question you are trying to answer and determine which variables are most important to answer that question. If your goals are exploratory, then use AIC.

$\endgroup$
4
  • $\begingroup$ Thanks for your answer. The ultimate goal is to classify, and I think it is an added bonus to be able to clearly identify relationships among the predictors that influence the classification. $\endgroup$ – cgo Mar 25 '20 at 3:12
  • $\begingroup$ Ah, then things are easier. My advice is to use a lasso model for classification. The penalty will take care of any separation problems, so the model will converge and you won't have to remove any variables. To identify strong predictors, I would encourage you to look at the coefficient paths, $\endgroup$ – Demetri Pananos Mar 25 '20 at 3:14
  • $\begingroup$ Thank you for your help. I will look into this. $\endgroup$ – cgo Mar 25 '20 at 3:37
  • $\begingroup$ If you’ve found this helpful, please upvote and accept the abswer $\endgroup$ – Demetri Pananos Mar 25 '20 at 4:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.