knowing which predictors are significant in a logistic regression model

Question

I am trying to make a logistic regression model based on 5 predictors: 2 of these are categorical and 3 are numerical. The output is simply 1 or 0, and upon performing the Matlab function glmfit(x,y,'binomial'), a warning message appears:

Warning: X is ill conditioned, or the model is overparameterised, and some coefficients are not identifiable. You should use caution in making predictions.

Seeing this message, the instinct is to take delete different predictors (blindly and randomly for now) and I saw that I indeed get no warning for other combinations of predictors.

Now there is a barrage of questions going through in my mind

1. How do I know which parameters are important? Do I simply look at the p-values of the model and decide from there? Or do I look at the coefficients of the original model where the error/warning appeared and check which coefficients are zero. The zero coefficients decide that those particular predictions can be expelled.

2. Now every time I playfully delete/add predictors, I am creating a 'new' model, am I not? Is there a way to rank the models from 'least' useful to 'most' useful using AIC or BIC? What then is the use of those I asked in question 1? If we can just use AIC/BIC and other criteria to rank models (seeing the predictors and interactions that are useful), why bother with the p-values in the first question?

I am so confused, I appreciate your insights.

Answer 1

How do I know which parameters are important? Do I simply look at the p-values of the model and decide from there?

No, for the reasons I talk about here.

I highly suspect you are experiencing seperation of your data. There likely exists some hyperplane which perfectly separates the positive and negative cases. This leads to unidentifiability, which I discuss in this blog post as a motivating topic. In this case, regularization is one approach you can take, or you can decide which variable to remove a priori (that is, without referencing model fits).

Now every time I playfully delete/add predictors, I am creating a 'new' model, am I not? Is there a way to rank the models from 'least' useful to 'most' useful using AIC or BIC?

People like to do this. It isn't my preference, but it has been done and seems to be inoffensive. Again, I would encourage you not to do this if your goal is inference. Instead, I would say think about the question you are trying to answer and determine which variables are most important to answer that question. If your goals are exploratory, then use AIC.