I am trying to make a logistic regression model based on 5 predictors: 2 of these are categorical and 3 are numerical. The output is simply 1 or 0, and upon performing the Matlab function glmfit(x,y,'binomial'), a warning message appears:
Warning: X is ill conditioned, or the model is overparameterised, and some coefficients are not identifiable. You should use caution in making predictions.
Seeing this message, the instinct is to take delete different predictors (blindly and randomly for now) and I saw that I indeed get no warning for other combinations of predictors.
Now there is a barrage of questions going through in my mind
How do I know which parameters are important? Do I simply look at the p-values of the model and decide from there? Or do I look at the coefficients of the original model where the error/warning appeared and check which coefficients are zero. The zero coefficients decide that those particular predictions can be expelled.
Now every time I playfully delete/add predictors, I am creating a 'new' model, am I not? Is there a way to rank the models from 'least' useful to 'most' useful using AIC or BIC? What then is the use of those I asked in question 1? If we can just use AIC/BIC and other criteria to rank models (seeing the predictors and interactions that are useful), why bother with the p-values in the first question?
I am so confused, I appreciate your insights.