5
$\begingroup$

I want to show statistical significant impact of about 15 different independent variable on a binary dependent variable (I am not a statistician). Some of my independent variables are term counts in text so they are likely correlated with the length of the text which is also one of the independent variables. I selected this set of variables because they seemed to impact the outcome at least individually. I performed matching on the length variable to try to control for this effect (length alone explains a lot of variance in my outcome/dependent variable).

Now I want to figure out which variables still matter in the presence of the other variables and what their relative importance is.

Question 1: What is the best way to deal with the possible collinearity of my independent variables?

Variant 1: Replace term_counts with residuals after regressing them on length.

Variant 2: First, do logistic regression model on all variables. All factors where the parameter is significantly different from zero are significant factors (this is a claim I would like to make). And then models where I remove a single variable and show that the fit gets significantly worse to show that every single one matters, and by how much (how to do this exactly leads to Question 2).

Question 2: What is a reasonable and standard way of assessing superior fit of the different logistic regression models? (also see Assessing logistic regression models)

What measure should I use to compare fit? Likelihood doesn't really work since it will always get better for more complex models. Classification seems stupid since our empirical estimates for p(Y=1) are only 20-40%. We were also thinking about plotting p_empirical versus p_model after binning multiple observations to get a clearer picture. I am sure people have done this before and I would be very happy if you could point me to how to visualize the fit/performance.

Question 3: What is the best way to normalize the features so I would be able to compare the model parameters (i.e. how much more important is one feature over another?)? Is it reasonable to scale everything between zero and one. Should it be "minus mean, divide by std". I have some ordinal, some binary, and a few continues dependent variables.

Thanks!

$\endgroup$
3
  • $\begingroup$ When you say 'related to this thread' at the start, which thread do you mean? $\endgroup$
    – Glen_b
    Nov 11, 2013 at 21:50
  • $\begingroup$ Sorry, this was confusing and I edited it. I realize that similar questions (like the one linked) have been asked before but I have been unable to really answer the questions stated above. $\endgroup$
    – Tim
    Nov 12, 2013 at 0:05
  • $\begingroup$ Yes, just in the wrong place :) $\endgroup$
    – Tim
    Nov 12, 2013 at 1:10

2 Answers 2

3
$\begingroup$

As far as figuring out which variables matter in the presence of other variables you might consider the Wald Test or the Likelihood-Ratio test.

$\endgroup$
2
$\begingroup$

Some thoughts. Hopefully something here is useful, but feel you've thought of all of this already.

(1) Collinearity: Three simple options. (A) Drop one of the problematic variables. Since the two collinear variables provide redundant information, this shouldn't impact the result much. (B) Combine collinear variables into new single predictor. (C) any good options you can think of given your knowledge of data (like term counts per 100 words of text rather than term count as an absolute number).

(2) AIC or BIC seem reasonable options to compare different models and help guide final model selection. But for model building would recommend a standard feature selection procedure (stepwise, stepwise using AIC, lasso - whatever) rather than just comparing AIC.

Once you have a final model classification (c-statistic for discrimination), and some measure of calibration usually make sense.

(3) As far as I know there is no good way of scaling features.

Scaling with mean 0 and sd+/-1 was/is popular (standardized regression coefficients). Still see it in sas output. I don't know why it is thought to be useful or if people really still use this. Seems meaningless. Doesn't account for binary predictors. But it must be popular for a reason. Many stats packages do it for you automatically.

Some people use "relative importance". There is an r package - relaimp - for linear regression. But in the academic setting no one uses this as far as I can tell. It's benefits unclear.

I'd recommend against normalizing/comparing. It seems like a good idea, but in the end rarely makes sense. Would be interested if someone has a different opinion.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.