2
$\begingroup$

I'm building a logistic regression model in which almost all of the input variables are categorical. There are multiple sets of categorical variables, for example, day of the week, age range buckets, occupation types, etc. So it may look something like this:

SAT,
SUN,
MON,
TUE,
WED,
THU,
(FRI is omitted as a base variable)

AGE_0_to_10,
AGE_11_to_20,
AGE_21_to_30,
AGE_31_to_40,
AGE_40_plus (omitted as base)

MANUFACTURING,
HEALTHCARE,
FINANCE,
RETAIL,
SERVICES,
TRANSPORTATION,
COMMUNICATIONS,
UTILITIES (omitted as base)

I am testing for multicollinearity using the VIF and I notice that there are usually high VIFs among the sets of categorical variables. For example, the days of the week would all have high VIFs and removing one or more fixes that.

Is this normal and expected since they do have some sort of relationship already?

NOTE: The actual variable categories are not the ones I am using, but just have provided textbook examples for ease of understanding. Not saying that days of the week are correlated in a logistic regression setting.

Improve this question
$\endgroup$
4
  • $\begingroup$ Exactly what do you mean by "usually high VIFs"? How large are they? What are your sample sizes? $\endgroup$
    – whuber ♦
    Commented Oct 8, 2014 at 17:28
  • 1
    $\begingroup$ Hi. Thank you for the comment. The VIFs can range from above 10 up to 25,000. I am using roughly 80 variables, of which ~70 are categorical and in various sub-groups (as described above). And I have roughly 20,000 observations. Hope this helps. $\endgroup$
    – vdiddy
    Commented Oct 8, 2014 at 19:37
  • $\begingroup$ What's your sample size? $\endgroup$
    – Jeremy Miles
    Commented Oct 8, 2014 at 19:51
  • 1
    $\begingroup$ Have you systematically cross-tabulated your categorical variables to check for close dependencies? This can be a useful thing to do during initial exploration of the data and often will reveal near-collinearities between variables. $\endgroup$
    – whuber ♦
    Commented Nov 8, 2014 at 17:35

1 Answer 1

Reset to default
1
$\begingroup$

A set of indicator (dummy) variables for the same categorical variable are always correlated: If the indicator variable for friday is 1 than the indicator variables for all other days are necessarily 0. The correlation will be higher if one category dominates the categorical variable.

Removing an additional indicator variable on top of the reference category is definately a bad idea: that way you change your reference category to a combination of your original reference category and the catagory you additionally left out, and what is that than mean?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.