To build a tweedie glm for categorical variables, the document suggested that I can use data['variable_name'].asfactor().

However, in the model output, there is no reference level, i.e., if I have gender = Male or Female as categorical variable, it will result in both categories with none-zero coefficient. Any idea how to resolve this?


The way categorical levels are handled is described in the documentation here, and if regularization is enabled then GLM doesn't drop a reference level - you will only see a level dropped if you disable regularization by setting lambda = 0 .

I will repost the relevant sections for your convenience:

Handling of Categorical Variables

If the response column is categorical, then a classification model is created. GLM supports both binary and multinomial classification. For binary classification, the response column can only have two levels; for multinomial classification, the response column will have more than two levels. We recommend letting GLM handle categorical columns, as it can take advantage of the categorical column for better performance and memory utilization.

We strongly recommend avoiding one-hot encoding categorical columns with any levels into many binary columns, as this is very inefficient. This is especially true for Python users who are used to expanding their categorical variables manually for other frameworks.

Handling of Numeric Variables

When GLM performs regression (with factor columns), one category can be left out to avoid multicollinearity. If regularization is disabled (lambda = 0), then one category is left out. However, when using a the default lambda parameter, all categories are included.

The reason for the different behavior with regularization is that collinearity is not a problem with regularization. And it’s better to leave regularization to find out which level to ignore (or how to distribute the coefficients between the levels).

| cite | improve this answer | |
  • 1
    $\begingroup$ Make perfect sense, thank you! $\endgroup$ – Jie Huang Apr 30 '19 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.