Linear regression including categorical variables with hundreds of levels [duplicate]

Question

I am trying to teach myself data science by solving some of the problems available on the internet.
Currently I am trying to predict a fraud event with the aid of 4 categorical variables. Each of the categorical variables have 100 of levels. Some of the levels occur frequently, some none at all.

Currently I have tried the following

• Throwing the categorical variable directly into linear regression. I get memory related errors
• Dummy encoding each categorical variable. Throw all of them into a linear regression. This also fails due to memory error
• Throwing the categorical variable into a Random Forest^TM. This fails as Random Forest can only handle 52 distinct levels

None of the approaches I tried above have worked. What else can I try?

Any help is appreciated.

Update #1: The categorical variables are groupings like zip code, county, etc. In addition to these the customers are grouped into different buckets based on presence or absence of certain factors.

Update #2: I was able to use memory.limit() and increase the maximum memory size available to R. This has solved my memory issues.

Update #3: The current approach I was to use linear regression with regularization to find average fraud rate per factor level (for example average fraud rate per zip code). Now the categorical variable with hundreds of factors has been reduced to a numeric variable. I follow the same approach to convert all the 4 categorical variables into numeric. I throw these 4 numeric variables into a random forest. My results with the approach are okay with lots of room for improvement.

Update #4: I am using the R ecosystem

Answer 1

I have had success in similar application (in a customer relationship modelling problem, we used as predictors registration data with text fields, zip codes etc inside), using the preprocessing approach, described here.

I didn't find it there, but it seems quite similar to what I did. The idea is to build a proper predictor matrix and throw it into glm.net to get a small number of significant features. Glmnet can handle sparse matrices with huge number of columns (>10000) and runs in (relatively) very little memory. You can use sparse.model.matrix from the Matrix package to one-hot encode all factors you're interested in. You can then cbind them, together with the numerical columns and have your final modelling matrix.

The blog post describes some specific problems that came up for me as well, and their solutions.

Answer 2

One approach is to try dimensionality reduction techniques, such as principal components analysis (PCA). But, maybe even better approach is to perform an exploratory factor analysis (EFA), which not only will reduce dimensionality, but will allow to hypothesize some (latent) factor structure and generate some theoretic model (based on EFA and subject domain). This theoretic model can then be tested by confirmatory factor analysis (CFA) on a real data set.

Further options would involve path analysis, which, along with CFA, is referred to under an umbrella term structured equation modeling (SEM). The difference between SEM and a closely related term latent variable modeling (LVM) is that SEM is generally considered as an exploratory and/or confirmatory approach, whereas LVM might be considered as a predictive SEM approach.

The R ecosystem provides a wealth of options in terms of tools for various types of multivariate analysis, which I have mentioned above. References to these tools are mostly clustered within these two CRAN Task Views: http://cran.r-project.org/web/views/Multivariate.html and http://cran.r-project.org/web/views/SocialSciences.html (though the lists are not always complete - for example they don't include lavaan package, which might be listed in some other CRAN Task View, if at all).

Answer 3

Try clustering your data first, build Finite Mixture Model, that can help

Detailed Answer

To begin with author is actually doing Binomial Logistic Regression to predict fraud 0 or 1. The names can be confusing its not Linear Regression.

Lots depends on subjective knowledge on categorical variables. Statistics cannot compete with domain knowledge. Increased number of levels increases chances of over-fitting. One can try reducing that.

We cannot use Factor Analysis or PCA as number of independent variables / Predictor are 4. Assuming author may already been tried this already, it didnt help.

He must have already tried calculating Variance Inflation Factors on this data but that would have not helped either. P-Value must be coming below 0.1.

Now lets looks at this data from data engineer perspective (not data scientist). When we apply logistic regression then categorical variables are getting factored. Now if you cannot reduce data Vertically then try reducing Horizontally meaning put data in several different clusters. The run regression separately on each cluster either apply FMM mode or some ensemble the cluster output.

I dont know if this is the right approach without looking at the data (infact sometimes after looking at data also).

Answer 4

Unless I'm missing something, it would make a lot of sense to try a generalized linear mixed model on this (e.g. lme4 package in R).

• it avoids the memory problem because the model matrix is coded as a sparse matrix (you can do this in any case to help with memory use, even if you choose to use some other approach -- e.g. see Matrix::sparse.model.matrix in R).
• it automatically does regularization on the underlying parameters associated with each categorical predictor, with the degree of regularization selected automatically (by assuming that the variation among categories is Normally distributed on the link scale). This assumption might be stronger than you want to make, but it's theoretically and computationally convenient.