2
$\begingroup$

I am trying to teach myself data science by solving some of the problems available on the internet.
Currently I am trying to predict a fraud event with the aid of 4 categorical variables. Each of the categorical variables have 100 of levels. Some of the levels occur frequently, some none at all.

Currently I have tried the following

  • Throwing the categorical variable directly into linear regression. I get memory related errors
  • Dummy encoding each categorical variable. Throw all of them into a linear regression. This also fails due to memory error
  • Throwing the categorical variable into a Random ForestTM. This fails as Random Forest can only handle 52 distinct levels

None of the approaches I tried above have worked. What else can I try?

Any help is appreciated.

Update #1: The categorical variables are groupings like zip code, county, etc. In addition to these the customers are grouped into different buckets based on presence or absence of certain factors.

Update #2: I was able to use memory.limit() and increase the maximum memory size available to R. This has solved my memory issues.

Update #3: The current approach I was to use linear regression with regularization to find average fraud rate per factor level (for example average fraud rate per zip code). Now the categorical variable with hundreds of factors has been reduced to a numeric variable. I follow the same approach to convert all the 4 categorical variables into numeric. I throw these 4 numeric variables into a random forest. My results with the approach are okay with lots of room for improvement.

Update #4: I am using the R ecosystem

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ How many observations you have? Is there, by any chance, a natural ordering in any of these variables? $\endgroup$ – Łukasz Kidziński Nov 4 '14 at 19:39
  • 3
    $\begingroup$ Unless you have (at least) several hundreds of observations, it would be unwise to control for a hundred dummy variables. You do not want to "overfit" your model. You need to collapse those hundred categories into fewer, more general categories before creating dummies from the categorical variables. I can't say how many because I don't know your sample size or exactly how detailed these categories are. Still, this sounds like a case where theory should be guiding how you recode and collapse these categories, not pure statistics. $\endgroup$ – two sheds Nov 4 '14 at 19:42
  • $\begingroup$ @ŁukaszKidziński unfortunately there is no ordering between these variable. There are about 32,000 observations $\endgroup$ – Abhi Nov 5 '14 at 15:24
  • $\begingroup$ @twosheds There are about 32,000 observations. How exactly do you mean "theory should be guiding"? Can you give some examples? $\endgroup$ – Abhi Nov 5 '14 at 15:26
  • 2
    $\begingroup$ Tell us what are this dummy variables, with hundreds of levels. What are the levels? theory-guided here probably refers to some theory about this kind of variables. Also, if you have $n=32000$, a total of a thousand dummys, so $p=1000$, then the design matrix size is only $np=32E6$ which in double precision needs about 256 MB, or a quarter of a gigabyte. A modern computer with 8 GB ram and good software (like R) should handle that! $\endgroup$ – kjetil b halvorsen Nov 6 '14 at 9:31
2
$\begingroup$

I have had success in similar application (in a customer relationship modelling problem, we used as predictors registration data with text fields, zip codes etc inside), using the preprocessing approach, described here.

I didn't find it there, but it seems quite similar to what I did. The idea is to build a proper predictor matrix and throw it into glm.net to get a small number of significant features. Glmnet can handle sparse matrices with huge number of columns (>10000) and runs in (relatively) very little memory. You can use sparse.model.matrix from the Matrix package to one-hot encode all factors you're interested in. You can then cbind them, together with the numerical columns and have your final modelling matrix.

The blog post describes some specific problems that came up for me as well, and their solutions.

|cite|improve this answer
$\endgroup$
1
$\begingroup$

One approach is to try dimensionality reduction techniques, such as principal components analysis (PCA). But, maybe even better approach is to perform an exploratory factor analysis (EFA), which not only will reduce dimensionality, but will allow to hypothesize some (latent) factor structure and generate some theoretic model (based on EFA and subject domain). This theoretic model can then be tested by confirmatory factor analysis (CFA) on a real data set.

Further options would involve path analysis, which, along with CFA, is referred to under an umbrella term structured equation modeling (SEM). The difference between SEM and a closely related term latent variable modeling (LVM) is that SEM is generally considered as an exploratory and/or confirmatory approach, whereas LVM might be considered as a predictive SEM approach.

The R ecosystem provides a wealth of options in terms of tools for various types of multivariate analysis, which I have mentioned above. References to these tools are mostly clustered within these two CRAN Task Views: http://cran.r-project.org/web/views/Multivariate.html and http://cran.r-project.org/web/views/SocialSciences.html (though the lists are not always complete - for example they don't include lavaan package, which might be listed in some other CRAN Task View, if at all).

|cite|improve this answer
$\endgroup$
1
$\begingroup$

Try clustering your data first, build Finite Mixture Model, that can help

Detailed Answer

To begin with author is actually doing Binomial Logistic Regression to predict fraud 0 or 1. The names can be confusing its not Linear Regression.

Lots depends on subjective knowledge on categorical variables. Statistics cannot compete with domain knowledge. Increased number of levels increases chances of over-fitting. One can try reducing that.

We cannot use Factor Analysis or PCA as number of independent variables / Predictor are 4. Assuming author may already been tried this already, it didnt help.

He must have already tried calculating Variance Inflation Factors on this data but that would have not helped either. P-Value must be coming below 0.1.

Now lets looks at this data from data engineer perspective (not data scientist). When we apply logistic regression then categorical variables are getting factored. Now if you cannot reduce data Vertically then try reducing Horizontally meaning put data in several different clusters. The run regression separately on each cluster either apply FMM mode or some ensemble the cluster output.

I dont know if this is the right approach without looking at the data (infact sometimes after looking at data also).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Can downvoter explain why this answer is bad? I'm not sure mixture models make sense here, but clustering data sounds like a good way to collapse hundreds of variables to a few more general. $\endgroup$ – ffriend Nov 6 '14 at 14:51
  • $\begingroup$ @ffriend, probably people think it looks a bit more like a comment than a full answer. Sohsum, given that these are dummies, you presumably mean latent class analysis rather than FMM, but why not expand your answer to say a bit more & explain why it might resolve the OP's problem? $\endgroup$ – gung - Reinstate Monica♦ Nov 6 '14 at 16:48
1
$\begingroup$

Unless I'm missing something, it would make a lot of sense to try a generalized linear mixed model on this (e.g. lme4 package in R).

  • it avoids the memory problem because the model matrix is coded as a sparse matrix (you can do this in any case to help with memory use, even if you choose to use some other approach -- e.g. see Matrix::sparse.model.matrix in R).
  • it automatically does regularization on the underlying parameters associated with each categorical predictor, with the degree of regularization selected automatically (by assuming that the variation among categories is Normally distributed on the link scale). This assumption might be stronger than you want to make, but it's theoretically and computationally convenient.
|cite|improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.