# Combining many sparse binary variables

Based on kjetil b halvorsen suggestion, I rephrased my problem:

My problem is analogous to the following: i am supposed to predict if a high school student will go to university (Yes/No).

I have some data (high school scores etc) and I can make a model with modest auc < 0.7. In addition to that, I have event data (student id, date, event id) and domain experts say it include very useful information. However, I can not find a way to improve my auc with the data.

I think the problem including the event data is that:

1. the data is very sparse. Most of students do not have any events and those who have, have only few event bits on
2. some of the have strong association so that e.g. events E1 and E221 are practically the same

So my question is: What is a typical and statistically sound way to combine sparse E1 - E500 binary events into e.g. five event clusters C1 - C5 to be used as features in classification problem?

## My approaches so far:

### glmnet/lasso

First I included all the events as binary features into glmnet / lasso and I got different events included or dropped from the model each time. I guess it is due to sparsity and cross correlation.

I did a hack in which I ran glmnet / lasso 1000 times, then summed up the coefficients of each event, and finally combined the events into five clusters C1 - C5 based on the coefficient sum of 1000 runs. Using C1 - C5 in my model gave better auc than using the original E1 - E500 so that makes me think that some kind of grouping/clustering would be useful.

### MCA

Somebody suggested MCA. I tried that but primary dimension explained only 4.9% of variance so maybe it is not very useful. I guess low explainability is due to data sparsity

### logic regression

Logic regression (not logistic regression) is a algorithm to combine binary variables into one binary predictor http://kooperberg.fhcrc.org/logic/documents/logic-regression.pdf

I tried it, but the result is not stable. It gave totally different variables to be included in predictor in every run

• Possible duplicate of Principled way of collapsing categorical variables with many levels? – kjetil b halvorsen Jul 11 '19 at 8:48
• @kjetil, I don't think that is a duplicate. The OP's data are many binary variables, not categorical variable(s) with many levels. – ttnphns Jul 11 '19 at 14:41
• I edited your question and tags because it appeared to me that your 500 correlated sparse features are binary (i.e. "events"). If that is true, please search for similar questions by tags "binary data" and "sparse". – ttnphns Jul 11 '19 at 14:46
• Are you interested in inference, or just prediction (i.e. do you want to draw conclusions about how your response variable is related to the events, or only to predict the response as well as possible)? – user20160 Jul 11 '19 at 23:38
• @user20160: very good point. My paying customer is interested in prediction whereas other key stakeholders in a project are interested in inference – Tikke Jul 15 '19 at 9:05