Combining many sparse binary variables

Question

Based on kjetil b halvorsen suggestion, I rephrased my problem:

My problem is analogous to the following: i am supposed to predict if a high school student will go to university (Yes/No).

I have some data (high school scores etc) and I can make a model with modest auc < 0.7. In addition to that, I have event data (student id, date, event id) and domain experts say it include very useful information. However, I can not find a way to improve my auc with the data.

I think the problem including the event data is that:

1. the data is very sparse. Most of students do not have any events and those who have, have only few event bits on
2. some of the have strong association so that e.g. events E1 and E221 are practically the same

So my question is: What is a typical and statistically sound way to combine sparse E1 - E500 binary events into e.g. five event clusters C1 - C5 to be used as features in classification problem?

My approaches so far:

glmnet/lasso

First I included all the events as binary features into glmnet / lasso and I got different events included or dropped from the model each time. I guess it is due to sparsity and cross correlation.

I did a hack in which I ran glmnet / lasso 1000 times, then summed up the coefficients of each event, and finally combined the events into five clusters C1 - C5 based on the coefficient sum of 1000 runs. Using C1 - C5 in my model gave better auc than using the original E1 - E500 so that makes me think that some kind of grouping/clustering would be useful.

MCA

Somebody suggested MCA. I tried that but primary dimension explained only 4.9% of variance so maybe it is not very useful. I guess low explainability is due to data sparsity

logic regression

Logic regression (not logistic regression) is a algorithm to combine binary variables into one binary predictor http://kooperberg.fhcrc.org/logic/documents/logic-regression.pdf

I tried it, but the result is not stable. It gave totally different variables to be included in predictor in every run

Answer 1

I conclude this thread by explaining how I finally (partly) solved this problem. I tried all the proposed approaches without too much success. Finally I asked one domain expert to manually cluster the events E1...E221 into four groups C1..C4 and I used them in my model. For various reasons, manual clustering was quite significant amount of work but my model improved to the satisfactory level of AUC == 0.80 and I was able to proceed with the project.

I want to thank all of you who contributed into this thread. I have learnt a lot on new cool statistical tools and I will study them further as a new project is lurking around and there the amount of events is beyond of manual clustering.

So, I am still looking for an algorithm to automatically cluster hundreds of sparse binary variables into ~dozen clusters :-)