Based on kjetil b halvorsen suggestion, I rephrased my problem:
My problem is analogous to the following: i am supposed to predict if a high school student will go to university (Yes/No).
I have some data (high school scores etc) and I can make a model with modest auc < 0.7. In addition to that, I have event data (student id, date, event id) and domain experts say it include very useful information. However, I can not find a way to improve my auc with the data.
I think the problem including the event data is that:
- the data is very sparse. Most of students do not have any events and those who have, have only few event bits on
- some of the have strong association so that e.g. events E1 and E221 are practically the same
So my question is: What is a typical and statistically sound way to combine sparse E1 - E500 binary events into e.g. five event clusters C1 - C5 to be used as features in classification problem?
My approaches so far:
First I included all the events as binary features into
lasso and I got different events included or dropped from the model each time. I guess it is due to sparsity and cross correlation.
I did a hack in which I ran
lasso 1000 times, then summed up the coefficients of each event, and finally combined the events into five clusters C1 - C5 based on the coefficient sum of 1000 runs. Using C1 - C5 in my model gave better auc than using the original E1 - E500 so that makes me think that some kind of grouping/clustering would be useful.
Somebody suggested MCA. I tried that but primary dimension explained only 4.9% of variance so maybe it is not very useful. I guess low explainability is due to data sparsity
Logic regression (not logistic regression) is a algorithm to combine binary variables into one binary predictor http://kooperberg.fhcrc.org/logic/documents/logic-regression.pdf
I tried it, but the result is not stable. It gave totally different variables to be included in predictor in every run