3
$\begingroup$

Based on kjetil b halvorsen suggestion, I rephrased my problem:

My problem is analogous to the following: i am supposed to predict if a high school student will go to university (Yes/No).

I have some data (high school scores etc) and I can make a model with modest auc < 0.7. In addition to that, I have event data (student id, date, event id) and domain experts say it include very useful information. However, I can not find a way to improve my auc with the data.

I think the problem including the event data is that:

  1. the data is very sparse. Most of students do not have any events and those who have, have only few event bits on
  2. some of the have strong association so that e.g. events E1 and E221 are practically the same

So my question is: What is a typical and statistically sound way to combine sparse E1 - E500 binary events into e.g. five event clusters C1 - C5 to be used as features in classification problem?

My approaches so far:

glmnet/lasso

First I included all the events as binary features into glmnet / lasso and I got different events included or dropped from the model each time. I guess it is due to sparsity and cross correlation.

I did a hack in which I ran glmnet / lasso 1000 times, then summed up the coefficients of each event, and finally combined the events into five clusters C1 - C5 based on the coefficient sum of 1000 runs. Using C1 - C5 in my model gave better auc than using the original E1 - E500 so that makes me think that some kind of grouping/clustering would be useful.

MCA

Somebody suggested MCA. I tried that but primary dimension explained only 4.9% of variance so maybe it is not very useful. I guess low explainability is due to data sparsity

logic regression

Logic regression (not logistic regression) is a algorithm to combine binary variables into one binary predictor http://kooperberg.fhcrc.org/logic/documents/logic-regression.pdf

I tried it, but the result is not stable. It gave totally different variables to be included in predictor in every run

$\endgroup$
  • 5
    $\begingroup$ Possible duplicate of Principled way of collapsing categorical variables with many levels? $\endgroup$ – kjetil b halvorsen Jul 11 '19 at 8:48
  • 2
    $\begingroup$ @kjetil, I don't think that is a duplicate. The OP's data are many binary variables, not categorical variable(s) with many levels. $\endgroup$ – ttnphns Jul 11 '19 at 14:41
  • 1
    $\begingroup$ I edited your question and tags because it appeared to me that your 500 correlated sparse features are binary (i.e. "events"). If that is true, please search for similar questions by tags "binary data" and "sparse". $\endgroup$ – ttnphns Jul 11 '19 at 14:46
  • 1
    $\begingroup$ Are you interested in inference, or just prediction (i.e. do you want to draw conclusions about how your response variable is related to the events, or only to predict the response as well as possible)? $\endgroup$ – user20160 Jul 11 '19 at 23:38
  • 1
    $\begingroup$ @user20160: very good point. My paying customer is interested in prediction whereas other key stakeholders in a project are interested in inference $\endgroup$ – Tikke Jul 15 '19 at 9:05
0
$\begingroup$

I conclude this thread by explaining how I finally (partly) solved this problem. I tried all the proposed approaches without too much success. Finally I asked one domain expert to manually cluster the events E1...E221 into four groups C1..C4 and I used them in my model. For various reasons, manual clustering was quite significant amount of work but my model improved to the satisfactory level of AUC == 0.80 and I was able to proceed with the project.

I want to thank all of you who contributed into this thread. I have learnt a lot on new cool statistical tools and I will study them further as a new project is lurking around and there the amount of events is beyond of manual clustering.

So, I am still looking for an algorithm to automatically cluster hundreds of sparse binary variables into ~dozen clusters :-)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.