0
$\begingroup$

I got a huge dataset, with 75 variables and over 800,000 observations. The target variable is "time to event", where the event is the withdrawal of the subject from a gambling website (days since joining). The independent variables vary, some are numeric, some categorical, for example: sum of winnings, gender, etc..I do not have censoring. My aim is to try and predict the number of days to withdrawal. The distribution of my target variable looks like this, and is clearly not normal:

enter image description here

This is a data mining problem from size of data point of view. How should I analyze this data? Which model should I use?

Thank you.

Improve this question
$\endgroup$
4
  • $\begingroup$ In what sense do you not have censoring- did you wait until each subject did finally withdraw from website (just checking) $\endgroup$
    – seanv507
    Commented Sep 20, 2016 at 19:57
  • $\begingroup$ Out of curiosity, how is "withdrawal" defined? Does withdrawal mean "withdraw money from the site?" Or does it mean something like, "runs out of money in their account?" Or does it mean something like, "doesn't login anymore" (which I would think isn't clearly observable.) $\endgroup$
    – Matthew Gunn
    Commented Sep 20, 2016 at 20:18
  • $\begingroup$ One thing to check is fit of the exponential distribution (which is often used to model inter-arrival times). Another thing to check might be what the distribution looks like in logs? $\endgroup$
    – Matthew Gunn
    Commented Sep 20, 2016 at 20:22
  • $\begingroup$ yes, all subjects eventually withdrawn from the website. Matthew, your first definition is closer to what I have. $\endgroup$
    – user3275222
    Commented Sep 21, 2016 at 4:15

1 Answer 1

Reset to default
1
$\begingroup$

The problem you have is called 'customer churn'. Look up retention and churn on Wikipedia and Google Scholar articles.

https://en.wikipedia.org/wiki/Churn_rate

https://en.wikipedia.org/wiki/Customer_retention

Strictly speaking you have a regression problem: you have a few dozen variables (features) and you want to predict a continuous dependent variable, i.e. the number of days while the customer is retained. This will be your target variable, you will regress this.

As a start I would recommend trying a Random Forest regressor, a multivariate linear regression using only the numeric features or a SVR (support vector regressor).

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html

http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html

If you have the actual meaning of the feature values I recommend spending ~80% of your time on feature engineering not on the model. This is a problem where good features are more important than smart algorithms.

P.S.: Yes, the distribution of retention periods is very not normal, it's like that in all online services. It's closer to a power law than to a normal distribution.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you Peter. How robust will the regression be when the data is so skewed? $\endgroup$
    – user3275222
    Commented Sep 21, 2016 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.