1
$\begingroup$

If I have a data set containing multiple users' data with features $X$ and some class (let's assume binary for simplicity) that I want to classify, $Y$. A user can generate multiple "events" and $X$ as well as $Y$ might not be the same for each event. Each "event" is a discrete moment in time but surely each user's events are correlated by the mere virtue of being generated by the same person.

How do I deal with this in building a machine learning classifier for $Y$?

I have some ID to distinguish between users. If I ignore the ID and treat the data as independent I loose some information and also perhaps bias the classifier to work well on users for whom I have a lot of events. But if I keep only one event per user I loose a lot of information for cases where their behaviour could have been different with different (or the same) $Y$.

Is there any literature on dealing with cases such as this or any best practices?

Perhaps adding a feature to $X$ indicating if it is a first-time or current user or perhaps a running count on the number of events generated with that user for each data point?

I'd also possibly want to stratify my train/validation/test set to not contain the same users or... be stratified over time so that the validation/test set is later in time than the train set.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

You have what is called , which is typically modeled using s. I'd encourage you to look through questions carrying these two tags and maybe look at some textbooks.

In your specific case, you have a binary response, so a mixed logistic regression would be appropriate. In R, you can use the lme4 package for this.

Alternatively, you could use ML tools like CARTs or Random Forests, including the ID for each observation as a feature. The question then is whether you want to classify a response from a new user or from one that you have seen in your training data. In the first case, your ID feature would contain a previously unobserved value, and your ML tool might choke on this. Mixed models can deal gracefully with such a situation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Stephan, thanks for the reply. You've given me a couple of terms to search for online. Ideally I would be able to handle lots of new users (with the occasional repeat user), so perhaps mixed models would be better than CARTs/Random Forests. In your opinion are mixed models still appropriate when the majority of events are new users? $\endgroup$
    – CFdV
    Nov 2, 2017 at 12:53
  • $\begingroup$ I'd say that mixed models would be especially useful in such a situation. If you can meaningfully cluster your IDs and have a new user that you don't know yet but know the cluster membership of, you might be able to use hierarchical models. $\endgroup$
    – Stephan Kolassa
    Nov 2, 2017 at 13:11
2
$\begingroup$

Stephan Kolassa already gave a great answer. I'd like to add some thoughts regarding the stratification. I can't stress enough that you must split data based on users (train/test/val), if you intend to evaluate performance for a new user. If you leak data of the same user between your different sets it will give you biased performance estimates.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.