How do I handle correlated data points in classification problem where users generate multiple events? If I have a data set containing multiple users' data with features $X$ and some class (let's assume binary for simplicity) that I want to classify, $Y$. A user can generate multiple "events" and $X$ as well as $Y$ might not be the same for each event. Each "event" is a discrete moment in time but surely each user's events are correlated by the mere virtue of being generated by the same person.
How do I deal with this in building a machine learning classifier for $Y$?
I have some ID to distinguish between users. If I ignore the ID and treat the data as independent I loose some information and also perhaps bias the classifier to work well on users for whom I have a lot of events. But if I keep only one event per user I loose a lot of information for cases where their behaviour could have been different with different (or the same) $Y$.
Is there any literature on dealing with cases such as this or any best practices?
Perhaps adding a feature to $X$ indicating if it is a first-time or current user or perhaps a running count on the number of events generated with that user for each data point?
I'd also possibly want to stratify my train/validation/test set to not contain the same users or... be stratified over time so that the validation/test set is later in time than the train set.
 A: You have what is called repeated-measures, which is typically modeled using mixed-models. I'd encourage you to look through questions carrying these two tags and maybe look at some textbooks.
In your specific case, you have a binary response, so a mixed logistic regression would be appropriate. In R, you can use the lme4 package for this.
Alternatively, you could use ML tools like CARTs or Random Forests, including the ID for each observation as a feature. The question then is whether you want to classify a response from a new user or from one that you have seen in your training data. In the first case, your ID feature would contain a previously unobserved value, and your ML tool might choke on this. Mixed models can deal gracefully with such a situation.
A: Stephan Kolassa already gave a great answer. I'd like to add some thoughts regarding the stratification. I can't stress enough that you must split data based on users (train/test/val), if you intend to evaluate performance for a new user. If you leak data of the same user between your different sets it will give you biased performance estimates.
