I am trying to tackle a problem that involves binary classification. However, my data is multi-level or hierarchical in its structure. This example illustrates its structure:
Time Group Response F1 F2 F3 F4
1 1 a 0 0.89 0.01 10 2
2 2 a 1 0.82 0.58 21 4
3 3 a 1 0.34 0.39 37 6
4 1 b 0 0.32 0.26 10 2
5 2 b 0 0.32 0.04 21 4
6 3 b 0 0.00 0.20 37 6
7 1 c 1 0.80 0.18 10 2
8 2 c 0 0.34 0.71 21 4
9 3 c 1 0.55 0.52 37 6
We have multiple groups which have a binary response (0, 1) that is measured over time. The are an array of features associated with each response in each group. Some of these features take unique values for each time point in each group (F1, F2), while others (F3, F4) represent the environmental "state" that was common across groups at each time point. All feature variables are continuous and numeric and I also do not tell the model that they belong to different groups as I want the model to predict the response based on the continuous features. This is important because over time some groups go extinct and disappear from the data set and other new groups come in so I want the model to be agnostic to group ID.
I use a rolling window walk-forward validation scheme to account for the temporal nature of the data. Until now, I have been running this data through a random forest model via the h2o package in R with y = Response and x = c(F1:F4). The results have been reasonable, however I am noticing that the environmental features (F3, F4) have the most importance and the model is good at predicting when the environment is right – e.g. conditions of F3 and F4 that enable discrimination between 0 and 1 in the Response. However the model is less good at distinguishing between groups on any given day given the environment.
This led me to think that this outcome may be due to not fully accounting for the hierarchical nature of the data. Instead I had the idea to fit random forest models in a 2 stage process. In stage 1, I train and validate the model using only the environmental features (F3, F4). I then use this model to generate predictions for the response variable and then keep only the positive responses (‘1s’) and throw out all negative responses (‘0s’). In stage 2 I then train and validate a new random forest model on this new dataset which includes only the group specific features (F1, F2) to make more granular predictions of when each group will exhibit a positive or negative response. I then finish in stage 3 by testing the two random forest models on the hold out test data in the same 2 stage process – i.e. test with model 1 on F3 and F4, keep positive predictions and then test with model 2 using F1 and F2 to generate the final set of granular predictions.
Does this approach seem reasonable and sensible? Or are there other/better ways to deal with the data structure I have?
