Multi-level / Hierarchical Machine Learning I am trying to tackle a problem that involves binary classification. However, my data is multi-level or hierarchical in its structure. This example illustrates its structure:
  Time Group Response   F1   F2 F3 F4
1    1     a        0 0.89 0.01 10  2
2    2     a        1 0.82 0.58 21  4
3    3     a        1 0.34 0.39 37  6
4    1     b        0 0.32 0.26 10  2
5    2     b        0 0.32 0.04 21  4
6    3     b        0 0.00 0.20 37  6
7    1     c        1 0.80 0.18 10  2
8    2     c        0 0.34 0.71 21  4
9    3     c        1 0.55 0.52 37  6

We have multiple groups which have a binary response (0, 1) that is measured over time. The are an array of features associated with each response in each group. Some of these features take unique values for each time point in each group (F1, F2), while others (F3, F4) represent the environmental "state" that was common across groups at each time point. All feature variables are continuous and numeric and I also do not tell the model that they belong to different groups as I want the model to predict the response based on the continuous features. This is important because over time some groups go extinct and disappear from the data set and other new groups come in so I want the model to be agnostic to group ID.
I use a rolling window walk-forward validation scheme to account for the temporal nature of the data. Until now, I have been running this data through a random forest model via the h2o package in R with y = Response and x = c(F1:F4). The results have been reasonable, however I am noticing that the environmental features (F3, F4) have the most importance and the model is good at predicting when the environment is right – e.g. conditions of F3 and F4 that enable discrimination between 0 and 1 in the Response. However the model is less good at distinguishing between groups on any given day given the environment.
This led me to think that this outcome may be due to not fully accounting for the hierarchical nature of the data. Instead I had the idea to fit random forest models in a 2 stage process. In stage 1, I train and validate the model using only the environmental features (F3, F4). I then use this model to generate predictions for the response variable and then keep only the positive responses (‘1s’) and throw out all negative responses (‘0s’). In stage 2 I then train and validate a new random forest model on this new dataset which includes only the group specific features (F1, F2) to make more granular predictions of when each group will exhibit a positive or negative response. I then finish in stage 3 by testing the two random forest models on the hold out test data in the same 2 stage process – i.e. test with model 1 on F3 and F4, keep positive predictions and then test with model 2 using F1 and F2 to generate the final set of granular predictions.
Does this approach seem reasonable and sensible? Or are there other/better ways to deal with the data structure I have?
 A: With the data set out how it is, I'm not sure it accounts for the non-independence of your data points. I would consider modelling each time point separately. This has the advantage that you could include previous state as a predictor - it depends what exactly you're modelling, but shouldn't the likelihood of being 1 or 0 at time point 3 depend strongly on whether that group was 1 or 0 at time point 2?
So for example, I would split your example data into three training data sets, split by the Time variable.
Response at Time point 1 would be modelled like this:
Response_at_time_point_1 ~ F1 + F2 + F3 + F4

Response at Time point 2 would be modelled like this:
Response_at_time_point_2 ~ F1 + F2 + F3 + F4 + Response_at_time_point_1 

Response at Time point 3 would be modelled like this:
Response_at_time_point_3 ~ F1 + F2 + F3 + F4 + Response_at_time_point_1 + Response_at_time_point_2 

You could add a helper column to determine which sub-model each row should go to, based on what time point they are at.
