# Random forest on multi-level/hierarchical-structured data

I am quite new to machine learning, CART-techniques and the like, and I hope my naivete isn't too obvious.

How does Random Forest handle multi-level/hierarchical data structures (for example when cross-level interaction is of interest)?

That is, data sets with units of analysis at several hierarchical levels (e.g., students nested within schools, with data about both the students and the schools).

Just as an example, consider a multi-level data set with individuals on the first level (e.g., with data on voting behavior, demographics etc.) nested within countries at the second level (with country-level data; e.g., population):

ID voted age female country population
1 1 19 1 1 53.01
2 1 23 0 1 53.01
3 0 43 1 1 53.01
4 1 27 1 1 53.01
5 0 67 0 1 53.01
6 1 34 1 2 47.54
7 0 54 1 2 47.54
8 0 22 1 2 47.54
9 0 78 0 2 47.54
10 1 52 0 2 47.54

Lets say that voted is the response/dependent variable and the others are predictor/independent variables. In these types of cases, margins and marginal effects of a variable (partial dependence) for some higher-level variable (e.g., population) for different individual-level variables, etc., could be very interesting. In a case similar to this, glm is of course more appropriate -- but when there are many variables, interactions and/or missing values, and/or very large-scale datasets etc., glm is not so reliable.

Subquestions: Can Random Forest explicitly handle this type of data structure in some way? If used regardless, what kind of bias does it introduce? If Random Forest is not appropriate, is there any other ensemble-type method that is?

(Question Random forest on grouped data is perhaps similar, but doesn't really answer this.)

• Hi @MikaelAndersson, did you find a solution to the questions you raised? I am facing a similar situation and hope to hear your experience. Thanks. – NoviceProg Sep 5 '15 at 3:45

Random Forests would work fine, but you have to be very careful when you tune the hyperparameters (especially if you want a realistic measure of generalization performance). The traditional OOB error estimates will be way optimistic since there is rampant "twinning" in your data.

To get proper tuning and generalization estimates you need to understand what are the characteristics of any new data you expect to encounter. If you want to extrapolate to new countries, then you'll need to set up some manner of re-sample based tuning (such as k-fold cross validation) that does stratified sampling by country.

You also need to be careful how you encode the data into a Random Forest. It appears that country is a categorical variable. Feeding it in as a numeric would be a bit rough, but not hopeless (especially if you ordered the IDs by something useful).

• Could you expand a little on why the OOB error estimates will be too optimistic? – dmartin Oct 23 '14 at 14:44
• I suppose their optimism depends on what new data might look like. If new data came from other counties, then this random forest would likely not perform as well as it's OOB errors indicate. This is because the OOB errors are still coming from samples from the same set of counties for example. – Shea Parkes Oct 23 '14 at 19:25

I'm actually working on a R package that runs randomForest as the local classifier along a pre-defined class hierarchy. The package can be found in R Forge under 'hie-ran-forest'. The package is already operational, although it is failing one of the cran test (for MAC), I'm not sure exactly why. In addition to actually running randomForest for each parent node down the hierarchy, the package also contains predict functions and performance functions. One of the performance measures actually accounts for the hierarchical class structure.

The multiplicative proportion of votes are comparable to the proportion of votes produced by a regular randomForest

• Can you clarify that your package addresses "cross-level interactions", & if so, how it does this? Just saying that a package exists isn't much of an answer (I don't mean to be too critical here, but CV is looking to build a permanent repository of high-quality ML information & the fact that a package exists doesn't quite meet that standard.) – gung - Reinstate Monica Mar 23 '15 at 15:51
In a single classification tree, these groups are coded the same as any other categorical variable. This is often done as either binary coding or just using an integer. There are different arguments for using either. In random forests if you are using binary coding, some groups will be included/excluded for any given tree. So you may have an indicator for country_2 but not country_3. If you leave the group variable as an integer then the ordering can affect the outcome as well. What does it mean for country > 5 and country < 12? How does that change if you randomly re-label the countries with new integers?
In your case, country and population are perfectly collinear. There is no information gained by using both variables in your model. So you can think about how a random forest model would treat these variables in your data.