# How to deal with data in which users_ids belong to more than one category (Multilevel) using Random Forest?

I know it sounds trivial, but I could not find any ready answers for this. Suppose we have this kind of data and we want to predict some target values for several users.

target user_id value1  category ...
0        1       .425    Ab|Be
0        2       .325     Ab
1        3       .26     Ab|Ck
1        4       .28     Be|Ck|Ju|Po
0        5       .56      Ab|Ck
1        6       .25       Ck


My issue is: how to deal with this kind of data using a statistical learning technique such as RandomForest.

I tried to extend the database this way and, after that, to calculate averages for cateogories, but I am unsure whether this is the appropriate way:

target user_id value1  category ...
0        1       .425    Ab
0        1       .425    Be
0        2       .325    Ab
1        3       .26     Ab
1        3       .26     Ck
1        4       .28     Be
1        4       .28     Ck
1        4       .28     Ju
1        4       .28     Po
0        5       .56     Ab
0        5       .56     Ck
1        6       .25     Ck


As a matter of fact, I couldn not get better results with this approach. And it looks wrong to me.

Just dropping the column is not an option either, because I am sure that it is important for the prediction.

I also know there are straightforward ways of modeling this with bayesian statistics and employing a software like Jags, but I am refering in this case to RandomForest.

What would you suggest?

• I would like to hear more of how the data was obtained and if users occur several times. To include user_id could be problematic if each sample is an observation of a given user, and over time several observations are acquired. At the first observation of an user, predictions based on a user_id is useless. Thus, although this is not a time series forecasting, if samples are collected over time, and predictions are made online, batch modelling may not be useful in practice and batch cross-validation (nFold-CV, OOB-CV) over-optimistic. If so, back testing is needed. – Soren Havelund Welling Feb 3 '16 at 21:51
• A person should only be judged by his past actions, not future, unless we're speaking of the Sci-Fi movie Minority Report :) – Soren Havelund Welling Feb 3 '16 at 22:21

In R, this is your dataset:

> dataset <- structure(list(
target = structure(c(1L, 1L, 2L, 2L, 1L, 2L), .Label = c("0", "1"), class = "factor"),
user_id = structure(1:6, .Label = c("1", "2", "3", "4", "5", "6"), class = "factor"),
value1 = structure(c(5L, 4L, 2L, 3L, 6L, 1L), .Label = c(".25", ".26", ".28", ".325", ".425", ".56"), class = "factor"),
category = structure(c(2L, 1L, 3L, 4L, 3L, 5L), .Label = c("Ab", "Ab|Be", "Ab|Ck", "Be|Ck|Ju|Po", "Ck"), class = "factor")),
.Names = c("target", "user_id", "value1", "category"), row.names = c(NA, -6L), class = "data.frame")


Each of your user_ids is a member of one or more classes. So just create dummy variables for all possible classes and put in a 1 (or TRUE) if that user_id is a member of that class. Again in R:

> foo <- strsplit(as.character(dataset$category),"\\|") > bar <- unique(unlist(foo)) > dataset.new <- cbind(dataset,t(sapply(foo,function(xx)table(factor(xx,levels=bar))))) > dataset.new target user_id value1 category Ab Be Ck Ju Po 1 0 1 .425 Ab|Be 1 1 0 0 0 2 0 2 .325 Ab 1 0 0 0 0 3 1 3 .26 Ab|Ck 1 0 1 0 0 4 1 4 .28 Be|Ck|Ju|Po 0 1 1 1 1 5 0 5 .56 Ab|Ck 1 0 1 0 0 6 1 6 .25 Ck 0 0 1 0 0  Now you can start classifiying, including membership in the various classes (and value1) as predictors. (Make sure your 0-1 target is interpreted as class memberships, not numerics.) > dataset.new$target <- factor(dataset.new$target) > library(randomForest) > model <- randomForest(target~value1+Ab+Be+Ck+Ju+Po,dataset.new) > model Call: randomForest(formula = target ~ value1 + Ab + Be + Ck + Ju + Po, data = dataset.new) Type of random forest: classification Number of trees: 500 No. of variables tried at each split: 2 OOB estimate of error rate: 50% Confusion matrix: 0 1 class.error 0 3 0 0 1 3 0 1 > model$importance

MeanDecreaseGini
value1       1.05986667
Ab           0.47546667
Be           0.11180000
Ck           0.45940000
Ju           0.09733333
Po           0.10820000