Handling categorical and ordinal data with highly imbalanced classes

Question

I have a dataset with approximately $70,000$ entries and $8$ features. Some of them are ordinal and the rest are nominal. The task is a binary classification task; however, the class I am interested in is represented only by $5\%$ in my dataset (highly imbalanced classes).

Some of the nominal features have many levels, so I have tried to group them (based on the relative frequency), such that I have at most 8 levels for every feature.

Moreover, I am interested in getting a fairly high precision in the minority class. I have tried RandomOverSample and RandomUnderSample using RandomForest, but in every case I get a precision of ~$8\%$.

I have implemented RandomForest, since there is no need to proceed to one-hot encoding and I know it generally performs well even in imbalanced data.

I don't know how else to proceed regarding this one. I was wondering if there is anything fundamental which I probably miss.

---

P.S. I can/will definitely try different classification algorithms, such as SVM (though I need to introduce one-hot encoding in my data).

Answer 1

Suppose you have a categorical variable that takes 6 possible values. One might be tempted to simply represent these values as integers 1 thru 6. But, if the random forest interprets these as numerical values, it will always group consecutively ordered values together when splitting a node. For example, the left and right nodes may contain values $(\{1,2,3\},\{4,5,6\})$ or $(\{1,2,3,4\},\{5,6\})$, etc. In contrast, values 1 and 6 will never be grouped together. However, because the variable is categorical, there is no underlying order, and it may actually be necessary to group non-consecutive values together when performing a split. Failing to do so prevents such splits from being considered, and could hurt performance.

One solution is to use an implementation of random forests that is explicitly aware of categorical variables, and splits them correctly. Some implementations have this feature, but others don't; rather, they treat all values as numerical. In this case, it's necessary to encode categorical variables numerically (e.g. using one-hot coding or an alternative).

Based on the comments, it sounds like you're not currently using either of these solutions, so I'd recommend looking into this. It might help performance, but no guarantees; there are plenty of other things that could be happening, including the possibility that your inputs simply don't contain much information about your outputs.

Using a categorical-aware RF implementation may work better than one-hot coding (e.g. see this blog post). The potential problems with one-hot coding stem from its extreme sparsity, combined with the fact that RFs split on a single feature at a time. Thus, if going the numerical encoding route, it may be worth looking into alternative coding strategies that produce less sparse representations (e.g. feature hashing).