# Handling categorical and ordinal data with highly imbalanced classes

I have a dataset with approximately $70,000$ entries and $8$ features. Some of them are ordinal and the rest are nominal. The task is a binary classification task; however, the class I am interested in is represented only by $5\%$ in my dataset (highly imbalanced classes).

Some of the nominal features have many levels, so I have tried to group them (based on the relative frequency), such that I have at most 8 levels for every feature.

Moreover, I am interested in getting a fairly high precision in the minority class. I have tried RandomOverSample and RandomUnderSample using RandomForest, but in every case I get a precision of ~$8\%$.

I have implemented RandomForest, since there is no need to proceed to one-hot encoding and I know it generally performs well even in imbalanced data.

I don't know how else to proceed regarding this one. I was wondering if there is anything fundamental which I probably miss.

P.S. I can/will definitely try different classification algorithms, such as SVM (though I need to introduce one-hot encoding in my data).

• Are you using a random forest implementation that is explicitly aware of nominal/categorical variables? I ask because some implementations are not, and require some kind of numerical encoding (and one-hot may not perform well in these cases). – user20160 Feb 28 '18 at 0:22
• @user20160 No, it's not aware of this, but in any case all the levels of every categorical feature are numbers, i.e. label encoding has been used. For the specific case of the random forest implementation, I haven't used one-hot encoding at all. – thanasissdr Feb 28 '18 at 0:36
• That is not highly imbalanced. Highly imbalanced is a fraction of a percent. You need to fit a probabalistic model and tune the decision threshold. – Matthew Drury Feb 28 '18 at 1:36

Suppose you have a categorical variable that takes 6 possible values. One might be tempted to simply represent these values as integers 1 thru 6. But, if the random forest interprets these as numerical values, it will always group consecutively ordered values together when splitting a node. For example, the left and right nodes may contain values $(\{1,2,3\},\{4,5,6\})$ or $(\{1,2,3,4\},\{5,6\})$, etc. In contrast, values 1 and 6 will never be grouped together. However, because the variable is categorical, there is no underlying order, and it may actually be necessary to group non-consecutive values together when performing a split. Failing to do so prevents such splits from being considered, and could hurt performance.