I have a dataset with approximately $70,000$ entries and $8$ features. Some of them are ordinal and the rest are nominal. The task is a binary classification task; however, the class I am interested in is represented only by $5\%$ in my dataset (highly imbalanced classes).

Some of the nominal features have many levels, so I have tried to group them (based on the relative frequency), such that I have at most 8 levels for every feature.

Moreover, I am interested in getting a fairly high precision in the minority class. I have tried RandomOverSample and RandomUnderSample using RandomForest, but in every case I get a precision of ~$8\%$.

I have implemented RandomForest, since there is no need to proceed to one-hot encoding and I know it generally performs well even in imbalanced data.

I don't know how else to proceed regarding this one. I was wondering if there is anything fundamental which I probably miss.

P.S. I can/will definitely try different classification algorithms, such as SVM (though I need to introduce one-hot encoding in my data).

  • $\begingroup$ Are you using a random forest implementation that is explicitly aware of nominal/categorical variables? I ask because some implementations are not, and require some kind of numerical encoding (and one-hot may not perform well in these cases). $\endgroup$
    – user20160
    Feb 28 '18 at 0:22
  • $\begingroup$ @user20160 No, it's not aware of this, but in any case all the levels of every categorical feature are numbers, i.e. label encoding has been used. For the specific case of the random forest implementation, I haven't used one-hot encoding at all. $\endgroup$ Feb 28 '18 at 0:36
  • $\begingroup$ That is not highly imbalanced. Highly imbalanced is a fraction of a percent. You need to fit a probabalistic model and tune the decision threshold. $\endgroup$ Feb 28 '18 at 1:36

Suppose you have a categorical variable that takes 6 possible values. One might be tempted to simply represent these values as integers 1 thru 6. But, if the random forest interprets these as numerical values, it will always group consecutively ordered values together when splitting a node. For example, the left and right nodes may contain values $(\{1,2,3\},\{4,5,6\})$ or $(\{1,2,3,4\},\{5,6\})$, etc. In contrast, values 1 and 6 will never be grouped together. However, because the variable is categorical, there is no underlying order, and it may actually be necessary to group non-consecutive values together when performing a split. Failing to do so prevents such splits from being considered, and could hurt performance.

One solution is to use an implementation of random forests that is explicitly aware of categorical variables, and splits them correctly. Some implementations have this feature, but others don't; rather, they treat all values as numerical. In this case, it's necessary to encode categorical variables numerically (e.g. using one-hot coding or an alternative).

Based on the comments, it sounds like you're not currently using either of these solutions, so I'd recommend looking into this. It might help performance, but no guarantees; there are plenty of other things that could be happening, including the possibility that your inputs simply don't contain much information about your outputs.

Using a categorical-aware RF implementation may work better than one-hot coding (e.g. see this blog post). The potential problems with one-hot coding stem from its extreme sparsity, combined with the fact that RFs split on a single feature at a time. Thus, if going the numerical encoding route, it may be worth looking into alternative coding strategies that produce less sparse representations (e.g. feature hashing).

  • $\begingroup$ So, as a rule of thumb when I have both ordinal and categorical data represented by numbers, I can understand that I should use one-hot encoding for the categorical data, and the ordinal data should be labeled according to the importance of the levels. The level with the lowest value ->0 , the middle one -> 1, the best one -> 2. And I guess then it's up to us if we want to handle the ordinal data as numerical or nominal. Right? $\endgroup$ Mar 1 '18 at 16:38
  • $\begingroup$ For categorical variables, either use a categorical aware RF implementation, or use a numerical encoding (one-hot or alternative). For ordinal variables, code as any numerical values that increase monotonically with the order (the exact values don't make a difference). These should be handled the same as numerical variables, not treated as categorical. $\endgroup$
    – user20160
    Mar 1 '18 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.