I have a dataset as below. It is a classification problem with multiple x variables and a y variable. Y variable has 5 levels. The below picture has y variable on x axis and one x variable on y axis. As you can see from the picture below, for each value of Y, a x variable distribution is not very different (i plotted similar chart for all x variables- one x variable at a time vs y and seeing the same trend). Due to this issue, most of the observations are getting classified as class 0 or 4. I built a randomforest with 500 trees. How could i improve accuracy? I thought of taking square or cube of each x so that distance between class means would increase but it won't help with variance of each class

distribution of 5 classes in my data is as below

   0    1    2    3    4 
5104 2639 2322 2661 5274 

enter image description here

Column means and standard deviations of my x variables are very similar :(

update 1

I have 26 x variables and 5 outputs. As mentioned above most of the datapoints are being predicted to class 0 or 5. I performed one additional model where for each Y class i found out mean of each x variable. That step provided me 26*5 means. Then for each observations and each x variable i found out squared distance between the datapoint and class centers and summed it for each observation. This would return me 5 distances for each observations and I assigned that observation to a class where distance is minimum. This approach is much better.

The approach in short:

  1. find mean of x variable for each Y class - for example i got x1_mean_for_y=0
  2. find squared distances for each value of x column - for example squared distance between (1st observation of x1 and x1_mean_for_y)
  3. for each datapoint some all distances by class
  4. assign a datapoint to a class where the distance is minimum

This approach assigns relatively less number of observations to class 0 and 4 and my overall accuracy has gone by 7%

how could i improve my model further?

  • $\begingroup$ Are all 26 features continuous? (I assume so, since you say you calculate class conditional expectations for each x) $\endgroup$ – Zhubarb Apr 18 '17 at 13:16
  • $\begingroup$ yes...all 26 features are continuos $\endgroup$ – user2543622 Apr 18 '17 at 17:42
  • $\begingroup$ Why not try a multi-layer neural network with dropout? If there is some deep structure in your data a RF might have some trouble picking that up. $\endgroup$ – MJW Apr 19 '17 at 9:33
  • $\begingroup$ what is dropout? i know multilayer neural network but dont know what dropout is $\endgroup$ – user2543622 Apr 19 '17 at 13:37

Transforming individual variables would not help in random forest, because you can transform distances, but order of observations would remain the same, and problem of inability to distinguish classes would remain. In fact in most decision trees distances does not matter at all and only order of observations matter.

Unfortunately, if for some groups of cases your data does not differ between different classes, it is impossible to predict them correctly.

However, there is a hope. Although, your classes does not differ on any single variable, there is a chance, that they differ in multidimensional space of those variables. Try using some algorithms that are multidimensional (does not use single variable splits like decision trees in random forest, but many variables at once) or uses distances. I would recommend trying: -k-means (this of course uses distances on multidimensional space, you should try with transforming variables with PCA before modelling and experiment with 5+ centroids), -multinominal logistic regression with quadratic terms on every variable (this method gives score basing on many variables at once, so maybe it would help), -support vector machine (quite complicated to explain it here, but it also uses kind of multidimensional distances).

If your problem is only the imbalance of predicted classes you can retrieve votes of every tree from your model and decide yourself about some custom rules what class should be predicted with certain votes share or you can weight votes for different classes. Some random forest implementations also allow to assign weights to model. If you would weight underrepresented classes with higher weights your model will predict them more often.

  • $\begingroup$ RF supports arbitrarily complicated interactions if you make your trees deep enough. $\endgroup$ – einar Apr 16 '17 at 11:18
  • $\begingroup$ I checked and my x variables are not heavily correlated...would PCA help in such case? $\endgroup$ – user2543622 Apr 16 '17 at 20:32
  • $\begingroup$ Yes, my intention was to secure against colinearity. If your variables form highly correlated groups of variables with most variability on the same single dimension, the k-means on raw data would be highly flawed, because distances on those single dimension would weight more then on other dimensions, that are poorly represented by variables. After using PCA on correlation matrix you can be sure, that you fit k-means to the real space that you data variation is distributed on, not to the space deformed by colinear variables. $\endgroup$ – PtrZlnk Apr 16 '17 at 22:16
  • $\begingroup$ please read my update. I wont perform PCA as correlation is not significant...how should I improve my model further? $\endgroup$ – user2543622 Apr 16 '17 at 23:55
  • $\begingroup$ I would still try PCA. However, even simple standarisation of variables can improve performance (standarisation is substracting mean and then dividing by standard deciavian). You can do it with scale() function. This would scale all your x variables to the same standard deviation making distances on each dimension weight the same in final distance measure. Now every x variable can have different standard deviation, therefore can have different weight in computing distance. What you have done is in fact very similar to k-means with k=5. You can try k-means with more k, like 10, 15, 50. Maybe $\endgroup$ – PtrZlnk Apr 17 '17 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.