3
$\begingroup$

I have a target which can take on 3 classes. For the features I have only 2, a count variable and a continuous. Right now, it is just a simple Decision Tree model, I did not boostrap/aggregate anything yet. And right now, the issue is that this decision tree model does not make very good decision splits. I suppose I could always ratchet up the max depth, but right now it is at 5 and it is already making a few strange seemingly random splits that don't make sense. So I stopped there so as to not over-fit. Here is a picture of the decision surface plot:

enter image description here

As we can see, the decision boundaries (bounding rectangles) tend to not really make sense of the actual targets (represented in scatter plot form). Max depth of 5 is fairly generous for a model with only 2 features, so I'm thinking perhaps the model specification is the culprit. Let me go over one example quick:

Imagine if you tried to classify oranges, grapefruit and tangerines but you only had color and months to ripe from gestation as variables. Since we don't know size in this example, the model would really struggle as months to ripeness and color have some variation, but not nearly enough. This missing information may be what is going on in my real model as well. If we refer back to the graph, we will notice most of the decision areas have a high impurity, I've measured the highest to be .51 and .45.

My question is: Given that I don't have access to any more features (that would be the obvious fix), is there any point in trying to train a "better" aggregated model through random forests or would it just wind up getting stuck too? And if it doesn't work, is there anything else I can do (aside from getting more features) to bring to bear?

Further Clarification

  • Number of classes: 3
  • Number of features: 2
  • Number of observations: 560
  • Max node depth: 5
|cite|improve this question|||||
$\endgroup$
3
$\begingroup$

Random Forests shine in situations where there are many predictors, in particular if they are highly correlated. In addition, the speedup through the bagging (not all predictors are considered for potential splitting, only a randomly preselected small number of candidates) gets interesting if you have a large dataset.

You have only two predictors. That's not many. In particular, the random choice of, e.g., mtry=5 candidate predictors (to use R's randomForest) nomenclature is meaningless if you only have two predictors altogether.

I don't think Random Forests will be helpful. Then again, fitting one involves just a few lines of code in R, so you could just take a look.

One thing you might be able to do if you can't increase the number of predictors is to collect more observations. Perhaps there is a subtle interaction between your two predictors that only becomes visible with enough data. (I wouldn't be overly optimistic, though, judging from your description.)

Sometimes, your machine learning problem simply is hopeless.

|cite|improve this answer|||||
$\endgroup$
1
$\begingroup$

Don't expect miracles, but the random forest may have better error rate than just one decision tree. If your cross-validated error rate is, say, 40%, and you want to decrease it to, say, 25%, it may do the trick. Also, k nearest neighbours, with properly scaled features (that is, how much one feature is scaled versus the other) may be good for your problem.

|cite|improve this answer|||||
$\endgroup$
  • $\begingroup$ Could you link to an example or other reference so I can look into scaled features? I'm not sure if it's part of a cost function or if its a kind of pre-processing transformation. $\endgroup$ – Arash Howaida Jun 14 '17 at 14:20
  • 1
    $\begingroup$ Scaled features means following. When you do knn, you should calculate the distance between two observations. For two 2d points (x1,y1) and (x2,y2), it is sqrt((x1-x2)**2+(y1-y2)**2). But what is the right measure for x and y? The distance should be sqrt((x1-x2)**2+a*(y1-y2)**2), where a is metaparameter of your model. $\endgroup$ – user31264 Jun 14 '17 at 17:07
  • $\begingroup$ ohh, I see what you mean now, knn clustering as opposed to random forests. I might consider that at some point. Perhaps transforming would even help with random forests too. $\endgroup$ – Arash Howaida Jun 14 '17 at 17:30
1
$\begingroup$

Most probably yes. How much? Try it out!

The expected improvement will be a smoother classification as well as getting some extra correlations which I suspects from your plot. Honestly, the splitting looks good, but I fear I could even do better by eyes...

Anyway, it is worth to give it a try. Just don't overfit! This should not take more then 10 minutes to swap two lines of code and run it.

|cite|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.