I have a target which can take on 3 classes. For the features I have only 2, a count variable and a continuous. Right now, it is just a simple Decision Tree model, I did not boostrap/aggregate anything yet. And right now, the issue is that this decision tree model does not make very good decision splits. I suppose I could always ratchet up the max depth, but right now it is at 5 and it is already making a few strange seemingly random splits that don't make sense. So I stopped there so as to not over-fit. Here is a picture of the decision surface plot:
As we can see, the decision boundaries (bounding rectangles) tend to not really make sense of the actual targets (represented in scatter plot form). Max depth of 5 is fairly generous for a model with only 2 features, so I'm thinking perhaps the model specification is the culprit. Let me go over one example quick:
Imagine if you tried to classify oranges, grapefruit and tangerines but you only had color and months to ripe from gestation as variables. Since we don't know size in this example, the model would really struggle as months to ripeness and color have some variation, but not nearly enough. This missing information may be what is going on in my real model as well. If we refer back to the graph, we will notice most of the decision areas have a high impurity, I've measured the highest to be .51 and .45.
My question is: Given that I don't have access to any more features (that would be the obvious fix), is there any point in trying to train a "better" aggregated model through random forests or would it just wind up getting stuck too? And if it doesn't work, is there anything else I can do (aside from getting more features) to bring to bear?
Further Clarification
- Number of classes: 3
- Number of features: 2
- Number of observations: 560
- Max node depth: 5