Thanks in advance for the help.
I have moderately large dataset (around 7000 samples) with numerous categorical predictors and a single binary response. All of the predictors are categorical. Through trial and error I have found that using a random forest model produces the best predictive accuracy of the response.
In order to increase the accuracy of my model I decided to include pairwise interaction terms. However, doing so led to a decrease in accuracy. I initially assumed this to be because including interaction terms adds a significant number of predictors to my training set. Specifically, before adding interaction terms I have 12 predictors and after adding them around 60. With dummy encoding I have approximately 60 predictors and a few hundred, respectfully.
With this in mind I performed feature selection and picked the 5,10,15,etc best predictors to train my model. This had little effect on the accuracy of my model, namely it wasn't as good as the model with no interaction terms. Notice that I optimized for trees in the forest and the number of features used in random selection.
At this point one might conclude that interaction terms don't add any anything to my model. However, I have good evidence that they should be very strong predictors of the response. Ideally I would be able to have interactions more complicated than just pairwise, but I don't have enough samples to do this.
I am wondering now if this behavior is a result of the method I am using, namely I am using a forest of trees. Suppose for the moment that I am using only one tree. Is it possible that using categorical data to train a tree type model inherently encodes interactions? If so, this might explain why including the interaction terms leads to a decrease in the predictive performance of my model.
Performing feature selection results in picking almost exclusively interaction predictors. When I produce a random forest using these predictors I am then limiting my "path". I'll motivate this with an example. Suppose I have predictors $w$,$x$,$y$,$z$ and using feature selection I settle on the interaction predictors $wx$ and $yz$. I can no longer produce a tree that which branches at $x$ and then at $y$. I can, however, produce such a tree using the original predictors without interactions.
Here is my hypothesis. Producing a random forest with just the original predictors allows me to "encode" interactions through the branching behavior of trees. Using only the results of feature selection does not perform as well since the features selected are predominately interactions. This limits the the branching behavior of the generated trees. Finally, including everything doesn't perform as well since this results in just way too many predictors. I don't have enough samples for the model to train properly. I should also note that I looked at using the original predictors and a handful of the best interactions, as determined by feature selection. This also did not perform as well as using only the original predictors.
Here is my question. Am I looking at this the right way? More specifically, does generating a tree using only categorical predictors inherently encode the interactions between those predictors? Also any insight or advice on how to alternatively tackle this problem (predicting the binary response) would much appreciated.