I'm using the randomForest package in R to analyse a genetic dataset of ca 100 samples with 10.000 genes. The samples are grouped into 5 classes, the smallest being only of 5 samples.
I'm ultimately interested in the proximity matrix, confusion matrix and the variable importances, which is why I don't wish to start by reducing the dataset with PCA or similar. Rather, I'd tune the model using OOB estimates to handle the high dimensionality and variable class - if possible at all!
What's the best parameters to tune for such high dimensional data? After some initial tests, raising the minimum end-node size seems to improve OOB scores. I was thinking the main parameters of interest should be:
- Number of trees in forest (ntree)
- Number of features sampled (mtry)
- End-node size (nodesize)
I don't know the best way to control for very small class sizes. The small classes are never predicted correctly in my initial tests.
What's the best way to solve this problem? Simply to run all possible combinations of parameters, and pick the best based on OOB estimates, or is there a smarter way?