I am trying to classify cases according to binary response variable. For this purpose I chose random forest and classification tree methods. So, as it is usually done, I randomly split the data into training and test datasets (prop 0.8). Train the model with train dataset and test it with test dataset.
1) The question is... how can I be sure that test dataset is good for testing and that the model and tests results are objective?
This question arises, because when I rerun the division of data into train and test datasets several times, each time I got a tangible variation in the results: different variable importance, tree structures and sizes for classification tree and different AUC (0.92 - 0.97) for random forest.
So to get the best test result, I can create a function, which would try out different seeds for data division into test and train samples. Then I would choose the seed which would maximise my test score. However, I consider this approach unwelcome in academical society.
So my preliminary thoughts about how I could solve this issue are:
1. Compute similarity between test and train datasets and publish it with test results. The idea is simple. The better test results are and the more dissimilar datasets are, the better model it is, as it can explain truly new data.
2. Publish several most common classification tree structures produced using different subsamples of the data. Each substructure of a tree and each tree should have assigned its rate of appearance (for example, if root variable's "X" rate of appearance is 60%, when second split variable's "Y" rate of appearance is 90%, that means that "X" variable was the root variable in 60% of produced trees, when "Y" variable was used as a second split in 90% of those 60% trees. If tree structure's rate of appearance is 25%, that means that the exact same tree structure appeared 25% of the time).
However, these are just some thoughts. I do not know if am looking at the right direction and if this approach is the right one. So I would appreciate if someone would expand on the topic of 1) question or/and on my suggested or other possible solutions of the problem, preferably with the examples of R code.